Why Your Vector Embeddings Need Proper Protection – and How to Actually Do It

Intro

You’ve got a vector database full of embeddings. Question: are they sensitive?

The gut answer is no. They’re not text, not images, not records — they’re arrays of floats. A few thousand numbers per chunk. Nobody can read a list of decimals like [0.0241, -0.118, 0.0907, …] and learn anything. They feel like a one-way street: meaningful text goes in, meaningless numbers come out. So we store them in a managed vector DB, run them through a cloud RAG pipeline, and don’t think twice.

That intuition is wrong, and it’s wrong in a way that should change how you handle embeddings.

 

The Hidden Risk of Vector Embeddings: Vector Embedding Inversion

Embeddings can be turned back into the text they came from. This is called embedding inversion, and it’s not a thought experiment — it’s a demonstrated attack.

The reason it works is baked into what an embedding is for. The whole point of an embedding is to preserve the meaning of the input so that similar inputs land near each other in vector space. But “preserves the meaning” and “can be reversed back to the meaning” are two sides of the same coin. If the vector didn’t retain the information, search wouldn’t work. Because it does, the information is recoverable.

In 2023, researchers at Cornell (Morris et al., Text Embeddings Reveal (Almost) As Much As Text) built a method called vec2text that treats inversion as iterative refinement: guess the text, re-embed it, compare to the target, correct, repeat. The result is uncomfortable. It reconstructed 92% of 32-token inputs exactly — not approximately, exactly — and pulled full names out of clinical notes. It did this against production embedding models, including OpenAI’s text-embedding-ada-002. The original text was never seen by the attacker. Only the vectors were.

And it’s getting easier, not harder. Early attacks needed white-box access and a decoder trained for the specific encoder. Newer work performs zero-shot inversion against black-box embedding APIs — no training a custom decoder, no internals, just query access. The barrier to pulling text back out of “just numbers” keeps dropping.

So here’s the corrected mental model: an embedding is not anonymization. It’s a reversible, lossy encoding of your source data. Treat a leaked embedding the way you’d treat the leaked document behind it.

Don’t take our word for it — see it happen. We built a live inversion demo where you can watch text get reconstructed from its embedding: https://inversion.heaan.land/inversion.heaan.land

What This Means To You

If the data you’re embedding is yours and not sensitive, none of this matters. Embed away.

But most of us aren’t in that situation. If you’re building on customer records, patient data, internal documents, or anything a user entrusted to you, you are the custodian of data that isn’t yours. And the obligations that come with that — GDPR, HIPAA, your own contracts and DPAs — attach to the information, not to its file format. Turning a clinical note into a 1,536-dimensional vector does not discharge your duty to protect it, any more than zipping a file would. If the vector can be inverted back to the note, you should assume regulators and courts will treat it as the note.

This creates a real dilemma.

To do vector search well, the vectors have to be searched somewhere — typically a managed service or a shared cluster, because that’s what gives you scale and operational sanity. But searching means computing over the vectors in plaintext, in someone else’s memory, on infrastructure you don’t fully control. Encryption at rest and in transit doesn’t help, because similarity search needs the cleartext vectors to do the math. So you’re stuck choosing between two bad options: use the cloud tooling and accept that your invertible embeddings sit exposed at search time, or lock everything into an isolated environment and give up the scale and convenience you reached for the vector DB to get.

You want both. You shouldn’t have to pick.

 

How Homomorphic Encryption Can Solve This

Homomorphic encryption (HE) is encryption that lets you compute on the ciphertext directly. You encrypt your data, hand the encrypted data to a server, the server does math on it — adds, multiplies, computes similarity — and hands back an encrypted result. The server produces the right answer without ever decrypting anything. It never sees your data, and it never holds the key.

Apply that to vector search and the dilemma dissolves.

You encrypt your embeddings on the client, before they leave your control. You send ciphertext to the search service. The service computes similarity over the encrypted vectors and returns encrypted scores. Only you, holding the key, can decrypt the result. At no point — not at rest, not in transit, and crucially not during the search itself — does a plaintext embedding exist on the server.

The query and the stored vectors are searched in ciphertext on the server. The key — and the only point where anything becomes plaintext — stays on the client.

The vectors are always encrypted, and are never in plaintext. The thing an attacker would need to invert simply isn’t there to steal. Custodial obligation, satisfied; cloud-scale search, kept.

Problem solved.

…mostly. The obvious objection is the one you’re already thinking: isn’t homomorphic encryption slow? It’s a fair question, and the honest answer is that it deserves its own post. That’s exactly what the next one is about — so hold that thought.

Summary

1. Embeddings feel like harmless numbers. They aren’t — they’re an invertible encoding of your source data, and inversion attacks recover the original text, including names and other personal information.
2. If you handle data that isn’t yours, your duty to protect it follows the information into vector space. Embedding it doesn’t make it safe.
3. The catch is that ordinary encryption can’t help during search, because search needs plaintext.
4. Homomorphic encryption breaks the tradeoff: the vectors stay encrypted even while they’re being searched, so there’s no plaintext on the server to leak.

Vector embeddings need proper protection, and homomorphic encryption is how you get it without giving up cloud-scale search.

Is it slow? That’s the next post.

—————————————————————————————–

enVector runs similarity search directly over homomorphically encrypted vectors — your embeddings are encrypted client-side and never decrypted on the server, not even during search. Deploy it from Google Cloud Marketplace: https://console.cloud.google.com/marketplace/product/heaan-public/envector-sm

Reference: Morris, Kuleshov, Shmatikov, Rush. “Text Embeddings Reveal (Almost) As Much As Text.” EMNLP 2023. arxiv.org/abs/2310.06816