What an Embedding Is
Run the sentence “the pump is leaking” through an embedding model and it returns a list of numbers: 384 of them, or 1,024, or 3,072, depending on the model. That list is the embedding. It neither summarises nor compresses the sentence, and no position in it is readable: position 47 does not mean “mechanical” and position 300 does not mean “urgent”. Not even the people who trained the model know what a single number stands for.
The vector is good only for comparison. Ask how close it sits to the vector for “hydraulic unit is losing oil” and you get a number back, usually cosine similarity, between −1 and 1. Everything embedding-based systems do rests on that number.
A vector is therefore comparable only to vectors from the same model. Coordinates from multilingual-e5-large and from bge-m3 both have 1,024 dimensions and still describe two unrelated spaces. Comparing across them does not crash; it returns a plausible-looking number that means nothing. The rule comes back further down as the most expensive trap in this article.
Why Similar Meaning Lands Nearby
Nobody designed that space; it emerges from training. The model saw enormous numbers of text pairs that belong together, such as a question and its answer, a headline and its article, or a German sentence and its English translation. Training pushed it to place each pair close together and to push randomly drawn pairs apart. Run that for long enough over a large part of the internet, and text that occurs in the same contexts ends up in the same neighbourhood. The geometry follows from the statistics.
The model did not learn meaning in the process; it learned which texts are about the same thing. “Similar” therefore means “these two would plausibly turn up in the same conversation”. It does not mean “these say the same thing”, and certainly not “these are both true”.
“Water damage is covered under this policy” and “water damage is not covered under this policy” share a topic and a vocabulary and differ by one word, and an embedding model puts them almost on top of each other. For a regional insurer building search across its policy wordings, that is the core of the job, not an edge case. In exclusions, exceptions, and safety instructions that difference decides the answer, and similarity does not capture it.
What That Buys You: Retrieval and Classification
Retrieval is the familiar half. You split documents into chunks, embed each chunk, store the vectors, and at query time embed the question and fetch its nearest neighbours. Every RAG system starts that way; the full pipeline contains considerably more, and the database underneath holds the engineering of approximate search. Embeddings are one stage of that pipeline, and rarely the one that fails.
Almost nobody uses the second half, although it is often the better first step: embeddings make an excellent classifier. Every company has a thousand emails sitting in a shared mailbox that a human already sorted into “spare part enquiry”, “complaint”, “invoice question” and “other”. Embed all of them and fit a logistic regression on top in about ten lines of scikit-learn. That router answers in milliseconds, costs nothing per call, runs on a CPU on your own hardware, and yields a confusion matrix you can show the department that owns the process.
Teams now reach for an LLM prompt by reflex. Sometimes that is right, namely when the categories are subtle, when you have no labelled examples, or when the label needs a justification next to it. But the LLM costs money per call, takes noticeably longer per call, and changes behaviour unnoticed when the vendor ships a new version, and you can evaluate its output only by reading it. You test the embedding classifier the way you test ordinary software. For high-volume, low-nuance routing it is the better engineering, and it takes an afternoon of work.
Choosing an Embedding Model
The choice of model matters less than the leaderboards suggest, and a leaderboard cannot settle it anyway. MTEB is a serious benchmark and a reasonable way to narrow the field to three candidates. It cannot tell you which of the three does best on your maintenance reports, because your maintenance reports were not in it. In practice the top of the list and the solid middle differ by less than chunking decisions do.
So take 50 real questions from the ticket system, not invented ones, and write down which document should come back for each. Run your three candidates and count how often the right document appears in the top five. That takes an afternoon and replaces the argument with measurements. It leaves you with an evaluation set you will still use in two years, every time someone proposes a change.
Do not assume that a model at the top of an English leaderboard also reads German competently. Tokenisers trained mostly on English break up German compounds: Betriebshaftpflichtversicherung becomes a handful of fragments that individually mean nothing, and the term your user typed is the one the model handles worst. A strong multilingual model such as multilingual-e5, bge-m3, jina-embeddings-v3, or Cohere's or OpenAI's multilingual offerings nevertheless often beats a small German-only model, though that is not a fixed rule. Use MTEB's German tasks to pick the candidates, then test them with your own 50 questions.
If a project depends on keeping data in-house, the embedding layer is the cheap part of going on-premise. Models like multilingual-e5-large or bge-m3 fit on a mid-range GPU and also run on CPU for overnight batch indexing. The generation model makes self-hosting expensive, not the embedding model.
Three Traps Without an Error Message
Each of these traps leaves you with a system that runs, returns results, and is wrong. They get expensive because nothing reports the failure.
Your vocabulary was not in the training data
An embedding model knows bolts because the internet is full of bolts. It does not know that at your company “K1 part” means a part on the critical path, or that DIN 912 and DIN 913 are different fasteners and not two spellings of the same string. The tokeniser splits part numbers, article codes, internal abbreviations, and project jargon into fragments, and the fragments resemble each other. A machine builder's service technician who searches for a spare part by its number is therefore the worst case for pure vector search.
A better embedding model does not fix this, because no model knows your parts list. Run BM25 keyword search next to the vector search instead, merge the hits, and keep the part number in a metadata field you can filter on exactly. For any corpus with identifiers in it, hybrid search is the starting point, not a later refinement.
One chunk, one point
Every chunk gets a single vector, however much text it holds. A tight paragraph about warranty periods lands in the warranty neighbourhood. A five-page chunk covering warranty, delivery terms and liability lands at the average of the three, a spot in the space where nothing lies. It therefore loses to a mediocre small chunk that is at least about the right topic.
Hence the advice to chunk small and along the structure. For the same reason, the proposal “let's use a model with an 8k context window, then we can embed whole documents” leads into a trap although it looks like progress. The limit is not how much text fits through the model; it is that one vector cannot point in three directions at once.
Two kinds of stale, and only one is obvious
In the obvious kind, the document changed and the index did not. The price list is from March, the answer quotes it, and the customer gets the March price. Every RAG project plans for this: it reindexes on change, timestamps everything, and expires what it cannot verify. The problem is solved once somebody owns it.
The other kind catches teams out: the model changed. You move to a newer embedding model, or your provider retires the version you indexed with, or someone swaps in an e5 model without noticing that the e5 family expects queries with the prefix query: and documents with passage:. That convention is stated in the model card, and no error enforces it. Cosine similarity then compares two vectors from two unrelated spaces and returns a reasonable-looking number. Quality does not drop to zero, which would at least show. It sinks to mediocre, and nobody notices, because nobody kept a baseline.
So write the model name, version, and prompt convention into the index metadata, and have the query path check that they match before it searches. Treat re-embedding as a migration with a cutover, not as a config change. This also argues for hosting the embedding model yourself: a vendor can retire a model on its own schedule, and then you rebuild your index on its schedule.
From our own build: the model fingerprint
We ruled out two embedding models in one index, one form of the third trap, by construction in TippelPi, our knowledge system on a Raspberry Pi. On ingest, every passage receives the fingerprint of the model that embedded it. If the server finds passages with a foreign fingerprint in the index, it refuses to search and returns an error instead of serving wrongly ranked hits, and the status endpoint reports how many passages are affected.
Changing the model also invalidates the markers by which synchronisation recognises unchanged files. The next sync therefore re-embeds the whole corpus and lifts the block by itself. The silent failure thus becomes a reported one, and the next run repairs it. More on the system is in the Lab.