What a Vector Database Does

Given a list of numbers, a vector database finds the other lists that point in roughly the same direction. Those lists are embeddings, chunks of your text turned into a few hundred or few thousand coordinates and positioned so that passages about the same thing land near each other. The database stores them and answers that nearest-neighbour question fast.

The search is only approximate, because comparing your query against every vector in the corpus would be exact but slow. Production systems instead use an approximate nearest-neighbour index: HNSW builds a navigable graph through the vectors, and IVFFlat sorts them into buckets and searches only the closest few. Both trade correctness for speed, answer in milliseconds, and quietly miss some good matches. A parameter sets how much search effort the index spends on this (in HNSW, how many candidates to keep in play during the search).

Your search index is lossy, and it never tells you what it lost. The index raises no error when it never visits the chunk that answers the question and that chunk is missing from the top 50, although an exact search would have ranked it near the top. The model receives a plausible context and writes a fluent answer from it. Vector search then returns worse results without failing.

pgvector Is Probably Enough

pgvector extends PostgreSQL with a vector column, the usual distance functions, and HNSW indexes. All of it lives in the database that already holds the row the chunk came from, with the transactions, backups, roles and monitoring your team knows. For most corpora that is the correct engineering decision and not a compromise.

Do the arithmetic before you go shopping, because vendors write their sizing pages for a different scale than yours. A 1024-dimension vector in float32 takes about 4 KB, so a million chunks come to roughly 4 GB of raw vectors plus graph overhead. A million chunks is a lot: at perhaps twenty chunks per document, that is around fifty thousand documents, every quote, manual, contract and policy from a decade. It fits in RAM on one ordinary server.

When a vendor says "scale," they mean billions of vectors and thousands of queries per second, because only that problem justifies a distributed system. You have millions of vectors and a few queries per minute from staff who are also doing other things. I would therefore not stand up a self-hosted Milvus cluster for 200,000 chunks. You would be taking on a distributed system's failure modes, namely sharding, rebalancing, and node ops, for a load a single Postgres server carries on the side.

The default. If your corpus is under a few million chunks, your query load is human-scale, and you already run Postgres, use pgvector. A project that chooses a dedicated vector database first and looks for a reason afterwards ends up operating an extra system for no measurable gain.

Three Selection Criteria

Three criteria speak against that default, and only one of them is about size.

Corpus size

In our assessment pgvector gets laborious in the tens of millions of vectors. That shows in index build time and in memory during rebuilds long before it shows in query latency. As a rule of thumb, comparing products is not worth it under roughly five million chunks with modest concurrency, because you would be optimising a problem you do not have. At fifty million and rising, as with a large document archive or a product catalogue across many markets, a purpose-built store pays off.

Filtering during the search

This criterion decides most selections in practice, and almost nobody raises it in the first meeting.

A regional insurer wants to build an assistant over policy wordings and claims guidelines. No query there is "search everything"; every query is "search the documents this user is allowed to see, in this line of business, valid on today's date." The pipeline therefore needs a metadata filter alongside the vector search, and where that filter runs decides whether the system works.

POST-FILTER — the naive variant 1M chunks whole index ANN top 50 by similarity then filter line of business, date ≈ 1 left recall collapses PRE-FILTER — what you want 1M chunks whole index filter first inside the index ANN top 50 within allowed set 50 good recall holds
Index, filter, and query are the same, and the order decides the answer. Filter after the search and a selective filter removes most of your results without anything reporting an error.

If you retrieve the top 50 by similarity and then filter in your application code, a filter that admits 2% of the corpus leaves you with roughly one usable chunk out of fifty. The model answers from that one mediocre passage in its usual tone, and the system looks like it is working. That is how an assistant that impressed everyone in the demo gives thin, subtly wrong answers in production.

The remedy is a filter that runs inside the index traversal, so the search only visits permitted vectors. Purpose-built stores such as Qdrant and Weaviate treat filtered search as a core feature; Elasticsearch and OpenSearch do it as an extension of the filtering they have always done. pgvector applies filters to approximate indexes only after the index scan. Since version 0.8.0, iterative index scans compensate by scanning further until enough permitted rows are found. So check on your own corpus whether the query planner chooses your index or falls back to a scan, and how many hits remain after the filter. If your access model is a hard requirement, and for an insurer it is not negotiable, filter behaviour is the selection criterion, ahead of throughput and well ahead of brand.

Hybrid search for exact strings

Semantic search does badly on exact strings. An embedding maps "DIN 933", "4-2291-B", or "§ 34c" to a vague neighbourhood of similar-looking strings, because a part number carries no meaning it could capture. A supplier whose engineers search by part number therefore cannot get by on pure vector search.

Hybrid search solves this. Keyword search (BM25) and vector search run in parallel, and you fuse the rankings. That speaks for Postgres, which has had full-text search for years and does both in one query against one system, although the built-in ranking is ts_rank and true BM25 comes only with an extension. On text-heavy corpora it speaks equally for Elasticsearch or OpenSearch, because a mature keyword engine that added vector search often fits this problem better than a vector engine that added keywords. Which retrieval mix you need is a pipeline design question, and it should be settled before anyone signs a contract.

Plan for Re-Indexing

A vector index is derived data and not a system of record. Treating it as one turns a small decision into an expensive one.

Vectors produced by one embedding model are meaningless in another model's space, and that weighs more than any benchmark table. No practical migration path exists, so after a model change you re-embed every chunk. You will change it, because a better or cheaper model ships or, far more often, because you change your chunking strategy after seeing how retrieval behaves on your actual documents. Re-indexing is therefore routine.

The question to ask a vendor. Do not ask the vendor for the p99 latency; ask what happens when you re-embed four million chunks. You want to build into a fresh collection and swap atomically, from chunk text and metadata you hold in your own durable store. If the answer involves a support ticket or a maintenance window nobody will approve, that counts against the product.

Retrieval Quality Is Decided Before the Database

Whether the right passage is findable depends on what got ingested, how it was chunked, whether metadata survived the import, and whether a reranker sits between retrieval and the prompt. The database only executes the search and does not know whether the answer is in there.

A project rarely fails because of the wrong vector database. It stalls more often because the chunks are cut wrong, because access control was added after the fact, or because nobody can rebuild the index without losing a weekend. The store is also one of the more reversible decisions in the stack, because chunks, embeddings and metadata are portable, and swapping it usually takes days and not a rewrite. The decision therefore needs little deliberation.

So choose the option that handles filters and hybrid search, that your team can back up and restore, and that you can rebuild from source at any time. The time saved is better spent on chunking, because how the chunks are cut helps decide whether retrieval finds the right passage.