Keyword search has a structural flaw that no amount of tuning fixes: it matches characters, and people remember meaning. Someone searching for "staff leaving rate" will not find a document that consistently says "employee attrition", however well written that document is.
Vector search addresses this directly, and it has become the default retrieval layer for a generation of knowledge products.
Matching meaning instead of spelling
An embedding model converts text into a vector — a list of numbers positioning that text in a high-dimensional space. The models are trained so that passages with similar meaning land near each other, regardless of the words used.
Retrieval then becomes geometry. Embed the query, find the nearest vectors, return the passages they came from. "Staff leaving rate" and "employee attrition" land close together, so the right document surfaces even with no shared vocabulary.
Distance is usually measured by cosine similarity, which compares the direction of two vectors while ignoring their magnitude — appropriate here, since it is the semantic direction that carries the meaning.
How the search actually happens
Comparing a query against every vector in a large corpus is accurate and too slow. Production systems use approximate nearest neighbour indexes, which trade a small amount of recall for a very large speed gain.
Two families dominate. Graph-based indexes such as HNSW build a navigable structure through the vectors and walk it toward the query, giving excellent recall and fast queries at the cost of memory and slower index construction. Partition-based approaches such as IVF divide the space into clusters and search only the nearest few, using less memory but requiring their partitioning to be tuned against the data.
The decision is a genuine engineering trade-off between recall, latency, memory and update cost. It should be made against measured requirements, not by reputation.
Why hybrid retrieval usually wins
Pure vector search has a predictable weakness: it is poor at exact matches. Product codes, error identifiers, surnames, part numbers — precisely the terms where the user wants that string and nothing else — are where lexical matching excels and semantic similarity blurs.
Hybrid retrieval runs both a lexical search and a vector search, then fuses the results. In practice this outperforms either approach alone across mixed workloads, because real queries are a mixture of conceptual questions and precise lookups.
A reranking pass over the fused candidates improves things further. A cross-encoder examines the query and each candidate together, rather than comparing pre-computed vectors, producing a more accurate ordering. It is too expensive to run across a whole corpus and well suited to reordering a shortlist.
Chunking is the hidden variable
Documents are split before embedding, and how they are split shapes retrieval quality more than most teams expect.
Chunks that are too large dilute the embedding — a vector averaging several topics sits near none of them. Chunks that are too small lose the context that made the passage meaningful, retrieving a sentence whose referents have been stripped away.
Splitting on document structure — sections, headings, natural boundaries — generally beats splitting on a fixed character count, because it preserves the author's own organisation of the material. Modest overlap between adjacent chunks helps avoid severing a point across a boundary.
Vectors are not a knowledge base
The important limitation: vector search retrieves passages that resemble a query. It does not know anything.
It cannot reliably answer questions requiring aggregation across many records, or traversal of explicit relationships, or reasoning over structure — how many, which ones connect to this, what changed between these two points. These are the questions a knowledge graph or a database answers well and a similarity index answers badly.
The strongest knowledge platforms therefore treat vector search as one retrieval strategy among several, routed to when the question is genuinely about semantic similarity over unstructured text, and routed past when it is not. Treating it as the whole architecture produces systems that are impressive in demonstration and unreliable in the specific, structured queries that daily work is actually made of.