Retrieval quality is the product in a production-ready RAG system. Generation is the easy half: any competent model writes fluent prose from whatever context you hand it.
The hard half is putting the right three paragraphs in front of the right user, at the right version, inside a latency and cost budget you can defend in a review. The surrounding service is ordinary system design work. The retrieval path is where the surprises live.
The original RAG paper made the case in 2020: combine parametric memory with a non-parametric index you can update, inspect and cite. Lewis and colleagues set state of the art on three open-domain QA tasks with a dense retriever feeding a seq2seq generator. The pattern survived because of the index, not the generator. You can fix retrieval at 2am without retraining a model.
Everything below assumes you already chose retrieval over long-context stuffing. If that decision is still open, the RAG vs CAG comparison owns it. This post starts after the choice, where the failures actually happen.
Why Retrieval Quality Is the Product in Production-Ready RAG Systems
Because the generator can only be as correct as the context it receives. A model handed the wrong passage produces a confident, well-formatted, wrong answer. No instruction recovers information that never arrived.
The benchmarks support that ordering. BEIR evaluated ten retrieval systems across 18 datasets and found BM25 a robust baseline, reranking and late-interaction models best zero-shot but expensive, and dense or sparse models cheaper and often weaker out of domain. MTEB reached a compatible conclusion across 58 datasets and 112 languages: no single embedding method dominates every task.
Read those results as a warning about leaderboard-driven design. A model that wins average nDCG across 58 tasks can still lose on your corpus, because your corpus has its own vocabulary.
Part numbers, internal codenames, a German compound noun, a column header that only makes sense next to the table above it. Retrieval owns all of that. So do permissions, freshness, and the format the generator eventually sees.
Generation, by contrast, is a commodity you can swap. That asymmetry is the whole economic argument for spending your engineering time on the left side of the pipeline, and it is where I would put the first hire and the first dashboard.
If I had one quality budget, it would go to recall. I would fund retrieval work before a bigger generator, quarter after quarter, until recall@20 on a labelled set stops moving.
How Do You Ingest Documents Without Inheriting Their Chaos?
You treat ingestion as a data pipeline with a contract, not as a script that runs once. Parse, normalise, deduplicate, version, attach metadata, chunk, contextualize, embed, index. Every stage needs to be idempotent, because you will run all of them again.
The cheap version of this pipeline is the one that sinks most projects. Files arrive as PDFs with two-column layouts, HTML with navigation chrome, spreadsheets where the header row is row four, and tickets written in shorthand nobody outside the team understands. Parse before you chunk. If your chunk boundaries cut through a table that lost its header in extraction, no embedding model repairs that.
I would reject any parser that cannot preserve table headers and heading hierarchy. Extraction quality caps everything downstream, and it is far cheaper to change a parser than to re-embed a corpus.
Chunking is the stage with the most measurable payoff. Chroma's evaluation of chunking strategies found that some strategies beat others by up to 9% in recall, by their measure, and that default settings for popular splitters can perform poorly.
In their runs with text-embedding-3-large and five retrieved chunks, a 400-token recursive split with no overlap reached 89.5 recall against 85.4 for an 800-token split with 400 tokens of overlap. The larger chunks were not better. They were just bigger.
Context loss at the boundary is the real cost of small chunks, and it has a fix that pays for itself. Anthropic's contextual retrieval prepends a short generated context to each chunk before embedding it, and indexes the same contextualized text for BM25.
On their evaluation, contextual embeddings cut the top-20 retrieval failure rate from 5.7% to 3.7%, a 35% reduction. Adding contextual BM25 took it to 2.9%, a 49% reduction. Reranking on top reached a 67% reduction.
The fix is cheap. Anthropic measured the one-time cost at $1.02 per million document tokens, assuming 800-token chunks, 8k-token documents, a 50-token context instruction, 100 tokens of context per chunk and prompt caching for the repeated document prefix. You pay once per document version, not per query.
Chunking Is a Retrieval Decision, Not a Formatting Step
Choose chunk size from your question shapes, then verify with recall, not from a blog post default. Questions that ask for a value, a name or a date want small, precise chunks. Questions that ask for an explanation want a section, or a parent block the small chunk points to.
Three rules hold up in practice. Split on structure first, meaning headings, list items, code blocks and table rows, then fall back to token windows. Keep a document header on every chunk so the source, title and section are never ambiguous. Store a parent reference so you can expand a hit into its surrounding section at generation time, which gives you small-chunk precision with large-chunk context.
My starting point is 300 to 500 tokens with structure-aware splits, and I would move it only when recall on the golden set says so. Smaller chunks sharpen exact answers and lose narrative context. Larger chunks read better and bury the match.
Overlap deserves scepticism. It doubles storage and cost per token, and Chroma's numbers show a 400-token split with zero overlap beating a 400-token split with 200 tokens of overlap on recall in their setup. Use overlap when sentences genuinely straddle boundaries. Do not use it as a substitute for contextual headers.
Metadata Is the Schema of Your Retrieval Contract
Every chunk carries the fields that retrieval, filtering and audit will need later. Source system, document ID, version, updated_at, section path, document type, tenant, and the access control list. Add these at ingestion or you will rebuild the index to get them.
The test is simple: can you answer why did this chunk appear
and who is allowed to see it
from the chunk alone? If not, your metadata is incomplete. Version and updated_at also give you a deletion path and a freshness metric later, which is why they belong in the first schema rather than the third migration.
Pipeline frameworks help mainly by making this repeatable. LlamaIndex's ingestion pipeline caches each node-and-transformation pair, so a rerun only reprocesses what changed. That property, not the API surface, is the reason to adopt one.
Why Is Hybrid Retrieval the Default, Not the Upgrade?
Because lexical and dense retrieval fail on different queries, and fusing them costs far less than either failure. BM25 finds the error code, the SKU, the surname and the rare token. Dense vectors find the paraphrase, the synonym and the question that shares no words with the answer.
The history matters here, because it explains why anyone believed vectors replaced keyword search. Dense passage retrieval beat a strong Lucene BM25 system by 9% to 19% absolute in top-20 passage accuracy across open-domain QA datasets.
That result was in-domain, on Wikipedia-style text. BEIR later showed the out-of-domain picture, where dense models often trail BM25 and rerankers. Your enterprise corpus is out of domain for every public model.
Hybrid search is how vendors resolve that. Azure AI Search runs the full-text and vector queries in parallel and merges them with reciprocal rank fusion, recommending k of 50 when the semantic ranker is enabled so the reranker gets enough candidates.
Pinecone stores dense and sparse vectors on one record and lets you set the balance client-side through alpha weighting, or fuses separate indexes with RRF. Weaviate makes relative score fusion the default since v1.24, with ranked fusion as the alternative. Qdrant expresses the same idea as prefetch queries plus an RRF or DBSF fusion step.
Reciprocal rank fusion is the safe default because it uses rank, not score. Elasticsearch's implementation exposes rank_constant, defaulting to 60, which sets how much influence lower-ranked documents retain. That one parameter changes behaviour more than most teams expect, and it is worth plotting before you tune anything else. I would fuse on rank first and only test score fusion once the evaluation set is large enough to trust a one-point difference.
Fusion Is Where Hybrid Systems Quietly Break
Raw BM25 and cosine scores are not comparable, so any fusion that adds them directly is a bug waiting for a corpus change. BM25 scores are unbounded and depend on document length and term statistics. Cosine similarity sits in a narrow band. Normalizing both min-max per query works until one retriever returns a flat score distribution.
Two defences are worth the code. Prefer rank-based fusion unless you have measured that score fusion wins on your evaluation set. And evaluate fusion per query type, not in aggregate: exact-match queries, paraphrase queries, multi-hop questions and unanswerable questions each move in different directions when you change alpha or the fusion method. An aggregate gain often hides a regression in the query class your users care about most.
I would rather lose a point of average relevance than ship a fusion rule nobody can explain during an incident. Rank fusion is boring, and boring is what you want in the path that gates every answer.
What Does Reranking Actually Buy You?
A reranker scores query and passage jointly, which is the most reliable accuracy gain per line of code in the whole pipeline. Bi-encoders embed the query and each chunk separately, so they cannot see the interaction between them. A cross-encoder reads both at once and returns a relevance score for that pair.
The evidence is old and consistent. Nogueira and Cho re-implemented BERT as a passage reranker and improved MS MARCO MRR@10 by 27% relative over the previous state of the art. Anthropic's numbers show the same shape at the system level: adding reranking on top of contextual embeddings and contextual BM25 took the retrieval failure reduction from 49% to 67%.
The pipeline shape is a funnel: retrieve broadly, rerank narrowly, generate from a handful. Retrieve 50 to 200 candidates, rerank the top 20 to 50, and send 5 to 20 chunks to the model. Every stage narrows, and each has its own cost curve.
Reranking has hard ceilings you need to design around. Azure's semantic ranker reranks only the top 50 results, no matter how many you return, and its summarisation step accepts up to 2,000 tokens per document. If your retriever buries the right chunk at position 60, reranking cannot rescue it. That is an argument for generous first-stage recall, not for a bigger reranker, and I would raise the first-stage k before I argued about reranker model choice.
Hosted rerankers expose the same economics through their API. Cohere's rerank endpoint takes a candidate list and a top_n, so latency and cost scale with how many documents you send it. On a p95 budget, the rerank stage is usually the first thing that has to shrink. I would set that budget backwards from the p95 target, never from the largest batch the vendor allows.
How Do Metadata Filters and Permissions Change Retrieval?
Filters must run inside the query, because filtering after retrieval silently starves recall. If you ask for the top 50 chunks and then discard the 47 the user cannot see, you answered a question with three candidates. On a large corpus, a selective filter plus approximate nearest neighbour search can return almost nothing relevant.
This is a correctness problem before it is a security problem. Azure AI Search's document-level access model lays out the choices: security filters that compare a user or group string against document metadata, ACL and RBAC scopes for ADLS and blob content, and Purview sensitivity labels extracted at indexing time. Whatever mechanism you pick, the identity has to reach the query, and the index has to carry the permission metadata on every chunk.
Design the tenant boundary before you design the filter. One shared index with a mandatory tenant_id filter is simpler to operate and easy to get wrong, because a single missing clause leaks across customers. Separate collections or namespaces per tenant cost more to manage and make cross-tenant search impossible. For most products, I would take the shared index plus a query builder that cannot emit a query without the tenant clause, and a test that asserts it.
Permissions change more often than content, so plan for two clocks. Document updates arrive on a content clock.
Revocations need a much shorter one, because a user who lost access should lose it in minutes. Keep ACLs out of the embedding text, store them as filterable metadata, and update them in place rather than re-embedding the chunk. I would rather pay for a second write path than debug a cross-tenant leak on a Friday.
How Do You Keep an Index Fresh Without Rebuilding It Every Night?
You make ingestion incremental and make index swaps atomic. Hash each document's normalised content, reprocess only what changed, and version the whole index so you can promote a new one without downtime.
Change detection is the first half. Azure's incremental enrichment keeps an enrichment cache and reprocesses only documents whose change or deletion detection policy flags them.
The docs also carry a lesson worth internalizing: changing the data source, the container, the credentials, the field mappings or the parsing mode invalidates the cache. Your pipeline definition is part of your index identity. Treat a config change as a rebuild.
Deletions are the failure everyone forgets. Absence in a source listing is not a deletion signal unless you make it one. Track document IDs per source and write tombstones, or your index will serve answers from documents that no longer exist. That is the worst class of retrieval bug, because the answer looks perfect and cites a page that returns 404.
The second half is the swap. Qdrant collection aliases let you build a second collection and move the alias atomically, and the docs are explicit that alias changes happen atomically so no concurrent request sees a partial state. That mechanism is what makes an embedding model upgrade a deployment rather than an outage. Build the new index, evaluate it against the old one, then switch.
Freshness needs a number. Measure the lag from source update to index visibility, publish it as a metric, and alert on the percentile rather than the average. I would treat the pipeline configuration as part of the index version, because a parsing change is a data change even when no source document moved.
This is ordinary distributed systems hygiene applied to a search index, and the operational failure modes are the ones covered in how to design a distributed system: partial writes, retries without idempotency, and consumers that lag without anyone noticing.
How Do You Evaluate a RAG System Before Users Do?
You build a labelled retrieval set first, measure recall before you measure prose, and run both in CI. Evaluation that starts with answer quality cannot tell you whether a bad answer came from a missing document or a lazy generator.
Retrieval evaluation needs a golden set: a few hundred real questions, graded relevance labels for the chunks that answer them, and strata for query type. Include exact-match questions, paraphrases, multi-hop questions, and questions the corpus cannot answer. That last class is the one teams skip and the one that produces the worst production behaviour, because a system that never abstains will always answer something.
The tooling is now decent. RAGAS introduced reference-free metrics for faithfulness, answer relevance and context quality, and the RAGAS metric documentation lists the concrete set: context precision, context recall, noise sensitivity, response relevancy and faithfulness. ARES trains lightweight judges on synthetic data and uses prediction-powered inference with a few hundred human annotations to keep them calibrated.
Model selection runs on the same principle. Use the MTEB leaderboard to shortlist embedding models, then run your own golden set. The leaderboard answers which model is good in general. Only your queries answer which model is good for you.
I would block a retrieval change on a golden-set delta the way you block a deploy on failing tests. Without that gate, every tuning session is an opinion, and the strongest opinion wins instead of the best evidence.
Measure Retrieval and Generation Separately
Recall@k explains a wrong answer; faithfulness explains a wrong sentence. If recall@20 is high and answers are still wrong, the generator is ignoring or misreading context. If recall is low, no prompt change will help.
The diagnostic split saves days, and I would not trade it for a better judge. Compute recall@k and context precision on the retrieval side, faithfulness and answer relevance on the generation side, and track them as separate series. A drop in one with the other flat localizes the regression immediately.
Two calibration traps are worth naming. Judges inherit position bias, and models use context unevenly, as Lost in the Middle showed: performance is highest when the relevant passage sits at the start or end of the context and degrades in the middle.
Order your chunks deliberately and test the ordering. The second trap is unanswerable queries. Track the abstention rate, because a system that confidently answers everything scores well on answer relevance and badly with users.
What Does a Realistic Cost and Latency Budget Look Like?
Budget the whole funnel, because the expensive stage is rarely the one you expect. Query embedding is trivial, retrieval is cheap, reranking is linear in candidates, and generation dominates the token bill once you send 15 chunks of context.
Start from the context you can afford. Anthropic tested 5, 10 and 20 retrieved chunks and found 20 most performant of the three, but 20 chunks at 800 tokens each is 16,000 tokens of prompt on every request. More context also raises the risk that the relevant passage lands in the middle, where models use it least. The right number is the smallest one that keeps recall high on your golden set.
Then attach numbers to each stage. A p95 target you can defend, say two seconds, has to cover query embedding, a hybrid query across two indexes, fusion, reranking 50 candidates, and time to first token. Reranking cost scales with candidate count, so treat 50 as a budget line rather than a constant.
Three levers move cost more than any prompt tweak. Cache the one-time work, as in contextual chunk generation; Anthropic's prompt caching reports more than 2x lower latency and up to 90% lower cost on cached prefixes.
Compress the vectors, where Qdrant's quantisation options trade a small recall loss for 4x compression with 8-bit scalar quantisation and up to 32x with binary quantisation. And cut the number of chunks you send, which reduces both prompt tokens and the odds of a diluted answer. My order is recall first, then latency bought back with quantisation and caching, never with fewer candidates.
The serving side has its own economics, and they are not subtle once context grows. KV cache and memory movement dominate long-context inference, so prompt length is a cost multiplier with a quality ceiling attached.
What Are the Failure Modes You Will Actually Hit?
A short list accounts for most production incidents: stale index, filter starvation, chunk boundary loss, embedding mismatch, and unanswerable queries answered anyway. All five degrade quality without throwing an error, which is why they survive for months. If I had to keep two alerts, they would be freshness lag and per-tenant recall.
| Failure | What you see | Fix |
|---|---|---|
| Stale index | Correct answers about an old version | Change detection, freshness SLO, deletion tombstones |
| Filter starvation | No resultsfor restricted users only |
Pre-filtering, partitioned indexes, per-tenant recall tests |
| Chunk boundary loss | Answers missing the qualifier or table header | Structure-aware splitting, parent expansion, contextual headers |
| Embedding mismatch | Sudden recall drop after a model upgrade | Pin model version in metadata, alias swap, re-embed everything |
| Unanswerable answered | Confident replies with no supporting chunk | Abstention threshold, grounded-answer checks, negative test set |
| Prompt bloat | Slower, vaguer answers at higher cost | Cap chunks, deduplicate near-identical hits, order by relevance |
Two of these deserve a standing test. Embedding mismatch is the one that ships quietly, because a new model version and old document vectors produce plausible but degraded neighbours. Pin the model name and version in your index metadata and refuse to serve a query whose embedding model differs from the document model. My first-month checklist is the table above, wired as tests rather than written as a document nobody reads.
The second is filter starvation. Write a test that runs your golden set as a restricted user, and assert that recall stays within an acceptable band. If you only ever evaluate as an administrator, you will ship a system that works perfectly in demos and fails for every customer with a large private corpus.
Which Decisions Should You Make First?
Build the evaluation set before you build the index. It is the only artefact that tells you whether every later decision helped, and it is cheap: a few hundred labelled questions from real users, not synthetic ones.
The order I would follow: golden set, then ingestion contract, then a hybrid plus rerank baseline, then permissions, then freshness, then cost tuning. Each step depends on the measurement from the previous one. Skipping ahead means tuning a system you cannot see.
Two anti-patterns dominate early decisions. Choosing a vector database before you know your filter and freshness requirements puts the constraint in the wrong place, because most of the operational difficulty lives in ingestion and permissions, not in the index type. And treating retrieval as a prompt problem guarantees a slow, expensive fix, since no instruction can inject a document that was never retrieved.
Ownership matters as much as order. Retrieval quality needs one owner with the authority to change the index schema, the evaluation thresholds and the ingestion pipeline. Split across three teams, the golden set rots, the freshness alert gets muted, and the failure table above becomes a list of things everyone knows and nobody fixes. My rule is simple: no retrieval change ships without a measured delta, and I would rather delay a launch by a week than ship retrieval nobody measured.
What this post is not — and what to read instead
This is the build guide, not the architecture debate. It stops at the boundary of the retrieval service: how chunks, filters and scores behave in production, and how you prove they still behave.
If your next step is agentic, read designing AI agents: an architectural guide for how retrieval becomes a tool call with its own failure modes. If you are deciding who should own this work and how to staff it, must-have LLM skills and agents for engineering managers covers the team side. If the cost curve above is your bottleneck, LLM clusters are becoming memory systems explains why long context is a memory problem before it is a retrieval one. And for the wider decision record around these choices, the same-batch piece on making better architecture decisions walks through how to write the trade-off down before you commit to an index design.
Related Reading
System Design: A Practical Guide for Software Engineers
Search system design and you get a diagram: a load balancer, a cache, a queue, a database, and a glossary.
RAG vs. CAG: Which Approach Powers Smarter AI Knowledge Bases?
Engineering teams deploying LLM applications face a recurring architectural dilemma: do we pull relevant knowledge on
How to Design a Distributed System: Six Decisions in Order
You design a distributed system by making six decisions in order — where state lives, how many copies exist, how those