Retrieval-augmented generation has a deceptive learning curve. A working prototype takes an afternoon: embed documents, store vectors, retrieve top-k, stuff them into a prompt. The demo convinces everyone. Then real users arrive with real questions and quality collapses in ways that are hard to diagnose.
The diagnosis is usually the same. Industry analysis through 2026 consistently finds that when RAG fails, the failure is in retrieval roughly 73% of the time — not generation. The 2023-era reference architecture of a single vector index with top-k similarity search produces retrieval failure rates around 40% on real enterprise corpora.
The model can only reason over what it is handed. If the answer-bearing passage never surfaces, no amount of prompt engineering or model upgrading recovers it.
Name the failure before fixing it
Most “RAG is broken” complaints resolve to one of three distinct problems, each with a different fix:
| Symptom | Underlying problem | Where to work |
|---|---|---|
| The answer exists in the corpus but never surfaces | Recall | Chunking, hybrid search, query rewriting |
| Relevant documents surface but the wrong one is chosen | Ranking | Reranking |
| The right chunk is retrieved but the answer is still wrong | Generation or grounding | Prompting, citation enforcement |
Teams that skip this triage spend weeks tuning prompts to compensate for a chunking problem. The distinction is only visible if you measure retrieval separately — which most teams do not.
Chunking is a modelling decision, not preprocessing
Fixed-size chunking with overlap is the default because it is easy, not because it is good. It cuts tables mid-row, separates headings from the clauses they govern, splits code mid-function, and strips the context that makes a passage interpretable.
Chunking directly controls two things: recall, meaning whether the answer-bearing text exists inside a single retrievable unit at all, and precision, meaning whether a retrieved chunk is tight enough to be useful without dragging in unrelated content. Too large and chunks become topic soup that looks relevant and answers nothing. Too small and you lose the definitions and dependencies that make the passage mean anything.
Two patterns that consistently outperform the default:
- Structure-aware splitting — split on document structure: headings for documentation, clauses for contracts, functions or classes for code. This aligns chunk boundaries with meaning boundaries.
- Semantic chunking — embed sentence by sentence and start a new chunk where cosine similarity between adjacent sentences drops below a threshold, splitting where meaning shifts rather than where the character count runs out.
Attach context to every chunk: source document, section, effective date, version. A retrieved passage that cannot be situated is a passage the model will misuse — and a citation you cannot render.
Hybrid search is the default architecture now
Dense embeddings capture semantic similarity well and exact terms badly. Enterprise queries are dense with exact terms: product codes, policy numbers, regulation references, internal acronyms. A user asking for clause 4471-B wants that clause, not a semantically adjacent paragraph.
The 2026 reference architecture runs BM25 keyword search and vector search in parallel — often with a graph or structured layer for entities and relationships — and merges results with reciprocal rank fusion, which combines on rank position and therefore needs no score normalisation or tuning.
The gains are large enough to justify the complexity. Reported figures put error reduction at roughly 69% when hybrid retrieval and contextual techniques are applied together. Applying metadata filters before hybrid search — restricting by document type, date range or department — reduces noise further by shrinking the search space before vector distance is computed at all.
Reranking: retrieve broadly, then be precise
First-stage retrieval optimises for recall; reranking optimises for precision. These are different jobs, and one stage does both poorly.
A cross-encoder reranker scores query and candidate chunk jointly, which is slower per pair than bi-encoder embeddings but materially more accurate. The working rule of thumb: retrieve 20–50 candidates, rerank, pass 3–5 to the model. Reranking 100-plus candidates rarely pays off — the head of the distribution carries the signal.
Reranking also mitigates the “lost in the middle” effect, where models attend poorly to information buried in the centre of a long context. Fewer, better-ordered chunks beat more chunks.
Long context does not remove the need for retrieval
Frontier models now offer very large context windows, prompting the reflex that RAG is obsolete. It is not, for three reasons: cost scales with tokens processed on every query; attention quality degrades across very long contexts; and dumping a corpus into a prompt provides no access control, no citations, and no freshness guarantee.
Use the long window where synthesis genuinely demands it — a long report, a whole codebase — not as a substitute for knowing which documents are relevant.
Measure retrieval separately from generation
This single habit most distinguishes teams that improve from teams that guess.
Build a set of questions paired with the passages that should be retrieved, then measure Recall@K and MRR independently of what the model does afterwards. Without this, a wrong answer is uninterpretable: did retrieval miss the passage, or did the model fumble a passage it had? Those need completely different fixes.
Downstream, score faithfulness — does the answer follow from the retrieved context — and answer relevancy. Every production RAG system in 2026 runs this continuously rather than spot-checking outputs, because spot-checking does not scale and does not catch drift.
Freshness and permissions: production concerns from day one
Two issues absent from every prototype and present in every production system.
Documents change. A system confidently citing a superseded policy is worse than one that says it does not know. Reindexing must be scheduled and monitored, with alerting on lag — because when embeddings go stale nothing errors. Retrieval simply returns vectors of text that no longer exists.
Retrieval must respect access control. If the index is flat and your users are not, RAG becomes an efficient mechanism for surfacing documents people should not see. Chunk-level access control applied during retrieval — not as a post-hoc filter, and never as a prompt instruction — is the only defensible pattern. Under India’s DPDP Act, with full enforcement expected by May 2027, this is the difference between a controlled system and a reportable incident.
The order of operations that works
- Build a retrieval test set with known correct passages before optimising anything.
- Fix chunking to respect document structure.
- Add hybrid search with reciprocal rank fusion.
- Add metadata filtering ahead of retrieval.
- Add cross-encoder reranking: retrieve 20–50, pass 3–5.
- Only then consider a larger or different generation model.
Teams routinely run this backwards, starting with the model because it is the most visible lever and the easiest to change. The model is rarely the constraint. Retrieval quality sets the ceiling; everything downstream operates beneath it.
Frequently asked questions
Why does my RAG system return irrelevant results?
Most often chunking or retrieval mode. Fixed-size chunking splits answers across boundaries, and pure vector search misses exact identifiers. Measure Recall@K against a test set of questions with known correct passages to establish whether the problem is recall or ranking before changing anything.
Is hybrid search worth the added complexity?
For enterprise corpora, generally yes. Reported error reductions of around 69% when hybrid retrieval is combined with contextual techniques substantially exceed what a model upgrade typically delivers, and reciprocal rank fusion requires no tuning.
How many chunks should I send to the model?
A common working pattern is retrieving 20–50 candidates, reranking with a cross-encoder, then passing 3–5 to the model. More chunks tend to dilute attention rather than improve answers.
Do long context windows make RAG unnecessary?
No. Long context costs tokens on every query, degrades in attention quality across very long inputs, and provides no access control, citations, or freshness guarantees. Use it for genuine synthesis tasks, not as a replacement for retrieval.
