RAG that actually retrieves: fix the retrieval layer first
Most retrieval-augmented generation problems are retrieval problems wearing a generation costume. Fix chunking, hybrid search and reranking before you touch the prompt.
Key takeaways
- If the right document is not in the context window, no prompt can save the answer.
- Hybrid search plus a reranker beats pure vector similarity on almost every business corpus.
- Measure retrieval separately from generation, or you will keep tuning the wrong layer.
Why do RAG systems return wrong answers?
Because retrieval failed before generation started. If the passage containing the answer is not in the context window, the model can only guess fluently. Fixing chunking, adding keyword search alongside embeddings and reranking the candidate set resolves more hallucinations than any prompt change.
Measure the two layers separately
A retrieval-augmented system has two failure surfaces and one output. When the answer is wrong, teams almost always debug the prompt, because the prompt is the part they can see.
Split the measurement. Retrieval gets recall at k against a labelled set of question-to-passage pairs. Generation gets faithfulness scored only on the passages that were actually supplied. Once those two numbers exist independently, the argument about what to fix ends in about ten minutes.
Chunk on structure, not on character count
Fixed-size chunking is the default in every tutorial and it is wrong for most real corpora. A contract clause split across two chunks retrieves as two half-answers. A specification table split mid-row retrieves as noise.
Chunk on the document's own structure where it has one — headings, clauses, table rows, list items — and attach the parent heading path to each chunk so it stays interpretable when it arrives alone in a context window.
Hybrid retrieval, then rerank
Embeddings are good at paraphrase and bad at identifiers. Ask a dense index for the warranty terms on model SX-4400 and it will happily return the warranty terms for a different model, because the sentences are nearly identical.
Run dense and sparse retrieval together, fuse the candidate lists, then rerank the top fifty with a cross-encoder and keep the top five. On the corpora we work with, that sequence moves recall at five from roughly 0.6 to above 0.9 without touching the generation prompt at all.
- 0.61
- Recall@5 before
- 0.92
- Recall@5 after hybrid + rerank
- -68%
- Unfaithful answers
Every hour spent on the prompt before retrieval is measured is an hour spent tuning the loudest layer instead of the broken one.
FAQFAQ
Frequently asked questions
About the author
Ravi builds the applied AI systems we ship: retrieval pipelines, evaluation harnesses and the guardrails that keep a model useful once real users reach it. He writes about the engineering that decides whether an AI feature survives production.
- LLM evaluation
- Retrieval-augmented generation
- Prompt engineering
- AI cost modelling
