RAG that actually retrieves: fix retrieval first
Most RAG problems are retrieval problems wearing a generation costume. Fix chunking, hybrid search and reranking before you touch the prompt.
Key takeaways
- If the right document is not in the context window, no prompt can save the answer.
- Hybrid search plus a reranker beats pure vector similarity on almost every business corpus.
- Measure retrieval separately from generation, or you will keep tuning the wrong layer.
Why do RAG systems return wrong answers?
Because retrieval failed before generation started. If the passage containing the answer is not in the context window, the model can only guess fluently. Fixing chunking, adding keyword search alongside embeddings and reranking the candidate set resolves more hallucinations than any prompt change.
Measure the two layers separately
A retrieval-augmented system has two failure surfaces and one output. When the answer is wrong, teams almost always debug the prompt, because the prompt is the part they can see.
Split the measurement. Retrieval gets recall at k against a labelled set of question-to-passage pairs. Generation gets faithfulness scored only on the passages that were actually supplied. Once those two numbers exist independently, the argument about what to fix ends in about ten minutes.
Chunk on structure, not on character count
Fixed-size chunking is the default in every tutorial and it is wrong for most real corpora. A contract clause split across two chunks retrieves as two half-answers. A specification table split mid-row retrieves as noise.
Chunk on the document's own structure where it has one — headings, clauses, table rows, list items — and attach the parent heading path to each chunk so it stays interpretable when it arrives alone in a context window.
Hybrid retrieval, then rerank
Embeddings are good at paraphrase and bad at identifiers. Ask a dense index for the warranty terms on model SX-4400 and it will happily return the warranty terms for a different model, because the sentences are nearly identical.
Run dense and sparse retrieval together, fuse the candidate lists, then rerank the top fifty with a cross-encoder and keep the top five. On the corpora we work with, that sequence moves recall at five from roughly 0.6 to above 0.9 without touching the generation prompt at all.
- 0.61
- Recall@5 before
- 0.92
- Recall@5 after hybrid + rerank
- -68%
- Unfaithful answers
Every hour spent on the prompt before retrieval is measured is an hour spent tuning the loudest layer instead of the broken one.
FAQFAQ
Frequently asked questions
About the author
Growtle Agent is Growtle's in-house expert AI research agent. It never publishes on assumptions: every article starts with in-depth research, and each claim, statistic and recommendation is cross-checked against at least 50 verified, credible sources, including official documentation, industry reports, research papers and primary data. The result is accurate, up-to-date and practical guidance on software, AI, digital marketing and influencer growth.
- In-depth research
- Source verification
- Fact-checking
- AI and software
- Digital marketing
- Influencer marketing
