Retrieval is an architecture decision, not a feature
In September 2024 Anthropic published a short post about a technique it called contextual retrieval, and the most interesting thing in it was a table of failure rates. Across codebases, fiction, arXiv papers and science papers, a plain embedding search failed to put the relevant chunk in its top 20 results 5.7% of the time. Add a sentence of context to every chunk before indexing it, add a keyword index next to the embeddings, rerank the candidates, and the failure rate dropped to 1.9%. That is a 67% reduction. The model that wrote the final answers was the same one throughout.
I take that table as the starting point for an argument. A retrieval-augmented generation system is a database with a language model as its query interface, and the part that decides most of its quality is the database half. The choices that fix a system's ceiling (how documents are cut, how they are indexed, who is allowed to see which piece, how anyone will know the system is right) deserve the review a schema change would get. They are cheap to make on day one and expensive to change on day ninety, and a project that picks a vector store in an afternoon and polishes prompts for a month has made them by default.
The ceiling is set before the model reads a word
The term comes from Patrick Lewis and colleagues at Facebook AI Research, who in 2020 paired a sequence-to-sequence generator with a dense index of Wikipedia and called the combination retrieval-augmented generation. Their framing was memory: the weights are parametric memory, the index is non-parametric memory, and the second can be inspected and updated without retraining the first. The framing still holds, and it explains why the retrieval half deserves the attention a storage layer would get.
Scott Barnett and his co-authors made the same point from the failure side. Their paper, presented at the 2024 International Conference on AI Engineering in Lisbon, drew on three case studies (research, education, biomedical) and listed seven places where a RAG system breaks. The first three never involve the generator. The corpus may not contain the answer at all, in which case the system tends to produce a plausible wrong one instead of saying so. The answer may exist but rank below the cutoff, so it is never passed on. Or it may be retrieved and then dropped while the results are merged to fit the prompt. Only after those three do the familiar model failures begin: the answer sits in the context but is not extracted, the requested format is ignored, the specificity is wrong, the response is incomplete.
For any question whose answer lives in a single passage, the arithmetic is simple. If the right chunk reaches the context on 94.3% of questions, which is what Anthropic's baseline implies, then no choice of generator will push accuracy past 94.3% on those questions. The generator can only lose ground from there. Moving the retrieval number to 98.1% raises the ceiling by almost four points, and a ceiling raised that way is available to every future model swap for free.
Chunking is a claim about your documents
Anthropic's explanation of why contextual retrieval works rests on one example. Take the chunk "The company's revenue grew by 3% over the previous quarter." Which company? Which quarter? Embedded alone, that sentence sits close to thousands of others like it. The fix was to have a model write a sentence or two situating each chunk in its document ("This chunk is from an SEC filing on ACME corp's performance in Q2 2023; the previous quarter's revenue was $314 million") and to prepend it before both embedding and keyword indexing. Under the assumptions in the post (800-token chunks, 8,000-token documents, prompt caching), the one-time cost came to about $1.02 per million document tokens. A corpus of 100 million tokens would cost roughly $102 to process.
That is a bargain, and it is also a reminder that a chunk is a modeling decision. Cutting a document every 500 tokens asserts that meaning is distributed evenly along the page, which is almost never true. A contract states a rule in clause 7 and its exception in clause 12. A runbook says "repeat step 3 with the secondary node." Tables span page breaks, and a changelog entry means nothing without the version heading three screens above it. Every one of these produces a chunk that is accurate, retrievable, and misleading.
There is a second reason chunk boundaries matter that has nothing to do with search. The chunk is also what the user sees as evidence. Whatever the system cites is the unit you cut, so a boundary drawn through the middle of a sentence becomes a visible defect in the product. A chunker written once as a preprocessing script produces this defect quietly. A customer says the cited passage "doesn't say that," and the customer is right, because the sentence that does say it begins in the previous chunk.
Embeddings blur exactly the things people search for
Dense embeddings are trained so that paraphrases land near each other. That is the right property for a question like "how do I get my money back," and the wrong one for "error E-4021," "clause 14.3(b)," a drug name, a part number, or a regulation cited by its number. Those are tokens whose whole value is that they are exact.
The BEIR benchmark, published at NeurIPS in 2021 by Nandan Thakur and colleagues, tested retrieval models across 18 datasets they had not been trained on. The plain keyword method BM25 turned out to be a hard baseline to beat out of domain. Reranking and late-interaction models scored best on average, at a much higher computational price. Embedding models have improved a great deal since 2021, but the pattern shows up in Anthropic's numbers too: contextual embeddings alone cut failures by 35%, and adding a contextual BM25 index took the reduction to 49%. The lexical index was worth another fourteen points of reduction.
The usual way to combine the two rankings is reciprocal rank fusion, introduced by Gordon Cormack, Charles Clarke and Stefan Büttcher in 2009. Each document gets 1/(k + rank) from every list that contains it, with k conventionally set to 60, and the scores are summed. Say document A is first in the keyword list and seventh in the embedding list, while document B is second in both. A scores 1/61 + 1/67, about 0.0313. B scores 2/62, about 0.0323. B wins without ever being first anywhere. The method rewards agreement between two imperfect rankers, which is a sensible thing to reward when you do not know which one is wrong on a given query. It also needs no training data and no score calibration, and Elasticsearch and Azure AI Search both ship it as the built-in way to merge keyword and vector results.
Reranking is bought with latency
The last step in Anthropic's pipeline fetched 150 candidates, passed them through a reranking model, and kept the best 20. Their tests of 5, 10 and 20 chunks in the final context found 20 worked best. A reranker is usually a cross-encoder that reads the query and one candidate together, so its cost grows with the number of candidates, and each call adds to the time before the user sees a first token.
So reranking is a budget line. A batch job that summarizes filings overnight can afford 150 candidates. A support chat with a two-second target probably cannot, and may do better with 30 candidates and a smaller reranker. What a team should not do is add the stage because a blog post said so, or leave it out because nobody measured it. Both mistakes come from the same place: the stage is being treated as a feature to toggle instead of a component with a cost and a measurable effect on recall.
Order and quantity inside the final context matter too. Nelson Liu and colleagues showed in the "Lost in the Middle" paper that accuracy on multi-document question answering follows a U shape: highest when the relevant passage is at the start or end of the context, lower when it is buried in the middle, and this held for models built for long inputs. That gives the context-assembly step real work to do. Placing the best-ranked evidence at the edges, and stopping at the number of chunks that evidence says helps, are parameters to tune against a test set. Padding the prompt with a few extra "just in case" chunks is not a free hedge.
A longer window does not retire the index
The strongest objection is that none of this will matter for long. Context windows now run to a million tokens, and the simplest architecture of all is to paste the corpus into the prompt.
For a small corpus, that objection is right. Anthropic's post says as much: if the knowledge base is under 200,000 tokens, about 500 pages, include all of it in the prompt and skip retrieval. With prompt caching this is cheaper and far simpler than a pipeline, and I would start there for any stable corpus of that size.
Past that size the numbers turn against it. Take an illustrative price of $3 per million input tokens (check your own rate card; this is only to show the shape). A 500,000-token prompt costs $1.50 per query. At 10,000 queries a day that is $15,000 a day. A retrieval pipeline that sends 20 chunks of 800 tokens, 16,000 tokens in all, costs about $0.048 per query, or $480 a day, roughly a thirtieth as much. Caching the static prefix at the up-to-90% discount Anthropic quotes brings the long-prompt option down to around $1,500 a day, which narrows the gap to about three times without closing it. Latency scales with prompt length as well, and the Liu results suggest that the extra tokens are not all being used equally well.
The other problem is permissions. A corpus that every user may read can be pasted into one shared prompt. Most corporate corpora are not like that: the HR folder, the deal room and the engineering wiki have different readers. Putting everything in the window for a user who may see a fraction of it hands access control to the model's good manners. A retrieval layer can filter on permissions before anything reaches the prompt, which is the only arrangement an auditor will accept.
The decisions you cannot cheaply take back
Some choices in a RAG system can be revised in an afternoon: the prompt template, the number of chunks, the reranker. Others touch the whole corpus or the whole product when they change. The table lists the ones I would settle on purpose, early.
| Decision | Why reversing it hurts | What to do early |
|---|---|---|
| Embedding model | Every vector must be recomputed, and scores from different models cannot be compared | Keep raw text and chunk IDs so a re-index is a job, not a project |
| Chunk identity | Citations, user feedback and test labels all point at chunk IDs | Derive IDs from source and position, and keep a mapping when re-chunking |
| Metadata schema | Filters on source, version, date, language and permissions cannot be added to vectors that lack them | Decide the fields before the first ingestion |
| Permission checks | A filter applied after the top-k cut removes results and leaves gaps, or leaks them | Filter inside the retrieval query, by the caller's identity |
| Deletion path | Data protection requests (LGPD, GDPR) reach derived indexes too | Track which chunks came from which record |
| Evaluation set | Without one, every change is a guess | Build it before choosing the vector store |
The last row is the one that makes the rest testable.
Evidence before infrastructure
A useful evaluation set for retrieval is modest in size. A hundred to two hundred real questions, each labeled with the passage or passages that answer it, is enough to compute recall at k, the share of questions for which a correct passage appears in the top k results. Add a handful of questions the corpus cannot answer, labeled as such, so the first failure point has a number too. Separately, score the final answers. Keeping the two numbers apart is what tells you which half of the system to fix, and it maps onto Barnett's list: the first three failure points show up in the retrieval number and the unanswerable questions, the last four in the gap between that number and answer accuracy.
If recall at 20 on your own questions is 85%, polishing the prompt is tuning the part of the system that is not failing. If recall is 98% and answers are still wrong, the opposite holds, and a better chunk format or a stronger generator is where to look. Neither conclusion is available to a team that has no labeled questions.
The first deliverable of a RAG project, then, is a spreadsheet and a script. The spreadsheet holds the questions and the passages that answer them. The script prints recall at 5, 10 and 20 for whatever pipeline is currently running. Choosing the vector database comes after that, and it will be a smaller decision than it looked.
Sources
- Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems 33. arXiv:2005.11401.
- Anthropic (19 September 2024). Introducing Contextual Retrieval. anthropic.com/news/contextual-retrieval.
- Barnett, S., Kurniawan, S., Thudumu, S., Brannelly, Z., Abdelrazek, M. (2024). Seven Failure Points When Engineering a Retrieval Augmented Generation System. 3rd International Conference on AI Engineering (CAIN), Lisbon. arXiv:2401.05856.
- Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., Gurevych, I. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS 2021, Datasets and Benchmarks Track. arXiv:2104.08663.
- Cormack, G. V., Clarke, C. L. A., Büttcher, S. (2009). Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. Proceedings of SIGIR 2009.
- Liu, N. F., Lin, K., Hewitt, J., et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics. arXiv:2307.03172.
- Elastic, Reciprocal rank fusion (RRF) reference, and Microsoft Learn, Relevance scoring in hybrid search using Reciprocal Rank Fusion (Azure AI Search). Product documentation.