RAG in Production: Retrieval Is the Hard Part

RAG got popular because it's the cheapest way to make an LLM talk about your data without fine-tuning: retrieve relevant snippets, stuff them into the prompt, and let the model answer from them. The demo takes an afternoon. The production version takes a lot longer, and almost every problem traces back to one place — retrieval.

Chunking is a real decision, not a default

Splitting your documents into fixed 500-token blocks is the number one reason RAG feels dumb. You slice a table in half, orphan a heading from its paragraph, and then wonder why the model can't answer. Chunk on semantic boundaries — sections, paragraphs, list items — and keep a little overlap so context doesn't fall off a cliff.

Embeddings alone aren't search

Pure vector similarity is great at 'find me something about this vibe' and surprisingly bad at exact matches — product codes, names, error strings. Combine it with old-fashioned keyword search (BM25) and you cover both. Then run the candidates through a reranker so the best three land at the top of the prompt, not buried at position nine.

Ground it, or it drifts

Tell the model to answer only from the retrieved context and to say 'I don't know' when the context doesn't cover the question. Ask it to cite which snippet each claim came from. This is the cheapest hallucination-reduction technique there is, and it doubles as a debugging tool when an answer looks off.

Evaluate retrieval separately

Before you touch the prompt, measure whether the right chunk even made it into the top-k. If recall is bad, no amount of prompt-wrangling saves you. Build a small set of question-to-source pairs and track retrieval quality like the metric it is.

The takeaway

When a RAG answer is wrong, resist the urge to rewrite the system prompt. Nine times out of ten the fix is upstream: better chunks, hybrid search, a reranker. The model is usually fine — it just never saw the right paragraph.