A retrieval-augmented generation (RAG) prototype can be built in an afternoon: chunk some documents, embed them, put them in a vector store, and pass the top results to a language model. The demo usually looks great. The trouble starts when real users ask real questions.
These are the areas we work through before calling a RAG system production-ready.
1. Chunking is a product decision
How you split documents decides what the model can see. Fixed-size chunks are a fine starting point, but they often cut a table in half or separate a heading from the paragraph it explains.
- Split on the document's own structure — headings, sections, list items — before falling back to size limits.
- Keep useful metadata with every chunk: source, section title, date, access level.
- Store a small amount of surrounding context so an answer can be traced back to its place in the original.
2. Retrieval needs its own evaluation
If the right passage never reaches the model, no amount of prompt engineering will fix the answer. Measure retrieval separately from generation:
- Build a set of real questions with the passages that should be retrieved.
- Track whether those passages appear in the top results.
- Compare plain vector search with hybrid search (vector + keyword) and re-ranking. Keyword matching still matters for product codes, names and exact phrases.
3. Answers must show their sources
Users trust answers they can check. Every response should cite the passages it used, and the interface should make those citations easy to open. When the retrieved context doesn't contain the answer, the system should say so rather than guess.
4. Permissions belong in retrieval, not in the prompt
If some users shouldn't see certain documents, filter them out before retrieval. Asking a model to "ignore documents the user can't access" is not access control.
5. Plan for change
Documents get updated, models get replaced, and embeddings drift. Keep ingestion repeatable, version your indexes, and re-run your evaluation set whenever a component changes.
RAG works best when it's treated as a search problem first and a generation problem second. Get retrieval right, measure it, and the language model has a much easier job.