Retrieval-augmented generation is the default way to make an AI answer from your own documents, and it is deceptively easy to prototype. A weekend gets you something that demos well. Then you point it at real users and real documents, and the cracks appear. These are the failures we see most often, and what actually fixes them.
Retrieval is the real problem
When a RAG system gives a wrong answer, the model is usually not the culprit. The retrieval step handed it the wrong context. If the right passage never makes it into the prompt, no amount of model quality will save the answer. Most production wins come from fixing retrieval, not swapping models.
The failure modes that bite
The same handful of issues account for most poor answers in production:
- Chunking that splits an idea across boundaries so no chunk is complete
- Stale indexes that answer from documents which have since changed
- No citations, so users cannot tell a real answer from a confident guess
- Missing evaluation, so quality drifts silently over time
Fixes that hold up
Reliable RAG comes from unglamorous engineering: thoughtful chunking tied to document structure, hybrid retrieval that combines keyword and semantic search, a re-ranking pass, and always returning citations so answers are checkable. Then wrap it in evaluation so you catch regressions before your users do.
A RAG answer without a citation is just a confident guess wearing a suit.
Treat it like a product, not a demo
The organisations that get durable value from RAG treat it as a living product with owners, evaluation, and a feedback loop, not a one-off build. The demo earns the meeting. The engineering earns the trust.
