Shipping Your First RAG Feature Without Regret

Every company I have worked with in the last two years has wanted the same feature: ask questions of our own documents. Retrieval-augmented generation is the sensible way to build it, and it is also where teams burn a quarter learning the same three lessons.

Chunking is a product decision

How you split documents determines what the model can ever cite. Fixed 500-token windows are easy and wrong for structured content — they cut tables in half and orphan headings from the text beneath them. Split on the document's own structure first, then pack to a size budget.

Build the evaluation set before the pipeline

Thirty real questions with known-good answers, written by someone from the business, is worth more than any benchmark. Without it you cannot tell whether last week's prompt change helped or quietly broke citation accuracy.

• Measure retrieval and generation separately — most failures are retrieval
• Log the retrieved chunks with every answer, in production, from day one
• Track refusal rate; a system that never says 'not in the documents' is lying somewhere

A confident wrong answer costs more trust than ten honest refusals.

Know what not to build

If the answer must be exactly right every time — pricing, compliance, medical dosing — do not put a language model between the user and the source of truth. Use retrieval to find the document and show it. Generation is for synthesis, not for arithmetic.