Module 9 of 11 · Expert · 10 min read

RAG and grounding with sources

Answer from your documents, cite them, and say 'not found' instead of guessing.

Key takeaways

1Retrieval-augmented generation (Lewis et al., 2020) retrieves relevant text, then generates from it.
2Retrieval quality usually matters more than prompt wording.
3Give sources IDs, require citations by ID, and allow 'not in the sources'.
4Check groundedness: does each claim trace to a retrieved passage?

01What RAG is

Retrieval-augmented generation, introduced by Lewis et al. (2020), combines a retriever with a generator: for each question, find relevant passages in a document collection, put them in the prompt, and have the model answer from them. It is how most chat-with-your-documents tools, help-centre bots and research assistants work.

The benefits are fresher and more specific knowledge than the model's training data, and answers you can check against sources. The AI Engineer path covers building retrieval systems; this module focuses on prompting them.

02The grounding prompt

Make sources addressable and make citing them the rule.

Example

Answer the question using only the sources below. Each source has an id. <sources> <source id="S1" title="{{title}}">{{text}}</source> <source id="S2" title="{{title}}">{{text}}</source> </sources> Rules: - After each claim, cite the supporting source id in brackets, e.g. [S2]. - If the sources do not contain the answer, say 'The sources don't cover this' and stop. Do not use outside knowledge. - If sources conflict, say so and cite both. Question: {{question}}

03Quote first, then answer

For high-stakes answers, ask the model to extract the exact quotes that answer the question before it writes its response. Quotes are easy to verify mechanically (does the string appear in the source?), and answering from quotes keeps the response close to what the sources actually say.

04Retrieval is the real bottleneck

When answers are wrong, check what was retrieved before changing the prompt.

If the right passage isn't retrieved, no prompt can save the answer.
Rewrite vague user questions into good search queries before retrieving.
Retrieve a few more passages than you need, then let the model or a reranker pick.
Chunk documents along natural boundaries such as sections and paragraphs, and keep titles with chunks.
Log questions answered with 'not covered' to find gaps in your documents.

05Checking groundedness

A grounded answer is one where every claim is supported by a cited passage. Check it with a second prompt that reviews each sentence against its cited sources, or with a programmatic check that quoted text appears verbatim. The share of answers that pass is one of the most useful metrics for any RAG system.

Watch out

Citations can be hallucinated too: a real source id attached to a claim it doesn't support. Check the claim against the passage, not just that a citation exists.