Why grounded answers still go wrong

Search and Retrieval, lesson 12

Missed passages, conflicting sources, and claims the text doesn't support.

"Measuring search with recall, MRR and NDCG" and "Checking answers against sources": scores for the search and for the answer. This lesson is a field guide to what's behind those scores when they drop.

Part 1. Find which half broke

A user reports a wrong answer. The tempting move is to change the prompt or try a bigger model. First, answer one question: was the passage with the right answer in the prompt?

Example 1.1

Triage for one bad answer

bad answer→look at the retrieved chunks
Every RAG system should log which chunks went into each prompt. Without that log, you're guessing.
right chunk missing→retrieval miss
If the answer wasn't in the prompt, no prompt change can fix it. Look at search, chunking and filters.
right chunk present→generation problem
If it was there, the model misread it, mixed it with something else, or added to it. Look at the context and the prompt.

Your turn

The log shows the enterprise refund chunk at rank 2 in the prompt, and the answer still says 30 days. Where do you look?

Retrieval did its job, so the search knobs aren't the problem. Something on the generation side went wrong, most likely the annual-plan chunk sitting next to it. That's context confusion, coming up in this lesson.

Show the answer

D. Retrieval did its job, so the search knobs aren't the problem. Something on the generation side went wrong, most likely the annual-plan chunk sitting next to it. That's context confusion, coming up in this lesson.

Sorting failures into two piles, in Python

cases = [
    {"q": "enterprise refund window", "gold": "p2", "retrieved": ["p1", "p5"], "faithful": True},
    {"q": "train ticket claims", "gold": "p5", "retrieved": ["p5", "p1"], "faithful": False},
    {"q": "laptop replacement", "gold": "p4", "retrieved": ["p4", "p3"], "faithful": True},
    {"q": "CEO home address", "gold": None, "retrieved": ["p3", "p8"], "faithful": False},
]

def diagnose(case):
    if case["gold"] and case["gold"] not in case["retrieved"]:
        return "retrieval miss"
    if not case["faithful"]:
        return "answer not supported by context"
    return "ok"

for case in cases:
    print(f'{case["q"]}: {diagnose(case)}')
# enterprise refund window: retrieval miss
# train ticket claims: answer not supported by context
# laptop replacement: ok
# CEO home address: answer not supported by context

The last case has no right passage at all. Nothing supports any answer, so the system should have said it didn't know.

In practice

Barnett and colleagues (2024) collected the failures of three RAG systems they built and found seven recurring points: the content wasn't in the documents at all, the right passage ranked too low, it was cut when the prompt was assembled, the model didn't pull the answer out of it, the answer came in the wrong format, it was too vague or too specific, or it was incomplete. Each one lands in one of the two piles above, which is why the first question is always whether the passage was in the prompt.

Part 2. Retrieval misses

Most retrieval misses fall into a few patterns, and each one points back to an earlier lesson. The user says "train ticket" and the handbook says "travel expenses" (a wording gap). The fact was cut away from its subject ("Chunk size and overlap"). The page was updated but not re-indexed (a stale index). A filter was stricter than the question needed ("Filters and metadata"). Or the answer needs two passages, and only one came back.

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Retrieval misses
Context confusion
Hallucination from context

See Pro plans I have Pro: sign in

Next lessonKnowledge graphs and GraphRAGStore the links between facts, and know when a graph beats vectors. Start lesson 13

Search and Retrieval overviewSpotted a mistake? Tell us.

12 · Why grounded answers still go wrong