Why your chatbot retrieves the right document but still answers wrong

Recall answers one narrow question: did the gold document appear anywhere in the candidate set? Between that and a correct answer there are at least four more gates, and each one leaks.

Multiply five gates at 90% each and you are at 59% end to end. That is roughly the number users experience, and it is why “our recall is great” and “nobody trusts it” coexist so comfortably.

1. The chunk is retrieved but insufficient

A threshold table split across a chunk boundary is the classic case. The model receives “$5,000 — director approval” without the row above it that scoped the rule to capital expenditure. The citation is correct. The answer is wrong. No retrieval metric catches this, because retrieval did its job.

2. A stale duplicate outranks the current version

Corpora accumulate archives. Old policies often use tighter, more distinctive language than their replacements, so they score higher on lexical similarity. Without recency or status metadata in the filter, the archive wins the rank and the current document never makes the cutoff.

3. The system has no way to say “I don't know”

Given weak context, most deployed assistants answer anyway. An abstention path — triggered by a context-sufficiency score rather than by model self-report — converts a confident wrong answer into a survivable one.

The fix is not a better prompt. It is scoring each stage independently so the failure has an address. A minimal per-query record looks like this:

With gold_rank and survived_cutoff recorded, a ranking problem stops looking like a hallucination problem. Aggregate the root_cause field and you have a failure distribution — which is what tells you where to spend the next sprint.

Most teams discover in that fourth step that their “hallucination problem” is a metadata problem. It is cheaper to find out with 50 labelled queries than with a model migration.