A policy assistant that retrieved the right document and still gave the wrong answer
Sample structure showing how findings, changes, and before/after metrics are presented in a published Retridge case study.
A mid-size lender with an internal assistant over policy documents, procedures, and product terms. Around 1,200 employees; roughly 400 daily queries at peak.
“It cites the right policy and then gets the number wrong.” Operations staff had stopped trusting it for approval thresholds — the exact task it was built for.
78 real queries from support logs, scored on recall@5, ranking position of the gold chunk, groundedness, completeness, plus 40 security probes and full token accounting.
Superseded versions had no recency metadata and scored higher on lexical overlap.
Threshold tables were cut mid-row, so the model saw a figure without its condition.
With weak context the system guessed instead of saying it could not answer.
Quote slot — filled with the client's own words once a case study is approved for publication. Retridge does not draft client quotes.
Role · Company (or “Head of Engineering, anonymized”)