Hook 40 · Version B · Client question answered
We helped a UK startup team improve its reported RAG accuracy from 65% to 96% after a two-day workshop.
Not scheduled Europe/Chisinau
Score 100/100Alternative
Show postHide post
We helped a UK startup team improve its reported RAG accuracy from 65% to 96% after a two-day workshop.
“Can our engineers get the same result?” There is no honest way to promise that from one engagement. The metric definition, dataset and causal breakdown for that owner-supplied case are not available here.
What your team can do is build a comparison it trusts.
Hypothetical procurement RAG example: a parts agent retrieves the current stock record and correctly says three units are available, but omits the remaining order shortfall. Retrieval succeeded; task completeness failed. A useful eval records the expected source and both required answers, then scores evidence coverage and response completeness separately.
If you automate that review with a model judge, ask domain experts to label passing and failing examples first. Inspect false passes, choose any score threshold on development cases, and leave untouched cases for the final check.
After one focused improvement, rerun the same set and inspect what got worse. Keep sampling new traces because a curated offline set will not reveal every production failure.
The most important part of production RAG is a team-owned eval loop. Engineers need to label hard cases, calibrate the judge and revisit false passes before users find them. That capability can serve several AI products, not just one demo.
Comment “RAG” if you want the short guide to separating retrieval from answer quality.
P.S. We'll start sending it once at least 10 people ask for it here.
P.P.S. Connect with me so I can send it by DM after that.
Copy post includes bold formatting when the destination supports it. Copy with bold letters uses special Unicode characters; they may be less accessible to screen readers.