Hook 11 · Version A · Client question answered
CTOs: how many AI failures can your current team reproduce?
2026-10-31 · 11:30 Europe/Chisinau
Score 100/100Done
Show postHide post
CTOs: how many AI failures can your current team reproduce?
The useful answer is not a large number. It is whether an engineer can take a reported failure and run it again with enough context to find the cause.
Suppose a field engineer says a RAG assistant gave the wrong pressure limit. Can the team recover the question, the recorded unit, the table it retrieved, the model input, the answer and the expected behavior? Without that trail, a prompt edit may improve the screenshot while leaving the actual cause untouched.
Start with a small sample of failures your team already knows. For each one, ask:
→ Can we reproduce it?
→ Can we tell which stage failed?
→ Can we define what a correct result would be?
→ Can we rerun it after a change?
If the answer is “no” at any step, that is a capability to build before debating which model to buy or whom to hire.
For a CTO, evals are the most important AI capability because they turn vague failures into engineering decisions. Coach the existing team to reproduce, label and investigate cases; that shared method can support a second and third product, not only rescue the first demo.
Our coaching focuses on helping existing engineers practise this process across projects.
Comment “TRACES” if you want the short guide to reproducing AI failures from traces.
P.S. We'll start sending it once at least 10 people ask for it here.
P.P.S. Connect with me so I can send it by DM after that.
Copy post includes bold formatting when the destination supports it. Copy with bold letters uses special Unicode characters; they may be less accessible to screen readers.