Show scriptHide script
A shopping agent recommends a laptop before learning the customer’s budget or intended use. In this example, I’ll show you a tempting fix, why it fails, and the expert workflow we can test instead.
The obvious fix is to instruct the agent: “Always ask about budget and intended use.” That catches missing details, but only if the customer hasn’t already given them.
Now a customer says, “I need a laptop for school, under 800 dollars.” The agent asks for the budget and use again. It has the information, but ignores it.
A sales expert gives the team a better workflow: first identify what the customer has already said. Ask about important missing requirements only when they could change the recommendation. Then explain why the option fits and any relevant trade-offs.
The team turns that workflow into eval cases. In one conversation, the customer says only, “I need a laptop,” so the expected next step is to clarify budget and use. In another, both are already given, so the agent should use them without repeating the questions.
An expert labels the expected behavior and reviews the agent’s full conversations. Then engineers rerun separate cases after a change: does it clarify real gaps, use known details and recommend something suitable?
Coaching engineers to turn expert workflows into evals gives them a way to catch these failures in production—and reuse the process for their next AI product.