Show scriptHide script
Your LLM judge says your agent did a great job. But would a human expert agree? Here’s how to find out and improve both.
Run your agent on representative requests across its features, and save the interactions and tool results. These are the examples you’ll review.
Ask an expert who knows how to do the job correctly without AI to label those same examples pass or fail and explain why. Compare those labels with the judge’s decisions, then count how often they agree.
For example, a customer asks to cancel an order. The agent says it’s canceled, and the judge passes it, but the tool shows no successful cancellation. The expert marks it fail.
Use that disagreement to improve the judge: require a successful cancellation before it passes the claim. Rerun the reviewed cases, then fix the agent so it handles the action correctly.
Run the improved agent again, collect new representative interactions, and have the judge and expert review them. Repeat the comparison and aim for over 90 percent agreement, while checking missed failures and false alarms separately.
When the judge is stable, freeze it and test on fresh expert-labeled cases kept aside. Review, compare, improve, and retest. Comment JUDGE and I’ll send you the full guide.