Hook 1 · Judge scores
Public reading view · SmartAItomation
Video hook ideas
88 starting points for spoken videos on evals, AI agents and RAG, including openings from your script review. Each hook should promise a lesson the script actually delivers.
Source ideas: Hamel’s LLM-judge guide and your educational reel review. Hypothetical stories are examples, not observed results. Visual cues make the example visible; they are not evidence of a real client outcome.
Judge scores
11 hooks
Hook 4 · Judge scores
Your AI scores 3.5 out of 5. What should your team fix before release?
Hook 5 · Judge scores
What makes an answer a 3 instead of a 4? If nobody agrees, what is that score measuring?
Hook 6 · Judge scores
Your dashboard says “correctness: 4 out of 5.” Can it tell you which customer request failed?
Hook 7 · Judge scores
Two reviewers gave the same answer a 3 and a 4. Here is a clearer question to ask first.
Hook 8 · Judge scores
An average can improve while an important customer task still fails. Here is what to inspect.
Hook 9 · Judge scores
If your judge gives everything a grade, how do you know which answers should never pass?
Hook 59 · Judge scores
AI evals help teams ship agents, but an undefined 1-to-5 score can hide the very failure you need to fix.
On screen: Show a 3 versus 4 beside the same incorrect refund date, then a clear policy criterion.
Hook 62 · Judge scores
Teams want a single AI quality score, but averaging failure types can hide a critical mistake in one customer workflow.
On screen: Show a high overall bar masking a low handoff bar.
Hook 63 · Judge scores
A judge can agree with experts on most examples, but overall accuracy can still hide the failures it misses.
On screen: Show many passing rows and a few critical failures all labeled pass.
Hook 64 · Judge scores
A bigger model can make a stronger judge, but model size cannot fix an unclear definition of success.
On screen: Show two judge models receiving the same missing budget rule.
Judge calibration
1 hook
Hook 2 · Judge calibration
Your LLM judge says your agent did a great job. But would a human expert agree? Here’s how to find out and improve both.
Specialist checks
4 hooks
Hook 3 · Specialist checks
Your LLM judge has to cover lots of different use cases, and one broad check keeps missing important failures. Here’s when specialized judges can help.
Hook 35 · Specialist checks
One broad LLM judge misses three different failure types. When should you add a focused check?
Hook 36 · Specialist checks
Do you need another LLM judge, or can code check this requirement exactly?
Hook 37 · Specialist checks
A specialist judge catches unsupported claims. Who checks whether the user’s whole task was completed?
Expert review
9 hooks
Hook 10 · Expert review
Your judge passed the answer. The support expert failed it. Which one should you trust?
Hook 11 · Expert review
Your support expert knows what a good answer looks like. Does your AI judge know too?
Hook 12 · Expert review
A customer asks to cancel an order. The agent says “done,” but the cancellation never happened.
Hook 13 · Expert review
The judge and your domain expert disagree on a cancellation. Use that disagreement to improve the judge.
Hook 14 · Expert review
Your team changed the judge prompt. Did it fix the disagreement or just memorize the example?
Hook 15 · Expert review
How do you teach an LLM judge the rules your domain expert applies without thinking?
Hook 16 · Expert review
Your agent changed after an eval. Here is why you need to review new interactions again.
Hook 17 · Expert review
You reached 90% judge–expert agreement. Which failures can still hide inside the other 10%?
Hook 18 · Expert review
A judge agrees with your expert on almost everything. Did you check whether it misses the failures?
Datasets
7 hooks
Hook 19 · Datasets
Your eval dataset has 1,000 examples. Does it include a customer with three active orders?
Hook 20 · Datasets
Your order agent works when the order number is present. What happens when the customer leaves it out?
Hook 21 · Datasets
More test rows will not help if they all describe the same easy customer request.
Hook 22 · Datasets
Before generating another 500 test cases, map the features, situations and users your product serves.
Hook 23 · Datasets
The demo customer asks one clear question. Real customers change their minds halfway through.
Hook 24 · Datasets
Your synthetic dataset looks realistic. Does it exercise the cases that matter to your users?
Hook 25 · Datasets
A customer says “change my delivery.” Which missing detail should the agent ask for first?
Agent actions
4 hooks
Hook 26 · Agent actions
Your agent called the right tool with the wrong customer ID. A polished answer will not reveal that.
Hook 27 · Agent actions
Your agent says an order was canceled. What evidence would prove the action actually succeeded?
Hook 28 · Agent actions
The tool returned an error, but the agent told the customer everything worked. Here is the eval to add.
Hook 29 · Agent actions
Your AI agent passes the final-answer test. Does it also pass the action test?
RAG
5 hooks
Hook 30 · RAG
Your RAG assistant has a citation. Is it the current policy, and does it support the answer?
Hook 31 · RAG
The assistant found the right document but answered from an old paragraph. Where did the failure start?
Hook 32 · RAG
A customer asks a question your knowledge base cannot answer. Does your assistant admit the gap?
Hook 33 · RAG
Your answer cites a real source. What if that source says 30 days and the answer says 60?
Hook 34 · RAG
Your retrieval test passes, but the final answer ignores the evidence. Which eval catches that?
Engineering teams
3 hooks
Hook 38 · Engineering teams
Your engineering team has an impressive AI demo. How will it find the failures before customers do?
Hook 39 · Engineering teams
CTOs: can your team turn one user complaint into a test it can rerun on the next AI product?
Hook 40 · Engineering teams
An AI workshop should leave your engineers with a repeatable eval process, not just a better demo.
Product recommendations
5 hooks
Hook 41 · Product recommendations
A shopping agent recommends a laptop before learning the customer’s budget or intended use. In this example, I’ll show you a tempting fix, why it fails, and the expert workflow we can test instead.
On screen: Show the customer request, the premature laptop recommendation and two empty fields: Budget and Intended use.
Hook 42 · Product recommendations
A shopping agent asks ten questions before recommending one product. I’ll show you why limiting questions alone can fail, and how to test a better expert-defined workflow.
On screen: Show ten question bubbles, two already answered, and one important missing requirement highlighted.
Hook 43 · Product recommendations
A customer reports a delayed order, and the agent pitches a premium subscription before helping. Here’s the expert workflow that fixes the sequence—and how to test it.
On screen: Show a delayed-order complaint, a premature upgrade card, then a resolution-before-offer sequence.
Hook 44 · Product recommendations
A customer wants a waterproof camera for a child, but the agent checks only waterproofing. Let’s test the questions and trade-offs a suitable recommendation needs.
On screen: Show a camera card checked for waterproofing and an unanswered child-suitability requirement.
Hook 58 · Product recommendations
A customer asks whether an upgrade is necessary, and the agent lists benefits without answering. Here’s how to test a direct answer and a relevant offer.
On screen: Show the customer’s yes-or-no question beside a generic upgrade pitch, then a direct answer and optional offer.
Order changes and support
6 hooks
Hook 45 · Order changes and support
An agent cancels an order without identifying which one or discussing its shipped status. Here’s how to test the clarification and confirmation steps before cancellation.
On screen: Show two orders, one marked shipped, and the missing selection and confirmation steps.
Hook 46 · Order changes and support
A vague “Are you sure?” leads an agent to cancel the wrong order in this example. Let’s test what a specific confirmation needs to include.
On screen: Show two orders and a vague Yes branching to the wrong order, then a named-order confirmation.
Hook 47 · Order changes and support
An agent quotes the refund policy without checking the purchase date it depends on. Here’s a test for the questions and policy decisions it skips.
On screen: Show a refund policy timeline and a blank purchase-date field beside the reply.
Hook 48 · Order changes and support
A customer asks twice for a person, but the agent repeats its troubleshooting script. Here’s how to test when it should escalate and what it should hand over.
On screen: Show the two human-agent requests followed by a repeated troubleshooting bubble and an escalation path.
Hook 49 · Order changes and support
An agent escalates a complaint but leaves out everything the customer already tried. Let’s build a test for a handoff that lets the next person continue.
On screen: Show an empty handoff card contrasted with a summary of the issue, prior steps and unresolved question.
Hook 57 · Order changes and support
A customer says, “Wait, keep the order,” but the agent acts on the earlier cancellation request. Here’s how to test changing intent before an action.
On screen: Show the customer reversing the request and the earlier cancellation path stopping before confirmation.
Booking and scheduling
2 hooks
Hook 50 · Booking and scheduling
A customer says Tuesday might work, and the agent books before confirming the time or fee. Here’s the booking sequence we can test.
On screen: Show tentative Tuesday, missing time and fee, and a booking confirmation step.
Hook 51 · Booking and scheduling
Friday is unavailable, so an agent offers another location despite the customer’s travel restriction. Here’s how to test alternatives against the customer’s constraints.
On screen: Show a fixed location requirement and an invalid distant slot, then a prompt asking which constraint is flexible.
Business workflows and agents
2 hooks
Hook 52 · Business workflows and agents
An agent fills every application field but submits two contradictory answers. Here’s how to test the consistency check before submission.
On screen: Show two completed fields with conflicting answers and a pre-submit review checkpoint.
Hook 53 · Business workflows and agents
An agent drafts a service proposal before checking whether the service is available in the customer’s region. Here’s how to test the prerequisite first.
On screen: Show a proposal document blocked by a region-availability check.
Expert-defined evaluations
2 hooks
Hook 54 · Expert-defined evaluations
A product description is correct, but the agent never checks the buyer’s budget or use. Here’s how an expert can make the eval test the full recommendation workflow.
On screen: Show an accurate product description beside missing budget and intended-use requirements.
Hook 55 · Expert-defined evaluations
An agent follows last month’s cancellation workflow because its evals still check the old policy. Here’s how to update the cases and catch regressions.
On screen: Show old and current policy cards, then a revised test case and regression checks.
Different users and context
1 hook
Hook 56 · Different users and context
A new employee asks for step-by-step help and receives the same unexplained shorthand used for experts. Here’s how to test whether the response fits the user.
On screen: Show a new-user question beside jargon-heavy output, then an explanatory step-by-step response.
Synthetic datasets
4 hooks
Hook 60 · Synthetic datasets
Teams know AI agents need test data, but synthetic questions generated only from documents can miss how real customers actually ask.
On screen: Show a policy document generating similar questions, then a matrix of user situations.
Hook 65 · Synthetic datasets
Synthetic datasets can expand test coverage, but asking an LLM for random test questions often creates the same easy case repeatedly.
On screen: Show near-duplicate rows turning into a feature-by-scenario matrix.
Hook 66 · Synthetic datasets
Generating synthetic test inputs is useful, but generating the assistant’s answer too can mean you never test your actual agent.
On screen: Show synthetic question leading to a fictional answer, then reroute it through the real agent.
Hook 67 · Synthetic datasets
Document-based test generation is convenient, but the right source material is only one part of a useful eval dataset.
On screen: Show one clean policy branching into normal, missing and conflicting cases.
RAG quality
1 hook
Hook 61 · RAG quality
Teams use faithfulness and correctness scores for RAG, but a high generic score does not prove the assistant completed the user’s job.
On screen: Show a high score next to an archived contract and a current version.
Production traces
3 hooks
Hook 68 · Production traces
Teams know they need production feedback, but looking only at complaints misses users who quietly leave.
On screen: Show complaint tickets beside silent abandoned sessions entering review.
Hook 69 · Production traces
Capturing every AI trace is useful, but a giant log no one reviews will not tell the team what to fix.
On screen: Show a long trace with the first failed step highlighted.
Hook 70 · Production traces
Teams keep a gold eval set to prevent regressions, but a frozen dataset can become outdated as the product and policies change.
On screen: Show old and new policies with versioned expected outcomes.
Domain expert workflows
3 hooks
Hook 71 · Domain expert workflows
Engineers can write excellent code, but they cannot infer every business decision a support agent needs to make.
On screen: Show customer request branching through identify, explain and confirm.
Hook 72 · Domain expert workflows
A domain expert can tell you an answer is wrong, but a bare fail label may not tell engineers which rule to fix.
On screen: Show a red fail label expanding into budget, software and evidence reasons.
Hook 73 · Domain expert workflows
A checklist can define an agent’s workflow, but forcing every step in every conversation can make the experience worse.
On screen: Show a checklist asking already-answered questions, then conditional steps.
RAG retrieval
4 hooks
Hook 74 · RAG retrieval
Teams know RAG needs a retrieval system, but an answer-quality score alone cannot tell you whether retrieval found the right evidence.
On screen: Show question → old page → wrong answer and highlight the first failure.
Hook 75 · RAG retrieval
A RAG answer can contain citations, but a citation does not guarantee that the source supports the claim.
On screen: Show a claim and a cited passage with different policy topics.
Hook 76 · RAG retrieval
The retriever can find the correct document, but that does not mean it found the right section for this user.
On screen: Show one document with a standard clause and regional exception.
Hook 77 · RAG retrieval
RAG can make answers grounded in documents, but grounding in an outdated document can still produce a confidently wrong answer.
On screen: Show a green grounded badge on an archived policy beside a newer one.
Agent workflows
5 hooks
Hook 78 · Agent workflows
An AI agent can call the correct tool, but the right tool with the wrong account can still harm the customer.
On screen: Show correct tool icon with a mismatched account identifier.
Hook 79 · Agent workflows
Agents need multi-step tests, but testing every tool call in isolation can miss whether the customer’s job was completed.
On screen: Show green tool steps leading to an unconfirmed appointment.
Hook 80 · Agent workflows
A final answer can look correct, but an agent’s intermediate actions may violate the workflow that makes the answer safe.
On screen: Show cancellation occurring before the confirmation step.
Hook 81 · Agent workflows
Agents should ask clarifying questions, but asking the same question twice can be a real workflow failure.
On screen: Show highlighted customer details and duplicated agent questions.
Hook 82 · Agent workflows
Escalating to a human can protect customers, but a handoff without context can force the customer to start over.
On screen: Show an empty transfer card transformed into a concise handoff summary.
Judge validation
5 hooks
Hook 83 · Judge validation
An LLM judge can save review time, but a persuasive explanation does not prove its decision agrees with experts.
On screen: Show a polished judge explanation beside the ignored order identifier.
Hook 84 · Judge validation
Giving an LLM judge more context seems helpful, but a long trace can distract it from the evidence needed for one decision.
On screen: Show a long trace narrowed to customer budget and recommendation evidence.
Hook 85 · Judge validation
A judge threshold turns a score into a decision, but the default cutoff may miss the failures your team cares about.
On screen: Show scores below 0.5 and a lower threshold separating cases.
Hook 86 · Judge validation
A cheaper judge may reduce eval cost, but it is only a saving if it catches enough of the failures that matter.
On screen: Show cost beside separate missed-critical-failure counts.
Hook 87 · Judge validation
Tuning an LLM judge against expert examples can improve it, but testing on the same examples can overstate how well it generalizes.
On screen: Show labeled examples split into rubric development and protected final check.
Production monitoring
1 hook
Hook 88 · Production monitoring
Passing evals before release matters, but a fixed test suite cannot represent every way customers will use the product later.
On screen: Show release test set alongside new production cases flowing into review.