Public reading view · SmartAItomation

Video scripts & schedule

Short educational videos about AI evals, agents and RAG. Each draft teaches one concrete lesson. Scripts and schedules are for review and manual posting.

The three initial scripts come from your video review; examples and scores are illustrative. Guide delivery is manual and is not configured automatically.

Unlock editing on this device to schedule, mark done or comment.

0/35marked done

Scores are your own editorial ratings, not predicted performance. New drafts have no score.

Video 1 · Review draft · earlier scores video rendered

Why vague judge scores are hard to use

Hook: Your AI evals show 3.5 out of five for correctness, and four out of five for faithfulness. Is your assistant ready to deploy?

UnscoredPlanned
Show scriptHide script
Your AI evals show 3.5 out of five for correctness, and four out of five for faithfulness. Is your assistant ready to deploy? These scores are hard to use. What makes an answer a three instead of a four? Domain experts have to define every level, and the LLM judge has to apply them consistently. Start with one simpler question: did the assistant achieve what the user needed? Have a domain expert mark pass or fail and explain why. Imagine a user asks an AI query assistant to find the slowest trace. The generated query returns the maximum duration, but leaves out the trace ID. The expert marks fail: the user gets a number, not the actual trace to investigate. That critique makes the missing requirement clear. Use reviewed examples to teach a judge: read the user request and query, give a pass or fail, and explain the decision. Then compare it with expert labels on different cases. Check which failures the judge misses and which good answers it rejects. Fix disagreements, then test on fresh held-out cases. Now you know what your system gets wrong and why. Comment RUBRIC and I’ll send you the full guide.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 2 · Review draft · not recorded

Improve your judge with expert feedback

Hook: Your LLM judge says your agent did a great job. But would a human expert agree? Here’s how to find out and improve both.

UnscoredPlanned
Show scriptHide script
Your LLM judge says your agent did a great job. But would a human expert agree? Here’s how to find out and improve both. Run your agent on representative requests across its features, and save the interactions and tool results. These are the examples you’ll review. Ask an expert who knows how to do the job correctly without AI to label those same examples pass or fail and explain why. Compare those labels with the judge’s decisions, then count how often they agree. For example, a customer asks to cancel an order. The agent says it’s canceled, and the judge passes it, but the tool shows no successful cancellation. The expert marks it fail. Use that disagreement to improve the judge: require a successful cancellation before it passes the claim. Rerun the reviewed cases, then fix the agent so it handles the action correctly. Run the improved agent again, collect new representative interactions, and have the judge and expert review them. Repeat the comparison and aim for over 90 percent agreement, while checking missed failures and false alarms separately. When the judge is stable, freeze it and test on fresh expert-labeled cases kept aside. Review, compare, improve, and retest. Comment JUDGE and I’ll send you the full guide.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 3 · Review draft · not recorded

When to add a specialist judge

Hook: Your LLM judge has to cover lots of different use cases, and one broad check keeps missing important failures. Here’s when specialized judges can help.

UnscoredPlanned
Show scriptHide script
Your LLM judge has to cover lots of different use cases, and one broad check keeps missing important failures. Here’s when specialized judges can help. Start by having a domain expert review representative outputs. Group mistakes by type, such as unsupported claims, incorrect actions, or missing information. Prioritize recurring or high-impact failures. You don’t need a separate judge for every scenario; focus on checks that add useful coverage. For each group, choose a suitable check. Code can verify a document exists; a focused LLM judge can compare a claim with the source when meaning needs interpretation. For example, a citation exists, but the answer says 60 days while the policy says 30. The focused check should fail and show the conflicting evidence. Validate each new judge against separate expert-reviewed examples, including passes and failures. Check missed failures and false alarms, then fix the product and retest. Keep overall task review alongside targeted checks, and add more only when the errors justify it. Comment CHECKS and I’ll send you the full guide.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 4 · Review draft · hypothetical example · not recorded

Ask before recommending a laptop

Hook: A shopping agent recommends a laptop before learning the customer’s budget or intended use. In this example, I’ll show you a tempting fix, why it fails, and the expert workflow we can test instead.

UnscoredPlanned
Show scriptHide script
A shopping agent recommends a laptop before learning the customer’s budget or intended use. In this example, I’ll show you a tempting fix, why it fails, and the expert workflow we can test instead. The obvious fix is to instruct the agent: “Always ask about budget and intended use.” That catches missing details, but only if the customer hasn’t already given them. Now a customer says, “I need a laptop for school, under 800 dollars.” The agent asks for the budget and use again. It has the information, but ignores it. A sales expert gives the team a better workflow: first identify what the customer has already said. Ask about important missing requirements only when they could change the recommendation. Then explain why the option fits and any relevant trade-offs. The team turns that workflow into eval cases. In one conversation, the customer says only, “I need a laptop,” so the expected next step is to clarify budget and use. In another, both are already given, so the agent should use them without repeating the questions. An expert labels the expected behavior and reviews the agent’s full conversations. Then engineers rerun separate cases after a change: does it clarify real gaps, use known details and recommend something suitable? Coaching engineers to turn expert workflows into evals gives them a way to catch these failures in production—and reuse the process for their next AI product.
Unlock scheduling & Done

Hook details

Video 5 · Review draft · hypothetical example · not recorded

Ask enough questions—not ten

Hook: A shopping agent asks ten questions before recommending one product. I’ll show you why limiting questions alone can fail, and how to test a better expert-defined workflow.

UnscoredPlanned
Show scriptHide script
A shopping agent asks ten questions before recommending one product. I’ll show you why limiting questions alone can fail, and how to test a better expert-defined workflow. The first fix sounds simple: tell the agent to ask no more than two questions. That reduces the count. But it doesn’t guarantee they’re the right questions. Imagine a student already gave a budget and said the laptop must fit in a backpack. The agent uses its two questions to ask for those details again, then recommends a laptop without checking required software. In this example, that software determines which laptop will work. So we try another instruction: “Don’t repeat questions.” Better—but the agent can still skip an essential question. Avoiding repetition is only part of the job. A sales expert defines the full workflow: use what the customer already said, identify missing details that could change the choice, and clarify important gaps or conflicts. When there’s enough information, recommend a suitable product and explain the trade-offs. The team builds eval conversations around that workflow. If the customer gave everything needed, proceed without repeated questions. If required software is missing, ask. If requirements conflict, clarify before choosing. An expert labels examples, and engineers review the full conversation on separate cases after each change. That shows both unnecessary questions and missed requirements. Coach your engineers to build these evals with experts. They can use the same method on the next AI product too.
Unlock scheduling & Done

Hook details

Video 6 · Review draft · hypothetical example · not recorded

AI evals help teams ship agents

Hook: AI evals help teams ship agents, but an undefined 1-to-5 score can hide the very failure you need to fix.

UnscoredPlanned
Show scriptHide script
AI evals help teams ship agents, but an undefined 1-to-5 score can hide the very failure you need to fix. Consider this hypothetical example: A support agent gives a wrong refund date, and one reviewer calls it a three while another calls it a four. Neither number explains the customer harm. The tempting fix is to add more scoring dimensions. That gives the dashboard more numbers without defining what they mean. A better approach: Ask a domain expert whether this specific answer follows the current refund rule, record pass or fail with a reason, then compare a judge against separately labeled cases. A well-defined scale can still help in other tasks; the problem is an uncalibrated one.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 7 · Review draft · hypothetical example · not recorded

Teams know AI agents need test data

Hook: Teams know AI agents need test data, but synthetic questions generated only from documents can miss how real customers actually ask.

UnscoredPlanned
Show scriptHide script
Teams know AI agents need test data, but synthetic questions generated only from documents can miss how real customers actually ask. Consider this hypothetical example: Imagine a returns assistant tested only with polished questions paraphrased from the policy. Real shoppers mention damaged packaging, missing receipts and frustration in the same message. Generating ten times more generic questions will repeat the same blind spot. A larger set is not automatically more representative. A better approach: Review privacy-appropriate real interactions, map features, situations and user types with experts, then generate diverse user inputs to fill known gaps. Run those inputs through the actual agent and review its behavior.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 8 · Review draft · hypothetical example · not recorded

Teams use faithfulness and correctness scores for RAG

Hook: Teams use faithfulness and correctness scores for RAG, but a high generic score does not prove the assistant completed the user’s job.

UnscoredPlanned
Show scriptHide script
Teams use faithfulness and correctness scores for RAG, but a high generic score does not prove the assistant completed the user’s job. Consider this hypothetical example: An employee asks which current contract they can use. The answer faithfully summarizes an obsolete contract and receives a reassuring score. A generic judge that sees only the answer and retrieved text can miss that the source is outdated. A fluent citation can make the error look safer than it is. A better approach: Define the actual task with the business expert: identify the authoritative current contract, apply it to this user, and explain any missing information. Test those checks against expert-reviewed examples.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 9 · Review draft · hypothetical example · not recorded

Teams want a single AI quality score

Hook: Teams want a single AI quality score, but averaging failure types can hide a critical mistake in one customer workflow.

UnscoredPlanned
Show scriptHide script
Teams want a single AI quality score, but averaging failure types can hide a critical mistake in one customer workflow. Consider this hypothetical example: A booking assistant handles common questions but mishandles requests to speak with a human. Its overall average stays high because those requests are less frequent. A prettier dashboard or a higher overall threshold still hides which users are affected. The business needs to see the failure category. A better approach: Label handoff cases separately, inspect false passes and report results by important workflow and severity. Use the aggregate as context, not as the only launch decision.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 10 · Review draft · hypothetical example · not recorded

A judge can agree with experts on most examples

Hook: A judge can agree with experts on most examples, but overall accuracy can still hide the failures it misses.

UnscoredPlanned
Show scriptHide script
A judge can agree with experts on most examples, but overall accuracy can still hide the failures it misses. Consider this hypothetical example: Suppose most test conversations pass, while a small number contain unsafe order changes. A judge that passes everything looks accurate on the imbalanced set. Adding up correct labels alone does not reveal missed bad cases. The team may trust a judge that never catches the failure it was built for. A better approach: Compare judge and expert labels by error type, inspect false passes and false alarms, and report recall and precision where appropriate. Keep separate cases for a final check.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 11 · Review draft · hypothetical example · not recorded

A bigger model can make a stronger judge

Hook: A bigger model can make a stronger judge, but model size cannot fix an unclear definition of success.

UnscoredPlanned
Show scriptHide script
A bigger model can make a stronger judge, but model size cannot fix an unclear definition of success. Consider this hypothetical example: A judge reviews product recommendations without knowing whether the customer’s budget is mandatory. It scores an expensive recommendation highly because the explanation sounds good. Switching to a larger judge before defining the requirement leaves it guessing more confidently. A model comparison cannot replace a rubric. A better approach: Ask a sales expert to define required constraints and examples of acceptable trade-offs. Test candidate judges against expert labels, then choose a model that meets the team’s quality and cost needs.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 12 · Review draft · hypothetical example · not recorded

Synthetic datasets can expand test coverage

Hook: Synthetic datasets can expand test coverage, but asking an LLM for random test questions often creates the same easy case repeatedly.

UnscoredPlanned
Show scriptHide script
Synthetic datasets can expand test coverage, but asking an LLM for random test questions often creates the same easy case repeatedly. Consider this hypothetical example: For a shopping agent, a generated list contains many versions of “recommend a laptop” but no conflicting budgets, angry users or accessibility needs. More paraphrases feel like variety, but they may exercise the same decision path. That gives the team little evidence about other behaviors. A better approach: Map product features, customer situations and personas, then intentionally combine them into test inputs. Run the real system, inspect its traces and have experts review the cases that matter.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 13 · Review draft · hypothetical example · not recorded

Generating synthetic test inputs is useful

Hook: Generating synthetic test inputs is useful, but generating the assistant’s answer too can mean you never test your actual agent.

UnscoredPlanned
Show scriptHide script
Generating synthetic test inputs is useful, but generating the assistant’s answer too can mean you never test your actual agent. Consider this hypothetical example: A team asks a model to invent both customer questions and polished assistant answers, then judges those answers without running its deployed workflow. The resulting score describes fictional outputs, not the current retrieval, tools or conversation logic. It cannot tell engineers where their system fails. A better approach: Generate or collect the user inputs, feed them through the real agent, record the actual trace and review its behavior. Label illustrative cases separately from real production interactions.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 14 · Review draft · hypothetical example · not recorded

Document-based test generation is convenient

Hook: Document-based test generation is convenient, but the right source material is only one part of a useful eval dataset.

UnscoredPlanned
Show scriptHide script
Document-based test generation is convenient, but the right source material is only one part of a useful eval dataset. Consider this hypothetical example: An HR assistant’s generated tests all cover policies that exist in the knowledge base. None asks what to do when the answer is missing or two policies conflict. Scoring only questions with neat source passages misses uncertainty, missing evidence and conflicting documents. More of the same cases will not cover those situations. A better approach: Add expert-reviewed scenarios for missing, stale and conflicting evidence alongside normal requests. Check whether the assistant asks, abstains or escalates according to the workflow.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 15 · Review draft · hypothetical example · not recorded

Teams know they need production feedback

Hook: Teams know they need production feedback, but looking only at complaints misses users who quietly leave.

UnscoredPlanned
Show scriptHide script
Teams know they need production feedback, but looking only at complaints misses users who quietly leave. Consider this hypothetical example: A product team reviews support tickets and fixes the loudest issue. It never inspects conversations where users abandoned the recommendation flow. Waiting for complaints biases the dataset toward people who report problems. Silence is not a passing label. A better approach: With appropriate privacy controls, sample a mix of production traces, including abandoned sessions and common successful journeys. Have experts identify failures and turn representative examples into regressions.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 16 · Review draft · hypothetical example · not recorded

Capturing every AI trace is useful

Hook: Capturing every AI trace is useful, but a giant log no one reviews will not tell the team what to fix.

UnscoredPlanned
Show scriptHide script
Capturing every AI trace is useful, but a giant log no one reviews will not tell the team what to fix. Consider this hypothetical example: An agent produces long conversations with retrieval results, tool calls and customer replies. The team stores all of them but only reads final answers. Adding more telemetry without a review process can leave the failure buried. The final answer may hide the first wrong step. A better approach: Make traces reviewable, sample representative sessions and ask reviewers to identify the user outcome and first important failure. Group the resulting problems before choosing a fix.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 17 · Review draft · hypothetical example · not recorded

Teams keep a gold eval set to prevent regressions

Hook: Teams keep a gold eval set to prevent regressions, but a frozen dataset can become outdated as the product and policies change.

UnscoredPlanned
Show scriptHide script
Teams keep a gold eval set to prevent regressions, but a frozen dataset can become outdated as the product and policies change. Consider this hypothetical example: A refund agent is tested against last month’s return policy after the company changes its eligibility rules. The old set rewards the old behavior. Blindly preserving every old expected answer can penalize the correct update. Deleting all old cases would also lose useful history. A better approach: Version the policy and test cases, review which labels changed with a domain expert, and maintain a current holdout plus historical cases when they still matter.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 18 · Review draft · hypothetical example · not recorded

Engineers can write excellent code

Hook: Engineers can write excellent code, but they cannot infer every business decision a support agent needs to make.

UnscoredPlanned
Show scriptHide script
Engineers can write excellent code, but they cannot infer every business decision a support agent needs to make. Consider this hypothetical example: A customer says an order should be cancelled. The agent cancels immediately, while the actual workflow requires identifying the order and confirming after explaining relevant consequences. Checking only whether the final message contains “cancelled” misses the required conversation and can reward the wrong behavior. A better approach: Have the support expert map clarification, policy checks and confirmation. Engineers turn that workflow into labeled multi-turn cases and review whether the full job was done correctly.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 19 · Review draft · hypothetical example · not recorded

A domain expert can tell you an answer is wrong

Hook: A domain expert can tell you an answer is wrong, but a bare fail label may not tell engineers which rule to fix.

UnscoredPlanned
Show scriptHide script
A domain expert can tell you an answer is wrong, but a bare fail label may not tell engineers which rule to fix. Consider this hypothetical example: A reviewer marks a laptop recommendation as failing. The team cannot tell whether the problem was budget, required software or an unsupported product claim. Training a judge on unexplained pass and fail labels alone can obscure the decision rule. The same label may represent several distinct failures. A better approach: Ask the expert for a concise critique naming the violated requirement and supporting evidence. Group similar critiques, revise the rubric, then check the judge on separate reviewed cases.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 20 · Review draft · hypothetical example · not recorded

A checklist can define an agent’s workflow

Hook: A checklist can define an agent’s workflow, but forcing every step in every conversation can make the experience worse.

UnscoredPlanned
Show scriptHide script
A checklist can define an agent’s workflow, but forcing every step in every conversation can make the experience worse. Consider this hypothetical example: A shopping agent asks for budget and intended use even when the customer already supplied both. It satisfies a rigid checklist while wasting the customer’s time. Adding “always ask these questions” fixes missing information in one case and breaks another. The rule ignores context. A better approach: Ask the domain expert which facts are required and when to clarify missing ones. Test cases where facts are missing, already known and contradictory before recommending.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 21 · Review draft · hypothetical example · not recorded

Teams know RAG needs a retrieval system

Hook: Teams know RAG needs a retrieval system, but an answer-quality score alone cannot tell you whether retrieval found the right evidence.

UnscoredPlanned
Show scriptHide script
Teams know RAG needs a retrieval system, but an answer-quality score alone cannot tell you whether retrieval found the right evidence. Consider this hypothetical example: A warranty assistant gives the wrong period. The correct page exists, but the retriever returned an old page and the answer followed it. Rewriting the answer prompt may not help when the missing or stale evidence is upstream. A single final score hides the failing stage. A better approach: Review the end-to-end task first, then inspect the retrieved passages and their source versions. Add separate diagnostic checks for retrieval and grounded answers on expert-reviewed cases.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 22 · Review draft · hypothetical example · not recorded

A RAG answer can contain citations

Hook: A RAG answer can contain citations, but a citation does not guarantee that the source supports the claim.

UnscoredPlanned
Show scriptHide script
A RAG answer can contain citations, but a citation does not guarantee that the source supports the claim. Consider this hypothetical example: A support answer cites a return policy, but the cited paragraph describes exchanges rather than refunds. The answer sounds plausible. A rule that only checks whether a link appears will pass this failure. The link is present but does not establish the claim. A better approach: Give reviewers and the judge the actual claim and cited passage. Ask whether the passage supports that claim, validate decisions against expert labels, and retain examples of misleading citations.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 23 · Review draft · hypothetical example · not recorded

The retriever can find the correct document

Hook: The retriever can find the correct document, but that does not mean it found the right section for this user.

UnscoredPlanned
Show scriptHide script
The retriever can find the correct document, but that does not mean it found the right section for this user. Consider this hypothetical example: A product policy contains a standard warranty and a regional exception. The agent retrieves the document but answers from the wrong section. Treating document retrieval as automatically successful misses the user-specific constraint. A relevance score alone may not reveal the mismatch. A better approach: Create cases with regions and exceptions, inspect the actual retrieved passages and ask experts whether the answer applies the right rule. Fix chunking or retrieval when the relevant evidence is absent.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 24 · Review draft · hypothetical example · not recorded

RAG can make answers grounded in documents

Hook: RAG can make answers grounded in documents, but grounding in an outdated document can still produce a confidently wrong answer.

UnscoredPlanned
Show scriptHide script
RAG can make answers grounded in documents, but grounding in an outdated document can still produce a confidently wrong answer. Consider this hypothetical example: The agent retrieves an archived warranty page and accurately quotes it, even though a newer policy governs the product. A faithfulness check against the retrieved page may pass because the answer follows the wrong source. Grounding and authority are different questions. A better approach: Add source freshness and authority to the task-specific criteria. Build reviewed old-versus-current cases, then check retrieval and the final answer separately.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 25 · Review draft · hypothetical example · not recorded

An AI agent can call the correct tool

Hook: An AI agent can call the correct tool, but the right tool with the wrong account can still harm the customer.

UnscoredPlanned
Show scriptHide script
An AI agent can call the correct tool, but the right tool with the wrong account can still harm the customer. Consider this hypothetical example: A customer asks to update their own address. The agent selects the address tool but supplies an identifier for a different account. A test that checks only the tool name will pass the failure. A polite final message will not show the incorrect argument. A better approach: Use a safe test environment and cases with similar accounts. Verify identity, tool arguments and the resulting state, alongside the expert-defined end-to-end outcome.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 26 · Review draft · hypothetical example · not recorded

Agents need multi-step tests

Hook: Agents need multi-step tests, but testing every tool call in isolation can miss whether the customer’s job was completed.

UnscoredPlanned
Show scriptHide script
Agents need multi-step tests, but testing every tool call in isolation can miss whether the customer’s job was completed. Consider this hypothetical example: A booking agent retrieves availability and prepares a slot correctly, then never confirms the actual appointment. Every individual tool test is green. A collection of passing component checks does not prove the end-to-end goal was achieved. The customer still has no booking. A better approach: Define task success with the domain expert and evaluate the full conversation and resulting state. Once you find failures, use step-level diagnostics to locate the cause.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 27 · Review draft · hypothetical example · not recorded

A final answer can look correct

Hook: A final answer can look correct, but an agent’s intermediate actions may violate the workflow that makes the answer safe.

UnscoredPlanned
Show scriptHide script
A final answer can look correct, but an agent’s intermediate actions may violate the workflow that makes the answer safe. Consider this hypothetical example: A cancellation agent identifies the right order, but cancels before explaining a relevant consequence or obtaining the required confirmation. Its final message sounds helpful. Judging only the final sentence loses the sequence of decisions. You cannot see whether the user actually authorized the change. A better approach: Record the multi-turn trace and relevant actions. Have the business expert define when clarification and confirmation are required, then test the sequence and final state.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 28 · Review draft · hypothetical example · not recorded

Agents should ask clarifying questions

Hook: Agents should ask clarifying questions, but asking the same question twice can be a real workflow failure.

UnscoredPlanned
Show scriptHide script
Agents should ask clarifying questions, but asking the same question twice can be a real workflow failure. Consider this hypothetical example: A customer already gives a budget and a use case. The shopping agent asks for both again before recommending a product. An eval that merely rewards the presence of a question will pass this interaction. It never checks whether the answer was already available. A better approach: Have the expert specify which information matters and when to ask. Test missing, already supplied and conflicting details across the full conversation.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 29 · Review draft · hypothetical example · not recorded

Escalating to a human can protect customers

Hook: Escalating to a human can protect customers, but a handoff without context can force the customer to start over.

UnscoredPlanned
Show scriptHide script
Escalating to a human can protect customers, but a handoff without context can force the customer to start over. Consider this hypothetical example: After several failed troubleshooting steps, the agent transfers the conversation without summarizing the issue or what it already tried. A binary check for “handoff occurred” misses the quality of the handoff. The next person has no useful starting point. A better approach: Define a handoff checklist with the support expert: goal, key facts, actions taken and unresolved issue. Review realistic traces and verify when escalation should occur.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 30 · Review draft · hypothetical example · not recorded

An LLM judge can save review time

Hook: An LLM judge can save review time, but a persuasive explanation does not prove its decision agrees with experts.

UnscoredPlanned
Show scriptHide script
An LLM judge can save review time, but a persuasive explanation does not prove its decision agrees with experts. Consider this hypothetical example: A judge explains why a support response is excellent while missing that the customer asked to cancel a different order. The prose sounds convincing. Accepting explanations without comparing decisions to expert labels can hide systematic false passes. Confidence is not validation. A better approach: Collect expert pass/fail judgments and concise critiques, compare the judge on separate cases, inspect disagreements and measure the kinds of errors that matter.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 31 · Review draft · hypothetical example · not recorded

Giving an LLM judge more context seems helpful

Hook: Giving an LLM judge more context seems helpful, but a long trace can distract it from the evidence needed for one decision.

UnscoredPlanned
Show scriptHide script
Giving an LLM judge more context seems helpful, but a long trace can distract it from the evidence needed for one decision. Consider this hypothetical example: A judge checks whether a product recommendation respected a budget, but receives a lengthy unrelated chat and misses the customer’s earlier limit. Dumping the entire conversation into a vague prompt may make review harder. Removing too much context can also make the decision impossible. A better approach: Identify the criterion and necessary evidence with experts, present relevant trace sections clearly, and compare judge decisions with human labels on cases requiring longer context.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 32 · Review draft · hypothetical example · not recorded

A judge threshold turns a score into a decision

Hook: A judge threshold turns a score into a decision, but the default cutoff may miss the failures your team cares about.

UnscoredPlanned
Show scriptHide script
A judge threshold turns a score into a decision, but the default cutoff may miss the failures your team cares about. Consider this hypothetical example: Imagine a failure detector assigns bad cases scores around 0.4. A 0.5 cutoff flags none of them, even if those scores separate them from passing cases. Accepting the default without checking human labels leaves the team unaware of missed failures. Tuning only for overall accuracy can also hide that risk. A better approach: Use labeled examples to choose a threshold for the desired precision and recall trade-off, then measure it on held-out cases. If no threshold meets the requirements, say so.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 33 · Review draft · hypothetical example · not recorded

A cheaper judge may reduce eval cost

Hook: A cheaper judge may reduce eval cost, but it is only a saving if it catches enough of the failures that matter.

UnscoredPlanned
Show scriptHide script
A cheaper judge may reduce eval cost, but it is only a saving if it catches enough of the failures that matter. Consider this hypothetical example: Two candidate judges review the same order traces. One costs less but misses cancellation-confirmation failures the team considers critical. Choosing by price or a single aggregate score can hide an unacceptable error pattern. A costly model is not automatically the right choice either. A better approach: Compare candidates on the same expert-labeled held-out cases, including per-category false passes, latency and cost. Pick the least expensive option that meets your actual requirements.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 34 · Review draft · hypothetical example · not recorded

Tuning an LLM judge against expert examples can improve it

Hook: Tuning an LLM judge against expert examples can improve it, but testing on the same examples can overstate how well it generalizes.

UnscoredPlanned
Show scriptHide script
Tuning an LLM judge against expert examples can improve it, but testing on the same examples can overstate how well it generalizes. Consider this hypothetical example: A team edits its judge rubric until it agrees with the same small set of reviewed shopping conversations. The reported agreement looks excellent. Rechecking the examples used for every edit measures fit to those examples, not performance on unseen cases. A better approach: Separate development examples from expert-labeled holdout cases. Iterate on the development set, then evaluate the final judge on the protected cases and inspect false passes and false alarms.
Unlock scheduling & Done

Source: Evals reference · Hook details

Video 35 · Review draft · hypothetical example · not recorded

Passing evals before release matters

Hook: Passing evals before release matters, but a fixed test suite cannot represent every way customers will use the product later.

UnscoredPlanned
Show scriptHide script
Passing evals before release matters, but a fixed test suite cannot represent every way customers will use the product later. Consider this hypothetical example: A recommendation agent passes its reviewed release set. Later, customers introduce a new product type and a question the original cases did not cover. Running the same tests forever can preserve yesterday’s confidence without revealing new failure modes. That does not make the original tests worthless. A better approach: Keep release regressions, sample production traces with privacy controls, review new patterns with experts and add representative cases. Coach engineers to repeat that loop as products change.
Unlock scheduling & Done

Source: Evals reference · Hook details