AI Evals for Product Managers and Builders: Write Your First Eval in an Afternoon (2026)

Your agent demo worked. That is the problem: a demo is one example, and an eval is twenty.

An AI eval (short for evaluation) is a repeatable test that runs your AI system on a fixed set of real cases and marks each result pass or fail against criteria you wrote down before you looked at the output. The pass rate tells you whether a change made the system better or worse, and the failures tell you what to fix. You can build your first one in a spreadsheet in an afternoon. This guide gives you the five steps, the 20-case sheet itself, the graders, the metrics, the traps and a brief for your coding agent.

It is written for product managers, technical product managers and builders. The method needs no code. The part that does need code is the harness that runs the cases, and a later section shows how to hand that to a coding agent without letting it grade its own homework.

If you only remember one line from this guide, make it this: "11 of 20 cases pass" is a score, but "four of the nine failures skipped the same check" is a fix, so an eval is only worth running if you read every failure and give it a name.

Six questions people ask about AI evals (quick answers)

• What is an eval? A set of real cases, each with a pass or fail rule written in advance, run against your system every time you change it. It is the AI version of a test suite, with one difference: you read the failures, because the output varies and the rules are about meaning.
• How many cases do I need to start? About 20 real ones. Anthropic's engineering team calls 20 to 50 simple tasks drawn from real failures a great start. Twenty is enough to find your failure categories and too few to measure a precise pass rate.
• Who writes the pass or fail rules? A person who knows the domain, before looking at any output. Not the model under test, and not the coding agent that builds the system.
• Can an LLM (Large Language Model) grade the outputs? Yes, once you have shown that it agrees with human labels. Judge models have known biases, including favouring longer answers and favouring an answer because of where it appears.
• What do I measure for an agent? Task success first. Then tool-call correctness, the number of steps, cost per successful task, and whether it passes every time you repeat the run.
• When does the eval run? Offline, on every change to the prompt, model, tools or retrieval, as a gate in CI (Continuous Integration, the automatic checks that run before a change is merged). Online, on a sample of live conversations that a person reads every week.

A scene you will recognise

This is an illustrative scenario, not a real company. A product manager at an online retailer demos a refund assistant. A customer writes, "My mixer arrived with a cracked lid." The assistant finds the order, checks the refund policy, issues the refund and apologises. The team tried it on three orders they picked themselves, and all three worked. It ships.

Two weeks later the support lead forwards three screenshots. A refund went out for an item bought 40 days earlier, outside the refund window. A customer with two orders in the same month had the wrong one refunded. A reply quoted a clause that is not in the policy. None of these was in the demo. Worse, nobody can say whether last week's prompt change made things better or worse, because there is no list of cases to run it on.

The fix is not a better model or a longer prompt. It is a list of 20 real cases with a rule for each, run before every change. The rest of this guide builds that list, and we come back to this assistant with a worked example. Every number in that example is made up for the walkthrough.

Three habits keep teams in the scene above: judging by the demo, writing the criteria after seeing the output, and counting passes without reading the failures.

What an eval is, and how it differs from a unit test, a benchmark and a demo

Hamel Husain, who has written widely about evals for AI products, describes three levels. Level 1 is unit tests: fast, cheap assertions that can run on every code change. Level 2 is human and model review of logged conversations. Level 3 is A/B testing (showing two versions to different users and comparing outcomes), which he says is usually only appropriate for more mature products. So a unit test is one kind of eval, the cheapest kind. What changes with AI is that the output varies from run to run and "correct" is often about meaning, not exact text. OpenAI's evaluation best practices page makes the same point: generative AI is nondeterministic, so traditional testing falls short. Husain adds that, unlike a traditional unit test, you do not necessarily need a 100% pass rate, because the pass rate you accept is a product decision.

A benchmark is a different animal. Husain's point is that product evals are tailored to your users' tasks, which is a different thing from general model benchmarks. A model that tops a public leaderboard can still get your refund policy wrong.

Anthropic's engineering team uses a small vocabulary worth borrowing, because it keeps product and engineering precise when they talk about the same thing:

• Task: one test case, with an input and a definition of success. One row of your sheet.
• Trial: one attempt at a task. You run several trials because outputs vary.
• Grader: the logic that scores a trial. A task can have more than one grader.
• Transcript (also called a trace): the full record of a trial: every model call, tool call and result.
• Outcome: the state of the world at the end of a trial, such as whether a refund exists in the database.
• Harness: the infrastructure that runs the evals from end to end.

Why an afternoon is worth it

Husain wrote in March 2024 that unsuccessful AI products usually share one root cause: a failure to build robust evaluation systems. OpenAI lists "vibe-based evals", meaning judging by "it seems to work" or adding evals only after shipping, among the common mistakes. Anthropic's engineering team makes a practical point about speed: when a new model arrives, teams without evals face weeks of testing, while teams with evals can upgrade in days.

There is a quieter benefit for a product manager. An eval turns an argument about taste ("I think the new prompt is better") into a list of cases that either pass or do not. It gives the team a shared definition of "working", and it gives you something to write in the requirements document before the pressure to ship arrives.

The five-step method

1. Collect about 20 real cases. From logs, tickets, complaints and your own experts, including the awkward ones.
2. Write the pass or fail criteria before you look at any output. One expected behaviour per case, specific enough that a colleague would mark it the same way.
3. Run every case through the real system, more than once if the output varies, and save the full trace of each run.
4. Read the failures and name the error categories. Group them, count them, and rank them by cost.
5. Fix, re-run the whole sheet, and keep the cases. Every fixed failure becomes a permanent regression case.

Step 1: collect about 20 real cases

Where the cases come from matters more than how many there are. In order of preference:

• Real traffic. Logs from a pilot or from the live system. Paste the exact text, typos and mixed Hindi and English included.
• Real complaints and bugs. Support tickets, forwarded screenshots, your bug tracker. Anthropic's engineering team recommends starting from manual checks, pre-release behaviours and user-reported failures.
• Your experts' questions. The ten things the support lead or the policy owner gets asked every week.
• Cases you write by hand, with structure. If you have no users yet, Husain and his co-author Shreya Shankar suggest defining the dimensions that matter (for example issue type and customer mood), writing about 20 combinations by hand, and then turning each into a natural request. They warn that synthetic data cannot tell you how common a failure is in production, only that it can happen.

Then check the balance. Anthropic's documentation says to mirror the real task distribution and not to forget edge cases: irrelevant or missing input, very long input, poor or harmful input, and ambiguous cases where humans would disagree. Their engineering team adds a rule that is easy to forget: test both the cases where a behaviour should occur and the cases where it should not. A refund assistant tested only on valid refund requests will learn to refund everything.

As a starting mix (my suggestion, not a sourced rule), use eight typical cases, four hard but legitimate cases, four cases where the right behaviour is to refuse or hand over to a person, two past failures and two traps. The sheet below builds that mix in.

Why 20? It is enough to see patterns and small enough to read every output in one sitting. Husain and Shankar describe reviewing 20 to 50 outputs as roughly 30 minutes of work. Treat 20 as the floor: the RAG (Retrieval-Augmented Generation) guide on this site suggests at least 30 to 50 real questions for a document assistant, and a repeatable eval set often grows past 100 cases over time.

The 20-case sheet (copy this into a spreadsheet)

Make one row per case. Fill the input and the expected behaviour first, and leave the last three columns empty until you have run the cases. When you are done, export it as a CSV (Comma-Separated Values) file so a script can read it later.

Two optional columns help once you start running: must never fail (yes or no), for cases where one failure matters more than all the others, and run count, for how many trials you ran.

Step 2: write the pass or fail criteria before you look at outputs

For each case, write one or two sentences describing the behaviour that counts as a pass. Write it before you run anything, because once you have seen an output it is very easy to decide that what the system did was what you meant. Anthropic's documentation says good criteria are specific, measurable, achievable and relevant. Its own example of a vague criterion is "the model should classify sentiments well"; the stronger version names a metric, a threshold and a test set.

Here is the difference for the refund assistant (a hypothetical criterion, written for this guide):

• Vague: "The reply should be helpful and follow policy."
• Specific: "The assistant checks the refund policy before taking any action, declines the refund because the order is outside the window, says so politely, offers the exchange the policy allows, and records no refund."

Five rules make criteria work:

• Pass or fail, not a score. Husain and Shankar say binary labels force clearer thinking and more consistent labelling, while scales such as 1 to 5 are hard because annotators often default to the middle. Start binary. If a case has several things that must all be true, split them into separate checks and call the case a pass only when every check passes.
• The two-expert test. Anthropic's engineering team says a good task is one where two domain experts would independently reach the same pass or fail verdict. Give three rows to a colleague who has not seen your answers and compare.
• Grade the outcome, not the path. Anthropic recommends grading what the agent produced rather than the exact route it took, because agents often find valid routes you did not expect. Check the path only for steps that must happen, such as "the policy check comes before the refund".
• Write what must never happen. A refund outside the window, a promise the company cannot keep, advice outside the assistant's remit. Mark those cases must never fail. They become your hard gate later.
• Expect to revise, in the open. Husain cites the idea of "criteria drift": people often cannot define their criteria until they have graded some outputs. So revising is normal. Change the criterion in the sheet, note why, and re-score. What you must not do is quietly rewrite a criterion so that an output you have already seen passes.

Step 3: run every case, and save the whole trace

Run each case through the real system, the same way a user would: the same prompt, tools, retrieval and settings, not a simplified copy. For each run, save the input, every model call and tool call with its arguments and result, the final answer, the number of steps, the tokens or cost, and the time taken. That record is the transcript, and you will read it in the next step.

If the output varies, run each case more than once. Anthropic's engineering team points out that a task that passed on one run may fail on the next, and recommends running multiple trials. Three trials per case is a reasonable default for the first sheet (my suggestion). For anything customer-facing, count a case as passing only if every trial passes. The metrics section explains why that stricter rule exists.

Twenty cases at three trials each is 60 runs. You can do that by hand in a product's chat window, with a short script from an engineer, or with the harness your coding agent builds from the brief later in this guide. For a first pass, doing a few by hand is useful in its own right: you see the system the way your users do.

Step 4: read the failures and name the error categories

This is the step that teams skip, and it is the one that produces the value. Husain calls error analysis the most valuable activity in AI development. Do it like this:

1. Mark every case pass or fail against the criterion you wrote in step 2.
2. Read each failing trace in full, not just the final answer. Find the first thing that went wrong. Husain and Shankar advise annotating the first upstream failure, because early errors cascade into later ones, and fixing early errors usually has more impact.
3. Write a plain-language note in the Notes column: "refunded without checking the policy", "picked the first order it found". Husain and Shankar call this open coding, and compare it to journaling.
4. Group the notes into categories and count them. They call this step, which they describe as building a failure taxonomy, the most important one. Name each category as a defect that someone can fix ("skipped the policy check"), not as a symptom ("bad answer").
5. Rank by count and by cost. A refund wrongly paid out is worse than a clumsy sentence, even if the sentence happens more often.
6. Check for a problem with the case or the grader, not the system. Anthropic's engineering post gives a striking example: a model scored 42% on one benchmark until the grading bugs were fixed, such as marking "96.12" wrong when "96.124991" was expected, and then scored 95%. It also notes that a 0% pass rate across many trials usually signals a broken task, not an incapable agent. If the system is right and your criterion is wrong, fix the criterion.

Agent failures tend to fall into a small number of families. Use these as a starting vocabulary, then rename them to fit what you actually see:

Step 5: fix, re-run the whole sheet, and keep the cases

Fix the highest-cost category first, and change one thing at a time so you know which change did what. Then re-run the entire sheet, not only the cases that failed. That is how you catch a fix that breaks something that used to work, and you will meet one in the worked example below. Compare run against run, case by case: fixed, still failing, newly failing.

Two habits keep the eval useful after the first round. First, every failure you fix, and every new failure that real users find, becomes a new row. Second, once you have tuned against these 20 cases they are no longer a fair exam, because you have shaped the system around them. The RAG guide gives the same advice for question sets: keep a held-out set that you never look at while adjusting. Set aside five to ten fresh cases that you did not use for the fixes and run them at the end.

Worked example: the refund assistant in four runs

This is an illustrative scenario, not a real company, and every number below was made up for the walkthrough. The refund assistant has four tools: look up an order, check the refund policy, issue a refund, and hand the case to a person. The team built a 20-case sheet using the mix above. Six of the rows, after run 1, looked like this:

Grading used the cheapest grader that could do each job. "No refund recorded for C09" is a code check on the database. "Hands the case to a person" is a code check on whether the hand-over tool was called. "Invents no rule" needs reading, so it uses a judge model given the policy text, which is covered in the next section.

Run 1: 11 of the 20 cases passed, and the nine failures fell into four categories:

That table is the point of the exercise. The team did not need a better model. The biggest category is a missing guard, so the first fix goes in code, not in the prompt.

Run 3 shows why you re-run the whole sheet. The new rule fixed the two wrong-order cases, but it also made the assistant ask a needless question of a customer with a single order, and case C07 started failing. Reading that trace led to the narrower rule in run 4. Had the team re-run only the failing cases, the regression would have shipped.

After run 4, three cases still fail: the two that needed a person and the one invented policy detail. Those are not prompt problems. The hand-over rules need to come from the support lead, and the invented clause needs a grounding check. That is a better conversation to take to a meeting than "it is 85% good".

What 20 cases can and cannot tell you

With 20 cases, each case is worth 5 percentage points, so the pass rate moves in big steps. A standard way to put error bars on a pass rate is the Wilson score interval. At 95% confidence, 11 passes out of 20 is a true rate somewhere between about 34% and 74%, and 17 out of 20 is between about 64% and 95%. Those ranges overlap. The jump from run 1 to run 2 is real in one sense, because you can read the four cases that were fixed. But the pass rate alone is weak evidence about how the system performs on all future traffic.

So use the 20-case sheet for what it does well: finding and ranking failure categories. To estimate a rate, or to detect a small change, you need more cases. Husain and Shankar note that a repeatable eval set often grows to 100 or more examples and that CI test sets are often 100 or more. For live traffic they suggest tracking confidence intervals and investigating when the lower bound crosses your threshold.

Three ways to grade: code, humans and an LLM judge

These three types and their trade-offs follow Anthropic's engineering post. The practical rule is to use the cheapest grader that can do the job: code first, a judge where meaning matters, and humans to validate the judge. Husain and Shankar say the same about CI, where they prefer deterministic checks over judge-based ones because CI runs often and cost adds up.

One tension is worth naming. Anthropic's documentation advises prioritising volume over quality: more questions graded automatically, with slightly lower signal, beat fewer questions graded by hand. Your first sheet is hand-graded on purpose, because the goal is to learn what goes wrong. Once you know your categories, automate the checks that a script can do and grow the set.

Using an LLM as the judge: what goes wrong

The research on judge models comes mostly from one 2023 paper, "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" by Lianmin Zheng and colleagues. On two public test sets, a strong judge reached over 80% agreement with human preferences, which the authors describe as the same level of agreement as between humans. That is why judges are useful. The same paper catalogued their weaknesses, and every one of them applies to your product.

How to calibrate a judge against your human labels

1. Get human labels first. Husain's guide has the domain expert give a pass or fail verdict plus a written critique for each case. The critiques become examples for the judge.
2. Write a judge for one criterion. Husain and Shankar recommend a separate judge for each type of failure, and Anthropic likewise suggests grading each dimension with its own judge. Give it the rubric, the source text if there is one, and two or three labelled examples. Allow the judge to answer "Unknown", which Anthropic recommends as a way out when it lacks the information.
3. Run it on labelled cases it has not seen.
4. Compute two numbers, not one. The true positive rate (TPR) is the share of real failures the judge catches. The true negative rate (TNR) is the share of good outputs it correctly passes. Husain and Shankar say to track both, because catching more failures can mean more false alarms. Raw agreement misleads: Husain points out that a judge that always says "pass" scores 95% agreement when only 5% of outputs are bad, while catching nothing.
5. Read the disagreements and fix the judge prompt. Repeat until the judge's mistakes are ones you can live with.
6. Re-check on a schedule and after any change to the judge, the model or the criteria.

An illustrative example: you have 100 human-labelled outputs, of which 20 are bad and 80 are good. The judge flags 14 of the 20 bad ones and passes 76 of the 80 good ones. Its TPR is 14 ÷ 20 = 70% and its TNR is 76 ÷ 80 = 95%. Raw agreement is (14 + 76) ÷ 100 = 90%, which sounds fine, yet the judge misses 6 of the 20 failures. Which number matters depends on the cost of a miss against the cost of a false alarm. Husain and Shankar give no universal target and say to choose by that cost.

Scale matters here. Husain suggests about 100 labelled examples per failure mode to validate a judge and notes that confidence intervals are often too wide below 60. With your 20 labelled cases you can tell whether a judge is obviously wrong. You cannot yet trust it with a decision.

What to measure: classification, retrieval and agents

Anthropic's documentation lists common success criteria such as task fidelity, consistency, relevance, tone, privacy, use of context, latency and price, and notes that most use cases need more than one. Pick the few that match your risks. Be careful with generic scores: Husain and Shankar warn that abstract metrics such as a helpfulness score waste time and create false confidence when used as quality measures.

Classification: routing, tagging, triage

An illustrative example: 200 support tickets, of which 20 are truly urgent. The model flags 25 as urgent, and 16 of those are truly urgent. That makes 16 true positives, 9 false positives, 4 missed urgent tickets and 171 correct non-urgent calls.

Decide which mistake costs more before you set a target. If a missed urgent ticket is worse than a false alarm, aim your threshold at recall. Anthropic's documentation gives an example of a measurable criterion in this style: an F1 score of at least 0.85 on a held-out test set of 10,000 posts. Notice that it names the metric, the bar and the test set.

Retrieval: does the search find the right passage?

If your assistant answers from documents, evaluate the search separately from the answer. Husain and Shankar advise debugging retrieval first with search metrics, then judging the generated answers with validated judges. The complete guide to RAG architectures explains why, since most of the pipeline runs before the model sees anything.

OpenAI's best-practices page uses context recall and context precision as example metrics for answering questions over documents. The names vary between tools, so define each metric in your own sheet.

Agents: did it finish the job, and at what cost?

An illustrative cost example: if 20 runs cost $1.80 in total and 17 succeed, the cost per successful task is $1.80 ÷ 17, which is about $0.106. Tracking this number, and not just cost per run, stops a change that makes the agent cheaper and worse from looking like a win.

Husain and Shankar suggest evaluating agents in two phases: end-to-end task success first, treating the agent as a black box, then step-level diagnostics such as tool choice, parameter extraction and error handling. OpenAI's best-practices page likewise adds tool selection and data precision to what you check for a single agent.

Why repeat the run: pass@k and pass^k. Anthropic's engineering post defines two measures. pass@k is the chance of at least one success in k attempts, and it rises as k grows. pass^k is the chance that all k attempts succeed, and it falls as k grows. Use pass@k when one success is enough, and pass^k when consistency matters, as with a customer-facing agent. Anthropic's example: an agent with a 75% success rate per trial passes three trials in a row about 42% of the time (0.75 × 0.75 × 0.75). By the same arithmetic, a 90% per-trial rate passes five in a row about 59% of the time. The tau-bench benchmark (Yao et al., 2024) introduced pass^k for tool-using agents and reported that the agents it tested then succeeded on under half of its tasks and passed eight trials in a row under 25% of the time in its retail domain. Models have moved on since 2024, so read that as a reason to repeat your trials, not as a statement about any current model.

Offline evals, online evals, regression sets and CI gates

You need both, and Anthropic's engineering post says why: no single evaluation layer catches every issue. It describes a mix of automated evals before launch and in CI, monitoring after launch, A/B tests where traffic allows, user feedback, manual transcript review and systematic human studies. Husain and Shankar describe production monitoring as sampling live traces and running evaluators asynchronously, and note that reference answers are usually missing there, so reference-free judges are more common.

A workable online habit for a small team: every week, pull 30 to 50 recent conversations, read them, and add the failures to the sheet. Husain and Shankar suggest starting open coding with 30 to 50 traces. Reading real conversations is how the sheet stays tied to what users actually do.

Regression sets

A regression set is the group of cases that must keep passing. Every failure you fix joins it. Anthropic's engineering post distinguishes capability evals, which ask what the agent can do well and should start at a low pass rate (a hill to climb), from regression evals, which ask whether it still handles what it used to and should sit near 100%. A capability case that your system now passes reliably can graduate into the regression set. Keep the two lists separate in your sheet, so a dip in a capability case does not look like a broken release.

CI gates

CI (Continuous Integration) is the automatic check that runs on every proposed change before it is merged. A CI gate is a rule that blocks the change when the eval result is not good enough. For an AI system, the changes worth gating are not only code: a new prompt, a new model version, a changed tool description or a new retrieval setting can each change behaviour.

Some design notes. Run cheap deterministic checks on every change, and run judge-based checks less often, perhaps nightly or before a release. Husain notes that cost rises from unit tests to judge-based review to A/B tests, and that this dictates how often each runs. Husain and Shankar report a "pass all" rate, where a case passes only if it passes every check, and suggest reporting it separately for release-blocking checks and for tolerable issues. They give no universal pass rate to require for release. That number is a product decision, so write it into the PRD (Product Requirements Document) before the pressure to ship arrives, with the reasoning, so it is not negotiated downward in the last week.

Do not chase 100% on everything. Husain and Shankar warn that a team passing 100% of its evals is probably not challenging its system enough, and Anthropic calls the same effect saturation: when an agent passes all the solvable tasks, the eval stops telling you anything. When your capability set saturates, add harder cases.

Finally, give the eval an owner. Anthropic says owning evals should be as routine as maintaining unit tests, with domain experts and product teams contributing most of the cases. An eval that nobody owns gets quietly ignored within a quarter.

When the eval and your users disagree

Sooner or later the eval says one thing and users say another. Treat that as information about the eval, then check each possibility in turn.

The rule underneath all five: users are right that there is a problem, and the eval is only right about whether you fixed it if the case is in the set. So turn every disagreement into new cases.

When your eval misbehaves: symptom, cause, fix

Your afternoon, hour by hour

A realistic budget for a first sheet, which is my estimate and not a measured figure:

1. 45 minutes: collect 20 cases. Pull from logs and tickets, add the awkward ones, and fill in the case types.
2. 45 minutes: write the expected behaviour for each case. Ask a colleague to mark three of them without seeing your answers.
3. 30 minutes: run the cases and save the traces. Three trials each if the output varies.
4. 60 minutes: grade, read the failures, write notes, then name and count the categories.
5. 30 minutes: choose the first fix and write it up for the engineer or the coding agent. Re-run the next morning.

That is 3 hours 30 minutes. For comparison, Husain and Shankar describe reviewing 20 to 50 outputs as about 30 minutes of work, so the time goes into writing good criteria and reading, not into tooling.

Brief your coding agent to build the harness

The harness that runs the sheet is plumbing, and a coding agent is good at plumbing. The cases, the criteria, the category names and the thresholds are judgement, and they must come from you. Here is the division of labour:

Do not let the agent that builds the system also write the cases that test it. The safe assumption is that it will write cases it can already pass, drawn from what it has just read, and nobody will have checked that they resemble real traffic. For the same reason, tell it not to edit the cases or the criteria to make a test pass. Anthropic's engineering post says to make graders resistant to hacks, so that passing requires solving the problem.

Build an evaluation harness for this assistant. I wrote the cases; you build the runner.

Inputs (I supply these; you must not edit them or add to them):
1. cases.csv with columns: case_number, case_type, input, expected_behaviour,
 must_never_fail (yes or no).
2. criteria.md: the pass or fail rule for each case_type, and for each check whether it is
 code-graded or judge-graded.
3. config.json: pass_rate_threshold, trials_per_case, max_steps, judge_model, and the price
 per 1,000 tokens (I will fill these in; never hard-code a price).

Build:
1. run_evals.py: send every input through the real assistant entry point (the same code path
 users hit), trials_per_case times.
2. For every trial, append one JSON (JavaScript Object Notation) line to results.jsonl with:
 case_number, trial, the full transcript (every model call and tool call with arguments
 and results), the final answer, step count, token counts, cost and seconds.
3. Code-graded checks only for the checks I list in criteria.md (database state, tool names
 and arguments, required phrases). Judge-graded checks use one judge prompt per criterion,
 return pass, fail or unknown plus a one-sentence reason, and never a 1 to 5 score.
 The judge model must differ from the model under test.
4. report.py: print the pass rate per case (a case passes only if every trial passes), the
 overall pass rate, the pass rate by case_type, each failing case_number with the failing
 check, median and p95 steps, cost per successful task, and a comparison with the previous
 run (fixed, still failing, newly failing).
5. If results contain an error_category column that I fill in by hand, count failures by it.
6. Exit with a non-zero code if any must_never_fail case fails or the overall pass rate is
 below pass_rate_threshold, so CI can block the change.
7. Add tests: three fake cases (one pass, one fail, one unknown) against a stub assistant,
 checking all the arithmetic.

Rules: do not change any prompt, tool or model setting. Do not write new cases. If a case or a
criterion looks wrong, list it in QUESTIONS.md and keep going. Never put the contents of
cases.csv into the assistant's prompt.

Then check the harness yourself before you trust a single number:

• Read the first 10 lines of results.jsonl. Confirm that the transcripts are complete and that a failed or crashed trial is recorded, not silently dropped.
• Break the assistant on purpose. Make it answer "OK" to everything and confirm the report shows failures and the exit code is non-zero. An eval that cannot fail is worse than no eval.
• Check five judge verdicts by hand against your own reading of the same outputs.
• Check the arithmetic on three cases with a calculator.

What evals cannot tell you

• Whether the feature is worth having. An eval measures quality, not value. For the value question, see the guide to measuring the return on AI investments, and for whether AI should be doing the task at all, the Kill It or Keep It test.
• How the system behaves on cases that are not in the sheet. Twenty passing cases cover twenty situations. They are not a guarantee about the twenty-first.
• How common a failure is in production. A hand-built or synthetic set shows what can go wrong, not how often. Husain and Shankar make this point about synthetic data in particular.
• Whether the criteria are the right ones. The eval faithfully measures what you wrote down, including your mistakes.
• Whether the grader is right. Graders have bugs and judges have biases, as the earlier sections showed.
• Whether a rare, severe failure will happen. A high pass rate on a small sheet says little about a harmful output that appears once in a thousand conversations. That calls for guardrails, human review and monitoring in addition to evals.
• Whether users will trust, adopt or pay for the feature. Behaviour and willingness to pay need A/B tests and conversations with users.
• How the next model update will behave. Re-run the sheet when anything underneath you changes.

None of this makes evals less useful. It makes them one layer of several. Husain's advice applies at every level: you can never stop looking at the data.

Tools come and go, so keep your cases in plain files

The method in this guide works in a spreadsheet and a script. Many tools can run evals, and Anthropic's engineering post names several of the frameworks teams use, including Harbor, Braintrust, LangSmith, Langfuse and Arize Phoenix. Our comparison of LangChain, LangGraph and LangSmith describes LangSmith as a platform for tracing, evaluating and deploying agents.

Tools also change. OpenAI's documentation, read on 4 November 2026, had announced that its Evals platform would become read-only for existing users on 31 October 2026 and shut down on 30 November 2026, and pointed to its Datasets feature for iterating. Whichever tool you pick, keep your cases, criteria and results in plain files that you own, so a vendor change costs you an afternoon and not a quarter.

Where this fits in the rest of your AI work

• Agents. What Are AI Agents? covers what you are testing. A loop that repeats until a check passes needs a good check, and Loop Engineering describes the evaluator-optimizer pattern, where one model generates and a second evaluates against criteria. The criteria in that loop are an eval written for runtime.
• RAG. The RAG architectures guide has the retrieval side, including why you should check the search before you tune the prompt.
• Product practice. AI-Native Product Development and AI for Product Managers cover where evals sit in the wider job.
• Interviews. Evaluating AI output is a recurring theme in the reported questions collected in the collection of 141 reported interview questions for product managers. Having run your own 20-case sheet gives you real answers.

Frequently asked questions

What is an AI eval?

An AI eval is a repeatable test that runs an AI system on a fixed set of real cases and marks each result pass or fail against criteria written beforehand. It gives you a pass rate to compare across changes and a list of failures to read and categorise.

How is an eval different from a unit test?

A unit