Professional, lesson 13

Eval datasets

A metric is a dataset plus a rule

The Lumen Supply team reports 94% accuracy on its support agent. The number comes from 150 tickets the prompt was tuned on, so it cannot show which kinds of ticket fail. You will watch one score hide a weak slice, then split the set and score each slice with a grader that fits it.

I doWatch the failure and its fix.

94% on the set we tuned on

Lumen Supply is an online electronics shop. Its agent reads a ticket and picks one action: refund, deny, escalate or ask for the order number. The eval set has 270 labelled tickets: 240 in four slices and 30 sealed for the end.

Three things go wrong, in this order. The score is measured only on the tuned tickets. No slice holds edge cases, old failures or hostile tickets, so one average hides the weak ones. And one grader scores every field, even where a rule could.

A score per slice

Claude answers all 270 tickets, and each answer is graded. The report has one line for each slice, so the weak slice shows up by name. The edge slice is red at 70%: 28 of 40 passed.

Open the edge card to see the failing tickets. The agent denied a refund of exactly $500, and denied an item delivered on day 30. The slice with 5 German tickets belongs here too, because a language is a group you can score on its own.

Eval set270 tickets answered by Claude, graded by a rule

Representative134 of 150 passed, 89%

Show the result

{
  "n": 150,
  "passed": 134,
  "rate": 0.8933333333333333,
  "failing rows": [
    {
      "ticket": "Hello Lumen team, my order 1615: the docking station will not switch on. It arrived 27 days ago. Please sort this out with a refund.",
      "expected": "refund",
      "agent chose": "deny"
    },
    {
      "ticket": "Hey there, my order 1501: the laptop stand will not switch on. It was delivered 23 days ago. Please refund me.",
      "expected": "refund",
      "agent chose": "deny"
    },
    {
      "ticket": "Hello Lumen team, my wireless mouse arrived broken in the box. It was delivered 24 days ago. Can you refund this?",
      "expected": "ask_info",
      "agent chose": "deny"
    }
  ]
}

Edge cases24 of 40 passed, 60%

Show the result

{
  "n": 40,
  "passed": 24,
  "rate": 0.6,
  "failing rows": [
    {
      "ticket": "Hello, my order #1628: the USB cable keeps shutting down by itself. Delivery was 30 days ago. I would like my money back.",
      "expected": "refund",
      "agent chose": "deny"
    },
    {
      "ticket": "Good morning, my order 1454: the SSD arrived broken in the box. I got it 30 days ago. I would like my money back.",
      "expected": "refund",
      "agent chose": "deny"
    },
    {
      "ticket": "Hello, my ref 1135: the SSD keeps shutting down by itself. It was delivered 30 days ago. Please refund me.",
      "expected": "refund",
      "agent chose": "deny"
    }
  ]
}

Regression29 of 30 passed, 97%

Show the result

{
  "n": 30,
  "passed": 29,
  "rate": 0.9666666666666667,
  "failing rows": [
    {
      "ticket": "Good morning, please refund my tablet, $520. Order 1206, delivered 9 days ago, it will not switch on.",
      "expected": "escalate",
      "agent chose": "refund"
    }
  ]
}

Adversarial20 of 20 passed, 100%

Show the result

{
  "n": 20,
  "passed": 20,
  "rate": 1.0,
  "failing rows": []
}

Held out26 of 30 passed, 87%

Show the result

{
  "n": 30,
  "passed": 26,
  "rate": 0.8666666666666667,
  "failing rows": [
    {
      "ticket": "Hey there, my order 1008: the tablet keeps shutting down by itself. It arrived 21 days ago. It cost $520. Please refund me.",
      "expected": "escalate",
      "agent chose": "refund"
    },
    {
      "ticket": "Hey there, my order 1318: the gaming laptop arrived broken in the box. It was delivered 9 days ago. It cost $899. I would like my money back.",
      "expected": "escalate",
      "agent chose": "refund"
    },
    {
      "ticket": "Hey there, my order #1669: the tablet stopped working after two days. I got it 17 days ago. It cost $520. Please refund me.",
      "expected": "escalate",
      "agent chose": "refund"
    }
  ]
}

Reported per slice. The weakest is edge cases at 60%.

Slices and graders, written as code

Two functions hold the design. slices names the kinds of ticket: representative, edge, regression and adversarial. A regression ticket is a past failure kept so it cannot return. An adversarial ticket is hostile input.

score decides who grades a row. A code grader checks what a rule can check, such as the action. A model judge grades what a rule cannot, such as tone. The starting code does neither.

starter/evalset.py

1"""EDIT THIS FILE. It has two planted defects.21. slices(): it returns one list, the tickets the prompt was tuned on, so3 one number hides the weak slices and the held-out slice is never kept.42. score(): it sends every row to the same grader, even where a rule can5 check the answer.67Four of the six tests fail at the start. The fixture test and the tone8test already pass, because the starter sends every row to the grader.910Run `./workshop check eval-datasets` until it is green, then11`./workshop watch eval-datasets`.12"""1314from collections.abc import Callable1516from kit.api import Json1718type Row = Json192021def slices(rows: list[Row]) -> dict[str, list[Row]]:22 """Split the rows into named slices and seal the held-out rows.23 The slices are representative, edge, regression and adversarial, and24 `held_out`. Each row has `kind` and `sealed`.25 """26 tuned = [27 r for r in rows if r["kind"] == "representative" and not r["sealed"]28 ]2930 return {"representative": tuned}313233def score(row: Row, grader: Callable[[Row], bool]) -> bool:34 """Return whether the agent passed this row.35 A row with field "action" has a right answer in `row["expect"]`, and36 `row["answer"]["action"]` is what the agent chose.37 """38 return grader(row)

Does 94% on the tuning set say the agent is ready?

The prompt was tuned on 150 representative tickets, and the agent passes 141 of them. The team wants to launch on that number.

Predict: Does 94% on the tuning set say the agent is ready?

ANo. It says nothing about edge or hostile tickets.
BYes. 94% is high.

Show the answer

A. A metric is a dataset plus a rule. A set built from the tuned tickets can only confirm the tuning.

One number from the tuned set

This is a real Claude Haiku 5.5 run, recorded once and replayed here. The report has one line: representative, 141 of 150, 94%. The outcome is red because that line is all the team sees.

Nothing in it says how the agent does on a $500 refund or a ticket that tells it to ignore the policy. The same agent scores 70% on the edge slice. The number was true and it hid the failure.

Eval set150 tickets answered by Claude, graded by a rule

Representative134 of 150 passed, 89%

Show the result

{
  "n": 150,
  "passed": 134,
  "rate": 0.8933333333333333,
  "failing rows": [
    {
      "ticket": "Hello Lumen team, my order 1615: the docking station will not switch on. It arrived 27 days ago. Please sort this out with a refund.",
      "expected": "refund",
      "agent chose": "deny"
    },
    {
      "ticket": "Hey there, my order 1501: the laptop stand will not switch on. It was delivered 23 days ago. Please refund me.",
      "expected": "refund",
      "agent chose": "deny"
    },
    {
      "ticket": "Hello Lumen team, my wireless mouse arrived broken in the box. It was delivered 24 days ago. Can you refund this?",
      "expected": "ask_info",
      "agent chose": "deny"
    }
  ]
}

One number: 89%. It says nothing about the other kinds of ticket.

Slices, graders and a sealed set

slices splits the rows by kind. It keeps sealed rows in a held_out list of their own. score checks an action with code and sends only tone to the grader.

Cost decides the split. A judge reads 25 rows a request, so judging all 270 rows is 11 requests. Checking the 246 action rows with code is free, and the 24 tone rows take 1 request.

Repeatability decides it too. A code check gives the same answer on every run, while a judge is a model whose score can move between runs. A judge can drift, so compare a sample of its scores with labels a human wrote.

The judge is Claude Haiku 5.5, the same model as the agent it grades, so the labels a human wrote are what keep it honest. It returns an integer from 1 to 5 as JSON held to a schema, and a row passes at 4 or more.

evalset.pygreen lines are what the fix adds

1"""Slices, graders and a sealed set: the solution.2slices() names four kinds of ticket and keeps the sealed rows apart.3score() checks an action with code and sends only tone to a grader.4"""56from collections.abc import Callable78from kit.api import Json910type Row = Json1112KINDS = ("representative", "edge", "regression", "adversarial")131415def slices(rows: list[Row]) -> dict[str, list[Row]]:16 """Split the rows into named slices and seal the held-out rows.17 The slices are representative, edge, regression and adversarial, and18 `held_out`. Each row has `kind` and `sealed`.19 """20 found: dict[str, list[Row]] = {kind: [] for kind in KINDS}21 found["held_out"] = []2223 for row in rows:24 found["held_out" if row["sealed"] else row["kind"]].append(row)2526 return found272829def score(row: Row, grader: Callable[[Row], bool]) -> bool:30 """Return whether the agent passed this row.31 An action has one right answer, so code checks it. Tone has none, so32 the grader decides.33 """34 if row["field"] == "action":35 return bool(row["answer"]["action"] == row["expect"])3637 return grader(row)

We doChange one setting at a time and predict.

The same input now ends well

Switch between the two sets with the same code grader. The representative line is 94% in both. The card marked "This is where the runs split" is the edge slice, at 70% (28 of 40), and only the sliced set has it.

The agent did not change, and the report did. Answers vary from run to run, so a gap of one ticket on three examples means nothing. A rate over 150 or 40 tickets does.

This part is interactive in the lab: run ./workshop watch eval-datasets.

Try it: set and grader

Change one setting at a time. On the tuned set with the judge, the score moves to 90% (135 of 150). A different grader gives a different number, and the set still has no edge slice. On the sliced set, edge stays at 70% with either grader.

With the judge on, a Model judge card shows one tone reply and its 1 to 5 score. Every combination is a recorded run, so this works with no internet.

This part is interactive in the lab: run ./workshop watch eval-datasets.

The sealed slice

The sealed rows are 30 tickets, drawn from every slice and used once, at the end. Here they score 97% (29 of 30). Nobody tunes on them, so that is the only honest final number.

Peek at them while you tune and they stop being sealed. A slice this small moves 3 points on one ticket, so read the number as a rough check. Pre-register the rule fixes the ship rule before you see the result.

app/data.pyfile: 358 lines

···316 lines not shown317 (twice, 8),318 (shouting, 8),319 (over_limit, 6),320 (abuse, 8),321 ],322 "adversarial": [323 (ignore_rules, 6),324 (owner, 5),325 (fake_order, 4),326 (abuse, 5),327 ],328}329SEALED = {"representative": 15, "edge": 8, "regression": 4, "adversarial": 3}330331332def seed(name: str) -> random.Random:333 """A generator with a fixed seed, so the set is the same everywhere."""334 return random.Random(name) # noqa: S311 - a fixed seed, not security335336337def build(kind: str, sealed: bool, rng: random.Random) -> list[Row]:338 """The rows of one slice, in a fixed shuffled order."""339 found: list[Row] = []340 for maker, count in SLICES[kind]:341 found += [maker(rng) for _ in range(count)]342 rng.shuffle(found)343344 if sealed:345 found = found[: SEALED[kind]]346347 return [{**r, "kind": kind, "sealed": sealed} for r in found]348349350def rows() -> list[Row]:351 """All 270 rows: 240 in four slices, then the 30 sealed held-out ones."""352 found: list[Row] = []353 for kind in SLICES:354 found += build(kind, False, seed(f"eval-{kind}"))355 for kind in SLICES:356 found += build(kind, True, seed(f"sealed-{kind}"))357···1 line not shown

You doFix the starting code, then test yourself.

You do: slice the set and pick the grader

Copy the starting code and open work/eval-datasets/evalset.py. slices() returns one list of tuned tickets, and score() sends every row to the model grader.

Terminal

./workshop start eval-datasets
./workshop check eval-datasets
./workshop watch eval-datasets

The checks want every unsealed row in one named slice and the held-out rows sealed apart. A tuned set must show the weak slice, an action is checked by a rule and tone goes to the grader.

Check your understanding

Answer each question, then read why every option is right or wrong. The wrong options are the mistakes engineers make on the job.

1. A support agent scores 94% on the 150 tickets its prompt was tuned on, and the team ships it. A week later one refund in four fails on split orders, where the delivery date differs from the order date. None of the 150 tickets was a split order. Which change to the eval would have shown the problem before launch?

ARaise the pass mark from 94% to 98% on the same 150 tickets, and block the launch below it.
BRun the same 150 tickets five times and average the five scores, to remove run-to-run noise.
CAdd an edge slice of split orders and other boundary cases, and report it on its own.
DTune the prompt on the 150 tickets until the score reaches 100%.

Show the answer

Answer: C

A A higher bar on the same tickets still measures only what the prompt was tuned on. No split order is in the set, so the score cannot move for them.
B Averaging repeats steadies the number and adds no new kind of ticket. The failure is a missing kind, not noise.
C Correct. A slice of edge cases puts split orders in the set, and a score of its own stops the 94% from hiding them.
D Tuning to 100% on the same tickets measures the tuning. Tickets the prompt never met stay untested.

2. A team grows its eval set from 240 to 440 rows by adding 200 more tickets of the same kind as the existing representative slice. The score stays at 94%, and refunds of exactly $500 and orders that arrived in two parcels still fail in production. What should the team add to the set next?

ASlices of a new kind, such as the $500 limit and split parcels.
BAnother 200 representative tickets, so that the set reaches 640 rows and the score firms up.
CA model judge that reads every ticket twice and reports where its two reads differ.
DAn alert that fires when production refunds fail, with the failing tickets sent to the team.

Show the answer

Answer: A

A Correct. Coverage comes from new kinds of ticket. Slices for the $500 limit and split parcels put those failures in the set.
B More rows of the same kind repeat what is already covered. The set grows and the odd cases stay missing.
C A second read by a judge changes who grades a ticket and adds no ticket to grade.
D An alert reports a failure after a customer met it. It is not a test the team can run before a release.

3. A refund bug was fixed in March, when the agent denied a faulty item delivered on day 30. In May a prompt change brought it back, and the 240-row eval did not notice because that ticket was never kept. The team wants the same failure caught next time, at the cost of one ticket. What should it do?

AWrite the fix and its cause in the changelog so that the next editor sees it before editing.
BWatch the complaint queue for denied refunds after each release and report any it finds.
CHave the judge re-grade the whole 240-row set each month and flag any score that moved.
DAdd the failing ticket to a regression slice that runs on every change.

Show the answer

Answer: D

A A changelog line is read by people when they remember to. Nothing runs it, so the May change passes without a check.
B A complaint arrives after a customer is hurt. The eval should fail before the release, not after.
C A judge re-grading the same set finds nothing new, because the failing ticket is still not in it.
D Correct. A regression slice keeps each fixed failure as a ticket, so a change that brings it back turns the slice red.

4. A reviewer reads an eval plan for the Lumen support agent. It has 150 tuning tickets and a model grader, and the agent will face 3,000 tickets a day from strangers. Which slices does every eval set for this agent need besides the representative one? Select two.

AEdge cases, such as a refund of exactly $500 or a delivery on day 30.
BA second copy of the tuning tickets, used as the test and scored with the same grader.
CAdversarial inputs, such as a ticket that tells the agent to ignore the policy.
DNothing more, because a hostile slice tests only the attacker and costs more than it finds.

Show the answer

Answer: A, C

A Correct. Boundary tickets sit where rules meet, and a representative slice rarely holds them.
B A copy of the tuning tickets measures the tuning, so it cannot tell the team what the prompt misses.
C Correct. Customers write the tickets, so a hostile slice shows whether the agent follows the policy or the customer.
D Hostile input arrives in production whether it is tested or not. A slice for it is part of the set.

5. An eval scores 250 tickets with one model grader. It checks that the action is one of four allowed words and that the reply is polite. Re-running the same set moves the score between 88% and 93%, and most of the movement is on the action check. What should change?

ARun the grader three times on each ticket and average the three scores for the report.
BCheck the action with a code rule and keep the judge for tone.
CSwitch to a larger judge model for both checks, and keep the same rubric prompt.
DAsk the judge in its prompt to be stricter and to answer with the allowed word only.

Show the answer

Answer: B

A Averaging hides the spread and keeps paying for a model on an exact check. The noise has a cause that a rule removes.
B Correct. An action is one of four words, so a rule checks it exactly and for free. The judge keeps tone, which a rule cannot check.
C A larger model is still a model grading an exact field, so it still costs money and can still vary.
D A stricter prompt changes how the judge leans, not whether it gives the same answer twice.

6. A team judges all 270 rows with a model grader at 25 rows a request, which is 11 requests on every run. Of the 270 rows, 246 have one right action that exact comparison can check, and 24 are about tone. The team needs runs that are cheaper and exact. What should it do?

AJudge only every second row, and double the passed count to estimate the full run.
BShorten the judge's rubric so that it reads fewer tokens on every one of the 270 rows.
CCompare the 246 actions with code and send only the 24 tone rows to the judge.
DSwap the judge for a larger model on all 270 rows and keep the same batch size.

Show the answer

Answer: C

A Half the rows are then never graded, so the score covers less and is still judged where a rule would be exact.
B A shorter rubric trims tokens and leaves the judge on rows a rule checks exactly, so the answer stays noisy.
C Correct. Code is exact and free for the 246 actions, and the judge handles only the 24 rows a rule cannot check.
D A larger judge costs more per row and still grades fields that code checks exactly.

7. A model grader replies to each row in free text, such as "mostly fine but a bit curt". The team needs a pass rate for each slice across 240 rows. Reading the sentences by hand takes hours, and two reviewers sort the same sentence into different piles. Which change fixes this?

AMake the judge return an integer score from 1 to 5 for each row, and pass at 4 or more.
BAsk the judge for shorter sentences, no more than ten words for each row it grades.
CAdd a second judge that reads the first judge's sentences and sorts them into piles.
DHave a person label all 240 rows by hand on every run, and count the labels for the report.

Show the answer

Answer: A

A Correct. A schema with a score gives each row a number, so a pass rate per slice is a count and two runs can be compared.
B A shorter sentence is still free text, and two reviewers still read it differently.
C A second reader of the same sentences adds a layer and no number, and it adds its own wording to sort.
D A person can label a sample to check the judge. Labelling 240 rows by hand every run does not scale.

8. A reliability review looks at the model grader that scores tone on the 24 rows a rule cannot check. The grader will gate a release, its scores feed a per-slice report, and the review has to decide this week. Which two properties make this grader trustworthy? Select two.

AIt gives a verdict as a long paragraph for each row, so that no detail is lost.
BIt returns a structured score on a fixed scale for each row.
CIt is the same model with the same prompt as the agent, so the two always agree.
DIts scores are compared with labels a person wrote for a sample of rows.

Show the answer

Answer: B, D

A A paragraph cannot be counted into a pass rate, and two reads of it can differ.
B Correct. A score on a fixed scale turns each verdict into a number, so slices can be counted and runs compared.
C A judge with the agent's own prompt shares its blind spots, and agreeing with it proves nothing about tone.
D Correct. A model grader is itself a measurement, so a sample checked against human labels shows whether to trust it.

9. A prompt technique lifts the pass rate from 94% to 96% on a 270-row set. On the 40-row edge slice the rate falls from 70% to 60%, and refunds at the limit are among the new failures. The team has to decide today whether to keep the change. What should it do?

AKeep it, because the overall score rose from 94% to 96%, and that is the number the team reports.
BRe-run the overall score until the two-point rise is stable, then keep the change if it holds.
CDrop the edge slice from the report, since it is 40 rows of 270, and keep the change.
DRead the change per slice, and treat the edge drop as a regression to explain first.

Show the answer

Answer: D

A The overall figure is mostly the large representative slice. A gain there can sit next to a drop in a small slice.
B A steadier overall number still hides the slice that fell.
C Removing the slice that shows the drop makes the report look better and the agent no safer.
D Correct. A score for each slice shows where the technique helped and where it hurt, so the edge drop is seen and weighed.

10. A prompt change is judged on three hand-picked tickets where it works. Across 5 runs the agent's answers on a ticket differ from run to run, and the owner wants to know whether the change really helps. The eval has a 150-row representative slice, a 40-row edge slice and a sealed 30-row held-out slice. How should the team judge the change?

AKeep it if any one of the three hand-picked tickets passes on the next run, since it works.
BScore it on the representative and edge slices and compare the rates.
CTry variants against the held-out slice until one of them wins, and keep that variant.
DRun the same three tickets 50 times, count the passes, and compare that to the old prompt.

Show the answer

Answer: B

A Three chosen tickets where the change works show only that it can work. They say nothing about the rest.
B Correct. Run-to-run variation averages out over many tickets, so a difference in rates on a large set means something.
C Choosing variants by the held-out score tunes on it, and then it no longer gives an honest final number.
D Fifty runs of three tickets measure the variation on three tickets and still miss every other kind.

11. A fairness check on 270 support tickets in two languages reports 93% overall. German tickets are at 70% and English tickets at 95%, and the overall figure hides the gap. The team has to show whether the agent serves both groups. What should it do next?

ASlice the eval by language, and report each group's rate.
BTranslate every German ticket to English before scoring, so that one grader reads them all.
CReport the overall 93%, because it clears a 90% bar and shows no failing group.
DRemove the German tickets the agent fails, and report the rate on those that remain.

Show the answer

Answer: A

A Correct. A slice for each group gives each its own rate, so a gap of 70% against 95% cannot hide in a 93% average.
B Translating first removes the thing being measured, how the agent handles German tickets.
C An overall figure above a bar says nothing about the group that is below it.
D Deleting failures lifts the German score and leaves the agent exactly as unfair.

12. One group has 12 rows in the set and scores 75%, and a group with 150 rows scores 95%. A manager says the agent is unfair to the small group and asks for a prompt change this week. One ticket moves the small group's rate by about 8 points. What is the right response?

AChange the prompt now, because 75% is below 95% and the gap is already clear to the manager.
BDrop the 12-row group from the report, since it is too small to count toward the score.
CSay 12 rows is too small to conclude, and add rows to that slice before acting.
DCopy the 12 rows until the group has 150, so that its rate is as firm as the other group's.

Show the answer

Answer: C

A With 12 rows, one ticket moves the rate about 8 points, so the gap may be chance.
B Dropping the group removes the one at risk. The fix is more rows in that slice.
C Correct. The sample is too small to show a gap, so the honest answer is that it is unproven until the slice has enough rows.
D Copies of 12 tickets are still 12 distinct cases. They add no information and make the rate look firmer than it is.

Play: predict the output

This rule checks the action with code, as it should.

python

def score(row: Row, grader: Callable[[Row], bool]) -> bool:
    if row["field"] == "action":
        return row["answer"]["action"] == row["expect"]
    return grader(row)

Predict: The agent answers "Refund", and the row expects "refund". Does the row pass?

ANo. The strings differ in one capital letter, so the rule fails a right answer.
BYes. The action is the same word.

Show the answer

A. A rule is exact, so it fails right answers written another way. Fix the form with a schema or an enum, and read a sample of failures before you trust a drop.

References

Create strong empirical evaluations: code graders, model graders and when each fits.
Using the evaluation tool: running a test set against a prompt in the console.
Demystifying evals for AI agents: slices, held-out sets and graders for agent tasks.
Define success criteria: the targets each slice is scored against.

Conclusions

A score is a dataset plus a rule, so report it per slice and never as one number.
Use code where a rule can check the answer, and a judge only for what a rule cannot.
Keep a sealed held-out slice, and read it once at the end.

You can now split an eval set into slices, pick a grader for each field and seal a final set.

Slice

A named part of the eval set, scored on its own

Edge case

A ticket at a boundary, such as exactly $500 or day 30

Regression slice

Past failures kept as tickets so they cannot return

Adversarial slice

Hostile tickets that try to move the agent off policy

Held-out set

Sealed rows used once, never tuned on

Code grader

A rule that checks an answer with no model call

Model judge

A model that scores what a rule cannot, such as tone