Foundations, lesson 21

Confidence routing

Check confidence against labels before it routes anything

The refund desk auto-approves every field Claude rates as confident. Scanned forms carry misread digits, and Claude rates them as high as clean ones. Wrong refunds go out with nobody looking. This is the refund desk at Lumen Supply. You will watch confidence route wrong rows, then fix the router so it trusts a form type only after it is measured against labels.

I doWatch the failure and its fix.

The router decides who reviews each row

Forms arrive typed and scanned. Claude reads each one and rates its own confidence. A router then decides which rows are approved and which go to a human.

The router sees a labelled set: rows where the right amount is known. That is the only way to learn which confidence to trust.

A proven form type runs on its own

The typed queue matches its labels on every row, so every row is approved. The router does not slow down a type that has proved itself. Open a card to see the row.

Claudetyped form: 18.0, confidence 0.97

Show the result

{
  "id": "t1041",
  "segment": "typed",
  "amount": 18.0,
  "confidence": 0.97,
  "truth": 18.0,
  "correct": true
}

Routerauto-approved, matches the label

Show the result

{
  "id": "t1041",
  "segment": "typed",
  "amount": 18.0,
  "confidence": 0.97,
  "truth": 18.0,
  "correct": true
}

Claudetyped form: 24.5, confidence 0.95

Show the result

{
  "id": "t1042",
  "segment": "typed",
  "amount": 24.5,
  "confidence": 0.95,
  "truth": 24.5,
  "correct": true
}

Routerauto-approved, matches the label

Show the result

{
  "id": "t1042",
  "segment": "typed",
  "amount": 24.5,
  "confidence": 0.95,
  "truth": 24.5,
  "correct": true
}

Claudetyped form: 12.0, confidence 0.95

Show the result

{
  "id": "t1043",
  "segment": "typed",
  "amount": 12.0,
  "confidence": 0.95,
  "truth": 12.0,
  "correct": true
}

Routerauto-approved, matches the label

Show the result

{
  "id": "t1043",
  "segment": "typed",
  "amount": 12.0,
  "confidence": 0.95,
  "truth": 12.0,
  "correct": true
}

Claudetyped form: 59.0, confidence 0.95

Show the result

{
  "id": "t1044",
  "segment": "typed",
  "amount": 59.0,
  "confidence": 0.95,
  "truth": 59.0,
  "correct": true
}

Routerauto-approved, matches the label

Show the result

{
  "id": "t1044",
  "segment": "typed",
  "amount": 59.0,
  "confidence": 0.95,
  "truth": 59.0,
  "correct": true
}

Claudetyped form: 37.25, confidence 0.95

Show the result

{
  "id": "t1045",
  "segment": "typed",
  "amount": 37.25,
  "confidence": 0.95,
  "truth": 37.25,
  "correct": true
}

Routerauto-approved, matches the label

Show the result

{
  "id": "t1045",
  "segment": "typed",
  "amount": 37.25,
  "confidence": 0.95,
  "truth": 37.25,
  "correct": true
}

Claudetyped form: 89.0, confidence 0.95

Show the result

{
  "id": "t1046",
  "segment": "typed",
  "amount": 89.0,
  "confidence": 0.95,
  "truth": 89.0,
  "correct": true
}

Routerauto-approved, matches the label

Show the result

{
  "id": "t1046",
  "segment": "typed",
  "amount": 89.0,
  "confidence": 0.95,
  "truth": 89.0,
  "correct": true
}

Every row was approved, and every one matches its label.

The starting router trusts the number Claude gives

The report holds one accuracy for everything, and a row is approved when Claude's confidence is high enough. Nothing compares the confidence with a known answer.

starter/route.py

1"""Build an accuracy report from labelled rows, then route each row."""23from typing import Any45TARGET = 0.9678def by_segment(rows: list[dict[str, Any]]) -> dict[str, float]:9 """Accuracy per form type.10 TODO(workshop) step 1 of 3:11 one number per segment, not one number for everything.12 """13 right = sum(1 for r in rows if r["correct"])1415 return {"all": right / len(rows) if rows else 0.0}161718def auto_approve(row: dict[str, Any], report: dict[str, float]) -> bool:19 """Whether this row skips a human.20 TODO(workshop) step 2 of 3:21 approve only when the row's own segment meets the target22 and the read was clear. A segment that was never measured is held.23 """24 return bool(row["confidence"] >= TARGET)252627def sample(rows: list[dict[str, Any]], per: int, seed: int = 0) -> list[Any]:28 """Approved rows to check against the labels.29 TODO(workshop) step 3 of 3: draw `per` rows at random from every segment.30 """31 return rows[:per]

Will scanned forms with high confidence be approved?

Take the scanned queue. Some digits were misread by the scanner, so the text Claude sees differs from the paper.

Predict: What confidence does Claude give the misread rows?

AThe same as the clean rows, because the text looks normal to it.
BLower than the clean rows, because Claude notices something is off.

Show the answer

A. Claude read what the text said, and the text looked fine. The confidence is a guess about the reading, not a check against the paper.

Wrong rows pass at high confidence

This is the scanned queue with the starting router. The red cards are rows that were approved and are wrong. Claude rated them the same as the right ones.

This is a real Claude Haiku 5.5 run, recorded once and replayed here. Across the nine recorded forms, 7 are right, one overall figure of 78 percent. Split by form type, typed forms are 6 of 6 and scanned forms are 1 of 3. The one number hides the weak type.

Claudescanned form: 128.0, confidence 0.95

Show the result

{
  "id": "s2051",
  "segment": "scanned",
  "amount": 128.0,
  "confidence": 0.95,
  "truth": 126.0,
  "correct": false
}

Routerauto-approved, but the label says 126.0

Show the result

{
  "id": "s2051",
  "segment": "scanned",
  "amount": 128.0,
  "confidence": 0.95,
  "truth": 126.0,
  "correct": false
}

Claudescanned form: 74.0, confidence 0.95

Show the result

{
  "id": "s2052",
  "segment": "scanned",
  "amount": 74.0,
  "confidence": 0.95,
  "truth": 74.0,
  "correct": true
}

Routerauto-approved, matches the label

Show the result

{
  "id": "s2052",
  "segment": "scanned",
  "amount": 74.0,
  "confidence": 0.95,
  "truth": 74.0,
  "correct": true
}

Claudescanned form: 96.5, confidence 0.95

Show the result

{
  "id": "s2053",
  "segment": "scanned",
  "amount": 96.5,
  "confidence": 0.95,
  "truth": 98.5,
  "correct": false
}

Routerauto-approved, but the label says 98.5

Show the result

{
  "id": "s2053",
  "segment": "scanned",
  "amount": 96.5,
  "confidence": 0.95,
  "truth": 98.5,
  "correct": false
}

2 wrong row(s) were auto-approved. Nobody looked.

Measure each form type, then route on the measurement

A row is also held when it has no amount or a confidence below a floor of 0.5, even in a proven type. A missing read is never approved.

Three changes, made in this order in route.py. The report gives accuracy per form type from the labelled rows. A row is approved only when its type has met the target and the row has an amount and some confidence. An unmeasured type is held.

A stratified sample of the approved rows is also checked: a random draw from every type, so a type that drifts is noticed.

route.pygreen lines are what the fix adds

1"""Build an accuracy report from labelled rows, then route each row."""23import random4from typing import Any56TARGET = 0.97FLOOR = 0.58910def by_segment(rows: list[dict[str, Any]]) -> dict[str, float]:11 """Accuracy per form type, measured against the labels."""12 report: dict[str, float] = {}1314 for segment in sorted({r["segment"] for r in rows}):15 pool = [r for r in rows if r["segment"] == segment]16 report[segment] = sum(1 for r in pool if r["correct"]) / len(pool)1718 return report192021def auto_approve(row: dict[str, Any], report: dict[str, float]) -> bool:22 """Approve a proven segment when the read was clear; hold the rest."""23 proven = report.get(row["segment"], 0.0) >= TARGET24 clear = row["amount"] is not None and row["confidence"] >= FLOOR2526 return bool(proven and clear)272829def sample(rows: list[dict[str, Any]], per: int, seed: int = 0) -> list[Any]:30 """`per` random rows from every segment, so a rare type is not missed."""31 rng = random.Random(seed) # noqa: S311 - a sample, not security32 found: list[Any] = []3334 for segment in sorted({r["segment"] for r in rows}):35 pool = [r for r in rows if r["segment"] == segment]36 found += rng.sample(pool, min(per, len(pool)))3738 return found

We doChange one setting at a time and predict.

The same queue now ends differently

Switch between routing by confidence and routing by segment. The card marked "This is where the runs split" is the first wrong row that got through.

The forms and Claude's readings did not change. Only the router did.

This part is interactive in the lab: run ./workshop watch confidence-review.

What goes to a human

Switch between the two routers on the scanned queue. A weak type, an unmeasured type and a row with no amount all go to a human. A field whose source is unclear, such as a faded digit, goes to a human with the scan attached.

The router holds a row for a reason it can name, and never because Claude sounded unsure.

When two forms give different totals for one order, keep both values, each with the form it came from. Pass every value on with its source and the date it was read, so the next reader can check it.

This part is interactive in the lab: run ./workshop watch confidence-review.

Try it: routing

Change one setting at a time. On the typed queue both routers approve everything. On the scanned queue only the measured router holds the rows.

Every combination is a recorded run, so this works with no internet.

This part is interactive in the lab: run ./workshop watch confidence-review.

The Agent SDK reads the same forms

The Agent SDK serves the same extract_form tool, and the same router gives the same routing.

This is a real Claude Haiku 5.5 run through the SDK, and the router holds the scans here too.

app/sdk_agent.pyfile: 62 lines

···19 lines not shown202122def build_server() -> McpSdkServerConfig:23 """Serve `extract_form` in-process."""2425 @tool(TOOL, "Record the refund amount and a confidence.", SCHEMA)26 async def handler(args: Json) -> Json:27 return {"content": [{"type": "text", "text": "recorded"}]}2829 return create_sdk_mcp_server(SERVER, tools=[handler])303132def fields_of(messages: list[Json]) -> Json | None:33 """The input of the first `extract_form` call."""34 for message in messages:35 for item in message.get("content", []):36 if item["type"] == "tool_use" and item["name"] == PREFIX + TOOL:37 return item["input"]3839 return None404142async def run(43 queue: str, route: ModuleType, llm: sdk.Live | sdk.Replay44) -> Handled:45 """Read every form in the queue on the SDK, then route them."""46 rows: list[Json] = []4748 for form in QUEUES[queue]:49 request = sdk.build_request(50 form.text, SYSTEM + ONCE, [PREFIX + TOOL], max_turns=251 )52 messages = await llm(request, mcp_servers={SERVER: build_server()})53 rows.append(row_of(form, fields_of(messages)))5455 return finish(queue, rows, route)565758def run_sync(59 queue: str, route: ModuleType, llm: sdk.Live | sdk.Replay60) -> Handled:61 """Run the SDK reading from synchronous code."""62 return asyncio.run(run(queue, route, llm))

Claudetyped form: 18, confidence 0.95

Show the result

{
  "id": "t1041",
  "segment": "typed",
  "amount": 18,
  "confidence": 0.95,
  "truth": 18.0,
  "correct": true
}

Routerauto-approved, matches the label

Show the result

{
  "id": "t1041",
  "segment": "typed",
  "amount": 18,
  "confidence": 0.95,
  "truth": 18.0,
  "correct": true
}

Claudetyped form: 24.5, confidence 0.97

Show the result

{
  "id": "t1042",
  "segment": "typed",
  "amount": 24.5,
  "confidence": 0.97,
  "truth": 24.5,
  "correct": true
}

Routerauto-approved, matches the label

Show the result

{
  "id": "t1042",
  "segment": "typed",
  "amount": 24.5,
  "confidence": 0.97,
  "truth": 24.5,
  "correct": true
}

Claudetyped form: 12, confidence 0.97

Show the result

{
  "id": "t1043",
  "segment": "typed",
  "amount": 12,
  "confidence": 0.97,
  "truth": 12.0,
  "correct": true
}

Routerauto-approved, matches the label

Show the result

{
  "id": "t1043",
  "segment": "typed",
  "amount": 12,
  "confidence": 0.97,
  "truth": 12.0,
  "correct": true
}

Claudetyped form: 59, confidence 0.97

Show the result

{
  "id": "t1044",
  "segment": "typed",
  "amount": 59,
  "confidence": 0.97,
  "truth": 59.0,
  "correct": true
}

Routerauto-approved, matches the label

Show the result

{
  "id": "t1044",
  "segment": "typed",
  "amount": 59,
  "confidence": 0.97,
  "truth": 59.0,
  "correct": true
}

Claudetyped form: 37.25, confidence 0.97

Show the result

{
  "id": "t1045",
  "segment": "typed",
  "amount": 37.25,
  "confidence": 0.97,
  "truth": 37.25,
  "correct": true
}

Routerauto-approved, matches the label

Show the result

{
  "id": "t1045",
  "segment": "typed",
  "amount": 37.25,
  "confidence": 0.97,
  "truth": 37.25,
  "correct": true
}

Claudetyped form: 89, confidence 0.95

Show the result

{
  "id": "t1046",
  "segment": "typed",
  "amount": 89,
  "confidence": 0.95,
  "truth": 89.0,
  "correct": true
}

Routerauto-approved, matches the label

Show the result

{
  "id": "t1046",
  "segment": "typed",
  "amount": 89,
  "confidence": 0.95,
  "truth": 89.0,
  "correct": true
}

Claudescanned form: 128, confidence 0.9

Show the result

{
  "id": "s2051",
  "segment": "scanned",
  "amount": 128,
  "confidence": 0.9,
  "truth": 126.0,
  "correct": false
}

Routerheld for a human

Show the result

{
  "id": "s2051",
  "segment": "scanned",
  "amount": 128,
  "confidence": 0.9,
  "truth": 126.0,
  "correct": false
}

Claudescanned form: 74, confidence 0.97

Show the result

{
  "id": "s2052",
  "segment": "scanned",
  "amount": 74,
  "confidence": 0.97,
  "truth": 74.0,
  "correct": true
}

Routerheld for a human

Show the result

{
  "id": "s2052",
  "segment": "scanned",
  "amount": 74,
  "confidence": 0.97,
  "truth": 74.0,
  "correct": true
}

Claudescanned form: 96.5, confidence 0.95

Show the result

{
  "id": "s2053",
  "segment": "scanned",
  "amount": 96.5,
  "confidence": 0.95,
  "truth": 98.5,
  "correct": false
}

Routerheld for a human

Show the result

{
  "id": "s2053",
  "segment": "scanned",
  "amount": 96.5,
  "confidence": 0.95,
  "truth": 98.5,
  "correct": false
}

Weak segments went to a human. Nothing wrong was approved.

You doFix the starting code, then test yourself.

You do: route by segment

Copy the starting code and open work/confidence-review/route.py. It has three steps. Accuracy is one overall number, approval reads only the model's confidence, and the audit sample takes the first rows.

Terminal

./workshop start confidence-review
./workshop check confidence-review
./workshop watch confidence-review

The checks want accuracy per segment that matches the labels. A weak or unmeasured segment is held even when confident, and the audit sample covers every segment.

Check your understanding

Answer each question, then read why every option is right or wrong. The wrong options are the mistakes engineers make on the job.

1. A refund router shows 97% accuracy over 900 forms a day, but scanned forms are right 61% of the time. What report would have shown this?

AOne accuracy figure over the whole day, recomputed every hour.
BThe average confidence Claude wrote on each form.
CAccuracy for each form type, measured against human labels.
DA count of how many forms a human reviewed.

Show the answer

Answer: C

A A fresher average still blends the typed forms with the scans, so the weak type stays hidden.
B Confidence is what Claude wrote about its own reading. It is not a check against the paper.
C Correct. A per-type figure separates the 61% from the 97%.
D Review counts say how busy people were, not how often a type is wrong.

2. A router approves any form with a confidence of 0.9 or more. Nobody has compared that number with the true amounts. What should the team do first?

AScore a labelled set and learn how often each confidence level is right.
BAsk Claude to explain how it chooses its confidence numbers.
CRaise the bar to 0.95, since a higher number is a safer number.
DSend every form to a human until the model improves.

Show the answer

Answer: A

A Correct. Labels are the only way to learn what a confidence level is worth.
B Claude can describe its number, but the description is not a measurement of how often it is right.
C A higher bar on an unchecked number is still a guess, and scans may score high anyway.
D This works, but it gives up the speed of forms that have proved themselves.

3. The desk adds a new form type, handwritten notes, and there are 0 labelled examples. Claude rates them at 0.96. How should the router treat them?

AApprove them, since the typed forms and the scans both passed review.
BApprove them, since 0.96 is above the bar the router already uses.
CApprove them while the overall accuracy stays above the target.
DHold them until the type has been measured against labels.

Show the answer

Answer: D

A Trust belongs to a measured type. A new type has not earned what another type earned.
B The number is a self-rating, and for this type nobody has checked it.
C The overall figure is dominated by the larger types and cannot vouch for a new one.
D Correct. An unmeasured segment is held, however confident Claude sounds.

4. Which two measurements should be in place before a form type skips review? Select two.

AThe overall accuracy across every form type, as one figure.
BAccuracy for that form type, against human labels.
CAccuracy for each field, such as the amount, within that type.
DThe average confidence Claude reports for that type.

Show the answer

Answer: B, C

A One blended figure can be high while a single type or field is poor.
B Correct. A type is trusted on its own measured accuracy.
C Correct. A type can be right on the name and wrong on the amount, so fields are measured too.
D This is Claude's opinion of its own work, not a measurement.

5. A scanned form reads 128.00 when the paper says 126.00. The value is a well-formed amount and Claude wrote 0.95. Which check catches it?

AA check that the amount is a number.
BA comparison with the labelled amount, reported for the scanned type.
CA second question asking Claude whether it is sure.
DHolding every form that has a decimal point in it.

Show the answer

Answer: B

A A well-formed number can still be wrong, so this check passes it.
B Correct. Only the label knows the paper said 126.00, and the type's accuracy follows from it.
C Claude rates the same text it just read, so it tends to repeat the same answer.
D Most decimal amounts are right. This holds good rows and does not target the weak type.

6. A scanned form has no amount at all, and Claude returns a confidence of 0.97 with the amount empty. What should the router do?

AApprove it, because the confidence clears the bar.
BAsk Claude to lower its own confidence and route again.
CHold every form in the queue until this one is resolved.
DHold this row for a human, since there is no value to approve.

Show the answer

Answer: D

A A high number beside an empty field is a mismatch, and approving it sends nothing to the ledger.
B Claude adjusting its own number does not add the missing amount.
C One empty row does not make the other rows wrong.
D Correct. A row with no value goes to a human even when the number beside it is high.

7. Approved rows are never looked at again, and a type that was 98% right last month has drifted. How does the team find out?

ASample a few approved rows from each type at random and compare them with labels.
BSample only the rows with the lowest confidence, since those are likeliest to be wrong.
CRecompute the overall accuracy from the rows that were reviewed.
DStop approving anything automatically until the month ends.

Show the answer

Answer: A

A Correct. A random sample of the approved rows keeps measuring the error the router cannot see.
B Approved rows are the high scores. Drift hides in rows that looked fine.
C Reviewed rows were the ones already held, so they say nothing about approved rows.
D This stops the drift by stopping the automation, and it is not a measurement.

8. Which two things must a review sample of approved rows include? Select two.

ARows from every form type that is approved.
BOnly the rows Claude rated the lowest.
CEvery approved row, so nothing is missed.
DRows chosen at random inside each type.

Show the answer

Answer: A, D

A Correct. A type left out of the sample is a type nobody is measuring.
B Low scores were probably not approved, so this sample misses the rows that need checking.
C Checking every row removes the saving the router was built to give.
D Correct. Random picks stop the sample from repeating the rows that were easy to choose.

9. Two forms give different totals for the same order, 84.00 and 48.00. Both are read at high confidence. What should the record keep?

AThe value with the higher confidence, and drop the other.
BNeither value, and ask a human to retype the order from scratch.
CBoth values, each with the form it came from.
DThe value Claude says it trusts more, after asking it.

Show the answer

Answer: C

A Two confidence numbers that are both high do not settle which form is right.
B Throwing away both loses what a human needs to resolve it.
C Correct. Keeping both with their sources lets a human decide with the evidence in front of them.
D Claude has no fresh information, so asking again repeats its guess.

10. A field in a faded scan cannot be told apart from a similar digit, and the source is unclear. What happens to that field?

AIt is approved at the confidence Claude gave it.
BIt goes to a human, with the scan attached.
CClaude is asked to guess again until two readings match.
DThe whole queue waits until every unclear field is settled.

Show the answer

Answer: B

A A confident guess about an unclear source is still a guess.
B Correct. A human can look at the paper, which the model cannot do better than it did.
C Two matching guesses from the same unclear text are not evidence.
D The other rows are clear, so holding them costs time and buys nothing.

11. A report says a refund total was 96.50 and does not say which form it came from or when it was read. What is missing?

AThe source and the date, kept next to the value.
BAn average accuracy figure for the month.
CA higher confidence score beside the total.
DA sentence from Claude saying the total is reliable.

Show the answer

Answer: A

A Correct. A value with no source or date cannot be checked or told apart from an older one.
B An average about the month says nothing about where this one total came from.
C Confidence is about the reading, and it does not say where the value came from.
D A claim of reliability is not provenance.

12. Which two items should survive with each extracted value when the record is passed on? Select two.

AThe confidence number, used as proof that it is right.
BThe source the value was read from.
CThe overall accuracy of the whole pipeline.
DThe date the value was read.

Show the answer

Answer: B, D

A The number is a self-rating and proves nothing on its own.
B Correct. The source lets the next reader check the value against the paper.
C The pipeline figure describes all rows, and it does not describe this one.
D Correct. The date shows whether a newer reading would replace it.

Play: predict the output

This audit draws 30 rows at random from 1,000 approved rows. 20 of them are handwritten forms.

python

def sample(rows: list[dict[str, Any]], n: int) -> list[Any]:
    return random.sample(rows, n)

Predict: How often does the sample hold no handwritten form at all?

AAbout half the time.
BAlmost never, since 30 rows is plenty.

Show the answer

A. Each draw misses a 2 percent segment 98 times in 100, so 30 draws miss it about 54 percent of the time. Draw a fixed number from every segment, so a rare type is always read.

References

Create strong empirical evaluations: how to build the labelled set your accuracy comes from.
Structured outputs: the form fields read back in a shape you can score.
Agent SDK custom tools: the routing rule served as a tool to an SDK agent.
Demystifying evals for AI agents: how teams grade agent output and keep the grades honest.

Conclusions

A confidence Claude reports is a self-rating. It routes nothing until it is checked against labels.
An overall accuracy hides a weak segment. Report accuracy by form type and by field.
A segment skips a human only after it meets its target. An unmeasured segment is held.

You can now route work on measured accuracy instead of a number Claude gives.

Calibration

Checking a confidence against known answers

Labelled set

Rows where the right answer is known

Segment

A slice of the data, such as a form type

Auto-approve

A row that skips a human

Stratified sample

A random sample drawn from every segment, to keep measuring approved rows