Foundations, lesson 20

Validate and retry

Retry with the exact error, and only when a retry can help

The invoice extractor fails a record, asks again, and asks again. One invoice has a date in the wrong form, and another has no date at all. Only one of them is worth a second call. This is the extractor at Lumen Supply, an online electronics shop. You will watch a blind retry lose a good record, then write a validator that decides when another call can help.

I doWatch the failure and its fix.

The validator decides whether another call can help

Claude fills in an invoice record through a tool. A validator checks the record against what the ledger accepts. When the record fails, the validator has to decide what happens next.

Three things go wrong in turn. With no retry, invalid output ends the run. A blind retry that sends only "Try again" loses the record. A retry on absent data has nothing to fix.

A format error is fixed by one retry

The invoice writes its due date as "3rd of September 2026". The ledger wants a year, month and day. The validator returns the exact error, and the second call fixes the date.

Open the cards to see what the second call was sent: the document, the failed record and the error.

SupplierInvoice from Brightline Freight Total: $310.00 Due date: 3rd of September 2026
Show the result
```
Invoice from Brightline Freight
Total: $310.00
Due date: 3rd of September 2026
```

Call 1{"vendor": "Brightline Freight", "total": 310.0, "due_date": "3rd of September 2026"}

Show the result

{
  "vendor": "Brightline Freight",
  "total": 310.0,
  "due_date": "3rd of September 2026"
}

Validatorretry

Show the result

{
  "vendor": "Brightline Freight",
  "total": 310.0,
  "due_date": "3rd of September 2026"
}

Call 2{"vendor": "Brightline Freight", "total": 310.0, "due_date": "2026-09-03"}

Show the result

{
  "vendor": "Brightline Freight",
  "total": 310.0,
  "due_date": "2026-09-03"
}

Validatoraccepted

Show the result

{
  "vendor": "Brightline Freight",
  "total": 310.0,
  "due_date": "2026-09-03"
}

The record was accepted after 2 call(s).

The starting retry says try again

The starting validator retries every error, and the retry is two words. It does not say what failed, and it does not even repeat the invoice.

It also cannot tell a value that is wrong from a value that is not there.

A good retry names the field and the error, shows the failed record, and puts the invoice text in front of the model again. You will write that retry in the fix below.

starter/validate.py

1"""Check a record, decide whether to retry, and write the retry."""23from typing import Any45from app.model import raw_errors678def check(data: dict[str, Any], document: str) -> list[dict[str, str]]:9 """Every error in `data`, each with a field, a kind and a message.10 TODO(workshop) step 1 of 4:11 kind is "absent" when the field is null and the document12 never mentions it, and "format" when the value is there but wrong.13 """14 return [{**e, "kind": "format"} for e in raw_errors(data)]151617def retryable(errors: list[dict[str, str]]) -> bool:18 """Whether asking Claude again can fix these errors.19 TODO(workshop) step 2 of 4:20 retry only format errors. Absent data stays null.21 """22 return bool(errors)232425def retry_prompt(26 document: str, data: dict[str, Any] | None, error: dict[str, str]27) -> str:28 """The next request.29 TODO(workshop) step 3 of 4:30 send the document, the failed output and the error.31 """32 return "Try again."333435def more_tokens(stop_reason: str, tokens: int, calls: int) -> int | None:36 """A larger token limit for a reply that was cut off, or None.37 TODO(workshop) step 4 of 4:38 when the reply stopped at max_tokens and this was the39 first call, return a larger limit. Otherwise return None.40 """41 return None

Will a retry find a due date the invoice never had?

One invoice says only that payment terms are to be agreed. It has no due date, and the validator reports an error.

Predict: What does a second call return?

AThe due date, since asking again gives the model another chance.
BAnother failure, or a made-up date, since the fact is not in the document.

Show the answer

B. The fact is not in the document, so no wording of a retry can find it. The recording shows what the blind retry did instead.

Blind retries lose the record

The first call was right: the date is null, because the invoice has none. The validator called that an error, and the retry said only "Try again." The second call had no invoice to read, and it returned placeholders for every field.

A retry without the document can turn a good record into a worse one.

SupplierInvoice from Tallmark Events Total: $5,000.00 Payment terms to be agreed
Show the result
```
Invoice from Tallmark Events
Total: $5,000.00
Payment terms to be agreed
```

Call 1{"vendor": "Tallmark Events", "total": 5000.0, "due_date": null}

Show the result

{
  "vendor": "Tallmark Events",
  "total": 5000.0,
  "due_date": null
}

Validatorretry

Show the result

{
  "vendor": "Tallmark Events",
  "total": 5000.0,
  "due_date": null
}

Call 2{"vendor": "<UNKNOWN>", "total": 0, "due_date": null}

Show the result

{
  "vendor": "<UNKNOWN>",
  "total": 0,
  "due_date": null
}

Validatorformat error on total: Input should be greater than 0

Show the result

[
  {
    "field": "total",
    "message": "Input should be greater than 0",
    "kind": "format"
  },
  {
    "field": "due_date",
    "message": "Input should be a valid string",
    "kind": "format"
  }
]

The run made 2 call(s) and did not end well.

Name the error, then retry only what can change

The validator now sorts each error. A date that is null and never mentioned in the document is absent. A value that is there but written wrong is a format error.

Only a format error is retried, once, and the retry carries the document, the failed record and the exact error. Absent data stays null and is flagged for a human to review.

Refining code with a model follows the same rule: give it information, not pressure. Add an input and expected output example for each shape that fails. Write tests first for requirements you can check, and iterate on their failures. When the requirement is unclear, ask the model to interview you before it writes code.

validate.pygreen lines are what the fix adds

1"""Check a record, decide whether to retry, and write the retry."""23from typing import Any45from app.model import LABELS, raw_errors67GROWTH = 108910def check(data: dict[str, Any], document: str) -> list[dict[str, str]]:11 """Each error, as absent when the document never mentions the field."""12 found: list[dict[str, str]] = []1314 for e in raw_errors(data):15 label = LABELS[e["field"]]16 gone = data.get(e["field"]) is None and label not in document.lower()17 found.append({**e, "kind": "absent" if gone else "format"})1819 return found202122def retryable(errors: list[dict[str, str]]) -> bool:23 """Only a format error can be fixed by asking again."""24 return any(e["kind"] == "format" for e in errors)252627def retry_prompt(28 document: str, data: dict[str, Any] | None, error: dict[str, str]29) -> str:30 """The document, the failed output and the exact error."""31 return (32 f"{document}\n\nYour last record was {data}.\n"33 f"It failed on {error['field']}: {error['message']}. "34 "Record the invoice again and fix only that field."35 )363738def more_tokens(stop_reason: str, tokens: int, calls: int) -> int | None:39 """A larger limit for a reply cut off at max_tokens, once."""40 if stop_reason == "max_tokens" and calls == 1:41 return tokens * GROWTH4243 return None

We doChange one setting at a time and predict.

The same invoice, now ending differently

Switch between the blind retry and the exact-error retry. The card marked "This is where the runs split" is the second call: a record of placeholders in one run, and no second call in the other.

The invoice did not change. The validator decided whether another call could help.

This part is interactive in the lab: run ./workshop watch validate-retry.

When a reply stops at max_tokens

A long invoice can use up the token limit before the record is finished. This loop starts every call with a limit of 100 tokens, small on purpose, so the long invoice shows the cut-off. The reply stops at max_tokens, and the record arrives with its last fields missing. The starting loop accepts it.

Switch to the retry with more room. The validator sees the stop reason, asks once more with a larger limit, and every field arrives. A prefill starts the reply for the model and keeps a plain-text answer in shape, though newer large models reject one. This loop forces a tool, so it raises the limit instead.

This part is interactive in the lab: run ./workshop watch validate-retry.

Try it: retries and invoices

Change one setting at a time. With no retry, the first invalid record ends the run. With a blind retry, the second call loses the record. With the exact error, the date is fixed, and an absent date is flagged after one call.

Every combination is a recorded run, so this works with no internet. A schema cannot tell a missing date from a badly written one. The validator can.

A schema checks shape only. It cannot tell a true value from a wrong one of the right form, and it cannot fill a fact the document lacks. Code checks a value against its source. Force the tool so a record always arrives, and make a field nullable when the source may lack it.

This part is interactive in the lab: run ./workshop watch validate-retry.

The same loop runs in the Agent SDK

The Agent SDK serves the same tool in-process, and the same validate.py decides whether to ask again. The SDK does not force a tool, so the system prompt asks Claude to call it once.

This is a real Claude Haiku 5.5 run through the SDK. The second call carries the exact error, and the date is fixed.

app/sdk_agent.pyfile: 73 lines

···49 lines not shown50 for _ in range(MAX_CALLS):51 request = sdk.build_request(52 prompt, SYSTEM + ONCE, [PREFIX + TOOL], max_turns=2, tag=tag53 )54 messages = await llm(request, mcp_servers={SERVER: build_server()})55 data = fields_of(messages)56 records.append(data)57 errors = validate.check(data or {}, ticket.text)5859 if not errors or not validate.retryable(errors):60 break6162 prompt = validate.retry_prompt(ticket.text, data, errors[0])6364 stops = ["end_turn"] * len(records)6566 return Handled(ticket, records, errors, stops, flagged=bool(errors))676869def run_sync(70 ticket: Ticket, validate: ModuleType, llm: sdk.Live | sdk.Replay, tag: str71) -> Handled:72 """Run the SDK loop from synchronous code."""73 return asyncio.run(run(ticket, validate, llm, tag))

SupplierInvoice from Brightline Freight Total: $310.00 Due date: 3rd of September 2026
Show the result
```
Invoice from Brightline Freight
Total: $310.00
Due date: 3rd of September 2026
```

Call 1{"vendor": "Brightline Freight", "total": 310, "due_date": "3rd of September 2026"…

Show the result

{
  "vendor": "Brightline Freight",
  "total": 310,
  "due_date": "3rd of September 2026",
  "line_items": []
}

Validatorretry

Show the result

{
  "vendor": "Brightline Freight",
  "total": 310,
  "due_date": "3rd of September 2026",
  "line_items": []
}

Call 2{"vendor": "Brightline Freight", "total": 310, "due_date": "2026-09-03", "line_items": []}

Show the result

{
  "vendor": "Brightline Freight",
  "total": 310,
  "due_date": "2026-09-03",
  "line_items": []
}

Validatoraccepted

Show the result

{
  "vendor": "Brightline Freight",
  "total": 310,
  "due_date": "2026-09-03",
  "line_items": []
}

The record was accepted after 2 call(s).

You doFix the starting code, then test yourself.

You do: retry what a retry can fix

Copy the starting code and open work/validate-retry/validate.py. It has four steps. Every error counts as a format error, every error is retried, the retry says only "Try again" and a cut-off reply is never given more room.

Terminal

./workshop start validate-retry
./workshop check validate-retry
./workshop watch validate-retry

The checks tell an absent field from a badly written one and retry only format errors. The retry carries the document, the failed output and the error, a cut-off reply is retried once, and no run makes more than two calls.

Check your understanding

Answer each question, then read why every option is right or wrong. The wrong options are the mistakes engineers make on the job.

1. A due date written as "3rd of September 2026" fails validation on 12 of 200 invoices. The value is on the page, only in the wrong form. What is the best response?

ADrop due_date from the required fields, so those records pass.
BSend the same request again, since output varies between runs.
CRetry once with the document, the failed record and the exact error.
DRaise the retry limit to five, so the same request gets more chances.

Show the answer

Answer: C

A Dropping the field hides the failure, and the ledger loses a date it could have had.
B An identical request has the same odds of the same mistake. Nothing tells the model what was wrong.
C Correct. The fact is in the document, so one retry with the exact error can fix the form.
D More attempts of a request that does not mention the error only repeat the error.

2. 20 invoices in a batch state no due date. The loop retries each one 5 times and then fails. What should the loop do instead?

AKeep the date null and flag the invoice for a person after the first call.
BRetry until a date appears, since every invoice has one somewhere.
CRetry with the exact error each time, so the message eventually gets through.
DRemove due_date from the record, so the validator stops complaining.

Show the answer

Answer: A

A Correct. The document has no date to find, so a retry cannot help and a person has to decide.
B Not every invoice has a date, and a loop that waits for one will eventually get an invented one.
C A clear error message does not add a fact the document lacks.
D The ledger needs to know the date is missing. Removing the field loses that.

3. The retry message is "Try again." and the same error comes back on 9 of 10 retried records. What change fixes it?

AAllow more retries per record, so one of them gets it right.
BKeep the message short, since the error text only adds noise.
CRaise the temperature, so the second reply differs from the first.
DSend the document, the failed output and the specific error.

Show the answer

Answer: D

A Each extra attempt carries the same two words, so it has the same odds.
B The error text is the information the model needs to change anything.
C More variety is not aimed at the mistake. It can turn a small error into a different one.
D Correct. With the document, the output and the error, the model can see what to change.

4. Which two items belong in a retry message after a format error? Select two.

AA request that the model tries harder this time.
BThe record that failed validation.
CThe specific validation error, such as the field and what was wrong with it.
DPermission to leave the field out if it is hard to fill.

Show the answer

Answer: B, C

A Effort is not the problem. The model needs to know what to change.
B Correct. The failed record shows the model exactly what it wrote.
C Correct. The field and the reason point the retry at the one thing to fix.
D Leaving the field out trades a fixable format error for missing data.

5. On 6 of 300 invoices the record matches the schema, but the total is 1,000 times too large. What catches this?

ATighten the schema until it forbids the wrong number.
BA check in code that compares the value with what the document says.
CRetry every record once to be safe.
DForce the tool, so the values are checked by the API.

Show the answer

Answer: B

A A schema says what shapes are allowed. A wrong number of the right shape passes it.
B Correct. A value is checked against its source by code that knows what the source says.
C A blind retry on every record doubles the cost and may produce the same wrong number.
D Forcing the tool fixes the shape of the reply. It has no say over what the values mean.

6. 5 of 100 replies arrive as prose, and the ledger needs a record every time. What should change?

ARetry the prose replies until one of them holds a record.
BPull the fields out of the prose with a regular expression.
CRun the validator on the prose, so it rejects anything that is not a record.
DSet tool_choice to the extraction tool, so a record always arrives.

Show the answer

Answer: D

A Each retry may answer in prose again, and each costs a call.
B Prose changes wording from run to run, so a pattern breaks on the first new phrasing.
C A validator rejects prose. It cannot turn it into a record.
D Correct. A named tool makes the reply the tool's input, in the shape of the schema.

7. Invented values appear in 15 of 100 records, all in a field the source often lacks. What is the fix at the source?

AMake the field nullable, so silence has a place to go.
BRetry each invented value until one passes validation.
CKeep the field required and add a pattern that checks its form.
DAsk for the field again with the exact error, so the model finds it.

Show the answer

Answer: A

A Correct. When the source may lack a fact, the honest answer has to fit the schema.
B The next value to pass validation is another invention that looks right.
C A pattern checks form. A made-up value of the right form passes.
D An error message cannot add a fact the document lacks.

8. Which two things can a schema not do? Select two.

ATell whether a value in the right form is the true one.
BReject a record whose total arrives as a string.
CRequire that a due_date field is present in the record.
DFill in a fact that the document never states.

Show the answer

Answer: A, D

A Correct. The schema checks the shape, so a wrong date in the right shape passes.
B A schema can do this. A type check is exactly what it enforces, so a string total fails.
C A schema can do this. A required field is part of the shape it checks.
D Correct. A schema describes what is allowed and adds no information.

9. A first fix for a date parser fails on 3 input shapes, and the description of the goal did not help. What do you add next?

ARepeat the goal in capital letters at the top.
BSend the same message again until a run passes.
CAn input and expected output example for each shape.
DA higher limit on attempts, from 2 to 6.

Show the answer

Answer: C

A Capital letters add emphasis, not information about the three shapes.
B The same message leaves the same gap, so runs repeat the same fix.
C Correct. Examples show the model what each shape looks like and what it should become.
D More attempts on a request that lacks the examples repeat the miss.

10. A change has 4 requirements, each easy to check. How do you build it so that progress is visible?

AWrite the code first and add tests at the end.
BWrite the tests first, run them, and iterate on their failures.
CDescribe the tests in prose and skip running them.
DDelete any failing test that the first version cannot pass.

Show the answer

Answer: B

A Tests written last check what the code happens to do, not what was wanted.
B Correct. Failing tests state the target, and each run shows what is left.
C A test that never runs says nothing about the code.
D A deleted test stops checking the requirement it stood for.

11. The requirement for a report is unclear, and the model has guessed wrong 3 times. What helps most?

ASend the same request again until a guess is right.
BPick the most likely guess and ship it.
CWrite a longer request that lists every detail you can think of.
DAsk the model to interview you with questions before it writes code.

Show the answer

Answer: D

A Each guess comes from the same missing information.
B A guess shipped is a guess that the users find.
C A longer list of details still misses the ones you did not think of.
D Correct. Questions bring out the requirements that nobody had written down.

12. Which two things make a refinement request work better? Select two.

AExamples of the output you want.
BThe same request with more emphasis.
CA failing test to iterate against.
DLeaving out the error text, so the model is not anchored on it.

Show the answer

Answer: A, C

A Correct. An example shows the target in a way a description cannot.
B Emphasis raises the stakes without adding information.
C Correct. A failing test says what is wrong and tells you when it is fixed.
D The error text is what the model needs, and leaving it out removes the signal.

Play: predict the output

This version gives a cut-off reply more room.

python

def more_tokens(stop_reason: str, tokens: int, calls: int) -> int | None:
    if stop_reason == "max_tokens":
        return tokens * 2
    return None

Predict: An invoice is so long that even double the limit cuts it off. What happens?

AIt doubles again on every call, with nothing to stop it.
BIt stops after one retry and reports the cut-off.

Show the answer

A. The rule never reads calls, so nothing bounds it. Give more room once, on the first call, and report the cut-off after that.

References

Define tools: the schema the record is checked against.
Structured outputs: what a strict schema guarantees and what it cannot check.
Pydantic validators: writing the field checks whose errors you send back.
Common workflows: the same check, fix and rerun loop in Claude Code.
Handling stop reasons: what max_tokens means and how to recover from it.

Conclusions

Validate the output, and retry with the exact error only when a retry can help.
A format error is worth one retry. Absent data stays null and is flagged for a human.
A reply cut off at max_tokens is retried once with more room.

You can now write a validator that knows when another call is worth making.

Validator

Code that checks a record against what the ledger accepts

Format error

A value that is there but written wrong, so a retry can fix it

Absent data

A value the document never states, so a retry cannot find it

Exact error

The validator's own message, sent back with the document and the failed record

Stop reason

Why a reply ended. max_tokens means it was cut off