Foundations, lesson 19

Schema extraction

A nullable schema and a forced tool make extraction trustworthy

Lumen Supply reads supplier invoices into its ledger. One vendor never prints a PO number, the schema demands one, and the model writes a value nobody wrote. You will watch that happen, then fix it. This is the extractor at Lumen Supply, an online electronics shop. The schema is the contract, and you will change what it allows.

I doWatch the failure and its fix.

The schema is the contract Claude writes to

The extractor gives Claude one tool. The tool's input schema is the shape of the record the ledger wants. Claude fills the tool in, and the ledger checks the result.

Three things go wrong in turn. The answer is free text that code cannot parse. A required field the invoice lacks gets a made-up value. A category list with no place for an odd invoice files it wrongly.

A clean invoice becomes a valid record

A supplier invoice goes in and a record comes out, with vendor, total, PO number and category. Open a card to see the data each side saw.

This works because every field is printed on the invoice. The harder cases are invoices that are silent about something.

SupplierInvoice from Northvale Cells PO number: PO-7741 Total due: $1,240.50 Items: 200 battery…
Show the result
```
Invoice from Northvale Cells
PO number: PO-7741
Total due: $1,240.50
Items: 200 battery cells
```

Claude{"vendor": "Northvale Cells", "total": 1240.5, "po_number": "PO-7741", "category": "parts"}

Show the result

{
  "vendor": "Northvale Cells",
  "total": 1240.5,
  "po_number": "PO-7741",
  "category": "parts"
}

Ledgeraccepted

Show the result

{
  "vendor": "Northvale Cells",
  "total": 1240.5,
  "po_number": "PO-7741",
  "category": "parts"
}

The record is valid and says what the invoice says.

The starting schema demands a value for everything

Every field is required, the tool choice is auto (Claude may answer in text instead of calling the tool), and the category list has no escape. Each of those is a reasonable default, and each one fails on a real invoice.

A field that is required when the source may lack it leaves Claude only one way to comply: write something.

starter/schema.py

1"""What the extractor asks Claude to fill in, and how it makes it answer."""23from typing import Any456def schema() -> dict[str, Any]:7 """The JSON schema of the `extract_invoice` tool input.8 TODO(workshop) step 1 of 3:9 let po_number be null when the invoice has none, add10 "other" to the category list with a category_detail field.11 """12 return {13 "type": "object",14 "properties": {15 "vendor": {"type": "string"},16 "total": {"type": "number"},17 "po_number": {"type": "string"},18 "category": {19 "type": "string",20 "enum": ["parts", "freight", "software"],21 },22 },23 "required": ["vendor", "total", "po_number", "category"],24 }252627def choice() -> dict[str, str]:28 """The `tool_choice` for the request.29 TODO(workshop) step 2 of 3: force the `extract_invoice` tool.30 """31 return {"type": "auto"}323334def examples() -> list[dict[str, Any]]:35 """Worked examples for the prompt: layout, invoice text, record.36 TODO(workshop) step 3 of 3:37 show two or three differently laid-out invoices.38 """39 return []

What happens with an invoice that has no PO number?

One Lumen vendor sends freight invoices with no PO number on them. The schema says the PO number is required.

Predict: What does the record hold for the PO number?

AA value the invoice does not hold, since the field must be filled.
BNull, since the invoice has none.

Show the answer

A. A required field the source lacks invites a guess. The recording shows what kind of guess.

The record holds a guess, or never arrives

With tool_choice on auto, Claude called the tool and wrote an empty string for the PO number. The schema accepts an empty string, so the ledger got a valid record that is not null.

Auto also lets Claude answer in text: on the odd invoice it replied in prose and the ledger got no record at all. Try forcing the tool, and Claude has to fill the field.

Forcing the tool kept the shape and still broke the value: the PO number became the placeholder <UNKNOWN>. A forced tool guarantees a record. It does not make the values true.

SupplierInvoice from Brightline Freight Total due: $310.00 Items: pallet shipping, Riga to Hamburg
Show the result
```
Invoice from Brightline Freight
Total due: $310.00
Items: pallet shipping, Riga to Hamburg
```

Claude{"vendor": "Brightline Freight", "total": 310.0, "po_number": "", "category": "freight"}

Show the result

{
  "vendor": "Brightline Freight",
  "total": 310.0,
  "po_number": "",
  "category": "freight"
}

Ledgeraccepted

Show the result

{
  "vendor": "Brightline Freight",
  "total": 310.0,
  "po_number": "",
  "category": "freight"
}

The record is valid but wrong. It should hold {'vendor': 'Brightline Freight', 'total': 310.0, 'po_number': None, 'category': 'freight'}.

This part is interactive in the lab: run ./workshop watch schema-extraction.

Force the tool, allow null, keep an escape

Three changes. The tool is forced, so a record always arrives. The PO number may be null, so silence on the invoice has somewhere to go. The category list ends with other and a detail field, so an odd invoice is not misfiled.

Vendor and total stay required, because every invoice has them. Those changes fill schema() and choice(), in that order: nullable PO number and other first, then the forced tool. The third stub, examples(), is filled in the examples step below.

tool_choice takes four values: auto, any, a named tool and none. A named tool forces that tool, which suits a first step that must be an extraction. Claude Haiku 5.5 accepts a forced tool. Some larger current models reject it with a 400 error, and there a structured output format (output_config.format) returns the record.

schema.pygreen lines are what the fix adds. file: 77 lines

···6 lines not shown7 """The JSON schema of the `extract_invoice` tool input."""···5 lines not shown13 "po_number": {14 "type": ["string", "null"],15 "description": "The PO number, or null if none is printed.",16 },···2 lines not shown19 "enum": ["parts", "freight", "software", "other"],20 },21 "category_detail": {22 "type": ["string", "null"],23 "description": "What the item is, when category is other.",···7 lines not shown31 """Force the tool, so the reply is always a record."""32 return {"type": "tool", "name": "extract_invoice"}333435Field = str | float | None363738def shot(layout: str, invoice: str, **record: Field) -> dict[str, Any]:39 """One worked example: the invoice as printed, and its record."""40 return {41 "layout": layout,42 "invoice": invoice,43 "record": {"category_detail": None, **record},44 }···3 lines not shown48 """Three layouts, each settling how a PO number is written: PO-####."""49 return [50 shot(51 "lines",52 "Invoice from Coldwater Cells\nYour ref: 1207\n"53 "Total due: $860.00\nItems: 40 cell packs",54 vendor="Coldwater Cells",55 total=860.0,56 po_number="PO-1207",57 category="parts",58 ),59 shot(60 "table",61 "VENDOR | Ashby Haulage\nREF | order 3310\n"62 "NET | 375.00\nTAX | 75.00\nTOTAL | 450.00 USD",63 vendor="Ashby Haulage",64 total=450.0,65 po_number="PO-3310",66 category="freight",67 ),68 shot(69 "letter",70 "Dear Lumen team, Fernhill Software bills you $2,400.00 for "71 "the annual licence, purchase order 6150.",72 vendor="Fernhill Software",73 total=2400.0,74 po_number="PO-6150",75 category="software",76 ),77 ]

We doChange one setting at a time and predict.

The same invoice now ends differently

Switch between the two schemas. The card marked "This is where the runs split" is the PO number: an empty string before, null after.

The invoice did not change. The schema gave the model an honest answer to give.

This part is interactive in the lab: run ./workshop watch schema-extraction.

Examples settle what a schema cannot

Some invoices print the PO number as a bare number, and the ledger wants the PO- form. A schema cannot say that. Worked examples of differently laid-out invoices can, and examples() returns them.

Switch between no examples and the examples. Without them the model copies the number as printed. With them it writes the form the ledger expects.

An example can also settle a name written three ways, with a one-line reason.

This part is interactive in the lab: run ./workshop watch schema-extraction.

Try it: schemas and invoices

Change one setting at a time. Pick the odd invoice with the starting schema and Claude answers in prose. Pick it with the fixed schema and it is filed under other, with a detail.

Every combination is a recorded run, so this works with no internet. A record can have the right shape and a wrong value.

This part is interactive in the lab: run ./workshop watch schema-extraction.

The same schema works in the Agent SDK

The Agent SDK serves the same tool in-process, with the same schema as its input. The SDK cannot force a tool, so the system prompt asks Claude to call it once.

This is a real Claude Haiku 5.5 run through the SDK. The PO number is null, as it should be.

app/sdk_agent.pyfile: 63 lines

1"""The same extraction on the Agent SDK, with the schema as a tool input.23The SDK cannot force a tool, so the system prompt asks for it. The schema4in `schema.py` still decides what shape Claude's input may take.5"""67from types import ModuleType89from claude_agent_sdk import McpSdkServerConfig, create_sdk_mcp_server, tool1011from kit import sdk12from kit.api import Json13from .agent import TOOL, Handled, is_valid, system14from .tickets import Ticket1516SERVER = "lumen"17PREFIX = f"mcp__{SERVER}__"18MAX_TURNS = 2192021def build_server(extract: ModuleType) -> McpSdkServerConfig:22 """Serve `extract_invoice` in-process with the learner's schema."""2324 @tool(TOOL, "Record the fields of one supplier invoice.", extract.schema())25 async def handler(args: Json) -> Json:26 return {"content": [{"type": "text", "text": "recorded"}]}2728 return create_sdk_mcp_server(SERVER, tools=[handler])293031def request_for(ticket: Ticket, extract: ModuleType, tag: str) -> Json:32 """The plain request a recording is keyed by."""33 return sdk.build_request(34 ticket.text,35 system(extract) + " Call the tool once, then stop.",36 tools=[PREFIX + TOOL],37 max_turns=MAX_TURNS,38 tag=tag + ":" + str(extract.schema()),39 )40···23 lines not shown

SupplierInvoice from Brightline Freight Total due: $310.00 Items: pallet shipping, Riga to Hamburg
Show the result
```
Invoice from Brightline Freight
Total due: $310.00
Items: pallet shipping, Riga to Hamburg
```

Claude{"vendor": "Brightline Freight", "total": 310, "po_number": null, "category": "freight"}

Show the result

{
  "vendor": "Brightline Freight",
  "total": 310,
  "po_number": null,
  "category": "freight"
}

Ledgeraccepted

Show the result

{
  "vendor": "Brightline Freight",
  "total": 310,
  "po_number": null,
  "category": "freight"
}

The record is valid and says what the invoice says.

You doFix the starting code, then test yourself.

You do: fix the schema and force the tool

Copy the starting code and open work/schema-extraction/schema.py. It has three steps. The schema requires a PO number and has no "other" category, the tool choice is auto, and the prompt has no examples.

Terminal

./workshop start schema-extraction
./workshop check schema-extraction
./workshop watch schema-extraction

The checks let a missing PO number be null, add "other" with a detail field and keep vendor and total required. They want the tool forced and examples from two different layouts.

Check your understanding

Answer each question, then read why every option is right or wrong. The wrong options are the mistakes engineers make on the job.

1. 40 of 800 weekly invoices reach the ledger with the PO number PO-0000. The vendor behind them prints no PO number at all. What is the fix at the source?

AAdd a prompt line that tells the model not to invent a PO number.
BLet po_number be null in the schema, so silence has a place to go.
CReject every record whose PO number is PO-0000 after extraction.
DKeep the field required, and add a regex that checks the PO format.

Show the answer

Answer: B

A A prompt line asks politely, while the schema still demands a value. The model has to comply with one of the two.
B Correct. A field the source may lack is nullable, so the honest answer fits the schema.
C This filters one placeholder. The model will write another one, such as N/A or 0.
D A format check cannot tell a real PO number from a made-up one that looks right.

2. The extractor offers its tool with tool_choice set to auto. In 5 of 100 runs the reply is prose that the ledger cannot parse. What should change?

AWrite a longer system prompt that insists on a tool call.
BRetry any prose reply until a tool call comes back.
CPull the JSON out of the prose with a regular expression.
DSet tool_choice to the extraction tool, so a record always arrives.

Show the answer

Answer: D

A Insisting is still a request. On auto the model may choose prose on any run.
B Retries hide the cause and cost a call each time. The next prose reply is only a matter of time.
C Prose has no stable layout, so a pattern breaks on the first reply that is worded differently.
D Correct. A named tool in tool_choice turns the reply into tool input that follows the schema.

3. A category list has six values. Each week 30 invoices for event hire land under freight, since nothing else fits. What is the change?

AAdd an other value and a category_detail field that says what the item is.
BMake category optional, so the model can leave it empty.
CAdd ten more categories until every invoice seen so far fits.
DTell the model in the prompt to pick the closest category.

Show the answer

Answer: A

A Correct. An escape value keeps the odd invoice out of a wrong category, and the detail keeps what it was.
B An empty category hides the problem from the ledger and loses the information.
C A longer list still has an end. The next odd invoice will not fit, and there is nowhere for it.
D Closest is how the misfiling happens. The record looks fine and is wrong.

4. Which two statements about extraction with a tool schema are true? Select two.

AA record can match the schema and still hold a value the invoice does not contain.
BA record that matches the schema holds values that match the invoice.
CA forced tool means the reply is a record rather than prose.
DA forced tool corrects wrong values in fields the invoice cannot fill.

Show the answer

Answer: A, C

A Correct. The schema checks the shape. A placeholder in a string field is valid.
B Shape and meaning are different checks. The schema only does the first.
C Correct. With a named tool the model has to reply with that tool's input.
D Forcing makes the model fill the field, which is when an invented value appears.

5. Totals are wrong on 12 of 100 scanned invoices. The total sits in a different place on each layout. What helps most?

ALower the temperature to zero so the model reads more carefully.
BAdd a regular expression that finds the word Total in the text.
CShow examples of invoices with the total in different places.
DShow one example of a typical invoice with the total at the bottom.

Show the answer

Answer: C

A Temperature changes variety, not whether a layout is understood.
B A pattern tied to one word breaks on a layout that says Amount due or puts the figure in a table.
C Correct. Examples of differing layouts show the model where the value hides.
D One layout teaches one place to look. The others stay as wrong as before.

6. Across 5 invoice layouts the PO number comes out as 5530, PO5530 or PO-5530. The ledger wants PO-5530. What settles the format?

AWrite a cleanup rule for each of the five layouts after extraction.
BShow examples whose records write the number in the PO- form.
CMake po_number required, so it is always filled in the same way.
DForce the tool, so the format is fixed by the tool choice.

Show the answer

Answer: B

A Five rules grow to six with the next layout, and each is code to keep.
B Correct. An example shows the target form once, and the model carries it across layouts.
C Required says the field must exist. It says nothing about how the value is written.
D Forcing settles the shape of the reply. The form of a string is not part of it.

7. One vendor appears as Harlow & Pike, Harlow and Pike Ltd, and H&P. The ledger wants one name. What gives the most consistent result?

ATurn vendor into an enum of every name seen so far.
BAdd a prompt line asking the model to be consistent.
CMerge the names with a regular expression after extraction.
DShow an example that maps the short form to the full name, with a one-line reason.

Show the answer

Answer: D

A A new vendor has no place in a closed list, and the model misfiles it into the nearest name.
B Consistent with what? The model has no way to know which form the ledger prefers.
C A pattern merges the names someone thought of. The next variant slips through.
D Correct. An example that settles the ambiguity, with its reason, carries to similar names.

8. Which two effects do worked examples have on extraction? Select two.

AThey let the model call tools that have no schema.
BThe format of a record becomes the same from one run to the next.
CEvery layout is handled, including layouts that were not shown.
DInvented values are less likely on layouts close to the ones shown.

Show the answer

Answer: B, D

A Tools come from the request. Examples only show inputs and outputs.
B Correct. A shown format is the one the model copies.
C Examples cover the layouts they resemble. A new layout is still a risk.
D Correct. Seeing where values sit on a layout cuts the guessing on similar ones.

9. A pipeline has three tools, and the first call must be extract_invoice. Some runs open with a different tool. What guarantees the order?

ASet tool_choice to the extract_invoice tool for the first request.
BWrite the first step in capital letters in the system prompt.
CSet tool_choice to any, so a tool is always called first.
DCheck the first call in code and rerun the request when it is wrong.

Show the answer

Answer: A

A Correct. Naming the tool forces that tool on the first turn, and later turns can go back to auto.
B Capital letters are emphasis, which is still a request.
C Any guarantees some tool is called. It may be the wrong one.
D A rerun costs another call and does not stop the next wrong opening.

10. A step must call one of three tools, and which one is the model's choice. Some replies are chat instead. What setting fits?

ASet tool_choice to auto, so the model calls a tool when it needs to.
BName one tool, so the model always has a tool to call.
CSet tool_choice to any, so a tool call is required and the model picks.
DSay in the prompt that a tool call is needed on every turn.

Show the answer

Answer: C

A Auto allows chat, which is the problem.
B Naming one tool takes the choice away from the model.
C Correct. Any forces a call and leaves the pick of tool to the model.
D A prompt line is a request, and the chat replies show it is not always followed.

11. An assistant has three tools and should answer in text when none of them applies. Which tool_choice fits?

AAny, so the model is sure to use a tool.
BOne named tool, so the model is never without one.
CPut the choice into each tool description instead of the request.
DAuto, so the model decides between a tool and plain text.

Show the answer

Answer: D

A Any requires a tool call, so a text answer is no longer possible.
B A named tool forces that tool on every turn, even when it does not apply.
C Descriptions help the model understand a tool. The request decides whether one is required.
D Correct. Auto leaves both the tool and the text option open.

12. Which two statements about tool_choice are true? Select two.

AIt takes auto, any, or the name of one tool.
BWith one named tool, the reply holds that tool's input.
CAuto makes the model call a tool on every turn.
DIt decides which model answers the request.

Show the answer

Answer: A, B

A Correct. Those are the three forms of the setting.
B Correct. A named tool is called, so its input is the answer.
C Auto lets the model choose, and a text reply is one of its choices.
D The model is set elsewhere in the request. The tool choice only shapes the reply.

Play: predict the output

The extractor offers one tool and asks for any tool.

python

tools = [{"name": "extract_invoice", "input_schema": schema()}]
tool_choice = {"type": "any"}

Predict: Does every reply call extract_invoice?

AYes. Any means some tool must be called, and there is only one.
BNo. Any lets Claude answer in prose when it is unsure.

Show the answer

A. With one tool, any and a named tool force the same call. Name the tool anyway, so adding a second tool later cannot change which one the first call picks.

References

Define tools: input schemas and how tool_choice forces one tool.
Structured outputs: strict schemas, so every reply parses.
Use examples effectively: why examples from different layouts settle a format.

Conclusions

A schema fixes the shape of a record. It does not make the values true.
A field the source may lack is nullable, and a category list ends with an escape and a detail.
Examples settle layouts and formats that a schema cannot express.

You can now write an extraction schema that stays honest when an invoice is silent.

Tool schema

The input schema of a tool, used as the shape of the output

Forced tool choice

Setting tool_choice to one named tool, so a record always arrives

Nullable field

A field allowed to be null when the source has no value

Escape value

An other category, paired with a detail field

Worked example

A sample input and its record, shown to the model