Foundations, lesson 17

Review criteria

Name what counts, then review one file at a time

A Lumen Supply release changes many files. A review told to be thorough reads them all in one request, flags files that are fine, and buries the real bugs. The team stops reading the comments. This is the review bot at Lumen Supply, an online electronics shop. You will watch it drown its own findings, then fix it with named criteria and one pass per file.

I doWatch the failure and its fix.

Say what counts, then look at one file at a time

A review has two parts you control. The criteria say what to report and what to skip. The passes say which files go into which request. The code splits the files, and the model reviews each one.

Three things go wrong. Being thorough flags style noise and real bugs alike. One pass over many files spreads attention thin. A fault between two files has no pass that looks for it.

Criteria and passes give findings you can trust

The fixed review names what to report and what to skip, with an example of each. It gives each file its own request, then one last request that reads all of them together.

Open the cards. The result line counts the bug files that were flagged and the clean files that were flagged.

All six bug files are flagged. Three clean files, config.py, slug.py and tax.py, are still flagged. Criteria cut the noise. They do not remove it, so a human still reads what is left.

PipelineReview the release: the changed files. (14 files)
Reviewer15 passes made

Reviewerflagged 9 files

Show the result

cart.py, config.py, dates.py, discount.py, orders.py, refunds.py, slug.py, stock.py, tax.py

Resultconfig.py: env_int calls int(value) without handling bad input, so a non-numeric or decimal override such…

Show the result

config.py: env_int calls int(value) without handling bad input, so a non-numeric or decimal override such as TIMEOUT=abc or PAGE=2.5 raises ValueError from load() and crashes startup.
slug.py: label returns title[: n - 1].rstrip() + "…", which is n characters total, but the docstring-implied budget and short aside, nothing wrong here; no finding.
slug.py: slug lambda splits on whitespace and joins with dashes but does not strip non-alphanumerics, so titles with punctuation yield non-clean slugs (e.g. "Desk, Lamp" -> "desk,-lamp"), contradicting the module's stated purpose of turning punctuation into dashes.
cart.py: subtotal adds l.price instead of line_total(l), dropping the quantity factor, so subtotal ignores multi-unit lines and summary shows the wrong total.
tax.py: add_tax hardcodes the 1.2 (20%) multiplier and ignores the category, so reduced-rate amounts get taxed at the standard rate.
refunds.py: auto_approve uses amount < limit, but its docstring says refunds up to and including the limit are paid without a person, so a refund of exactly 50_00 is wrongly sent to review.
stock.py: can_reserve checks qty against on_hand instead of available (on_hand minus reserved), so reserve() can over-reserve stock already held for other carts.
discount.py: apply_coupon uses max(percent_for(code), CAP), which picks the larger discount, so the cap is inverted: WELCOME10 becomes 50% off instead of 10%, and VIP60 is cut to 50% instead of 60%. It should be min(...).
discount.py: bulk_percent's docstring says 5% from 10 units and 10% from 50, which the code matches, but sale_price is documented as rounding to a cent while round() on float prices can give banker's rounding, not a wrong result. No finding there.
dates.py: is_expired returns True on the expiry date itself (today >= expiry), but the docstring says a voucher can still be used that day, so the check should be today > expiry.
orders.py: shipping_for passes weight_kg to quote, but shipping.shipping_cents expects grams, so shipping is undercharged (a 2.5 kg parcel is priced as 1 step instead of 5).
discount.py: apply_coupon uses max(percent_for(code), CAP), but the docstring says "never more than CAP percent off"; this should be min, so WELCOME10 takes 50% off.
cart.py: subtotal adds l.price and ignores l.qty, so multi-unit lines are undercounted, which flows into orders.goods_total.
stock.py: can_reserve checks qty <= item["on_hand"] while its docstring says it checks unreserved stock, and reserve relies on it, so stock can be over-reserved.
refunds.py: auto_approve uses amount < limit, but its docstring says amounts up to and including the limit are paid at once, and needs_review depends on it.
dates.py: is_expired uses today >= expiry, so a voucher is expired on its expiry date, contradicting the docstring that says it can still be used that day.

6 of 6 bug files flagged, 3 clean files flagged. The real bugs stand out.

The criteria and the passes as written

The starting criteria say "be thorough" and nothing else. The starting passes put every file into one request. A request of this size carries the whole release, and the reviewer spreads its attention across all of it.

The fixed review sends 15 requests: one per file, then one across all the files. Together they read about two and a half times the input tokens of the one request, and each file gets the reviewer to itself.

starter/review.py

1"""What the reviewer is told, and which files it sees in each pass."""234def criteria() -> str:5 """The instructions each pass starts with.6 TODO(workshop): say what to report and what to skip, with two or more7 examples of each, so style does not bury a real bug.8 """9 return "Be thorough. Report everything you notice."101112def passes(files: list[str]) -> list[list[str]]:13 """The files in each pass, one list per request.14 TODO(workshop): review one file per pass, then add a last pass over15 all the files, so the links between them are read.16 """17 return [files]

What does "be thorough" add?

The criteria name no category of problem. The release holds real bugs, and it also holds code that only looks odd: a TODO note, a print call, a short name.

Predict: What does the review add when it is told to be thorough?

AMore real bugs found.
BMore flags, mostly on files that are fine.

Show the answer

B. "Thorough" names no category, so a style nit counts as much as a wrong result.

One pass over the release flags every file

This is the same release with the starting review. A bug counts as found when its file is flagged. The one request flags all 14 files: the six bugs and eight files that are fine. Every bug is in the list, but a reader cannot tell which flags matter.

This is a real Claude Haiku 5.5 run, recorded once and replayed here.

PipelineReview the release: the changed files. (14 files)
Reviewer1 pass made

Reviewerflagged 14 files

Show the result

cart.py, config.py, dates.py, discount.py, formatting.py, greet.py, logs.py, notify.py, orders.py, refunds.py, shipping.py, slug.py, stock.py, tax.py

Resultconfig.py: load() omits free_shipping_from, coupon_cap, low_stock, RESERVE_MINUTES, BACKOFF and…

Show the result

config.py: load() omits free_shipping_from, coupon_cap, low_stock, RESERVE_MINUTES, BACKOFF and DEBUG inconsistently, so only some settings can be overridden from the environment.
config.py: env_int lets int() raise ValueError on a bad value, crashing startup instead of falling back to the default.
config.py: PAGE has no lower bound, so zero or negative values pass through min() unchecked.
config.py: REFUND_LIMIT, FREE_SHIPPING_FROM, COUPON_CAP and LOW_STOCK are duplicated in refunds.py, shipping.py, discount.py and stock.py, so the copies can drift apart.
config.py: USE_NEW_TAX_TABLE is read but the tax module never uses it.
slug.py: slug is a lambda assignment (E731) and it doesn't strip punctuation, unlike clean_slug, so the two produce different slugs for the same title.
slug.py: ascii_only drops every non-Latin character, so titles in other scripts produce an empty slug from clean_slug and unique_slug returns just "-2".
slug.py: short slices without stripping or adding an ellipsis, unlike label, which is inconsistent.
notify.py: import os is unused (the TODO is noted) and import sys is unused.
notify.py: customer names are interpolated unescaped, which is an injection risk if these messages are ever rendered as HTML.
notify.py: delayed says "0 days late" or negative days without checking.
notify.py: refunded converts cents to float dollars and omits the thousands separator used elsewhere.
logs.py: record, record_error and timed write with print instead of a logger, so levels and handlers are ignored.
logs.py: record_error has a commented-out debug line left in.
logs.py: line() lets fields overwrite ts, level and msg, and json.dumps raises on non-serializable values such as datetimes.
logs.py: redact() matches only exact lowercase keys, so card_number, api_key and authorization are not redacted, and mixed-case secret_keys passed by a caller never match.
logs.py: redact() does not recurse into nested dicts.
logs.py: the timed decorator's inner does not use functools.wraps, so the wrapped function loses its name and docstring.
cart.py: subtotal() adds l.price instead of line_total(l), so every quantity beyond one is ignored; this also breaks summary().
cart.py: add() accepts zero or negative qty without validation and ignores a differing price when the SKU is already in the cart.
cart.py: for l in uses the ambiguous name l (E741).
tax.py: add_tax hardcodes a 1.2 multiplier, ignoring RATES and the category, so it will go stale.
tax.py: describe uses int(rate * 100), which truncates floating-point results such as 28.999… to 28; round should be used.
tax.py: round() uses banker's rounding on half cents, which may not match the tax rules expected.
refunds.py: auto_approve uses amount < limit, but its docstring says refunds up to and including the limit are paid automatically, so an amount equal to the limit is wrongly sent for review.
refunds.py: partial does not check that returned_units is within 0..units, so it can refund more than the order total.
refunds.py: refundable allows status "shipped", so refunds can be issued before delivery.
refunds.py: restocking_fee floors the fee with //, so the fee is rounded down on each return.
formatting.py: m() renders negative amounts as "$-5.00".
formatting.py: q() pluralizes by appending "s", which gives wrong plurals for words such as "box".
formatting.py: receipt() column widths of 6 and 10 are exceeded by longer quantity or total strings, and pad does not truncate, so the table misaligns.
stock.py: can_reserve compares qty to on_hand instead of on_hand - reserved, so units already held for carts can be sold again (overselling). The docstring says "unreserved stock".
stock.py: ship decrements on_hand without checking for it going negative.
stock.py: LOW = 5 duplicates config.LOW_STOCK.
discount.py: apply_coupon uses max(percent_for(code), CAP), which should be min; coupons are forced up to at least 50% off, and unknown codes get 50% off.
discount.py: sale_price uses round() (banker's rounding) on cents, which can misprice half-cent results.
discount.py: CAP duplicates config.COUPON_CAP.
greet.py: birthday_note never matches a February 29 birthday in non-leap years.
greet.py: greeting and welcome_back both take the full name and first name separately, so banner is inconsistent with its own helpers.
dates.py: is_expired returns today >= expiry, which marks a voucher expired on its expiry date, contradicting its docstring.
dates.py: add_working_days skips weekends but not public holidays, despite the "working days" wording.
dates.py: delivery_window and add_working_days do not reject negative n, and the loop silently returns start for it.
shipping.py: shipping_cents expects grams (per its docstring), but orders.parcel_weight_kg supplies kilograms, so a 2 kg parcel is charged as weight 2 g.
shipping.py: zone_price silently falls back to the far-zone multiplier for unknown zones.
shipping.py: express ignores the free-shipping threshold, which may be intended but is undocumented.
orders.py: parcel_weight_kg raises KeyError for a SKU missing from weights, with no fallback or message.
orders.py: shipping_for passes kilograms to quote, which expects grams, so shipping is undercharged (see shipping.py).
orders.py: order_total applies tax only to goods and not to shipping, which may be wrong for the jurisdiction.
orders.py: order_total always uses the standard rate, so the per-category rates in tax.py are never applied to order totals.
orders.py: build() recomputes goods, shipping and total separately, so the stored goods and total can diverge if these functions change.

6 of 6 bug files flagged, 8 clean files flagged. The noise hides the bugs.

Write criteria with examples, and split the passes

The criteria now say to report a wrong result and to skip style, names, notes and magic numbers. Each line starts with Report or Skip and gives a small example.

The passes return one request per file, then a last request over all the files. A category that flags too many clean files is switched off until its criteria are fixed.

The recorded criteria are short. A harder review adds more, and each addition has a job:

Define each severity level by what puts a finding there, and show one finding at the level. A bare word such as high leaves the boundary to the model, and two runs can draw it differently.
Show the output format in an example, with the file, the issue, the severity and the fix. A review that comes back in five shapes cannot be read by a pipeline.
Give two to four examples that cover the ambiguous cases, and the reason each is right or wrong. The reason lets the model carry the rule to a new case. A long pile of examples costs tokens on every call.
For extraction, show documents in different layouts, including one where a field is missing, so a missing field stays empty and is not invented.

review.pygreen lines are what the fix adds

1"""What the reviewer is told, and which files it sees in each pass."""234def criteria() -> str:5 """Report wrong results, with examples of what to report and skip."""6 return (7 "Report a problem only when the code gives a wrong result, moves "8 "money the wrong way, or can crash. Skip style, names, missing "9 "docstrings, TODO notes and magic numbers. Most files have no "10 "problem; when you cannot point to the line and the wrong result "11 "it gives, report nothing.\n"12 "Report: arithmetic that drops a factor the code needs.\n"13 "Report: a check on the wrong side of a boundary.\n"14 "Report: a docstring that says one thing while the code does "15 "the opposite.\n"16 "Skip: a variable named `p`.\n"17 "Skip: a `print` call in a demo helper.\n"18 "Skip: a TODO note next to code that works."19 )202122def passes(files: list[str]) -> list[list[str]]:23 """One file per pass, then one pass over all of them for the links.24 A file on its own gets the reviewer's full attention. A fault between25 two files shows only when a pass reads both.26 """27 return [[name] for name in files] + [list(files)]

We doChange one setting at a time and predict.

The same release, two reviews

Switch between the starting review and the fixed review. The card marked "This is where the runs split" is the flags. The same files go in, and a different set of files comes out flagged.

This part is interactive in the lab: run ./workshop watch review-criteria-passes.

The small change shows the noise too

Both cards use the starting review: one on the hotfix, one on the release. A short change leaves the reviewer less to read, yet the hotfix reply still flags logs.py and tax.py, two files that are fine, beside the refund bug.

A smaller change does not remove the cause. The criteria are still vague, so noise arrives whenever a file looks odd.

This part is interactive in the lab: run ./workshop watch review-criteria-passes.

A fixed chain, or an adaptive plan

The arrows show files going in and flags coming out. A release review has known steps, so the code runs them as a fixed chain: one pass per file, then the pass across files. Nothing is decided by the model about which step comes next.

An open investigation is different. Its next step depends on what the last one found, so it uses an adaptive plan. Map the structure first, then plan the work by priority.

Measure by type before you automate

The result line of this run keeps two counts apart, the bug files flagged and the clean files flagged. One total would hide that. A job that is right 97 times in 100 may be wrong on 4 in 10 of one kind of form. Break accuracy down by type and by field.

Sample the output at random from each group. A labelled set is a batch of outputs a person has marked right or wrong. Use one to set a confidence cutoff for each field, the score below which an item goes to a person, and send unclear items there too.

A review that posts too many findings can add a verification pass. That is a second request that reads each finding again and attaches a confidence, a number for how sure it is. Route by that number only after checking it against labelled findings, because a score the model gives itself is not a measure.

PipelineReview the hotfix: the changed files. (3 files)
Reviewer4 passes made
Reviewerflagged 2 files
Show the result
```
refunds.py, tax.py
```

Resultrefunds.py: auto_approve uses amount < limit, but its docstring says refunds up to and including the limit…

Show the result

refunds.py: auto_approve uses amount < limit, but its docstring says refunds up to and including the limit are paid without a person, so a refund of exactly 50_00 is wrongly sent to review.
tax.py: add_tax hardcodes the 1.2 (20%) multiplier and ignores the category, so reduced-rate amounts get taxed at the standard rate.
refunds.py: auto_approve uses amount < limit, but its docstring says amounts equal to the limit are paid automatically; an amount of exactly 50_00 is sent to review by needs_review.

1 of 1 bug files flagged, 1 clean file flagged. The real bugs stand out.

Try it: change and review

Change one setting at a time. Pick the release or the hotfix, then the starting review or the fixed one.

Every combination is a recorded run, so this works with no internet.

This part is interactive in the lab: run ./workshop watch review-criteria-passes.

The same review in the Agent SDK

The Claude Agent SDK runs each pass as a one-turn query with no tools. A second instance that reviews a file has no memory of how the file was written, so that reasoning does not anchor it.

This is a real Claude Haiku 5.5 run through the SDK on the hotfix.

app/sdk_agent.py

1"""The same per-file review on the Claude Agent SDK.23The SDK runs each pass as a one-turn query with no tools, so a second4instance reviews a file with no memory of how it was written.5"""67from types import ModuleType89from kit import sdk10from kit.api import Json11from .agent import SYSTEM, Handled, flagged_in12from .tickets import FILES, Ticket131415def request_for(review: ModuleType, names: list[str]) -> Json:16 """The plain request a recording is keyed by."""17 body = "\n\n".join(f"### {name}\n{FILES[name]}" for name in names)18 prompt = f"{review.criteria()}\n\nReview these files:\n\n{body}"1920 return sdk.build_request(prompt, SYSTEM, tools=[], max_turns=1)212223def text_of(messages: list[Json]) -> str:24 """The last assistant text in a message stream."""25 final = ""2627 for message in messages:28 for item in message.get("content", []):29 if item["type"] == "text" and message["type"] == "assistant":30 final = item["text"]3132 return str(final)333435async def run(36 ticket: Ticket, review: ModuleType, llm: sdk.Live | sdk.Replay37) -> Handled:38 """Review each group of files the review names, on the SDK."""39 groups = review.passes(list(ticket.files))40 replies: list[str] = []4142 for names in groups:43 replies.append(text_of(await llm(request_for(review, names))))4445 text = "\n".join(replies)4647 return Handled(ticket, len(groups), flagged_in(text, ticket.files), text)

PipelineReview the hotfix: the changed files. (3 files)
Reviewer4 passes made
Reviewerflagged 2 files
Show the result
```
logs.py, refunds.py
```

Resultlogs.py: redact compares keys exactly against secret_keys, so keys such as card_number, cvv_code or…

Show the result

logs.py: redact compares keys exactly against secret_keys, so keys such as card_number, cvv_code or password_hash are not masked and their values pass through unredacted.
refunds.py: auto_approve uses amount < limit, but its docstring says amounts up to and including the limit are paid automatically, so a refund of exactly 5000 cents goes to manual review.
refunds.py: auto_approve uses amount < limit, so a refund of exactly LIMIT (5000 cents) goes to review, while its docstring says amounts "up to and including the limit" are paid at once.

1 of 1 bug files flagged, 1 clean file flagged. The real bugs stand out.

You doFix the starting code, then test yourself.

You do: write the criteria and the passes

Copy the starting code and open work/review-criteria-passes/review.py. criteria() says "be thorough", and passes() puts every file in one request.

Terminal

./workshop start review-criteria-passes
./workshop check review-criteria-passes
./workshop watch review-criteria-passes

The checks want criteria that name what to report and what to skip, with examples. Each file gets its own pass, none is left out, and a last pass reads the links between files.

Check your understanding

Answer each question, then read why every option is right or wrong. The wrong options are the mistakes engineers make on the job.

1. A review bot posts comments on 9 of 14 files, and 6 of those files are fine. The team has started to ignore the bot. What is the root fix?

ATell it to be more thorough, so it finds the bugs among the noise.
BName which problems to report and which to skip in the criteria.
CAsk it for a longer report, so each comment carries more detail.
DLower the severity of every comment so nobody is alarmed by them.

Show the answer

Answer: B

A A plea for thoroughness names no category, so style noise counts as much as a bug.
B Correct. Explicit criteria separate a wrong result from a matter of taste.
C A longer report gives the reader more to skip. It does not change what gets flagged.
D A softer label keeps the noise and makes the real bugs easier to miss.

2. The prompt already says to be conservative and only flag what matters. Even so, 60 percent of the flags on a release are noise. What works better?

AWrite conservative in capitals, so the instruction carries more weight.
BAsk the model to flag only when it is certain, and trust its answer.
CSend the review to a model with a larger window, so it sees more.
DName the categories that count, with a sample of each.

Show the answer

Answer: D

A Volume does not define the word. The model still has no list of what matters.
B Stated certainty is not a measure, and a wrong flag can sound as sure as a right one.
C A larger window holds more text. It gives no rule for what to report.
D Correct. Named categories tell the reviewer where the line is, and a sample shows it.

3. One review category flags 8 false positives in every 10 comments. The team now distrusts the other categories too. What should you do?

AKeep it on, because a noisy category still finds some real problems.
BRaise the severity of the other categories so they stand out from it.
CTurn that category off until it is fixed, and keep the rest.
DTurn the whole review off until every category is accurate.

Show the answer

Answer: C

A The real problems it finds are lost among the false ones, and readers stop looking.
B Louder labels do not remove the noise. The reader still sifts the noisy category.
C Correct. One noisy category costs the trust in all of them, so switch it off until it is fixed.
D This throws away the categories that work, along with the one that does not.

4. Select two things that make a severity level usable in a review prompt.

AA definition of when each level applies.
BA single word for each level, such as high or low.
CA short code example that shows each level.
DA sentence that asks the reviewer to use good judgement.

Show the answer

Answer: A, C

A Correct. A definition says what puts a finding at that level.
B A bare word leaves the boundary to the model, so two runs can split it differently.
C Correct. An example shows a finding at the level, so the line is concrete.
D Judgement with no rule is the gap the criteria are meant to fill.

5. Across 10 runs, a review comes back in 5 different shapes, and the pipeline cannot read half of them. What is the fix?

AShow the format in examples: file, issue, severity, fix.
BRun it again each time until the shape matches what the parser wants.
CWrite in the prompt that the format must be followed strictly.
DAdd 12 more examples of findings, so the shape is averaged out.

Show the answer

Answer: A

A Correct. An example of the exact format is the clearest way to fix the shape.
B Each rerun costs a call and still may not match.
C A stern line names no shape. The model still has to guess one.
D More examples raise the cost, and a long pile can pull the output toward the wrong one.

6. A print call in a demo helper is fine, but the review flags it in every run. How do you stop that without hiding real problems?

ALet it flag the print call, and have a person drop it each time.
BAdd an example of the print call as acceptable, with the reason.
CForbid the word print in the review comments.
DRemove every example from the prompt, so there is less to copy.

Show the answer

Answer: B

A This spends a person's time on the same noise every run.
B Correct. An acceptable pattern with its reason settles an ambiguous case.
C The call is still flagged, only under different words. The judgement was never made.
D Without examples the model has even less to go on.

7. An extraction job invents a value for a field when the document has none. The documents come in 4 different layouts. What helps most?

AMove to a model with a larger context window, so it holds more text.
BName every field in the prompt and add nothing else to the instructions.
CShow examples in other layouts, with one that has a missing field.
DSet the temperature to 0 so the model stops inventing values for fields.

Show the answer

Answer: C

A A larger window adds room, not a rule for a missing field.
B Names say what to extract. They do not show what to do when a field is absent.
C Correct. Examples across layouts show that a missing field stays empty.
D A lower temperature makes the model more repeatable, and it can repeat an invented value.

8. Select two rules for the examples in a review prompt.

AUse as many as will fit, because each one improves the review.
BUse two to four examples that cover the ambiguous cases.
CShow the output each time and leave the reason out.
DGive the reason each example is right or wrong.

Show the answer

Answer: B, D

A A long pile costs tokens on every call, and it can crowd out the instructions.
B Correct. A few well-chosen cases settle the ambiguity without swamping the prompt.
C Without the reason the model copies the surface and misses the rule.
D Correct. The reason is what lets the model carry the rule to a new case.

9. A single request reviews a release of 14 files. The two bugs in the middle files are missed. What is the root fix?

AKeep one request, since one call is cheaper than fourteen.
BUse a model with a larger window, so the whole release fits.
CAdd a line that asks the model to read the middle carefully.
DReview each file in its own request.

Show the answer

Answer: D

A The saving is real, but the bugs that slip through cost more.
B The release already fits. The problem is attention spread across many files.
C A request to read carefully does not widen attention across 14 files.
D Correct. A pass on one file gives it the reviewer's full attention.

10. Each of 14 files passes its own review, yet a parcel weight in kilograms is passed to a function that expects grams. Which step finds it?

AA last pass that reads all the files together.
BNothing more, since every file was reviewed.
CThe same 14 passes again, to see whether they agree.
DA pass over only the files that looked risky.

Show the answer

Answer: A

A Correct. A fault between files shows only when a pass reads both.
B Each pass saw one file, so none could see that two files disagree.
C The same single-file passes see the same single file, and they reach the same answer.
D The fault sits in files that look fine on their own.

11. A review posts findings that a person must read, and there are far too many to read. How should they be routed?

APost the ones that sound the most certain first, and trust the order they come in.
BAdd a verification pass, calibrated on labelled findings, and route by its score.
CPost every finding in one list, and let each reader sort the list by hand.
DPost the average accuracy of the whole review next to the list of findings.

Show the answer

Answer: B

A Tone is not a measure, and a wrong finding can sound sure. A score needs checking against labels.
B Correct. A second look attaches a confidence, and calibrating it on labelled findings makes it usable for routing.
C One list sends the noise to the person along with the signal.
D An average does not say which findings to read first.

12. Select two reasons to add a review pass that is separate from the pass that wrote the code.

AIt has no memory of how the code was written, so nothing anchors it.
BIt remembers the bugs from earlier runs and looks for them first.
CIt reruns the same check, so a bug is confirmed by two identical answers.
DIt can read across files, which a pass on one file cannot do.

Show the answer

Answer: A, D

A Correct. A fresh instance starts with none of the author's reasoning.
B An instance carries nothing from earlier runs.
C An identical rerun adds nothing. The value is in the different view.
D Correct. A pass that holds several files can see how they fit together.

13. A release review always takes the same steps over 14 files, in the same order. How should you build it?

AAs an open plan that the model redraws after every file.
BAs an adaptive plan, because adapting is always safer.
CAs one step, because fewer steps always means fewer errors.
DAs a fixed chain of passes that the code runs in order.

Show the answer

Answer: D

A An open plan spends calls deciding what the code already knows.
B Adapting helps when the steps are not known, and here they are.
C Fewer steps here means one pass over 14 files, which is the failure.
D Correct. A predictable job is a fixed chain, which is easy to test and to repeat.

14. An investigation of an unfamiliar code base keeps turning up modules nobody expected. How should the work be planned?

AAs a fixed chain written before the work starts.
BBy reading every file in the repository before doing anything else.
CAs an adaptive plan that is revised as each finding comes in.
DBy writing the full plan up front and never changing it.

Show the answer

Answer: C

A A fixed chain cannot follow modules it did not know about.
B This is slow and it fills the context before the work has begun.
C Correct. When each step depends on what the last one found, the plan has to adapt.
D A plan that never changes cannot follow what the work turns up.

15. A chain reviews each of 14 files alone, and nobody checks how the files call each other. What should be added?

AA last pass over all the files, reading the links.
BNothing, because each file was checked in full.
CA bigger context window on each single-file pass, so it sees more.
DFold every file into one pass, so the links are visible.

Show the answer

Answer: A

A Correct. The integration pass reads the calls across files that no single pass sees.
B Checking each file in full does not check how two files fit together.
C A single-file pass still holds one file, however large its window.
D This brings back the dilution the per-file passes removed.

16. Select two steps that start an adaptive plan for an unfamiliar project.

ARead every file in full before forming any plan.
BMap the structure of the project first.
CPlan the work by priority, from what the map shows.
DFix the order of all the steps before looking at the code.

Show the answer

Answer: B, C

A This fills the context with files that may not matter.
B Correct. A map shows where the code lives at a low cost.
C Correct. The map says which areas matter, so the work starts there.
D An order fixed in advance cannot respond to what the map shows.

17. An extraction job scores 97 percent overall, but one of its 6 form types scores 61 percent. What should you do before automating it?

AAutomate it, since 97 percent is above the bar.
BTest more of the form types that already pass.
CRaise the pass mark to 98 percent and stop there.
DBreak accuracy down by form type and field.

Show the answer

Answer: D

A The 97 percent hides a form type that fails 4 times in 10.
B More passing forms make the average look better and hide the weak one.
C A higher mark says nothing about which form type fails.
D Correct. An average hides a weak type, and the breakdown shows it.

18. The extractions marked high confidence are not reviewed. A check of 100 of them finds 4 wrong. How should such output be sampled?

AIt needs no sampling, because high confidence means correct.
BSample at random from each confidence and type group.
CRead the first 100 that arrive each day.
DSample only the low-confidence ones, since they are the risky ones.

Show the answer

Answer: B

A Four in a hundred were wrong, so high confidence does not mean correct.
B Correct. A stratified random sample shows the error rate in each group.
C The first 100 are not random, and they may all be one type.
D This leaves the high-confidence group, where the wrong answers were found, unchecked.

19. The team sets the confidence cutoff for each field by feel, and the cutoffs keep drifting. What makes them stable?

ACalibrate each cutoff on a set of labelled documents.
BTrust the model's own score, since it knows how sure it is.
CKeep the cutoffs by feel, and revisit them each week.
DUse one cutoff for every field, so there is nothing to drift.

Show the answer

Answer: A

A Correct. Labelled documents show how often each score is right, so a cutoff can follow.
B A model's score is not calibrated to the truth until it is checked against labels.
C This is the drift. A cutoff should be set from data.
D Fields differ in how hard they are, so one cutoff is wrong for most of them.

20. Select two kinds of extraction that should go to a person.

AEvery extraction, because a person must see them all.
BAnything from a source the model has read before.
CLow confidence on a field.
DA source that is unclear or contradicts itself.

Show the answer

Answer: C, D

A This removes the value of automating. Reserve a person for the cases that need one.
B A familiar source can still be wrong, and being familiar is not a reason to skip a check.
C Correct. Low confidence is the signal that a person should check the value.
D Correct. Where the source is unclear the model cannot be checked against it.

Play: predict the output

These criteria list only what to skip.

python

def criteria() -> str:
    return (
        "Report bugs. Skip style, names and missing docstrings.\n"
        "Skip: a variable named `p`.\n"
        "Skip: a TODO note next to code that works."
    )

Predict: A docstring says the function returns cents, and the code returns dollars. What is the reviewer likely to do?

ASkip it as a docstring note, since every example points that way.
BReport it, since a wrong unit is a bug.

Show the answer

A. Examples of one kind teach one direction. Pair each skip with a report that sits near it, such as a docstring that says the opposite of the code.

References

Be clear and direct: how to state what to report and what to skip.
Use examples effectively: why two examples of each kind settle a fuzzy rule.
Agent SDK overview: how the same review runs as an SDK agent.
Building effective agents: sectioning work into parallel calls, then joining the results.

Conclusions

Name what to report and what to skip, with an example of each.
Give each file its own pass, then add a pass that reads the links between files.
Break accuracy down by type before you trust an average.

You can now write review criteria and passes that flag wrong results and skip noise.

Criteria

The rules for what a review reports and what it skips

Pass

One request that shows the reviewer only the files it names

Integration pass

A last pass that reads several files together for the links between them

Fixed chain

Steps the code runs in a set order