Claude Architect WorkshopPDFEPUBlearnwithparam.comkaas.team

Foundations, lesson 16

Claude in CI

CI reads a fresh review as data, not as prose

The review job at Lumen Supply returns prose the pipeline cannot read, from the session that wrote the code. Nothing tells a rerun which comments the team has already closed. This is the pipeline at Lumen Supply, an online electronics shop. You will watch a review fail, then fix it with a schema, a fresh session and a list of what is closed.

I doWatch the failure and its fix.

A pipeline needs a result it can parse

A review job runs on every pull request. The CI runner starts it, the model reviews the change, and code reads the answer and posts inline comments.

Four things can go wrong. A job run without -p waits for input nobody types and hangs. The harness always passes -p, so you will not watch that one fail.

The other three are in review.py. The answer is prose that cannot be parsed, the review resumes the author's session, and a rerun is not told which comments are closed.

The CI runner starts claude -p. Claude reads the repo and prints JSON. A parser turns it into inline comments.CI runnerstarts the jobclaude -pa fresh sessionRepo filesCLAUDE.md, testsParserreads the schemaInline commentsfile, line, issuethe schema, brief and session are written in review.pyThe CI runner starts claude -p. Claude reads the repo and prints JSON. A parser turns it into inline comments.CI runnerstarts the jobclaude -pa fresh sessionRepo filesCLAUDE.md, testsParserreads the schemaInline commentsfile, line, issuethe schema, brief and session are written in review.py

A schema gives the pipeline findings it can read

The fixed job runs claude -p, which takes its prompt from the command and exits. It asks for JSON that matches a schema, so every finding has a file, a line and an issue.

Open the cards. The reviewer read the existing tests, found that the discount is subtracted instead of multiplied, and reported one finding on the line that holds it.

  1. Pull request--- a/src/pricing.ts
    Show the result
    --- a/src/pricing.ts
+++ b/src/pricing.ts
+export function applyDiscount(total: number, rate: number) {
+  return total - rate;
+}
  2. Jobclaude -p --json-schema --tools Read (fresh session)
  3. ClaudeRead src/pricing.ts
  4. ClaudeRead tests
  5. ClaudeRead tests/pricing.test.ts
  6. Reply1 finding(s) the parser can read
    Show the result
    src/pricing.ts:6 applyDiscount subtracts the rate from the total (total - rate) instead of applying it as a fraction (total * (1 - rate)). Per CLAUDE.md, a rate of 0.1 means 10 percent off. applyDiscount(100, 0.1) returns 99.9 instead of 90, so the existing test at tests/pricing.test.ts:7-9 ('a rate of 0.1 takes 10 percent off') fails.

The findings parse, name the bug and repeat nothing closed.

The job and the brief as written

The starting code asks for no schema, so its answer is free text. Its brief asks about style and documentation and passes nothing that is closed. Its session resumes the author's session for the review.

A hang costs the most: a job that waits for a prompt holds a runner until the runner's timeout, which can be hours. A print run takes seconds.

starter/review.py
1"""What the CI review job asks for, the answer shape and its session."""23from typing import Any456def schema() -> dict[str, Any]:7 """The JSON schema the answer must match.8 TODO(workshop): a `findings` list, each with file, line and issue, so9 the pipeline can read it.10 """11 return {}121314def brief(diff: str, prior: str) -> str:15 """The prompt for a reviewer that sees only the diff.16 TODO(workshop): give criteria and pass the findings already closed.17 """18 return f"Review this change for bugs, style and documentation:\n\n{diff}"192021def session(author: str) -> list[str]:22 """Flags for the session the review runs in.23 TODO(workshop): the review must start a fresh session, not resume the24 author's.25 """26 return ["--resume", author]

What does the pipeline get back?

The starting job has no schema. The reviewer is the session that wrote the change, resumed for the review.

Predict: What does the pipeline get back?

  1. AProse it cannot parse, from the author's session.
  2. BA findings list it can parse.
Show the answer

A. With no schema the answer is text, and the pipeline has no file or line to post a comment on. The review also resumes the author's session, so it starts from the author's reasoning.

The answer is text nobody can parse

This is the same change with the starting job. The job resumes the author's session, and the answer is a paragraph. It is not JSON, so nothing can be posted on a line.

This is a real Claude Haiku 5.5 run, recorded once and replayed here.

  1. Pull request--- a/src/pricing.ts
    Show the result
    --- a/src/pricing.ts
+++ b/src/pricing.ts
+export function applyDiscount(total: number, rate: number) {
+  return total - rate;
+}
  2. Jobclaude -p --tools Read --resume (the author's session)
  3. Authorauthor's session: It subtracts the rate from the total as if it were a dollar amount, so applyDiscount(100, 0.1) returns 99.9 instead of 90.
    Show the result
    It subtracts the rate from the total as if it were a dollar amount, so applyDiscount(100, 0.1) returns 99.9 instead of 90. It is not correct: under the project's convention, a rate is a fraction (0.1 means 10% off), so the function should be return total * (1 - rate);.

I haven't changed the file, and I can apply that fix. Per the CLAUDE.md rule that every money function has a test in tests/, I'd also add a test that calls applyDiscount once and checks a 10% discount on a known total. Should I go ahead?
  4. Replyprose: Bug: applyDiscount subtracts the rate instead of the discounted amount. src/pricing.ts, the…
    Show the result
    Bug: applyDiscount subtracts the rate instead of the discounted amount.

src/pricing.ts, the return total - rate; line. A rate is a fraction (0.1 means 10% off, per CLAUDE.md), so applyDiscount(100, 0.1) returns 99.9 instead of 90. Any caller that applies a discount gets a result that is almost the undiscounted total. The fix is return total * (1 - rate);.

No other bugs in this change. I haven't run it or checked any callers, since the repo has no git history and I only have this diff.

The pipeline cannot parse this answer.

Ask for a schema, a fresh reviewer and the closed findings

The fix has three parts. The schema asks for a findings list. The brief holds the diff, the rules to apply and the findings already closed. The session function returns no flags, so the review starts fresh and does not resume the author's session.

Team standards and test rules live in the project CLAUDE.md, which a print run loads like any other session. Memory files stack: user, project, then directory, and each adds to the last. A user-level file stays on one machine, so a rule the team needs belongs in the committed project file.

A long file splits into topic files, and each package imports the ones it needs with an @path line. Import a shared file rather than pasting a copy, because two copies drift apart.

How one review moves from the runner to the parser.CI runnerClaudeRepo filesParserclaude -pReadfileReadfileReadfileJSONinline comments
review.pygreen lines are what the fix adds
1"""What the CI review job asks for, how it wants the answer, its session."""23from typing import Any45FINDING = {6 "type": "object",7 "properties": {8 "file": {"type": "string"},9 "line": {"type": "integer"},10 "issue": {"type": "string"},11 },12 "required": ["file", "line", "issue"],13}141516def schema() -> dict[str, Any]:17 """A `findings` list the pipeline can read."""18 return {19 "type": "object",20 "properties": {"findings": {"type": "array", "items": FINDING}},21 "required": ["findings"],22 }232425def brief(diff: str, prior: str) -> str:26 """The prompt for a reviewer that sees only the diff and what is closed."""27 return (28 "Review this change for bugs that give wrong results. Read the "29 "existing tests for the changed file first. Skip style and "30 "missing doc comments.\n"31 f"Already reported, do not repeat: {prior}\n\n{diff}"32 )333435def session(author: str) -> list[str]:36 """Flags for the session the review runs in: none, so it starts fresh.37 The author's session holds the author's reasoning, and a review there38 starts from it instead of from the diff.39 """40 return []
We doChange one setting at a time and predict.

The same change, two review jobs

Switch between the starting job and the fixed job. The card marked "This is where the runs split" is the parse result. One job hands the pipeline prose and the other hands it a finding.

This part is interactive in the lab: run ./workshop watch claude-in-ci.

Why the author's session does not review

The session that wrote a change holds the reasoning that led to it, and the Author card shows what it held. In this recording the author's own session says the change has a bug, so the resumed review finds it too. That is the lucky case.

A fresh session is the safer default: it sees only the diff and the repository, so no earlier reasoning can anchor it. A review of many files spreads attention thin, so review each file on its own and add a pass for how the files connect.

A verification pass checks each finding and attaches a confidence. Send only the high tier to a person once that confidence is calibrated against labelled outcomes. A score the model gives itself is not a measure, and neither is tone.

How one review moves from the runner to the parser.CI runnerClaudeRepo filesParserclaude -pprosecannot parse

When CI should use a batch

A pre-merge review blocks a developer, so it runs now, as the review in the recording does. A batch costs half and takes up to 24 hours, so it fits an overnight report and not a blocking check.

Each batch request is a single Messages call, so a multi-turn tool loop cannot run inside it. Results come back in any order, and each carries the custom_id of its request. If some fail, resend only those by custom_id.

  1. Pull request--- a/src/pricing.ts
    Show the result
    --- a/src/pricing.ts
+++ b/src/pricing.ts
+export function applyDiscount(total: number, rate: number) {
+  return total - rate;
+}
  2. Jobclaude -p --json-schema --tools Read (fresh session)
  3. ClaudeRead src/pricing.ts
  4. ClaudeRead tests
  5. ClaudeRead tests/pricing.test.ts
  6. Reply1 finding(s) the parser can read
    Show the result
    src/pricing.ts:6 applyDiscount subtracts the rate from the total (total - rate) instead of applying it as a fraction (total * (1 - rate)). Per CLAUDE.md, a rate of 0.1 means 10 percent off. applyDiscount(100, 0.1) returns 99.9 instead of 90, so the existing test at tests/pricing.test.ts:7-9 ('a rate of 0.1 takes 10 percent off') fails.

The findings parse, name the bug and repeat nothing closed.

Try it: change and job

Change one setting at a time. Pick the shipping change and the starting job to see prose again. Pick the fixed job to see the boundary bug reported as one finding.

Every combination is a recorded run, so this works with no internet.

This part is interactive in the lab: run ./workshop watch claude-in-ci.

The same review on the command line

The job in this lesson is the command line itself. It passes -p, --output-format json and --json-schema. The parsed answer lands in the structured_output field of the JSON it prints.

The --tools Read flag limits the run to reading files. The --allowedTools Read flag only lets a read run without a permission prompt. The --bare flag starts faster by skipping CLAUDE.md, hooks and .mcp.json, so a job that relies on them leaves it off.

The author session is opened with --resume only in the starting job. Replay needs nothing installed. A live run (LIVE=1) needs the claude command-line tool installed and signed in.

app/cli_agent.pyfile: 142 lines
1"""The review job's command line: `claude -p` with a schema.23The job prints JSON with `--output-format`, checks it against4`--json-schema`, and may only read files, since `--tools` limits it. A run that5`--resume`s the author's session carries the author's reasoning with it.6"""78import json9import re10import subprocess11from pathlib import Path12from typing import override1314from kit import cli, spend15from kit.api import Json16···126 lines not shown
You doFix the starting code, then test yourself.

You do: fix the review job

Copy the starting code and open work/claude-in-ci/review.py. schema() is empty, brief() asks for everything and forgets closed findings, and session() resumes the author's session.

Terminal
./workshop start claude-in-ci
./workshop check claude-in-ci
./workshop watch claude-in-ci

The checks want a findings list where each finding has a file, line and issue. The brief must hold the diff and the closed findings, and the review must start a fresh session.

Check your understanding

Answer each question, then read why every option is right or wrong. The wrong options are the mistakes engineers make on the job.

1. A CI job runs claude on each pull request. It sits idle until the runner kills it after 6 hours. What is the cause and the fix?

  1. AA permission prompt is blocking it, so add --dangerously-skip-permissions.
  2. BThe runner is too strict, so raise its timeout to 12 hours.
  3. CThe model is too slow, so switch the job to a larger one.
  4. DWith no -p, claude waits for input nobody types, so run it with -p.
Show the answer

Answer: D

  • A The skip flag removes a prompt but leaves the run interactive. The job still waits for input.
  • B A longer timeout only holds the runner longer. Nobody will ever type the input.
  • C A bigger model answers no faster than a session that is waiting on a keyboard.
  • D Correct. Print mode takes the prompt from the command, runs once and exits.

2. A review bot must attach each finding to a file and a line in the pull request. Today the job prints a paragraph. What changes?

  1. APrint JSON with --output-format json and pass --json-schema for a findings list.
  2. BAsk for JSON in the prompt and parse the reply with a regular expression.
  3. CSplit the paragraph on new lines and treat each line as a finding.
  4. DPost the whole paragraph as a single comment on the pull request.
Show the answer

Answer: A

  • A Correct. The schema makes each finding carry a file, a line and an issue the bot can read.
  • B A request in the prompt is not a check. One odd reply breaks the regular expression.
  • C Lines of prose carry no file or line, so there is nothing to attach a comment to.
  • D One comment cannot sit on a line of code, which is what the bot was asked to do.

3. After a push, the review job posts 5 comments that the team already closed. What is the root fix?

  1. ARun the job on the same runner, so the model remembers the last run.
  2. BLower the temperature so the same review is not reworded.
  3. CDelete the repeated comments from the pull request after they post.
  4. DPass the closed findings into the brief and ask for only new ones.
Show the answer

Answer: D

  • A Each run is a new session. The model has no memory of the last run, wherever it runs.
  • B A lower temperature makes the repeats more stable, not rarer.
  • C Cleaning up after the post treats the symptom, and the next push posts them again.
  • D Correct. The run can skip only what it is told was closed.

4. Select two things a claude -p run in CI can use from the repository to shape its review.

  1. AThe project CLAUDE.md, which holds the team standards.
  2. BThe author's chat history from their laptop.
  3. CThe existing tests, which it can open with a read tool.
  4. DNothing, because print mode skips CLAUDE.md.
Show the answer

Answer: A, C

  • A Correct. Print mode loads the project CLAUDE.md like any other session.
  • B A CI runner has no access to a laptop's chat history.
  • C Correct. Tests say what the code should do, and a read tool can open them.
  • D Print mode changes how the run ends. It does not change what it loads.

5. The rule that money is in dollars lives in one teammate's user-level CLAUDE.md. Three reviewers in CI miss it. What is the fix?

  1. ACopy the file into each reviewer's home directory by hand.
  2. BMove the rule into the project CLAUDE.md and commit it.
  3. CPaste the rule at the top of every review prompt.
  4. DAsk that teammate to start every review.
Show the answer

Answer: B

  • A Hand copies drift, and a CI runner has no home directory to copy into.
  • B Correct. A project file travels with the repository, so every session loads it.
  • C A pasted rule is one more thing to keep in step, and it is easy to forget in a new job.
  • D This makes one person the gate. A user-level file still never reaches the runner.

6. One CLAUDE.md holds 9 topics and runs long. Sessions in each package load all of it. What is the fix?

  1. ADelete the topics that few packages use.
  2. BMove it to a user-level file so the repository copy is shorter.
  3. CSplit it into topic rule files and import only what each package needs.
  4. DShorten every topic to a single word.
Show the answer

Answer: C

  • A Deleting loses the rules those packages do need.
  • B A user-level file is not shared, so the team would lose the rules.
  • C Correct. Topic files load where they apply, and each package imports only its own.
  • D A single word cannot carry a rule, so the standard stops being enforceable.

7. The payments package needs the same money rules that sit in the shared standards file. The text is being pasted into both. What is better?

  1. AImport the shared file with an @path line.
  2. BKeep the paste and add a note to update both together.
  3. CMention the shared file by name in a sentence, and trust readers to open it.
  4. DKeep both copies, because a duplicate does no harm.
Show the answer

Answer: A

  • A Correct. An @path import loads the shared file, so there is one source of truth.
  • B A note does not stop the copies from drifting apart.
  • C A sentence is not loaded. The session never sees the rules.
  • D Two copies disagree the first time one is edited.

8. Select two facts about how CLAUDE.md files layer.

  1. AA file in a directory replaces the project file for that directory.
  2. BUser, project and directory files stack, and each adds to the ones above it.
  3. CThe project file loads only when the prompt asks for it.
  4. DA user-level file stays on one machine and is not in the repository.
Show the answer

Answer: B, D

  • A A directory file adds to the project file. It does not replace it.
  • B Correct. The layers add up, so a session sees the user, project and directory rules together.
  • C The project file loads at the start of the session, with no request.
  • D Correct. Only files in the repository reach the team and the CI runner.

9. The review that gates a deploy has a 2-minute budget, and it is set up to use the Batches API for the price. What is wrong?

  1. ANothing, since the half price is worth a short wait.
  2. BNothing, since the deploy step can wait until the batch ends.
  3. CSplit the review into 40 smaller batches so each ends sooner.
  4. DA batch has no latency promise, so a blocking check runs sync.
Show the answer

Answer: D

  • A The wait is not short. A batch may take up to 24 hours.
  • B A deploy that waits on an unbounded batch is the failure. Nobody can say when it ends.
  • C Smaller batches are still batches, and none of them has a promise.
  • D Correct. The API answers now only on the sync path.

10. A weekly job tags 10,000 old reviews with a topic, and the team reads the tags on Monday. It runs sync and the bill is high. What should change?

  1. AStart the sync run at night, when the price is lower.
  2. BMove it to the Batches API, which costs half the price.
  3. CKeep it sync, since a faster answer must cost less.
  4. DKeep it sync, since 10,000 is too many for a batch.
Show the answer

Answer: B

  • A The sync price does not change with the hour.
  • B Correct. Nobody waits on the tags, so the half price costs nothing you notice.
  • C Sync answers sooner and costs more. The price is for the wait you give up.
  • D Size does not decide the mode. A large overnight job is the usual batch.

11. A batch of 10,000 review requests ends and 40 results are errors. How do you recover?

  1. ASubmit the full batch again and use the newer answers.
  2. BCount the error lines in the results file and resend that many requests.
  3. CFind the 40 by their custom_id and send only those again.
  4. DIgnore the 40, since the other results are enough.
Show the answer

Answer: C

  • A This pays for 9,960 answers you already have.
  • B Results come back in any order, so a position or a count says nothing.
  • C Correct. Each result carries the custom_id of its request.
  • D An unreviewed change is a hole in the pipeline, and the error says it needs another try.

12. Select two facts about the Message Batches API.

  1. AA batch has a 24 hour window and promises nothing faster.
  2. BResults come back in the order the requests were sent.
  3. CA batch ends within the hour, so it suits a blocking check.
  4. DA batch request cannot run a multi-turn tool loop.
Show the answer

Answer: A, D

  • A Correct. The window is a limit, not a promise.
  • B Results can come in any order. The custom_id is what ties a result to its request.
  • C No such promise exists, so a blocking check cannot depend on a batch.
  • D Correct. A batch request is one call, with no turn after a tool result.

13. The session that wrote a change reviews it and finds 0 issues. A teammate who reads the diff cold finds 3. What is the fix?

  1. ATell the author's session to be more critical of its own work.
  2. BReview in a fresh session that sees only the diff and the repository.
  3. CGive the author's session more turns to look again.
  4. DKeep the author's session, since it knows the intent best.
Show the answer

Answer: B

  • A The session still holds its own reasoning, and a stricter tone does not remove it.
  • B Correct. A fresh session has none of the reasoning that led to the code.
  • C More turns give it more time to agree with itself.
  • D Knowing the intent is the problem. It reads the code as it meant it, not as it is.

14. A review of 14 changed files finds the local bugs but misses one that only shows between two files. What is the root fix?

  1. AAdd a line to the prompt that says to be thorough.
  2. BMove to a model with a larger context window.
  3. CReview each file on its own, then run a pass on how the files connect.
  4. DReview a sample of the files, so attention goes further.
Show the answer

Answer: C

  • A A plea for thoroughness does not widen attention across 14 files.
  • B A larger window holds more text, but attention still thins across it.
  • C Correct. Each file gets full attention, and a cross-file pass reads the links.
  • D A sample leaves files unreviewed, so it misses more.

15. A review job posts findings on three tiers of trust, and a human reads only the top tier. How should it route them?

  1. ATrust the model's tone, and post the findings that sound most confident first.
  2. BVerify each finding, score it on labelled outcomes, and route by that confidence.
  3. CPost every finding in one list, and let the human reader sort noise from signal.
  4. DMeasure overall accuracy once, and route the whole review by that single number.
Show the answer

Answer: B

  • A Tone is not a measure. A wrong finding can sound sure.
  • B Correct. A second pass checks each finding, and a confidence calibrated on labelled outcomes is one the router can trust.
  • C One list sends the noise to the human along with the signal.
  • D One overall number hides which findings are wrong, so it cannot route them.

16. Select two reasons to review with a second instance of Claude.

  1. AIt remembers earlier runs, so it repeats fewer of the comments already closed.
  2. BIt never saw the code being written, so the author's reasoning does not anchor it.
  3. CIt can catch subtler issues that the author's session argued past.
  4. DIt reuses the files the author already read, so it needs no reads of its own.
Show the answer

Answer: B, C

  • A An instance carries nothing from earlier runs. Closed findings must be passed in.
  • B Correct. The second instance starts without the generation context.
  • C Correct. A fresh reader is not committed to the author's decisions.
  • D A fresh instance reads for itself. Its independence is the point.

Play: spot the bug

This findings schema lists the three fields but never says which are required.

python
FINDING = {
    "type": "object",
    "properties": {
        "file": {"type": "string"},
        "line": {"type": "integer"},
        "issue": {"type": "string"},
    },
}

Predict: The reviewer returns a finding with a file and an issue but no line. Does the schema reject it?

  1. ANo. A property is optional unless it is listed in required, so the finding passes with nowhere to post it.
  2. BYes. Every listed property must be present.
Show the answer

A. Properties describe a field, and required makes it mandatory. List file, line and issue in required, so every finding the pipeline reads can be pinned to a line.

References

Conclusions

  • Print mode with a schema gives a pipeline data it can read.
  • A fresh session reviews better than the one that wrote the code.
  • Pass the closed findings in, so a rerun reports only new ones.

You can now run a review in CI that returns findings as data, from a session apart from the author's.

Print mode

claude -p, which runs once from the command and exits

Schema

The shape the answer must match, enforced with --json-schema

Independent review

A review in a fresh session that sees only the diff

Closed finding

A comment the team already settled, passed in so it is not repeated