Claude Architect WorkshopPDFEPUBlearnwithparam.comkaas.team

Foundations, lesson 12

Surfacing failures

A failed search must reach the report

A coordinator starts three researchers to search the recall notes, then a writer drafts the report. The regulator search times out and the merge drops it, so the report reads as complete and a whole section is missing. A reader trusts it. This is the agent at Lumen Supply, an online electronics shop. You will watch a failure vanish, then fix the record and the merge so the report names what was not covered.

I doWatch the failure and its fix.

A failure is a result the writer must see

Researchers search and report back. A search can fail in two ways: it times out, or it half works. Two sources that disagree are not a failure, but both figures must stay in the report. The researcher classifies the failure, the coordinator decides what to do and the writer says what is solid and what is not.

This design goes wrong in three ways. A timeout is swallowed, one failure aborts every angle, and a merge picks one of two conflicting figures. You will see the first fail, then fix it, then see the third.

A failed search is passed on, not dropped.Searchmay time outFailure recordkind, query, partial resultsWritersees every resultReportnames the gapthe record and the merge are written in report.pyA failed search is passed on, not dropped.Searchmay time outFailure recordkind, query, partial resultsWritersees every resultReportnames the gapthe record and the merge are written in report.py

A failure that travels gives an honest report

The coordinator starts three researchers with the Agent tool, called Task in older releases, one for each group of searches. The regulator search fails, and its result comes back with an error flag and a category. The harness reads that result as the researcher got it, and your failure and merge turn it into a line the writer reads.

Open the writer prompt card to read the failure record field by field: the kind, the query, the attempt and the partial results. The report names the regulator as a gap.

  1. ColleagueWrite the recall report for the power bank.
  2. Coordinatorstarted 3 researchers, one per group of searches
  3. Researchers5 searches, replies below
    Show the result
  4. Writer prompt1 gap line shown to the writer
    Show the result
    Write the recall report for the power bank.

- cause (Lab teardown): Cells swell when a separator is thin.
- scope (Batch log): Units made in March are affected.
GAP: the regulator search failed (timeout) on 'agency deadline' after 2 attempts. Partial results: nothing. Say this area is not covered.
- customers (Returns desk): Customers can swap a unit for free.
- safety (Safety test): A retest of the swap units passed.
Some searches failed. Do not call the report complete.
  5. Writer- Cause: Lab teardown found that cells swell when the separator is thin.
- Scope: Batch log shows that units…
    Show the result
    - Cause: Lab teardown found that cells swell when the separator is thin.
- Scope: Batch log shows that units made in March are affected.
- Customers: Returns desk reports that customers can swap a unit for free.
- Safety: A retest of the swap units passed the safety test.
- Regulatory: Not covered. The regulator search timed out on "agency deadline" after 2 attempts and returned no partial results.
- Coverage: This report does not cover the whole recall, because the regulatory area is missing and the search failures mean it should not be called complete.
GAPS_NAMED.
The failure record and the merge as written
failure builds the record of a failed search and merge turns every result into the writer's prompt. The starting failure returns nothing and the starting merge skips any result with no text, which is every failed search.


Aborting on one failed search would lose the other angles. Here 4 of the 5 searches worked, so about 80 percent of the work would go. A record that travels keeps all of it.


starter/report.py
1"""Failure handling: what a failed search says, and how results are merged."""23from typing import Any456def failure(7    kind: str, query: str, partial: list[str], attempt: int = 18) -> dict[str, Any]:9    """The record of a failed search.10    TODO(workshop): keep the kind, the query, the attempt and the partial11    results, so the writer can say what was not covered.12    """13    return {}141516def merge(results: list[dict[str, Any]]) -> str:17    """One line per result, for the writer."""18    seen: set[str] = set()19    lines = []20    for r in results:21        if not r["text"] or r["angle"] in seen:22            continue2324        seen.add(r["angle"])25        lines.append(f"- {r['angle']} ({r['source']}): {r['text']}")2627    return "\n".join(lines)
What does the writer say about a search that failed?
Take the recall request. The regulator search timed out, and the starting merge dropped it. The writer is given the four results that worked.


Predict: What does the writer say about the regulator?
  1. AThat the regulator section is missing.
  2. BNothing, because it never saw the failure, and the report reads as complete.
Show the answer
B. A swallowed failure looks like a finished report. The writer cannot name a gap it was never told about.
The report claims to be complete
This is the same request with the starting failure handling. Three researchers ran five searches and the regulator search timed out, but the writer was shown only the four good results. The outcome line is red.


This is a real Claude Haiku 5.5 run through the Agent SDK, recorded once and replayed here.


  1. ColleagueWrite the recall report for the power bank.
  2. Coordinatorstarted 3 researchers, one per group of searches
  3. Researchers5 searches, replies below
    Show the result
  4. Writer prompt0 gap lines shown to the writer
    Show the result
    Write the recall report for the power bank.

- cause (Lab teardown): Cells swell when a separator is thin.
- scope (Batch log): Units made in March are affected.
- customers (Returns desk): Customers can swap a unit for free.
- safety (Safety test): A retest of the swap units passed.
  5. Writer- Cause: Lab teardown found that cells swell when the separator is too thin.
- Scope: Batch log shows that…
    Show the result
    - Cause: Lab teardown found that cells swell when the separator is too thin.
- Scope: Batch log shows that units made in March are affected.
- Customers: Returns desk confirms customers can swap an affected unit for free.
- Safety: A safety retest of the swapped units passed.

No, this does not cover the whole recall. The safety retest covers only swapped units, not all March units, and the findings include no unit counts or deadlines.
CLAIMED_COMPLETE.
Keep the kind, the query and what came back
The fix gives a failure a record with four fields: the kind, the query, the attempt and the partial results. The merge writes a line for each failed search and tells the writer not to call the report complete. A transient failure is retried once where it happens, and only the rest is passed up.


An empty result is different. A search that ran and found nothing is a valid answer. Only a search that could not be done is a failure.


A gap can also start in the plan. An angle nobody owns is never searched, so the report has an empty section and no failure to show for it. Broaden the split so every topic has a researcher, instead of retrying the same narrow plan.


What happens when a search fails.ColleagueCoordinatorResearchersWriterreport requestone group eachresults and errorsmerged findingsreport
report.pygreen lines are what the fix adds
1"""Failure handling: what a failed search says, and how results are merged."""23from typing import Any456def failure(7    kind: str, query: str, partial: list[str], attempt: int = 18) -> dict[str, Any]:9    """The record of a failed search: what failed, on what, what was got."""10    return {11        "kind": kind,12        "query": query,13        "partial": partial,14        "attempt": attempt,15    }161718def merge(results: list[dict[str, Any]]) -> str:19    """One line per result. A failure and a conflict stay visible."""20    lines = []21    for r in results:22        error = r["error"]23        if error:24            got = ", ".join(error["partial"]) or "nothing"25            lines.append(26                f"GAP: the {r['angle']} search failed ({error['kind']}) "27                f"on '{error['query']}' after {error['attempt']} attempts. "28                f"Partial results: {got}. Say this area is not covered."29            )30        else:31            lines.append(f"- {r['angle']} ({r['source']}): {r['text']}")3233    angles = [r["angle"] for r in results if not r["error"]]34    if len(angles) != len(set(angles)):35        lines.append("Some sources disagree. Report every figure and source.")3637    if any(r["error"] for r in results):38        lines.append("Some searches failed. Do not call the report complete.")3940    return "\n".join(lines)
We doChange one setting at a time and predict.
The same trouble, two different reports
Use the switch to run the request with the starting handling, then the fixed one. The card marked "This is where the runs split" is the first difference between the two runs.


Try the half-working search and the two conflicting figures too. The fix holds on all three.


This part is interactive in the lab: run ./workshop watch failures-surface.
Errors need a category
A tool that fails with only "Operation failed" gives the researcher nothing to report. In the bare run the regulator researcher passes on just those two words. Switch to the run where the error carries a category, a retry flag, a message, the attempt and the partial results. There the researcher reports a timeout that cannot be retried.


An MCP tool, one served through the Model Context Protocol, sets isError on its result and puts the errorCategory in the content. A business rule that rejects a request is not retryable, and the error should say why, so the agent stops instead of retrying.


The writer should also mark how well the merged report is supported. Mark the claims that their sources support well, apart from the ones sources dispute, and show the sources of each dispute. Mark every gap where a search failed or no source was found, and never hide one inside the conclusion.


This part is interactive in the lab: run ./workshop watch failures-surface.
Try it: trouble and failure handling
Change one setting at a time. Pick a timeout, a half-working search or two conflicting figures, then switch between the starting handling and the fix. Swallowing a failure and aborting on it are both wrong.


Every combination is a recorded run, so this works with no internet.


This part is interactive in the lab: run ./workshop watch failures-surface.
Passing failures back in the Agent SDK
A failed search is an MCP tool result with its error flag set and an errorCategory in its text. The flag has two spellings: is_error in Python and isError in MCP.


Researchers are defined with AgentDefinition and started with the Agent tool, so that tool must be allowed. The harness runs both phases through query(). This is a real Claude Haiku 5.5 run.


A coordinator could also send a new task out for a gap. These recordings never loop back: the gap goes to the writer, and the report names it.


app/agent.pyfile: 360 lines
···43 lines not shown44WRITER = (45    "You write a recall report for a colleague from the findings you are "46    "given. Write one line per finding, then one line saying whether the "47    "report covers the whole recall."48)495051def text(words: str, error: bool = False) -> Json:52    """One text result; `is_error` is how the SDK sets isError."""53    return {"content": [{"type": "text", "text": words}], "is_error": error}545556def outcome(result: Result, bare: bool) -> Json:57    """A search result as a tool returns it, typed or bare."""58    error = cast("Json | None", result["error"])59    if not error:60        found = f"{result['source']} ({result['angle']}): {result['text']}"61        return text(found)6263    if bare:64        return text("Operation failed", True)6566    body = {67        "errorCategory": error["kind"],68        "isRetryable": False,···292 lines not shown
  1. ColleagueWrite the recall report for the power bank.
  2. Coordinatorstarted 3 researchers, one per group of searches
  3. Researchers5 searches, replies below
    Show the result
  4. Writer prompt1 gap line shown to the writer
    Show the result
    Write the recall report for the power bank.

GAP: the supplier search failed (timeout) on 'suppliers of the separator' after 2 attempts. Partial results: Northvale Cells. Say this area is not covered.
- customers (Returns desk): Customers can swap a unit for free.
- safety (Safety test): A retest of the swap units passed.
- cause (Lab teardown): Cells swell when a separator is thin.
- scope (Batch log): Units made in March are affected.
Some searches failed. Do not call the report complete.
  5. Writer- Customers: Customers can swap a unit for free at the Returns desk.
- Safety: A retest of the swap units…
    Show the result
    - Customers: Customers can swap a unit for free at the Returns desk.
- Safety: A retest of the swap units passed.
- Cause: Lab teardown found that cells swell when a separator is thin.
- Scope: The batch log shows that units made in March are affected.
- Coverage: Not complete. The supplier search for separator suppliers timed out after 2 attempts, with only a partial result (Northvale Cells), so that area is not covered.
GAPS_NAMED.
You doFix the starting code, then test yourself.
You do: keep the failures
Copy the starting code and open work/failures-surface/report.py. failure() returns an empty record, and merge() drops failed angles and keeps one source per angle.


Terminal
./workshop start failures-surface
./workshop check failures-surface
./workshop watch failures-surface




The checks want the kind, query, attempt and partial results in a failure record. The merge must name the failed search and warn the writer not to call the report complete. It keeps both conflicting sources, and an empty result is not a failure.


Check your understanding
Answer each question, then read why every option is right or wrong. The wrong options are the mistakes engineers make on the job.


1. A regulator search times out. The merge drops that angle, and the writer produces a report that reads as complete. 1 of 4 angles is missing. What is the root fix?
  1. AAsk the writer to sound less certain in the conclusion, since one of the four angles may be thin.
  2. BPass the failure to the writer with its kind and its query, so the report can name the gap.
  3. CAdd a second search for the regulator each time, so that a timeout becomes unlikely on the next run.
  4. DDrop the regulator angle from the plan for good, since its search is not reliable enough to keep.
Show the answer
Answer: B
  • A Hedged wording does not tell the reader which angle is missing or why.
  • B Correct. A failure that never reaches the writer cannot appear in the report, so it must travel as a result.
  • C A retry may help once, but a search that fails again is still dropped without a word.
  • D Removing an angle hides the gap instead of reporting it.
2. A researcher times out on one of 4 searches. The coordinator aborts the whole run, so the 3 good results, about 75 percent of the work, are lost. What should happen instead?
  1. ARetry the whole run from the very start until all 4 searches pass in the same attempt.
  2. BKeep the abort, but add a clearer error message for the person who reads the log later.
  3. CSkip the failed search quietly and report the 3 results as if they were the whole answer.
  4. DReturn the failure together with the 3 good results, and let the report name the gap.
Show the answer
Answer: D
  • A A repeat of a failing search may time out again, and every pass throws away the good work.
  • B A better message does not save the 3 results that were lost.
  • C Skipping quietly hides the gap, which is the opposite mistake.
  • D Correct. Partial results still have value. The report can use them and say what is not covered.
3. A subagent returns the string 'search unavailable'. The coordinator cannot tell whether to retry, try another source or stop. What should the subagent return?
  1. AThe kind of failure, what was tried, what came back before it failed and what to try next.
  2. BThe same string unchanged, since the coordinator can read it and judge for itself what it meant.
  3. CAn empty list, since no results were found and that is the honest answer to give here.
  4. DA longer sentence that says the search was unavailable for some unknown reason.
Show the answer
Answer: A
  • A Correct. A coordinator can decide only when it knows the kind, the query and the partial results.
  • B A bare message carries no kind and no query, so the coordinator is left guessing.
  • C An empty list looks like a valid search with no hits. A failure must not look like that.
  • D More words without a kind or a query still give the coordinator nothing to act on.
4. Select two wrong ways to handle a failed search in a research run.
  1. APass the failure up with its kind, query and partial results.
  2. BDrop the failed angle from the merge and call the report complete.
  3. CRetry once where the failure is transient, then pass up what remains.
  4. DAbort every researcher because one of them failed.
Show the answer
Answer: B, D
  • A This is the right way, because the failure can reach the report.
  • B Correct. The reader is told nothing, so a missing section looks finished.
  • C A bounded retry is sound. Only an unbounded loop would be wrong.
  • D Correct. The good results are lost, and the coordinator learns nothing about what worked.
5. Two sources give different failure rates for the same batch: 12 percent from a lab report and 15 percent from a claims desk. The merge keeps 15 percent. What is wrong?
  1. ANothing, since the higher figure is the safer one to report for a recall.
  2. BThe report should have averaged them to 13.5 percent, which is closer to the truth.
  3. CThe other figure and its source are gone, so the reader cannot see the conflict.
  4. DThe report is fine as long as it mentions that some sources were consulted.
Show the answer
Answer: C
  • A Safer is a judgement the reader should make with both figures in front of them.
  • B An average belongs to neither source and cannot be checked.
  • C Correct. Two figures from two sources are a finding. Keep both and say where each came from.
  • D A general note names no source, so the reader still cannot check anything.
6. A report lists 40 swollen units from a 9 March field report and 65 from an 18 March claims update. A reviewer calls it a contradiction. What explains it?
  1. AThe dates differ, so the later count may include units that swelled after the first.
  2. BOne source is wrong, so the writer should keep the newer figure and delete the other.
  3. CBoth figures are noise, so the writer should report only that some units swelled.
  4. DThe claims desk counted the same units twice, so its figure should be halved.
Show the answer
Answer: A
  • A Correct. A date often explains a gap, and only a report that shows both dates lets the reader see it.
  • B Nothing says either count is wrong. Deleting one throws away the trend.
  • C Dropping both loses the evidence the reader needs.
  • D That is a guess with no source. The report should show the figures, not invent a fix.
7. A synthesis of 14 findings mixes 11 settled claims with 3 that sources dispute. The reader cannot tell which is which. What should the writer do?
  1. AWrite all 14 in one list with the same wording, so the report reads evenly.
  2. BLeave out the 3 disputed claims, since a report should hold only what is certain.
  3. CPut the disputed claims first, since the reader should see the risks before anything else.
  4. DMark the settled claims and the disputed ones apart, and show the sources of each dispute.
Show the answer
Answer: D
  • A An even tone hides which claims are solid.
  • B Leaving them out hides the dispute, and a reader may need it.
  • C Order alone does not mark them as disputed or show the sources.
  • D Correct. A reader can weigh a disputed claim only when it is marked and its sources are shown.
8. A report merges findings from 5 researchers. Select two rules for marking the quality of the synthesis.
  1. AMark the findings that are well supported by their sources.
  2. BHide the gaps inside the conclusion, so the report reads as one piece.
  3. CMark the gaps where a search failed or no source was found.
  4. DGive every claim the same confidence, so the report stays consistent.
Show the answer
Answer: A, C
  • A Correct. A reader needs to know which claims are solid.
  • B A gap that cannot be found cannot be acted on.
  • C Correct. A marked gap tells the reader what the report does not cover.
  • D A uniform tone hides which claims are weak.
9. A coordinator never heard that one of its 3 researchers failed. The researcher wrote the failure to its own notes. What does the design need?
  1. ALet researchers message each other directly, so a failure reaches whoever needs it without delay.
  2. BAll messages pass through the coordinator, so a failure is a message to it.
  3. CHave the writer read every researcher's notes before it starts, including any that failed.
  4. DGive the researcher a much longer timeout so that it rarely fails at all in practice.
Show the answer
Answer: B
  • A Side channels skip the coordinator, which is where the gap needs to be handled.
  • B Correct. The coordinator is the one place that routes results and errors, so nothing can be lost on the side.
  • C The writer would read what was written, not what failed. The coordinator still would not know.
  • D A longer timeout reduces some failures. It cannot make the coordinator hear of the rest.
10. The synthesis shows that 1 of 4 angles has no findings. The coordinator can still start a researcher. What should it do next?
  1. AFinish the report and leave the gap for a reader to spot.
  2. BRestart the whole run with all 4 angles, since partial work is not trusted.
  3. CSend a new task back out for the uncovered angle, then merge what it returns.
  4. DAsk the writer to fill the missing angle from its own general knowledge.
Show the answer
Answer: C
  • A A gap the coordinator can close should not be shipped as it is.
  • B The 3 good results are thrown away to redo work that was fine.
  • C Correct. A coordinator can loop back out for a gap instead of reporting it as it is.
  • D The writer has no source for it, so the section would be unsourced.
11. A plan split a recall into 2 angles, and 2 topics have no owner. The report has two empty sections. What is the fix?
  1. ABroaden the split so every topic the report needs has an angle and a researcher.
  2. BGive each of the 2 researchers a longer brief, so they cover the extra topics.
  3. CAdd a line to the report that says the empty sections are out of scope.
  4. DRetry the 2 researchers until the empty sections fill in.
Show the answer
Answer: A
  • A Correct. An angle nobody owns is never searched, so the gap starts in the plan.
  • B A longer brief does not change which angles exist.
  • C Declaring it out of scope hides a gap the plan created.
  • D A retry repeats the same narrow plan and gets the same result.
12. A coordinator sits between 4 subagents. Select two things it owns.
  1. ALetting every subagent write to the report directly.
  2. BRouting every message between the subagents and the writer.
  3. CHandling the errors that the subagents report.
  4. DHiding failed searches from the writer so the report reads cleanly.
Show the answer
Answer: B, C
  • A Direct writes bypass the merge, where gaps and conflicts are marked.
  • B Correct. All traffic passes through it, so it is the one place that can route.
  • C Correct. Retrying, passing up or recording a failure is a decision for the coordinator.
  • D Hiding a failure is how a report ends up claiming to be complete.
13. A tool returns the text 'operation failed' for 6 different faults. The agent cannot decide whether to retry. What should the tool return?
  1. AThe same text, since the agent can read it and judge which fault it was.
  2. BAn empty result, so the agent treats it as a search with no hits.
  3. CA stack trace from the server, since it holds the most detail about the fault.
  4. DAn error flag with a category, whether it is retryable, and a readable message.
Show the answer
Answer: D
  • A The text is the same for all 6 faults, so there is nothing to judge.
  • B An empty result hides a failure behind a valid answer.
  • C A trace is for the operator. It is not a category the agent can act on.
  • D Correct. With a category and a retry flag, the agent can choose to retry, ask or stop.
14. A refund tool rejects a request under a business rule: the order is 90 days old and the limit is 30. The agent retries 5 times. What was missed?
  1. AThe limit should be raised to 90 days so the retry succeeds.
  2. BThe error should say it is not retryable and explain the rule, so the agent stops.
  3. CThe agent should retry more times, since some retries succeed after a delay.
  4. DThe tool should return an empty result, so the agent gives up on its own.
Show the answer
Answer: B
  • A The rule exists for a reason. The fault is that the agent was never told it is final.
  • B Correct. A business rule is not a transient fault. A retry cannot change the answer.
  • C A delay does not change a business rule.
  • D An empty result looks like a valid search. The agent needs the reason.
15. A customer search returns 0 rows for an email that is not in the system. The agent treats it as an error and retries 3 times. What is the right reading?
  1. AIt is an access failure, so the tool should raise an error for each empty result.
  2. BIt is a timeout, so the agent should wait and run the same search again.
  3. CIt is a valid empty result, not a failure, so the agent reports that nobody was found.
  4. DIt is a sign the search tool is broken, so the agent should stop all work.
Show the answer
Answer: C
  • A An error for every miss drowns the real failures in noise.
  • B Nothing timed out. The same search will return the same empty list.
  • C Correct. The search worked and found nothing. A failure is when the search could not be done.
  • D One empty answer says nothing about the tool, and stopping loses the rest of the work.
16. An MCP tool call fails. Select two things its result can carry.
  1. AAn isError flag that marks the result as a failure.
  2. BA promise that the next call will succeed.
  3. CNothing, because an error is shown only in the server log.
  4. DA category such as timeout, with a message the agent can read.
Show the answer
Answer: A, D
  • A Correct. The flag tells the agent the call failed, and the content says why.
  • B A tool cannot know that, and the agent should not expect it.
  • C The result reaches the agent, which is why it must carry the kind of failure.
  • D Correct. A category lets the agent choose between retrying, asking for help and stopping.
Play: predict the output
This failure record names what failed but keeps nothing it got.


python
def failure(kind: str, query: str, partial: list[str], attempt: int = 1):
    return {"kind": kind, "query": query, "attempt": attempt}





Predict: The regulator search times out after it fetched 2 of 4 notices. What can the coordinator do with those 2?
  1. AQuote them in the report, since the search fetched them.
  2. BNothing. They are lost, so a retry must fetch all 4 again.
Show the answer
B. The record drops the partial list it was handed. Keep partial results in the failure, so the report can use what arrived and say what is still missing.
References


Conclusions
    

  • A failed search is a result, so it travels with its kind, query and partial results.
    • 

  • Never swallow a failure and never abort everything for one. Pass it up and name the gap.
    • 

  • Two figures for one angle keep both sources, and an empty result is not a failure.
    • 



You can now write a failure record and a merge that keep a report honest about what it did not cover.


Failure record
The kind, query, attempt and partial results of a failed search
Partial result
What a search returned before it failed
Coverage gap
An angle the report could not research, named in the report
Transient failure
A fault worth one retry where it happens, such as a timeout