Professional, lesson 19
Decision records
One decision, two documents: an ADR and a memo
Lumen Supply picked a model tier and a retrieval design in a chat thread, and six months later nobody can say why. A new team lead asks for the reasons, the alternatives and the way back, and the answer is a search through messages. This lesson writes the decision down twice, once for the engineers and once for the sponsor. This is the support desk at Lumen Supply, an online electronics shop. You will read a decision thread and see a checker accept a memo with no cost figure and no time to undo the change. Then you write the ADR and a checker that scores each section.
A decision nobody can explain
A decision record is read by someone who was not in the room. An architecture decision record, an ADR, holds the context, the options, the decision, the consequences and the rollback in one place.
A memo is the one page a sponsor reads before approving the spend. The model drafts the memo, the checker scores it, and the team owns the decision.
Three things go wrong. The decision lives in chat, so its reasons and alternatives are lost. One document serves the sponsor and the engineers, so it suits neither. And no implementation guide or rollback plan sits beside the decision.
An ADR and the memo drawn from it
The model reads the ADR, drafts a memo for the sponsor, and asks the checker to accept it. The ADR holds three options and the one chosen: the small tier with hybrid retrieval, the top 5 passages. It cuts the cost from 6 cents to under 1 cent a ticket, about 500 dollars a day at 10,000 tickets.
Open a card to see the draft. The memo opens with the outcome, the cost and the ask. It then gives the risk and the rollback: one config change, about 10 minutes, with a trigger of wrong answers above 12 in 100.
- ReaderWrite the memo the sponsor approves the spend from.
- RecordRead the ADR
- ClaudeSupport agent: move first-line tickets to the small tier…
Show the result
{ "title": "Support agent: move first-line tickets to the small tier with hybrid retrieval", "text": "## Outcome, cost and ask\n\n- Outcome: First-line support moves to the small model with hybrid search (keyword plus vector, top 5 passages). Refunds above $500 still go to a person.\n- Cost: Falls from 6 cents to under 1 cent per ticket, about $500 a day saved at 10,000 tickets. This meets finance's 2-cent cap. The large tier breaks that cap.\n- Ask: Approve the change. Marcus makes the config change and builds the sync job for the two search indexes. Dana tells finance the new cost and the review date. Priya reviews the outcome on 12 March.\n\n## Risk\n\n- Answer quality is the main risk. On a 200-ticket sample, the small tier with hybrid search gave about 10 fewer right answers than the large tier (182 of 200). The record does not state the exact accuracy for this option. The sample gap points to roughly 14 wrong in 100, which is above the rollback line of 12 in 100. Confirm the measured accuracy before go-live.\n- Speed: About 1 second a reply, well inside the 5-second promise at the 95th percentile.\n- Operations: Two indexes must stay in sync. The sync job needs a named owner (Marcus). Part-number search depends on the keyword index, so it cannot be dropped later.\n\n## The way back\n\n- Set the tier back to the large tier and turn off the vector index. This is one config value and takes about 10 minutes.\n- Roll back if wrong answers exceed 12 in 100 for two days running, or if the 95th-percentile reply time passes 5 seconds.\n- Check for one week after the change: wrong answers under 12 in 100 and cost under 2 cents per ticket." } - ClaudeSupport agent: move first-line tickets to the small tier…
Show the result
{ "title": "Support agent: move first-line tickets to the small tier with hybrid retrieval", "text": "## Outcome, cost and ask\n\n- Outcome: First-line support moves to the small model with hybrid search (keyword plus vector, top 5 passages). Refunds above $500 still go to a person.\n- Cost: Falls from 6 cents to under 1 cent per ticket, about $500 a day saved at 10,000 tickets. This meets finance's 2-cent cap, which the large tier breaks.\n- Ask: Approve the change. Marcus makes the config change and builds the sync job for the two search indexes. Dana tells finance the new cost and the review date. Priya reviews the outcome on 12 March.\n\n## Risk\n\n- Answer quality is the main risk. On a 200-ticket sample, the large tier gave 182 right answers (18 wrong in 100). The record gives no hybrid answer count. The stated gap is 10 answers in 200, which implies about 14 wrong in 100 for the chosen option. That is my inference from the record, so confirm it before go-live.\n- The rollback line is tighter than the old setting. The trigger is more than 12 wrong in 100. The large tier already ran at 18 wrong in 100 on the sample. So rolling back may not restore quality, and the trigger could fire even on the old setting. Treat the 12-in-100 line as a target to check, not a clean test, and agree a threshold the sponsor accepts.\n- Speed: About 1 second a reply, well inside the 5-second promise at the 95th percentile.\n- Operations: Two indexes must stay in sync. The sync job needs a named owner (Marcus). Part-number search depends on the keyword index, so it cannot be dropped later.\n\n## The way back\n\n- Set the tier back to the large tier and turn off the vector index. This is one config value and takes about 10 minutes. It restores the old cost of 6 cents per ticket, so the finance cap is breached again.\n- Roll back if wrong answers exceed 12 in 100 for two days running, or if the 95th-percentile reply time passes 5 seconds.\n- Check for one week after the change: wrong answers under 12 in 100 and cost under 2 cents per ticket." } - Checkaccepted
Show the result
{ "accepted": true }
The memo can be acted on: it has every section.
The memo checks, written as code
A memo is for a decision maker, and an ADR is for the next engineer. The memo needs five sections: the outcome, the cost, the ask, the risk and the rollback. A check on each one is a rule the model cannot talk past.
The starting check_memo accepts any text with a title. A memo that gives no cost figure, or no time to undo the change, passes.
Is a chat thread a decision record?
The thread has forty messages. Priya, Marcus and Dana weigh the options, quote the numbers and agree on the small tier. Every reason is in there somewhere.
Predict: Is a chat thread a decision record?
- ANo. Nobody can find or trust it later.
- BYes. It holds every reason.
Show the answer
A. A record names the context, the options, the decision and the rollback in one place. A thread holds the reasons in no order, with the cost and the rollback missing.
The decision lives in chat
The model reads the thread and writes a memo for a new team lead. The checker accepts it, because it has a title. Judged by the real checker, the cost and the rollback are too weak to act on.
The memo says the small tier "is cheaper than the big tier", with no figure. Its How to Undo section says the time and the steps "are not recorded", and its trigger has no number. Nobody wrote either down in the thread. The new lead reads a confident page with the answer missing.
- ReaderWrite the memo a new lead needs: why, and how to undo it.
- RecordRead the thread
- ClaudeSupport agent v1: model tier and retrieval decision
Show the result
{ "title": "Support agent v1: model tier and retrieval decision", "text": "## Outcome\n\nSupport agent v1 moves to the small model tier with hybrid retrieval (keyword plus vector search), returning the top 5 passages. Config change ships Friday, after the sync job is ready Thursday. Owners: Priya (decision, sample evaluation), Marcus (sync job and config change), Dana (finance note). Refunds over $500 still go to a person; this does not change.\n\n## Cost and ask\n\nCost: the small tier is cheaper than the big tier. The thread gives no dollar figures, so the actual savings are unquantified. Finance was told the savings are real; Dana will send a short note. Ongoing cost includes keeping two search indexes in sync, which Marcus will own.\n\nAsk: confirm the savings figure with finance, and confirm who owns the sync job after Marcus.\n\n## Why this choice\n\n- Sample of 200 tickets: small tier 172 correct (86%), big tier 182 correct (91%). Gap of 10 tickets.\n- Latency: small tier about 1 second, big tier about 3 seconds. Both are within the contract's 5-second p95.\n- Retrieval was the main source of errors. Keyword alone found the right article 61% of the time, vector 74%, hybrid 83%.\n- Top 10 passages were slower without better answers, so top 5 was kept.\n\n## What could go wrong\n\n- The team assumed the small tier's errors were mostly retrieval misses and that hybrid would close the gap. That was not tested directly on the small tier in the thread. Watch answer accuracy after launch, especially on long tickets, where the slowdown was first seen.\n- Spare-part questions depend on keyword matching for part numbers. If the keyword index falls out of sync with vector data, those answers will degrade quietly.\n- Holiday peak traffic arrives soon after launch.\n\n## How to undo\n\nNot discussed in the thread. The change is a config switch back to the big tier and retrieval settings, so rollback should be a config change, not a rebuild. Estimated time and the exact steps are not recorded; confirm them with Marcus before peak. Undo if answer accuracy drops below the big-tier baseline on a fresh sample, or if latency approaches 5 seconds at p95. Decide the threshold before launch, not after." } - Checkaccepted
Show the result
{ "accepted": true }
Accepted, but it cannot be acted on: cost, rollback.
Two documents, one checker
check_memo returns the sections that are missing or too weak: the outcome, the cost, the ask, the risk and the rollback. Each part is found by the words in its heading, so "How to Undo" and "Rollback" get the same verdict. A refused draft goes back to the model with the sections to fix.
A section is weak when it is a few words or says nothing was recorded. The cost needs a money figure, and the rollback needs a time and a trigger. The ADR carries both, so the model has something to say.
The arithmetic is plain. Rebuilding a lost reason takes a week of asking around, and writing it down takes an hour.
The same input now ends well
Switch the record for the new team lead. With the chat thread, the memo is accepted and cannot be acted on. With the ADR, the checker refuses a draft with no size for the outcome, then one with no cost figure. The third is accepted: its rollback takes about 10 minutes, triggered by wrong answers above 12 in 100.
The decision did not change. Only the record did: the ADR holds the figures and the chat thread does not. The checker scores sections, not facts. This draft works out about 14 wrong answers in 100 from the ADR figures, and nothing checks that sum, so a human still reads the memo.
This part is interactive in the lab: run ./workshop watch decisions-and-handoff.
Try it: record and reader
Change one setting at a time. Pick the record, then the reader. Every combination is a recorded run, so this works with no internet.
An implementation guide sits beside the ADR, with steps and an owner. With the ADR, the sponsor's memo is accepted at once. The engineer's first draft has no risk section and is refused; the second adds one and is accepted. The new lead's memo is refused twice, for the outcome and then the cost, and the third draft is accepted.
This part is interactive in the lab: run ./workshop watch decisions-and-handoff.
The implementation guide
The ADR has a guide that lists the steps in order. Each step names an owner, and a check says whether the change did what the ADR promised. A guide with no owner has nobody to call when a step fails. Enforce in managed settings and hooks ships it to the team.
The ADR also has a life after launch. It names a review date, 12 March, and an owner, Priya, who compares the measured outcome with the promise. A record that is never reviewed cannot be corrected when the numbers drift.
A reversed decision is never deleted. The new ADR marks the old one superseded and links back, so a later reader sees what was tried. The lifecycle plan also covers the end: retiring the agent names an owner, a date and what happens to the stored tickets.
You do: refuse a memo nobody can act on
Copy the starting code and open work/decisions-and-handoff/memo.py and adr.md. check_memo() accepts any memo that has a title, so one with no cost and no rollback passes.
./workshop start decisions-and-handoff
./workshop check decisions-and-handoff
./workshop watch decisions-and-handoff
The checks name each missing or weak section. The cost needs a money figure and the rollback needs a time and a trigger. A renamed heading gets the same verdict, and the record names every part of a decision.
Check your understanding
Answer each question, then read why every option is right or wrong. The wrong options are the mistakes engineers make on the job.
1. Lumen chose a model tier and a retrieval design in a chat thread of 40 messages, and the thread has since scrolled out of search. Six months later a new team lead asks why the small tier was picked and what else was considered. Nobody can say, and the review is on Friday. What should the team have written when the decision was made?
- AA pinned chat message that links to the thread and says it holds the reasons.
- BA decision record with the context, options, decision and rollback.
- CA short note that says the decision was made and who was in the room.
- DA summary written at the review, from what people remember by then.
Show the answer
Answer: B
- A A pinned link still points at a chat. The reasons stay scattered across 40 messages, and nobody can trust which ones held.
- B Correct. A record names the context, the options, the decision and the rollback in one place that a new lead can read.
- C A note with no reasons or alternatives repeats the problem. The question was why, and the note does not say.
- D Memory at the review is the source that already failed. A record is written while the options are fresh.
2. The sponsor of the Lumen support project has 5 minutes and must approve a spend. The two engineers who will run it need the design detail and the steps. The team wrote one 12 page document and both groups say it does not help them. What is the right change to how the decision is written down?
- AShorten the document to 3 pages so that it reads faster for everyone.
- BAdd a code listing to the document so that the sponsor sees how it works.
- CPost the document in chat so each reader can ask for the part they need.
- DWrite an ADR for the engineers and a one page memo for the sponsor.
Show the answer
Answer: D
- A A shorter single document still drops detail the engineers need or keeps detail the sponsor skips.
- B Code shows the sponsor how it works, and the sponsor asked for the outcome, the cost and the ask.
- C A chat answers one question at a time and leaves no record. Neither reader gets a document to act on.
- D Correct. Two readers make two decisions, so each gets a document built for that decision.
3. A memo to the finance sponsor opens with three paragraphs on how hybrid retrieval ranks results and ends with a list of file names. The sponsor must approve about 500 dollars a day of spend by Friday and cannot find the ask or the cost in the memo. What should the memo lead with?
- AThe outcome, the dollar cost and the ask, then risk and rollback.
- BThe ranking method, so that the sponsor understands what is being approved.
- CThe full ADR pasted in, so that the sponsor has every detail to hand.
- DNothing until the work is finished, since a memo can wait for the launch.
Show the answer
Answer: A
- A Correct. A sponsor decides on the outcome, the cost and the ask, so those come first and the detail follows.
- B A ranking method is design detail. It belongs in the ADR for the engineers who build it.
- C Pasting the ADR is the one document for all readers again. The ask is still buried.
- D Waiting leaves the sponsor with no way to approve by Friday, and the reasons fade before launch.
4. A reviewer audits the decision records of a support agent that cost 500 dollars a day to run last month. Each record must let a new engineer judge the decision without asking the authors. Select two parts that a decision record must hold for that to work. The audit is next week.
- AThe options that were rejected, and why each lost.
- BA copy of the chat thread, so that nothing is left out.
- CThe consequences, both the gains and the costs.
- DThe source code of the change, line by line.
Show the answer
Answer: A, C
- A Correct. The rejected options show what was weighed, so a later reader does not reopen a settled question.
- B A copy of the chat has no structure. The reader still hunts through it for the reason.
- C Correct. The consequences state what the decision costs, which is what a reviewer checks later.
- D The code lives in the repository. A record explains why, which the code cannot say.
5. The team ships the small tier change on a Thursday and moves on. On Monday the support team's on-call engineer is asked to roll it back, but there is no list of steps and nobody knows who owns the sync job. The change touched 3 systems. What should have shipped beside the ADR?
- AA note saying the change is simple and the code is easy to read.
- BA rollback section to be written after the first incident shows what broke.
- CAn implementation guide with ordered steps, an owner for each, and a check.
- DA chat message to the on-call engineer with the names of the three systems.
Show the answer
Answer: C
- A A note about simplicity does not tell an on-call engineer what to do or who to call.
- B A rollback written after an incident is written under pressure, with the details already lost.
- C Correct. A guide lists the steps, names an owner for each and says how to check the change worked.
- D Three names in a chat give no order, no owner and no check, and the message soon scrolls away.
6. A team plans to write the ADR for a retrieval change after launch, once the system has settled. The change ships in 2 weeks, three people made the decision, and one of them leaves next month. The record must let a new hire judge the choice without asking anyone. When should the ADR be written?
- ABefore the change ships, while the options and the reasons are fresh.
- BOnly if the change causes an incident, since otherwise nobody reads it.
- CIn the chat thread, where the three people can add notes as they remember.
- DAfter the rollback plan is needed, when the team knows what went wrong.
Show the answer
Answer: A
- A Correct. The reasons and rejected options are known now, and one of the three people will not be there later.
- B Documentation that waits for an incident is written when the details are gone and time is short.
- C Notes in chat are scattered and unstructured, so the next reader cannot rely on them.
- D A rollback plan is needed before the change, since the plan is what makes the change safe to try.
7. To save effort, a team merged the runbook, the ADR and the sponsor memo for a 10 minute rollback into one 20 page file. The on-call engineer skips it during an incident, and the sponsor never gets past page 1. The team wants both groups to use the documents. What should they do?
- AAdd a table of contents so that each reader can jump to a section.
- BDrop the sponsor memo, since the ADR already holds every fact.
- CRewrite the memo as a short list of the files that changed.
- DKeep one document per audience, and link them to each other.
Show the answer
Answer: D
- A A contents page still leaves one file that serves nobody well, and the on-call engineer will not scroll in an incident.
- B The memo holds the outcome, cost and ask. The ADR holds none of them in a form a sponsor reads.
- C A list of changed files describes the code. The sponsor needs the outcome, the cost and the ask.
- D Correct. Each reader gets a document sized for their decision, and links keep the set connected.
8. A handoff review checks the implementation guide for a change that 3 teams will run. The guide must be usable at 2 in the morning by someone who missed the design meetings, and it must fit on one screen. Select two things the guide must hold. The review closes on Friday.
- AA promise to add detail to the guide after the first incident.
- BThe ordered steps, so a reader never has to guess what comes next.
- CA list of every message from the design thread, for context.
- DThe owner of each step, so a reader knows who to call.
Show the answer
Answer: B, D
- A Detail promised for later is missing exactly when it is needed. The guide is complete at handoff.
- B Correct. A reader at 2 in the morning follows steps in order, so the order is part of the guide.
- C The design thread is context for the ADR, and it adds noise to a guide that must be read in a hurry.
- D Correct. A step with no owner has nobody to call when it fails.
9. A retrieval design has run for six months. The ADR said wrong answers would stay under 12 in 100, and nobody has compared that with what is measured now. The model tier will change again in 3 months, and the engineer who wrote the ADR has moved teams. What should the team schedule for the ADR?
- ANothing, since the decision was made and the record is complete.
- BA review against the measured outcome, with a date and an owner.
- CA rewrite of the ADR from scratch after the next tier change.
- DA thread in chat where people can say if the decision still looks right.
Show the answer
Answer: B
- A A decision that is never compared with its outcome cannot be corrected when it drifts from what was promised.
- B Correct. A scheduled review puts the measured number beside the promised one while there is time to act.
- C A rewrite after the next change comes too late and repeats the work, with the old reasons lost.
- D A chat thread gathers opinions with no measured number and no owner, and then scrolls away.
10. The team reversed its decision and went back to the large tier after wrong answers passed 12 in 100. A cleanup script deletes the old ADR so that the repository only holds current decisions. A new lead later asks why the small tier was ever tried. What should happen to the old ADR?
- AIt stays in the chat history, where anyone can search for it.
- BIt is deleted, since a reversed decision is no longer useful to read.
- CIt is marked superseded and linked to the new ADR that replaced it.
- DIt is rewritten to match the new decision, so only one version exists.
Show the answer
Answer: C
- A Chat history is neither a record nor reliably searchable, so the reasons would be lost again.
- B The old record holds what was tried and why it failed. Deleting it invites the same mistake.
- C Correct. A superseded record keeps the history, and the link shows what replaced it and why.
- D Rewriting the old ADR erases what was believed at the time, which is what a later reader needs.
11. The lifecycle plan for a support agent has a build phase and a run phase, with a review every 6 months. In 2 years the agent will be replaced, and it stores 40,000 customer tickets. A governance review asks what the plan says about the end. What should be added?
- AA line that says the agent will be reviewed if anyone raises a concern.
- BA rollback step that returns the agent to the previous tier.
- CA note to plan the retirement when the replacement is nearly ready.
- DA retirement step with the owner, the date and a plan for the stored data.
Show the answer
Answer: D
- A Waiting for a concern leaves the end of the agent to chance, and the stored tickets stay where they are.
- B A rollback undoes a change in the run phase. It is not a plan for taking the agent out of service.
- C A plan written when the replacement is nearly ready is written under time pressure and often skipped.
- D Correct. A lifecycle covers the end, so retirement names who acts, when, and what happens to the data.
12. A governance review of an agent lifecycle plan checks that the plan covers more than the build. The agent runs for 18 months, and its data is kept for 2 years under a policy. Select two lifecycle phases that the plan must cover beyond the build. The review is on Monday.
- AOngoing review of the decision against the measured outcome.
- BA rollback to be designed later, once a failure has shown the cause.
- CDocumentation that is optional and added when time allows.
- DRetirement, with the owner, the date and the data plan.
Show the answer
Answer: A, D
- A Correct. Review keeps the decision honest after launch, by comparing it with what is measured.
- B A rollback designed after a failure is missing when the failure happens.
- C Optional documentation is the first thing dropped, and the plan needs it at every phase.
- D Correct. Retirement closes the lifecycle, and the data plan keeps the 2 year policy.
13. Three stakeholders read one Lumen status report each Friday. The sponsor wants the cost against the 500 dollar a day budget, the support lead wants wrong answers per 100 tickets, and the engineer wants latency in seconds. Each says it is not for them. How should the reporting change?
- ALead each report with the number its own reader decides on.
- BAdd every number to one longer report, so nobody is left without theirs.
- CSend the code diff, since it shows exactly what changed that week.
- DMove the report into a chat channel, so each reader can ask questions.
Show the answer
Answer: A
- A Correct. Each stakeholder decides something different, so each report leads with the number that decision needs.
- B A longer report with every number buries each reader's figure under the others.
- C A diff describes the code and answers none of the three questions.
- D A chat channel gives no standard figure and no record, so the readers still disagree on the numbers.
14. The memo for the sponsor states the outcome and the cost, but says nothing about how the change would be undone. In week 3 wrong answers rise above 12 in 100, the sponsor learns that a rollback was possible only from a chat, and trust falls. The next memo goes out Friday. What should it add?
- AA line that documentation will follow once things are stable.
- BA promise to design a rollback if the numbers get worse.
- CThe rollback plan, with its trigger, time to undo and owner.
- DA list of the files changed, so the sponsor can judge the effort.
Show the answer
Answer: C
- A A promise of documentation later is the habit that left the sponsor without the rollback.
- B A rollback designed when numbers get worse is too slow. The sponsor needs the plan before the change.
- C Correct. A stated trigger, time and owner tell the sponsor how the change is undone, which rebuilds trust.
- D Changed files describe the code. The sponsor asked how the change would be undone.
Play: predict the output
This checker accepts a section that has at least six words.
def weak(body: str) -> bool:
return len(body.split()) < 6
rollback = "We can switch back to the large tier in 10 minutes."
Predict: Does this rollback section pass?
- AYes. It has eleven words, though it never says when to switch back.
- BNo. The checker sees that no trigger is named.
Show the answer
A. A way back needs a time and the signal that starts it. Without the signal, nobody rolls back until it is too late. The lab's checker wants both in the rollback.
References
- Define success criteria: the numbers a rollback trigger is written in.
- Building effective agents: the trade-offs a decision record should name.
- Documenting architecture decisions: the short record format behind
adr.md. - Architectural decision records: templates and tools for keeping records next to the code.
Conclusions
- Write the decision down as an ADR before the change ships, with the options and the rollback.
- Give each reader a document for their decision: a memo for the sponsor, an ADR for the engineers.
- Score the memo by section, and give the guide a step, an owner and a check.
You can now turn a chat thread into a decision record that a new lead can act on.
Decision record
One place that holds the context, the options, the decision and the rollback
Stakeholder
A reader who decides or pays, and needs a document made for them
Rollback
The steps, the time and the trigger for undoing a change
Implementation guide
Ordered steps, an owner for each, and a check