Professional, lesson 15

Diagnosing a regression

Ask what changed first, then check the layers in order

Lumen Supply answers get worse after a Friday release. A developer adds rules to the prompt and the wrong answers stay. You will run 40 cases before and after a change, name the cases that flipped, and write the ladder that points at the layer to check first.

I doWatch the failure and its fix.

Worse after Friday, and a prompt that grew

Lumen Supply is an online electronics shop. Its agent answers a policy question from the passage that the help centre returns for it.

The Friday release touched the layers that can break an answer: the index, the lookup tool, the model setting and the prompt. On Monday the answers are worse. A developer adds "do not make things up" to the prompt, and it grows by twelve lines of rules. The wrong answers continue.

Three things go wrong, in this order. A prompt rule is added for a fault that sits in retrieval. No baseline run exists, so nobody can say which cases flipped. And the fix is not kept as a regression case, so the fault can come back.

A baseline and a flip list

The team runs the same 40 cases before the change and after it. Before, all 40 are right. After the new index, 30 are right, and the flip list names the 10 that changed.

Open the flip list to see one case. The question is the same and the agent is the same. The passage is different, because the new index returned the retired page with the old value. The answer follows the passage.

Releasea new index, built on Thursday, that also holds retired pages
Baseline run40 of 40 right before the change
Run after the change30 of 40 right, 10 cases got worse

Flip list10 cases named

Show the result

{
  "flipped cases": [
    {
      "case": "c04",
      "question": "What is the standard return window?",
      "expected": "30 days",
      "passage before": "Returns: full refund within 30 days of delivery.",
      "passage after": "Standard policy. Returns: full refund within 14 days of delivery.",
      "answer before": "30 days",
      "answer after": "14 days"
    },
    {
      "case": "c08",
      "question": "What is the standard refund limit for an agent?",
      "expected": "$500",
      "passage before": "Refund approval: when a customer asks for money back on an order, an agent can refund up to $500 without a manager.",
      "passage after": "Standard policy. Refund approval: when a customer asks for money back on an order, an agent can refund up to $200 without a manager.",
      "answer before": "$500",
      "answer after": "$200"
    },
    {
      "case": "c12",
      "question": "What is the standard laptop warranty?",
      "expected": "24 months",
      "passage before": "Warranty: 24 months from the purchase date on laptops.",
      "passage after": "Standard policy. Warranty: 12 months from the purchase date on laptops.",
      "answer before": "24 months",
      "answer after": "12 months"
    },
    {
      "case": "c16",
      "question": "What is the standard free shipping minimum?",
      "expected": "$75",
      "passage before": "Free shipping: delivery is free for every order in the home zone once the basket total reaches $75.",
      "passage after": "Standard policy. Free shipping: delivery is free for every order in the home zone once the basket total reaches $50.",
      "answer before": "$75",
      "answer after": "$50"
    },
    {
      "case": "c20",
      "question": "What is the standard restocking fee?",
      "expected": "10%",
      "passage before": "Restocking fee: 10% on opened items.",
      "passage after": "Standard policy. Restocking fee: 15% on opened items.",
      "answer before": "10%",
      "answer after": "15%"
    },
    {
      "case": "c24",
      "question": "What is the standard price match period?",
      "expected": "14 days",
      "passage before": "Price match: requests are accepted within 14 days of purchase.",
      "passage after": "Standard policy. Price match: requests are accepted within 7 days of purchase.",
      "answer before": "14 days",
      "answer after": "7 days"
    },
    {
      "case": "c28",
      "question": "What is the standard support reply time?",
      "expected": "4 hours",
      "passage before": "Support reply time: every message sent to the help desk on a working day gets its first answer within 4 hours.",
      "passage after": "Standard policy. Support reply time: every message sent to the help desk on a working day gets its first answer within 24 hours.",
      "answer before": "4 hours",
      "answer after": "24 hours"
    },
    {
      "case": "c32",
      "question": "What is the standard gift card validity?",
      "expected": "36 months",
      "passage before": "Gift cards expire 36 months after purchase.",
      "passage after": "Standard policy. Gift cards expire 12 months after purchase.",
      "answer before": "36 months",
      "answer after": "12 months"
    },
    {
      "case": "c36",
      "question": "What is the standard trade-in quote validity?",
      "expected": "10 days",
      "passage before": "Trade-in quote: valid for 10 days.",
      "passage after": "Standard policy. Trade-in quote: valid for 5 days.",
      "answer before": "10 days",
      "answer after": "5 days"
    },
    {
      "case": "c40",
      "question": "What is the standard free battery replacement period?",
      "expected": "6 months",
      "passage before": "Battery replacement: free within 6 months of purchase.",
      "passage after": "Standard policy. Battery replacement: free within 3 months of purchase.",
      "answer before": "6 months",
      "answer after": "3 months"
    }
  ]
}

Ladderchecks the index first
After the fix40 of 40 right, 0 of the cases that got worse still fail

Regression slice10 cases kept, 10 right after the fix

Show the result

{
  "kept cases": [
    {
      "case": "c04",
      "question": "What is the standard return window?",
      "expected": "30 days"
    },
    {
      "case": "c08",
      "question": "What is the standard refund limit for an agent?",
      "expected": "$500"
    },
    {
      "case": "c12",
      "question": "What is the standard laptop warranty?",
      "expected": "24 months"
    },
    {
      "case": "c16",
      "question": "What is the standard free shipping minimum?",
      "expected": "$75"
    },
    {
      "case": "c20",
      "question": "What is the standard restocking fee?",
      "expected": "10%"
    },
    {
      "case": "c24",
      "question": "What is the standard price match period?",
      "expected": "14 days"
    },
    {
      "case": "c28",
      "question": "What is the standard support reply time?",
      "expected": "4 hours"
    },
    {
      "case": "c32",
      "question": "What is the standard gift card validity?",
      "expected": "36 months"
    },
    {
      "case": "c36",
      "question": "What is the standard trade-in quote validity?",
      "expected": "10 days"
    },
    {
      "case": "c40",
      "question": "What is the standard free battery replacement period?",
      "expected": "6 months"
    }
  ]
}

Diagnosed: the ladder named the index, and the fix cleared 10 of 10 cases that got worse.

The ladder, written as code

A release can change several layers at once, so ladder takes the list of layers that changed. It returns the one to check first: the index, then the tool, then the model, and the prompt last.

The order follows the path of the data. A wrong passage ruins every layer after it, so the index comes first. The prompt is the easiest layer to edit, and that is why teams reach for it first.

Change one layer per run. Then a flip has one cause, and the ladder's answer can be tested.

starter/diagnose.py

1"""EDIT THIS FILE. It has three planted defects.21. ladder(): it names the prompt for every change, the easiest layer to edit.3 The order to check is index, tool, model, prompt: the path of the data.42. flips(): it compares nothing, so no flipped case is ever named.53. regression_cases(): it keeps nothing, so a fixed fault can return unseen.67Six of the ten tests fail at the start. The fixture, the stale index, the8empty-flips and the prompt ladder tests already pass.910Run `./workshop check diagnose` until it is green, then11`./workshop watch diagnose`.12"""131415def ladder(change: list[str]) -> str:16 """Return the layer to check first for a release that changed `change`.17 `change` lists the layers that changed: index, tool, model or prompt.18 """19 return "prompt"202122def flips(before: dict[str, bool], after: dict[str, bool]) -> list[str]:23 """Return the ids of the cases whose result changed, in sorted order.24 `before` and `after` map a case id to whether it was answered right.25 """26 return []272829def regression_cases(30 flipped: list[str], rows: list[dict[str, str]]31) -> list[dict[str, str]]:32 """Return one regression case for each flipped id, in the order given.33 A case is a dict with the row's `id`, `question` and `expect`.34 """35 return []

Where should the team look first?

Answers worsen after a release, and nobody has compared the failing cases with the last good run. The prompt is the layer a developer can edit in a minute.

Predict: Answers worsen after a release. Where should the team look first?

AAt what changed in the release.
BAt the prompt, since it is the easiest to change.

Show the answer

A. A diagnosis starts from the change and works down the layers. The prompt is the last layer, not the first.

A prompt rule for a retrieval fault

This is a real run on Claude Haiku 5.5, recorded once and replayed here. The starting ladder names the prompt for every change, so the team adds twelve lines of rules. The run after them is still at 30 of 40, and the same 10 cases are wrong.

The starting flips compares nothing, so the flip list is empty. No one can name the 10 cases, and the rules cannot reach them. Each of those answers is correct for the passage it was given, and the passage is the old page.

Releasea new index, built on Thursday, that also holds retired pages
Baseline run40 of 40 right before the change
Run after the change30 of 40 right, 10 cases got worse
Flip listno case named
Ladderchecks the prompt first, but the index changed
After the fix30 of 40 right, 10 of the cases that got worse still fail
Regression sliceno case kept, so the fault can return unseen

Misdiagnosed: the prompt was blamed for a fault in the index, and 10 of 10 cases that got worse still fail.

Find the layer, then keep the case

ladder names the index, so the team checks the passages the agent was shown. The 10 flipped cases all got the retired page, which the new index also holds and which matches "standard" best. The lab repairs the named layer and gets 40 of 40, while naming the prompt would leave 30. A chunking change is an index change, so fix it and measure recall at 3, the share of questions with the right passage in the top 3.

flips lists the ids whose result changed. A flip can go either way. Judge flipped cases against a limit set before the run. regression_cases keeps each flipped id as its own case, so the same fault cannot return unseen.

The cost is easy to count. If a prompt try takes an hour, a day of them is eight tries that all leave the same 10 cases wrong. One run against the baseline points at the index the first time.

diagnose.pygreen lines are what the fix adds

1"""Ask what changed first, then name the flipped cases: the solution.2ladder() checks the layers from the retrieval side down to the prompt.3flips() lists the cases whose result changed, and regression_cases()4keeps each of them as a case, so the same fault cannot return unseen.5"""67LADDER = ["index", "tool", "model", "prompt"]8910def ladder(change: list[str]) -> str:11 """Return the layer to check first for a release that changed `change`.12 `change` lists the layers that changed: index, tool, model or prompt.13 """14 found = [layer for layer in LADDER if layer in change]15 if not found:16 raise ValueError("nothing changed, so there is no layer to check")1718 return found[0]192021def flips(before: dict[str, bool], after: dict[str, bool]) -> list[str]:22 """Return the ids of the cases whose result changed, in sorted order.23 `before` and `after` map a case id to whether it was answered right.24 """25 return sorted(i for i in before if before[i] != after.get(i))262728def regression_cases(29 flipped: list[str], rows: list[dict[str, str]]30) -> list[dict[str, str]]:31 """Return one regression case for each flipped id, in the order given.32 A case is a dict with the row's `id`, `question` and `expect`.33 """34 by_id = {r["id"]: r for r in rows}3536 return [37 {k: by_id[i][k] for k in ("id", "question", "expect")} for i in flipped38 ]

We doChange one setting at a time and predict.

The same input now ends well

Switch between the two checks with the same index change. The baseline and the changed run score the same in both: 40 and 30. The card marked "This is where the runs split" is the ladder, and only the ladder points at the index.

The data did not change and neither did the agent. The habit did. A team that asks what changed reaches the fault in one run, and a team that starts with the prompt never reaches it.

This part is interactive in the lab: run ./workshop watch diagnose.

Try it: change and check

Change one setting at a time. The index change breaks 10 cases, the trimmed tool 12, and the pasted prompt 8. Every combination is a recorded run, so it works offline.

The model setting flipped no case here: moving the effort from medium to low left all 40 right. A model change is rarely the cause of a fault that follows a release.

To tell the layers apart, compare the passages shown. An index fault shows the wrong page, the retired "14 days" one. A tool fault shows the page cut short, so the reply is "Not specified in passage". A prompt fault shows the right page and a wrong answer, "14 days" instead of "30 days".

This part is interactive in the lab: run ./workshop watch diagnose.

Keep it as a regression case

The run lists the flips, names the layer, and then keeps every flipped case with its question and expected value. After the fix it checks that each kept case is right again.

A guardrail is a check with an address puts a check at each layer, so a fault like this one fails a test on the day it appears. Until then, the flip list and the regression slice are the record of what changed.

starter/diagnose.py

You doFix the starting code, then test yourself.

You do: name the layer and keep the flipped cases

Copy the starting code and open work/diagnose/diagnose.py. ladder() names the prompt for every change, flips() compares nothing and regression_cases() keeps nothing.

Terminal

./workshop start diagnose
./workshop check diagnose
./workshop watch diagnose

The checks want the ladder to check the index before the tool, the model and the prompt. Every case whose result changed is named, and each one is kept as a regression case.

Check your understanding

Answer each question, then read why every option is right or wrong. The wrong options are the mistakes engineers make on the job.

1. Answers from the support agent got worse after a Friday release. Accuracy fell from 40 of 40 to 30 of 40 cases, and the release also rebuilt the retrieval index on Thursday and added four lines to the prompt. The team has one afternoon and a saved baseline run. What should they do first?

ARewrite the four new prompt lines, since the prompt is the layer the team can edit fastest and test today.
BAdd a rule telling the agent never to answer when it is unsure, and run the 40 cases again.
CRe-run the 40 cases with the old index, then compare the passages in the failing cases.
DSwitch to a larger model for the afternoon and see whether accuracy returns to 40 of 40.

Show the answer

Answer: C

A The prompt is the last layer on the ladder. Editing it first can hide an index fault for a day and leave the cause in place.
B A rule about unsure answers does not reach a fault where the agent was given the wrong page. It answers correctly for that page.
C Correct. The index is the first layer that changed, so the team checks it against the baseline and reads the passages that the failing cases were shown.
D A larger model reads the same wrong page and gives the same wrong value. The model is rarely the cause of a fault that follows a release.

2. A developer added "do not make things up" and eleven other lines to the support prompt after answers got worse. The next run still gets 30 of 40 cases right, and the same 10 cases are wrong. Every wrong answer matches the passage the agent was given. What does this result show about the fault?

AThe fault is upstream of the prompt, so check what was retrieved for those 10 cases.
BThe prompt needs more rules, because 10 cases is a small number and the rules so far were too general to help.
CThe agent ignores the prompt, so the next step is a model that follows instructions better.
DThe 40 cases are at fault, so the 10 cases should be removed from the set and the run repeated.

Show the answer

Answer: A

A Correct. The answers follow the passages, so the prompt did its job. The fault is upstream, and the passages for those 10 cases show whether retrieval found the right page.
B More rules add lines and leave the wrong passage where it was. The same 10 cases stay wrong because the prompt never changed what the agent was shown.
C The agent followed the prompt and answered from the passage. Every answer matches its passage, so it did not ignore anything.
D Removing failing cases raises the score and keeps the fault. The 10 cases are the evidence that points at the layer.

3. One Friday release changed the lookup tool, the prompt and the model setting together. Answers are worse, with 28 of 40 cases right against 40 of 40 on the last good run. The team wants to know which change caused it and has the same 40 cases and the old version of each layer. How should they find out?

ARead the diff of all three changes, pick the one that looks riskiest and revert it.
BRevert all three changes, ship the old version and leave the question of the cause open.
CAdd a rule to the prompt for each symptom in the failing cases and run the 40 again.
DApply the changes one at a time to the old version and run the 40 cases after each.

Show the answer

Answer: D

A A guess from a diff is not a measurement. The riskiest-looking change is often not the one that flipped the cases.
B Reverting everything restores the old answers and teaches nothing. The same mix of changes will be shipped again later.
C Rules for symptoms grow the prompt and leave the cause alone. The team would not know which change was at fault.
D Correct. One change per run gives each flip one cause, and each run is compared with the baseline on the same 40 cases.

4. In a postmortem after a Friday incident, the index change flipped 10 of 40 cases and a fix restored all 40 after two days. The team wants the postmortem to stop the same fault from returning unseen. Which two things should the team keep from the diagnosis? Select two.

AThe list of flipped cases, with each question and its expected value.
BA list of the prompt rules that were tried and did not help, with the date of each try.
CA regression case for each flipped case, run with every later release.
DThe name of the developer who built the new index, so the team knows who to ask.

Show the answer

Answer: A, C

A Correct. The flipped cases are the exact evidence of what broke, and each has a question and a value that can be checked again.
B Rules that did not help are a record of what was tried, and they keep no fault from returning. They also add lines to the prompt.
C Correct. A fixed fault kept as a test fails on the day it returns, so the fix does not depend on anyone remembering it.
D A name may help a follow-up conversation, and it checks nothing in the next release.

5. A bad run of 40 cases shows 12 wrong answers that all say the value is not stated. The team has the run, the passage shown for each case and the raw result of the lookup tool, and only an hour before the next release. Which two things should the team read first in this bad run? Select two.

AThe prompt text, line by line, for a rule that might suppress answers.
BThe passage the agent was shown for each of the 12 cases.
CThe accuracy of the same agent on another dataset of questions.
DThe raw result of the lookup tool for one failing case.

Show the answer

Answer: B, D

A The prompt is the last layer to check. A rule that suppresses answers is possible, and it is the most likely reading of the symptom only after the earlier layers are cleared.
B Correct. The passage is the input the agent answered from. A passage without the value explains an answer that says the value is not stated.
C Another dataset shows how the agent does elsewhere and says nothing about these 12 cases. The evidence is in this run.
D Correct. The tool result shows whether the tool handed over the full passage or a cut one, which is a tool fault and not a prompt fault.

6. A trace of a failing case shows the lookup tool returned an empty result for a refund limit question. The agent then answered with a plausible dollar figure that is wrong, and 4 of the 40 cases fail in the same way. The prompt has not changed since the last good run. Where is the fault?

AThe prompt, because it should forbid a guess when the tool returns nothing.
BThe tool, because it handed over nothing and the agent had nothing to read.
CThe model, because it should refuse instead of inventing a number.
DThe dataset, because 4 of 40 is too few failures to draw a conclusion.

Show the answer

Answer: B

A A rule may make the symptom rarer and leaves the empty result where it is. The tool still fails on the next question.
B Correct. The first wrong thing in the trace is the empty result. The agent answered without a passage, and fixing the tool removes the cause.
C A guess after an empty result is the symptom. The earlier layer, the tool, is the first place to check.
D Four failures that share a cause on the same kind of question are a pattern. Comparing with the baseline shows them as flips.

7. After the index was rebuilt, 10 of 40 answers cite a policy page that was retired in March, and each of them gives the old value. The prompt and the model setting did not change. The retired pages were added to the new index by mistake. What should the team check first?

AWhether the prompt tells the agent to prefer the newest page when two pages differ.
BWhether a larger model reads the two pages and picks the right value by itself.
CWhether the new index still filters out retired pages when it ranks the passages.
DWhether the 10 questions are phrased more loosely than the other 30.

Show the answer

Answer: C

A The agent is shown one passage, so it has no second page to prefer. The wrong page is chosen before the prompt is read.
B The agent never sees both pages, so a stronger model cannot choose between them. The fault happens at retrieval.
C Correct. The rebuilt index is the layer that changed. Without a version filter the retired page competes with the current page and can win.
D Wording decides which page wins only because the index has no filter. Checking the filter addresses the cause.

8. A chunking change on the index lowers recall at 3 from 95% to 80% on the 40 questions. The right passage is now missing from the top 3 results for 6 more questions. The prompt is unchanged, and answers for those 6 questions are wrong. What should the team do?

AAdd a prompt rule that says to answer only when the value is in the passage, and run the 40 again.
BRaise the number of passages in the prompt until the answers look right again on the 6 questions.
CMove the six questions to a separate dataset and report recall on the rest of the set.
DFix or revert the chunking, then measure recall at 3 on the same 40 questions.

Show the answer

Answer: D

A The rule would make the agent refuse when the value is missing, and the value is missing because retrieval lost it. The wrong answers become no answers.
B More passages can hide the loss on a few questions and raise cost and noise everywhere. Without a measurement the team cannot tell whether the loss was fixed.
C Moving the six questions away removes the evidence and keeps the fault. The recall figure would rise without any fix.
D Correct. The chunking is the change, so the team fixes it and measures recall again on the same 40 questions to show the loss is gone.

9. A team ships a new index and reads the score a week later. The average looks steady at 38 of 40, and nobody listed which cases changed. Support then reports 5 wrong refund answers that were right before the release. What should the team have done before shipping?

AWaited a week more, so the average would settle into a clearer number.
BList the flipped cases against the baseline run, and keep each fixed one as a case.
CRead the average again with a second reviewer, so a drop would be seen by two people.
DAdded a prompt rule for refund answers, since refunds are where the errors appeared.

Show the answer

Answer: B

A Waiting gives more time for wrong answers and no better view of which cases changed. The average hides cases that flip in both directions.
B Correct. A list of flips against the baseline run names the 5 wrong refund answers before shipping, and each becomes a regression case.
C A second reviewer of the same average sees the same average. The 5 cases are visible only when the two runs are compared case by case.
D A rule aimed at refunds treats one symptom and leaves the index that caused it. The list of flips would have pointed at the cause.

10. A change to the lookup tool raises the average from 34 to 36 of 40 cases. The team wants to keep it. The comparison with the baseline shows that 4 cases that were right before are now wrong, and 6 cases that were wrong are now right. How should the team decide?

AName the 4 flipped cases and decide against the limit written before the run.
BKeep the change, because 6 cases improved and 4 got worse, and the net result is better.
CReject the change, because any case that gets worse is a reason to revert.
DRun the 40 cases three more times and keep the change if the average stays above 34.

Show the answer

Answer: A

A Correct. The 4 flipped cases are a list a person can read, and the limit written before the run says whether they are acceptable.
B A net gain can hide 4 cases that matter more than the 6 that improved. The decision needs the names of the flipped cases.
C A rule that allows no flip would block most useful changes. The limit must be written first, so the decision is not made by reflex.
D More runs of the same average show that it is stable and say nothing about the 4 flipped cases.

Play: predict the output

The run after the release crashed on three cases, so they have no result. This flip list reads only the cases both runs have.

python

def flips(before: dict[str, bool], after: dict[str, bool]) -> list[str]:
    both = before.keys() & after.keys()
    return sorted(i for i in both if before[i] != after[i])

Predict: Before has 40 cases and after has 37. How many of the three crashed cases are in the flip list?

ANone. A case with no result after the release is skipped.
BAll three, because they were right before and have no answer now.

Show the answer

A. A case that did not run is a failure, not a pass. Loop over every case in the baseline, and read a missing result as changed, as the lab's flips() does.

References

Create strong empirical evaluations: the fixed case set you run before and after a change.
Prompt engineering overview: why a prompt edit comes after a test, not before it.
Reduce hallucinations: what a prompt rule can fix, and what it cannot.
Contextual retrieval: how an index change moves recall, the first rung of the ladder.

Conclusions

Ask what changed, then check the layers in order: index, tool, model, prompt.
Change one layer per run, and compare every run with a baseline.
Name the flipped cases, and keep each fixed fault as a regression case.

You can now run a baseline, name the flipped cases and write the ladder that points at the layer to check first.

Baseline

A run of the same cases before the change, kept to compare with

Flip

A case whose result changed between two runs

Ladder

The order in which layers are checked: index, tool, model, prompt

Regression case

A fixed fault kept as a test so it cannot return unseen

Retrieved passage

The text the index hands the agent for a question