Claude Architect WorkshopPDFEPUBlearnwithparam.comkaas.team

Professional, lesson 14

A/B tests and guard metrics

Pre-register the rule; pair every lever with its own guard metric

A Lumen Supply change cuts what the support agent costs, and accuracy falls with it. The team decides afterwards that the loss is fine. You will watch a rule written after the result ship a change that should be held back. Then you will write the rule first and give every lever its own guard metric.

I doWatch the failure and its fix.

A 22% saving and a 2 point loss

Lumen Supply is an online electronics shop. Its agent reads a ticket and picks one action: refund, deny, escalate or ask for the order number.

Two changes are proposed: a cheaper tier and a shorter prompt. Each runs on the same 100 recorded tickets next to the current setup, and the customer sees none of it.

Three things go wrong, in this order. The ship rule is written after the result, so it can always be met. A lever has no guard metric, so a gain in one place hides a loss in another. And the change goes straight to all traffic with no shadow run.

Two variants answer the same tickets in a shadow run. The rule, saved first, decides.Tickets100 recordedCurrent setupthe baselineLevercheaper or shorterShip rulethreshold and guardDecisionship or hold backthe amber box is written before the runTwo variants answer the same tickets in a shadow run. The rule, saved first, decides.Tickets100 recordedCurrent setupthe baselineLevercheaper or shorterShip rulethreshold and guardDecisionship or hold backthe amber box is written before the run

A lever with one guard metric

The cheaper tier sends the same prompt to Claude Haiku 5.5 instead of the larger tier. Both answer 100 of 100 tickets correctly. The cost for 10,000 tickets falls from $3.39 to $0.17.

The rule was saved before the run: cost down at least 10%, accuracy down at most 1 point. One point is one ticket, because a wrong refund costs more than the saving. The saving is 95% and accuracy held, so the lever ships. Open a card to see the two variants in detail.

  1. Ship rulecost down at least 10%, accuracy down at most 1 points
    Show the result
    {
  "metric": "cost",
  "threshold": 0.1,
  "guard": 0.01
}
  2. Current setupthe larger tier, long prompt: 100% correct, $3.39 for 10,000 tickets
    Show the result
    {
  "accuracy": "100 of 100",
  "typical tickets": "100%",
  "edge tickets": "100%",
  "input tokens a ticket": 123,
  "output tokens a ticket (estimated)": 9,
  "missed": []
}
  3. LeverHaiku 5.5, long prompt: 100% correct, $0.17 for 10,000 tickets
    Show the result
    {
  "accuracy": "100 of 100",
  "typical tickets": "100%",
  "edge tickets": "100%",
  "input tokens a ticket": 123,
  "output tokens a ticket (estimated)": 9,
  "missed": []
}
  4. DecisionShipped: saves 95% and accuracy held within the guard

Shipped: the saving is real and accuracy held.

The ship rule, written as code

rule writes down the metric, the threshold and one guard metric before anyone sees a number. The threshold is the smallest saving that counts. Guard metrics are second metrics the lever must not hurt, here accuracy. decide reads both from the saved rule and returns a verdict with its reason.

Cost is tokens times price. The current setup uses about 123 input tokens and 9 output tokens a ticket on Haiku 5.5, at $0.10 and $0.50 per million tokens. Times 10,000 tickets that is about $0.17.

Three levers cut that cost: a cheaper tier, a shorter prompt, and a cached prefix. A cached prefix bills the long policy text at a lower rate when the same text opens every request. This lab runs the first two, and each lever gets its own guard metric.

The larger tier's prices are this lab's figures, so check the pricing page before you use them. Output tokens are estimated from the reply size.

starter/ab.py
1"""EDIT THIS FILE. It has two planted defects.21. decide(): it builds its own bar from the result, so any saving clears it.32. decide(): it never reads the guard, so a lever that loses accuracy ships.45Three of the seven tests fail at the start. The fixture, the saved rule,6the boundary and the shipped lever already pass.78Run `./workshop check ab-and-levers` until it is green, then9`./workshop watch ab-and-levers`.10"""1112from kit.api import Json131415def rule(metric: str, threshold: float, guard: float) -> Json:16 """Write the ship rule before the run.17 `metric` is what the lever improves, `threshold` the smallest saving18 that counts, and `guard` the most accuracy the lever may lose.19 """20 return {"metric": metric, "threshold": threshold, "guard": guard}212223def decide(result: Json) -> Json:24 """Return whether to ship the lever, and why.25 `result` has `rule`, `baseline` and `lever`; the last two hold `cost`26 per ticket and `accuracy` as a fraction.27 """28 saving = 1 - result["lever"]["cost"] / result["baseline"]["cost"]29 late = rule("cost", saving / 2, 1.0)30 shipped = saving >= late["threshold"]31 words = "clears" if shipped else "misses"3233 return {34 "ship": shipped,35 "reason": f"the saving of {saving:.0%} {words} the bar",36 }

Can a team decide what counts as good after it sees the result?

A shorter prompt costs 22% less and gets 2 more tickets wrong out of 100. Nobody wrote down how much loss was acceptable. The team looks at the numbers and calls 2 points fine.

Predict: Can a team decide what counts as good after it sees the result?

  1. AYes. Then it has the context.
  2. BNo. The rule must exist first.
Show the answer

B. A rule written after the result can always be met. Writing it first removes that room.

A rule written after the result

This is a real run on Claude Haiku 5.5, recorded once and replayed here. With the full prompt it answers 100 of 100 tickets correctly. With the shorter prompt it answers 98, and the cost for 10,000 tickets falls from $0.17 to $0.13.

Both misses are tickets with no order number that got a refund instead of a request for one. The starting decide builds its bar from the result, so any saving clears it. It never reads accuracy, so the shorter prompt ships. Over 10,000 tickets that is about 200 refunds paid with no order number.

  1. Ship ruleNone written before the run. The bar is built from the result, so any saving clears it.
  2. Current setupHaiku 5.5, long prompt: 100% correct, $0.17 for 10,000 tickets
    Show the result
    {
  "accuracy": "100 of 100",
  "typical tickets": "100%",
  "edge tickets": "100%",
  "input tokens a ticket": 123,
  "output tokens a ticket (estimated)": 9,
  "missed": []
}
  3. LeverHaiku 5.5, short prompt: 98% correct, $0.13 for 10,000 tickets
    Show the result
    {
  "accuracy": "98 of 100",
  "typical tickets": "95%",
  "edge tickets": "100%",
  "input tokens a ticket": 86,
  "output tokens a ticket (estimated)": 9,
  "missed": [
    {
      "ticket": "Hello, my docking station arrived with a cracked case. I got it 20 days ago. Please refund me.",
      "expected": "ask_info",
      "agent chose": "refund"
    },
    {
      "ticket": "Hello Lumen team, my USB cable arrived broken in the box. Delivery was 15 days ago. Please refund me.",
      "expected": "ask_info",
      "agent chose": "refund"
    }
  ]
}
  4. DecisionShipped: the saving of 22% clears the bar

Shipped, and accuracy fell 2 points. No guard was read.

Pre-register, guard, then decide

rule is written and saved before the run. decide reads the saved threshold and guard, and returns not-ship with the reason when its guard metric is breached. The same run now reads "guard breached: accuracy fell 2.0 points, the limit is 1.0".

Every lever needs its own guard metric. A cheaper tier needs an accuracy limit. A change that lifts accuracy needs a latency limit and a cost limit, because it can double the wait.

Fix the sample size and the rule before the run, and read the result once. A team that checks the gap each morning and ships on the first good day is peeking. Every extra look is another chance to catch a lead that is only noise.

Messages between the tickets, Claude, the rule and the decision.TicketsClaudeRuleDecision100 tickets, twicetwo sets of decisionscost and accuracyship or hold back
ab.pygreen lines are what the fix adds
1"""Pre-register the rule, then decide with the guard: the solution.2rule() is written before the run and saved in the result.3decide() reads the threshold and the guard from it, never from the result.4"""56from kit.api import Json78POINTS = 10091011def rule(metric: str, threshold: float, guard: float) -> Json:12 """Write the ship rule before the run.13 `metric` is what the lever improves, `threshold` the smallest saving14 that counts, and `guard` the most accuracy the lever may lose.15 """16 return {"metric": metric, "threshold": threshold, "guard": guard}171819def decide(result: Json) -> Json:20 """Return whether to ship the lever, and why.21 `result` has `rule`, `baseline` and `lever`; the last two hold `cost`22 per ticket and `accuracy` as a fraction.23 """24 saved = result["rule"]25 base, lever = result["baseline"], result["lever"]2627 saving = round(1 - lever["cost"] / base["cost"], 4)28 drop = round(base["accuracy"] - lever["accuracy"], 4)29 points = round(POINTS * drop, 1)3031 if drop > saved["guard"]:32 limit = round(POINTS * saved["guard"], 1)33 reason = (34 f"guard breached: accuracy fell {points} points, "35 f"the limit is {limit}"36 )37 return {"ship": False, "reason": reason}3839 if saving < saved["threshold"]:40 reason = (41 f"the saving of {saving:.0%} is below {saved['threshold']:.0%}"42 )43 return {"ship": False, "reason": reason}4445 reason = f"saves {saving:.0%} and accuracy held within the guard"4647 return {"ship": True, "reason": reason}
We doChange one setting at a time and predict.

The same input now ends well

Switch between the two rules with the same shorter prompt. The two variants score the same in both runs: 100% and 98%. The card marked "This is where the runs split" is the decision, and only the pre-registered rule holds the change back.

The data did not change. The rule did. A saving of $0.04 for every 10,000 tickets is real, and so are the 200 refunds paid with no order number. The rule says which matters more before anyone is tempted.

This part is interactive in the lab: run ./workshop watch ab-and-levers.

Try it: rule and lever

Change one setting at a time, and only the shorter prompt splits the two rules. The cheaper tier ships under both, because Claude Haiku 5.5 matched the larger tier on all 100 tickets. The rule written after the result shipped it without reading accuracy, so a loss would have shipped too. Every combination is recorded and works offline.

Read a small difference with care. On 100 tickets one ticket is one point, so two variants that differ by less than a point differ by less than one ticket. That is inside the noise, so add data before you name a winner.

A test of significance tells you whether a gap is real. The rule tells you whether the gap is enough.

This part is interactive in the lab: run ./workshop watch ab-and-levers.

Run it in shadow first

In production, a shadow run answers live tickets with both variants and sends the customer only the current one. Guard metrics are then measured on real load. This lab uses a recorded replay of 100 tickets as its stand-in. Ask what changed first asks what changed when a case flips.

Read a change per step, not as one mean. A tier change can move the mean from 96% to 95% while the review step falls from 94% to 81%. That is a breach on review, and the smaller tier ships for classify only. Read p95 and failing slices too.

When the SLA lets either latency or cost win, state the trade-off and let the owner choose. A change 40% faster that doubles the bill passes both limits, and only the owner knows what a second is worth. Record the baseline first, then show value as money per ticket against the target.

app/report.pyfile: 95 lines
···75 lines not shown76 """Write the rule, run both variants as a shadow run, then decide."""77 rule = ab.rule(METRIC, THRESHOLD, GUARD)7879 before, after = LEVERS[lever]80 measured = {81 "baseline": measure(rows, llm, before),82 "lever": measure(rows, llm, after),83 }84 shown = {85 k: {"cost": m["cost"], "accuracy": m["accuracy"]}86 for k, m in measured.items()87 }88 decision = ab.decide({"rule": rule, **shown})8990 events: list[Json] = [91 {"kind": "call", "name": "run_variant"},92 {"kind": "call", "name": "decide"},93 ]9495 return Result(lever, rule, measured, decision, events, len(rows))
You doFix the starting code, then test yourself.

You do: decide by the rule you wrote first

Copy the starting code and open work/ab-and-levers/ab.py. decide() builds its bar from the result, so any saving clears it, and it never reads its guard metric.

Terminal
./workshop start ab-and-levers
./workshop check ab-and-levers
./workshop watch ab-and-levers

The checks want the limits read from the saved rule. A lever that breaches its guard metric or saves too little is not shipped, and a drop equal to the limit still ships.

Check your understanding

Answer each question, then read why every option is right or wrong. The wrong options are the mistakes engineers make on the job.

1. A change cuts the support agent's cost per ticket by 22% and accuracy falls 3 points. The team looks at both numbers after the run and decides that a 3 point drop is acceptable. Next quarter they ask why the decision process let a loss through. What should the team do differently before the next experiment?

  1. AAsk a second reviewer to approve the 3 point drop once the result arrives, and log the approval.
  2. BWrite the ship threshold and the accuracy guard down and save them before the run.
  3. CLower the saving needed to ship, so that more changes pass the review.
  4. DReport cost and accuracy on one chart, so the trade-off is easier to see for the whole team.
Show the answer

Answer: B

  • A A second reviewer who sees the result still chooses the rule after it. Any result can be defended once it is known.
  • B Correct. A threshold and guard metrics saved before the run cannot be bent to fit the result. The same drop is held back or shipped by a rule nobody tuned.
  • C A lower bar makes it easier for a lever to ship. The problem was that the bar moved after the result, not its height.
  • D A chart shows both numbers and still leaves the decision to the room. The limit must exist before the chart does.

2. A team runs a shorter-prompt test on live traffic and checks the gap every morning. On day 6 the new prompt is ahead by 2 points, so they stop and ship it. The sample grew each day and nobody wrote down when the test would end. What would have made this decision sound?

  1. AChecking the gap twice a day, so a real lead is noticed sooner and the test can stop.
  2. BRunning the test for a fixed week and shipping on the first day that the gap exceeds 1 point.
  3. CKeeping the test running while the gap stays under 3 points, then shipping when it is wider.
  4. DFixing the sample size and the rule first, and reading the result once at the end.
Show the answer

Answer: D

  • A More looks give more chances to catch a lead that is pure noise. The team would stop even sooner on a false win.
  • B Shipping on the first good day is still stopping when it looks good. The gap on one day can be noise, and the rule must say when the decision is read.
  • C The team would keep choosing the stopping point from the running result, which is a rule chosen after the fact.
  • D Correct. A sample size and a rule fixed first mean the result is read once, so it cannot be stopped at a lucky moment.

3. Two prompt variants each answer the same 100 tickets. Variant A gets 91.0 tickets right on average and variant B gets 91.4. The sponsor wants to ship B today because it is ahead, and the pre-registered rule asks for a gain of at least 2 points. What should the team say?

  1. AThe 0.4 point gap is inside the noise and below the rule, so more data is needed.
  2. BShip B, because any gain counts when the sponsor has asked for one and B is ahead.
  3. CShip B and watch its accuracy for a month, since a test this small is too small to hurt.
  4. DRe-run the test on the first 50 tickets, where B looks stronger, and report that result.
Show the answer

Answer: A

  • A Correct. On 100 tickets one ticket is one point, so 0.4 points is less than one ticket. It is noise and it misses the 2 point rule.
  • B A gain that the sample cannot show is not a gain. The rule asked for 2 points and the gap is 0.4.
  • C Shipping first and watching later skips the check the rule exists for, and puts the change in front of customers.
  • D Choosing the half of the tickets where B looks best is choosing the data after the result.

4. A reviewer reads an experiment plan for a change to the support agent. The plan names the variants and the 100 tickets and says a meeting will decide after the result, and the reviewer wants it pre-registered. Which two parts must the plan add before the run? Select two.

  1. AThe threshold, such as a saving of at least 10%.
  2. BA list of reasons the team may use to explain a disappointing result after the run.
  3. CA second metric the lever must not hurt, with its limit, such as accuracy down at most 2 points.
  4. DThe name of the person who will make the final call on the day of the meeting.
Show the answer

Answer: A, C

  • A Correct. The threshold is the smallest gain that counts, and it is written before anyone sees a number.
  • B Reasons prepared in advance to explain a bad result are a way to write the rule afterwards.
  • C Correct. That guard metric is the loss the lever may not cause, with its limit, so a saving cannot hide a harm.
  • D A named decider still decides by judgement after the result. The rule has to decide, with no room to move.

5. A shorter prompt cuts the support agent's input by 18% of its tokens. The pre-registered rule asks for a saving of at least 10% with accuracy down at most 2 points. On the 100 recorded tickets, accuracy is flat at 95 of 100, and the customer sees none of the test. What should the team do?

  1. AHold it back, because a prompt change needs a larger saving than a tier change does.
  2. BHold it back until the prompt is tested again on a different set of tickets.
  3. CShip it, because the saving clears the threshold and the accuracy guard holds.
  4. DShip it only after the shorter prompt scores above the current one.
Show the answer

Answer: C

  • A The rule sets one threshold for every lever. Asking for more on a prompt change is a rule nobody wrote down.
  • B The shadow run on the recorded tickets is the test the rule asked for. A second set is a new condition added after the result.
  • C Correct. The saving is 18% against a threshold of 10%, and accuracy did not fall, so both parts of the rule are met.
  • D That guard metric asks that accuracy not fall by more than the limit. It is a limit on loss, and a gain is not a second goal.

6. A cheaper tier saves 30% of the monthly bill, and the threshold was 10%. Accuracy on refund tickets falls 4 points, and the pre-registered guard allows 1 point. The sponsor says the saving is large and the loss is small, and the change would reach all traffic next week. What does the rule say?

  1. AShip it, because the saving is three times the threshold.
  2. BDo not ship it, because that guard metric is breached.
  3. CShip it for half the traffic, so the loss is cut in half.
  4. DShip it and ask the sponsor to sign off on the 4 point loss.
Show the answer

Answer: B

  • A A saving well over the threshold does not cancel a breached guard metric. Guard metrics are separate conditions.
  • B Correct. That guard metric allowed 1 point and the loss is 4, so the rule returns not-ship however large the saving.
  • C Splitting the traffic halves the harm and still ships a change the rule refuses. It is a compromise chosen after the result.
  • D A sign-off after the result moves the limit to fit the result, which is the failure its guard metric exists to stop.

7. A team plans a change that raises the agent's accuracy by 2 points by adding a second model call to review each answer. In a trial, p95 latency doubles and exceeds the support SLA, and the bill grows too. Which two guards should be paired with this lever in the ship rule? Select two.

  1. AA latency guard, such as p95 under the SLA.
  2. BA second accuracy target, set a little higher than the first.
  3. CA cost guard, such as cost per ticket under a set limit.
  4. DOne guard metric on the number of tickets in the test.
Show the answer

Answer: A, C

  • A Correct. A second call adds wait, and a latency guard is what catches a gain in accuracy bought with a slow reply.
  • B One guard metric protects what the lever may harm. A second accuracy target is another goal for the same metric.
  • C Correct. The review call also costs more per ticket, so a cost guard stops a gain that the bill cannot carry.
  • D The size of the test belongs in the plan. It does not protect latency or cost, which is where this lever can harm.

8. A cost review asks for ways to lower what the support agent costs per ticket without changing what it is asked to do. Each lever will get its own guard metric and a shadow run before it ships. Which two levers lower cost and belong on the list? Select two.

  1. AA cached prefix, so the long policy text is billed at a lower rate.
  2. BDropping the accuracy guard, so more changes clear the rule and the review goes faster.
  3. CA smaller tier on the steps where it passes that guard metric.
  4. DShipping each change to all traffic at once, with no shadow run, to get the saving sooner.
Show the answer

Answer: A, C

  • A Correct. A cached prefix cuts what a repeated policy costs, and it still needs its own guard metric and a shadow run.
  • B Removing that guard metric makes the rule easier to pass without lowering any cost. It hides a loss rather than a cost.
  • C Correct. A smaller tier saves the most where it holds, so it is tried step by step against that guard metric.
  • D Sending a change to all traffic first skips the shadow run. That is a risk, and it lowers no cost.

9. The agent has a classify step and a review step. A smaller tier is tried on the classify step only, where accuracy holds at 97% and cost falls 60%. The review step is left on the larger tier, because it was never tried on the smaller one and a wrong review costs the shop a refund. Where should the smaller tier go?

  1. AMove both steps to the smaller tier, since it is cheaper on one of them.
  2. BKeep both steps on the larger tier until the smaller tier wins on both.
  3. CMove the review step, where a mistake costs more, to the smaller tier first.
  4. DShip the smaller tier for the classify step only.
Show the answer

Answer: D

  • A The test showed the smaller tier holds on classify only. Nothing was measured for review, so moving it is a guess.
  • B A lever can ship for the step where it holds. Waiting for a win on both steps gives up a saving that is proven.
  • C The review step is the one where an error costs more, so it needs proof first and is not the place to start.
  • D Correct. The smaller tier held on classify and cut its cost 60%, so it ships there and the review step stays put.

10. A tier change is judged on the mean accuracy across all steps, which falls from 96% to 95%, inside the 2 point guard. A reviewer finds that the review step fell from 94% to 81% while the easy classify step rose. The release is due on Friday. How should the team read this result?

  1. AAccept the mean, since it is inside that guard metric and the easy step improved.
  2. BRaise the limit to 3 points so that the mean is comfortably inside it, and ship.
  3. CAverage the review step with the other steps twice as heavily, and read that number.
  4. DRead the change for each step, and treat the review step's drop as a breach.
Show the answer

Answer: D

  • A The mean is the one number that hides the loss. Any guard metric read on it only checks the average.
  • B Moving the limit after seeing the result is writing the rule afterwards, and the review step still loses 13 points.
  • C A different weighting of one mean is still one number. It does not show which step got worse.
  • D Correct. A mean hides a step that got worse. The 13 point drop on review is a breach when the change is read per step.

11. A change speeds up one step of the support agent by 30% on average. Accuracy on the long-ticket slice, the slowest 5% of requests, falls from 92% to 80%, and p95 latency is unchanged. The mean accuracy looks flat. How should the team read this result?

  1. AAs a pass, because the mean is flat and the average is 30% faster.
  2. BAs noise, because only 5% of requests are in the slice.
  3. CAs a breach on the slice, judged per slice and not on the mean.
  4. DAs a pass, once the average latency is checked again in a week.
Show the answer

Answer: C

  • A A flat mean can hide a failing slice. The speed-up is an average, and the tail got worse where customers wait.
  • B A 12 point drop on a slice is not noise because the slice is small. Customers in that slice meet the failures.
  • C Correct. The tail and the slice carry the harm, so its guard metric is read there and the change is held back.
  • D Checking the same average again tells the team nothing new. The loss is in the slice, which the mean cannot show.

12. A change improves latency by 40% and doubles the cost per ticket. The SLA allows either a slower reply or a higher bill, and both options are within the pre-registered guards. The team cannot say which is better without knowing what the business values. What should they do?

  1. AShip whichever variant the engineers prefer, since both pass the limits.
  2. BState the trade-off in numbers and let the owner of the budget choose.
  3. CSplit the difference by shipping to half of the traffic.
  4. DAdd a third metric that combines latency and cost, and ship on its score.
Show the answer

Answer: B

  • A Engineers choosing by preference is a decision made after the result. The owner sets what speed or cost is worth.
  • B Correct. When both pass, the choice is a business trade-off, so the team states it with numbers and the owner picks.
  • C Half the traffic gets half of each outcome and answers nothing. It avoids the choice rather than making it.
  • D A combined score hides the trade-off inside a weighting nobody agreed. The weights are the decision.

13. A sponsor asked the team for value, and a change saves cost on the support agent. The team has the tokens used per ticket and the prices of both tiers. The sponsor reads money, not tokens, and wants the saving compared with a target. How should the team report the result?

  1. AAs the percentage drop in tokens, which is the number the model reports.
  2. BAs money saved per ticket, set against the target the sponsor gave.
  3. CAs the accuracy of the new variant, which shows the agent still works.
  4. DAs the number of tickets in the test, which shows how strong the result is.
Show the answer

Answer: B

  • A A token count leaves the sponsor to convert it. Two tiers with the same tokens cost very different amounts.
  • B Correct. Tokens times price gives money per ticket, and the target turns it into a yes or no for the sponsor.
  • C Accuracy is protected by its own guard metric and it matters, but it is not the value the sponsor asked for.
  • D The size of the test says how much to trust the number. It is not the value of the change.

14. A project reports that a change raised accuracy by 4 points and claims it as value. The baseline was never recorded, because the old version was replaced before it was measured. A sponsor asks how anyone can know the gain is real. What should the team do on the next project?

  1. AReport the gain with a larger font and a clear chart, since the number is clear.
  2. BEstimate the old accuracy from memory and show it as the baseline in the report.
  3. CRun the new version for longer, to make the 4 points more certain for the sponsor.
  4. DRecord the baseline before any change, then report the gain against it.
Show the answer

Answer: D

  • A How the number is shown does not make it checkable. Without a baseline there is nothing to compare it with.
  • B A baseline written after the change is a number chosen after the result, and nobody can test it.
  • C A longer run makes the new number steadier and still has no old number to compare with.
  • D Correct. A gain is a difference, so the old figure is measured before the change, and only then is the claim checkable.

Play: predict the output

The rule was written before the run, with an accuracy guard metric of 2 points.

python
saved = rule("cost", threshold=0.10, guard=2)
result = {
    "rule": saved,
    "baseline": {"cost": 0.010, "accuracy": 0.96},
    "lever": {"cost": 0.007, "accuracy": 0.90},
}
decide(result)

Predict: Accuracy falls 6 points and cost falls 30 percent. Does the lever ship?

  1. AYes. The drop is 0.06, and 0.06 is under 2.
  2. BNo. A 6 point drop breaks a 2 point guard metric.
Show the answer

A. The limit is in points and the accuracy is a fraction, so the check compares two units. Write every limit in the unit the result uses, and have rule() refuse a limit above 1.

References

Conclusions

  • Write the ship rule before the run, so no result can be made to pass.
  • Pair every lever with one guard metric, and hold back a lever that breaches it.
  • Run a change in shadow on recorded tickets before it reaches traffic.

You can now write a ship rule, set guard metrics for each lever and read a shadow run.

Pre-registered rule

A threshold and guard metrics saved before the result is seen

Guard metric

A second metric a lever must not hurt

Lever

A change that moves cost or speed, such as a cheaper tier or a shorter prompt

Shadow run

A variant answers real tickets without reaching the customer

Baseline

The measured current setup, recorded before any claim of gain

Noise

A difference too small for the sample to show as real