Claude Architect WorkshopPDFEPUBlearnwithparam.comkaas.team

Professional, lesson 4

Service levels

A service level row turns a goal into a breach

Lumen Supply tells its board that support is faster and better, and the dashboard shows an average wait of 3 seconds in green. The contract promises 5 seconds, and some customers wait 40. This lesson writes the row that catches it. This is the support desk at Lumen Supply, an online electronics shop. You will measure 2000 tickets, see a green average hide a slow tail, then write the percentile and the row that says what happens on a breach.

I doWatch the failure and its fix.

One table, five promises

A service level row has a threshold, a place where it is measured, and an action. The model answers tickets, the trace measures each one, and the code decides whether a row is breached.

Three things go wrong. The goal is "faster and better", with no number. Latency is reported as an average, which hides the slow tail. The row has no breach action and no spend cap, so a breach changes nothing.

A run is measured, reported, and judged against a row.Trafficsamplesp95()SLA rowa written promisebreach()the action, or nonethe rule is written in sla.pyA run is measured, reported, and judged against a row.Trafficsamplesp95()SLA rowa written promisebreach()the action, or nonethe rule is written in sla.py

A latency table with two numbers

The run measures 2000 tickets and reports the mean and the 95th percentile side by side. The mean is 3.17 seconds. The 95th percentile is 40.12, far over the 5 second promise, so the code flags the row and names its action and owner.

Open a card to see what each step found. The samples are a fixture of 110 slow tickets in 2000, built so the tail is easy to see.

Fifty real Haiku 5.5 calls were also timed. Their mean was 1.44 seconds and their 95th percentile 4.41, with the slowest at 10.7. Real traffic has a tail too.

  1. TrafficFixture: 2000 tickets, latency in seconds
  2. Codemeasure: 2000 samples
    Show the result
    {
  "samples": 2000
}
  3. Codereport: mean 3.167, p95 40.115
    Show the result
    {
  "mean": 3.167,
  "p95": 40.115
}
  4. Codeflag: latency, then page on-call (support lead)
    Show the result
    {
  "row": "latency",
  "action": "page on-call",
  "owner": "support lead"
}

BREACH: p95 40.115 against latency. Mean 3.17, p95 40.12.

The service level row, written as code

A row holds the promise (sla), a tighter internal goal (slo), where it is measured, the action on a breach and the owner. The service level agreement is what the contract promises. The objective is what the team aims for, and it sits inside the promise so a miss is seen early.

The 95th percentile, p95, is the value that 95 of every 100 tickets meet or beat.

starter/sla.py
1"""EDIT THIS FILE. p95() returns the mean, so a slow tail never shows.2The table also has no breach action, no owner and no spend row. Run3`./workshop check value-and-sla` until it is green, then4`./workshop watch value-and-sla`.5"""67import statistics8from dataclasses import dataclass91011@dataclass(frozen=True)12class SlaRow:13 """One promise: how much is allowed, where it is read, and what follows."""1415 name: str16 unit: str17 sla: float18 slo: float1920 measured_at: str21 action: str22 owner: str232425ROWS = [26 SlaRow("latency", "seconds", 5, 5, "dashboard", "", ""),27 SlaRow("wrong answers", "per 100", 10, 10, "dashboard", "", ""),28 SlaRow("failed calls", "per 1000", 5, 5, "dashboard", "", ""),29 SlaRow("escalations", "per 100", 20, 20, "dashboard", "", ""),30]313233def p95(samples: list[float]) -> float:34 """Return the value that 95 percent of the samples are at or under."""35 return statistics.mean(samples)363738def breach(row: SlaRow, value: float) -> str | None:39 """Return the row's action when the value is over its SLA, else None."""40 return row.action if value > row.sla else None

Does a 3 second average meet a 5 second promise?

The dashboard shows a mean of 3 seconds. The contract says tickets are answered within 5. It looks as if the promise is met with 2 seconds to spare.

Predict: A 3 second average latency against a 5 second promise: is it met?

  1. ANot necessarily. The slow tail can still breach it.
  2. BYes. 3 is under 5.
Show the answer

A. An average hides the slow calls. The promise is met only if the agreed percentile is under the threshold.

The average looks fine while tickets wait 40 seconds

With the starting code, p95 returns the mean. On the long-history tickets, a slice of the same 2000, the run reports a mean of 3.19 seconds against a promise of 5, and the result is green.

No row names an action, so even a breach would change nothing. The fixture holds 110 tickets in 2000 that wait about 40 seconds, and the dashboard cannot see them.

  1. TrafficFixture: 400 long-history tickets, latency in seconds
  2. Codemeasure: 400 samples
    Show the result
    {
  "samples": 400
}
  3. Codereport: mean 3.19, p95 3.19
    Show the result
    {
  "mean": 3.19,
  "p95": 3.19
}

GREEN: mean 3.190 against latency. Mean 3.19, p95 3.19.

The percentile breaches, and the row says what to do

p95 sorts the samples. It returns the value that 95 of every 100 samples are at or under.

breach returns the row's action when the value is over the promise. Each row also gains an owner, a place it is measured, and an objective tighter than the promise.

The table gains a row for cost per ticket. At 10,000 tickets a day, this fixture's rate of 110 in 2000 is 550 slow tickets, each one a breach.

How one slice is judged, in order.TrafficCodeTablesampleslook up rowthresholdflag and act
sla.pygreen lines are what the fix adds. file: 79 lines
1"""The SLA table, the percentile and the breach rule."""···1 line not shown3import math···18 lines not shown22 SlaRow(23 "latency",24 "seconds",25 5,26 4,27 "trace, per ticket",28 "page on-call",29 "support lead",30 ),31 SlaRow(32 "wrong answers",33 "per 100",34 10,35 8,36 "weekly review sample",37 "pause the prompt change",38 "quality owner",39 ),40 SlaRow(41 "cost per ticket",42 "dollars",43 0.02,44 0.015,45 "trace, per ticket",46 "switch to the smaller model",47 "finance partner",48 ),49 SlaRow(50 "failed calls",51 "per 1000",52 5,53 3,54 "gateway log",55 "fail over to the backup",56 "platform owner",57 ),58 SlaRow(59 "escalations",60 "per 100",61 20,62 15,63 "ticket system",64 "review the routing rule",65 "support lead",66 ),···4 lines not shown71 """The value that 95 percent of the samples are at or under."""72 ordered = sorted(samples)7374 return ordered[math.ceil(0.95 * len(ordered)) - 1]···3 lines not shown78 """The row's action when the value is over its SLA, else None."""···1 line not shown
We doChange one setting at a time and predict.

The same tickets, two metrics

Switch between the two metrics on the long-history tickets, a slice of the 2000. Its mean is 3.19, a little above the 3.17 of all 2000, and its 95th percentile is 40.11 against 40.12. With the mean, the run is green. The 95th percentile is 40.11, and the run is a breach that names an action.

Nothing about the tickets changed. The only difference is which number is read.

This part is interactive in the lab: run ./workshop watch value-and-sla.

Try it: metric and slice

Change one setting at a time. Pick the metric, then the slice. Every combination is a recorded run, so this works with no internet.

The all-traffic run and the long-history run give almost the same numbers, because the slow tickets are spread evenly. A real slice can breach while the total passes, so keep a row for each slice that matters. Watch the spend slice too: the 95th percentile is 5 cents against a cap of 2 cents.

This part is interactive in the lab: run ./workshop watch value-and-sla.

Spend is a row, not a report

Cost per ticket breaches the same way latency does. The mean is 1 cent, under the cap, and the 95th percentile is 5 cents, over it. The row names its action, switching to the smaller model, and its owner.

The same breach function judges both rows. A spend cap that lives in a monthly report changes nothing. A row with an action does.

app/agent.pyfile: 76 lines
···32 lines not shown33 p95: float = 0.034 value: float = 0.035 count: int = 0363738def log(result: Result, name: str, args: Json, out: Json) -> None:39 """Record one step in the order it happened."""40 result.events.append(41 {"kind": "tool", "name": name, "input": args, "output": out}42 )434445def run(metric: str, kind: str, sla: ModuleType) -> Result:46 """Measure one slice and judge it against the row for that slice."""47 result = Result(metric, kind)48 data = samples(kind)4950 name = ROW_OF.get(kind, "latency")51 found = next((r for r in sla.ROWS if r.name == name), None)52 row = cast("SlaRow | None", found)5354 result.count = len(data)5556 result.mean = statistics.mean(data)57 result.p95 = sla.p95(data)···19 lines not shown
  1. TrafficFixture: 2000 tickets, cost in dollars
  2. Codemeasure: 2000 samples
    Show the result
    {
  "samples": 2000
}
  3. Codereport: mean 0.01, p95 0.05
    Show the result
    {
  "mean": 0.01,
  "p95": 0.05
}
  4. Codeflag: cost per ticket, then switch to the smaller model (finance partner)
    Show the result
    {
  "row": "cost per ticket",
  "action": "switch to the smaller model",
  "owner": "finance partner"
}

BREACH: p95 0.050 against cost per ticket. Mean 0.01, p95 0.05.

You doFix the starting code, then test yourself.

You do: measure the tail and own each row

Copy the starting code and open work/value-and-sla/sla.py. p95() returns the mean, so a slow tail never shows. The rows have no breach action and no owner, and cost per ticket is missing.

Terminal
./workshop start value-and-sla
./workshop check value-and-sla
./workshop watch value-and-sla

The checks want a slow tail to breach while the mean looks fine, and a value inside the SLA to pass. Every row needs an action, an owner, a named place it is read and an SLO tighter than its SLA. Cost per ticket must be a row.

Check your understanding

Answer each question, then read why every option is right or wrong. The wrong options are the mistakes engineers make on the job.

1. A support agent must answer fast, cost little and be right. The contract caps p95 latency at 5 seconds, finance caps spend at 2 cents a ticket, and the quality lead wants 90 percent correct. Design cannot meet all three on the current model, and the launch is in 30 days. What do you do first?

  1. ASet one blended score across the three and aim to keep the total above an agreed line.
  2. BRank the three limits and name the one that decides a tie.
  3. CPick the fastest model on the list, since speed is the easiest of the three to demonstrate.
  4. DReport the average of each metric and call the design balanced once all three look close.
Show the answer

Answer: B

  • A A blended score lets a miss on one limit hide behind a gain on another. Each limit is a separate row.
  • B Correct. A design with three competing limits needs a written ranking, so a tie has one decider.
  • C Speed is one of the three, and choosing it alone breaks the cost and quality limits without saying so.
  • D Averages hide the tail that the contract is written against, and they rank nothing.

2. A design meets a 2 second p95 only with a model that costs 5 cents a ticket, and finance caps spend at 2 cents. The product owner wants both limits kept, and nobody has said which one gives way. The design review is on Friday. What should the design record before it ships?

  1. AThat the faster model is used, because speed is the target that matters most to customers.
  2. BThat spend is a report for month end, so the cap can be looked at after launch.
  3. COne number that covers both limits, so that there is nothing left to trade between them.
  4. DWhich limit gives way, why, and who signed off the trade.
Show the answer

Answer: D

  • A Faster is not a target by itself. The trade still breaks the spend cap without anyone agreeing to it.
  • B Spend is a row with a cap and an action. A report shows a breach only after the money is gone.
  • C One number for two limits hides which one broke, and each has its own owner and action.
  • D Correct. Two limits that cannot both hold need a named trade and a signature, or the breach is a surprise.

3. A dashboard shows a mean latency of 3 seconds over 2000 tickets, and the team calls it green. About 1 ticket in 18 waits 40 seconds. The contract names the 95th percentile and promises 5 seconds at that percentile. Which number should the team report against the contract?

  1. AThe 95th percentile, since the contract names it and it shows what the slow tail waits.
  2. BThe mean, since 3 is well under 5 and most of the tickets are answered quickly.
  3. CThe fastest ticket, since it shows what the system can do when everything goes right.
  4. DThe mean of one busy slice, on the view that every slice shares the same number.
Show the answer

Answer: A

  • A Correct. The percentile the promise names is the number a breach is judged on.
  • B The mean averages 40 second waits away. It stays under 5 while customers break the promise.
  • C The fastest ticket says nothing about the promise. It ignores every customer who waits.
  • D Slices differ, and one slice's mean does not stand for the others.

4. A team is writing the rows of a service level table, and each row must be actionable when a number crosses it. The table is reviewed by finance and by the on-call lead, and the review is written down so that an auditor can read it later. Which parts must every row name? Select two.

  1. AA wish to be faster than last quarter, stated without a number.
  2. BA threshold, and the place where it is measured.
  3. CThe same value for the objective and the agreement, to keep the table short.
  4. DAn action to take on a breach, with an owner.
Show the answer

Answer: B, D

  • A A wish with no number cannot be missed, so it cannot be a row.
  • B Correct. A threshold without a measuring place cannot be checked, and two places give two numbers.
  • C An objective equal to the agreement gives no early warning. It sits tighter, inside the promise.
  • D Correct. A breach with no action and no owner is a chart, and nobody acts on it.

5. A sponsor is told the new support agent is faster and better, and the sponsor will judge the project at a review in 30 days. The team has no baseline yet, and the project plan currently holds only that one sentence about speed and quality. What should replace it?

  1. AA promise that the agent will be faster than a person on the desk.
  2. BAn average wait, measured once on a normal day and reported at the review.
  3. CA measured target with a percentile, a baseline and a date.
  4. DOne satisfaction score that is used for every kind of ticket.
Show the answer

Answer: C

  • A Faster than a person is still a wish. It has no number, so nobody can miss it.
  • B One average measured once hides the tail and gives no baseline to compare with.
  • C Correct. A target with a percentile, a baseline and a date can be missed, so the sponsor can judge it.
  • D One score for every kind of ticket blends slices that differ, and it hides where it fails.

6. A business case must justify a build that costs 40,000 dollars. About 500 slow tickets a day each cost a customer 6 minutes of waiting. Finance will check every number in the case against the trace, and the case goes to the budget meeting on Monday. How should the saving be stated?

  1. AAs an average wait across the whole desk, in place of the slow tail.
  2. BAs 500 tickets times 6 minutes a day, from the trace.
  3. CAs a line in the month-end spend report, to be reviewed once the build is live.
  4. DAs one satisfaction score for the whole desk, with a note on what it covers.
Show the answer

Answer: B

  • A An average spreads the 500 slow tickets over the desk, and the saving disappears in it.
  • B Correct. The saving is 3,000 minutes a day, from numbers that the trace can confirm.
  • C A report after the fact is not a case. The saving is what the build changes, stated before it is built.
  • D A score for the whole desk has no source in the trace, so finance cannot check it.

7. Two stakeholders read different charts of one run. The product lead reads a 4 second line as the promise, and the platform lead reads a 5 second line as the promise. The contract says 5, and the customer review is next week. How do you stop the mix-up?

  1. AUse one line on both charts, so the two charts agree about the promise.
  2. BDraw the average on both charts, so that the lines look close to each other.
  3. CLeave both lines and name an owner for whichever line gets crossed first.
  4. DDefine both. The agreement is 5 seconds, and the objective is 4 and internal.
Show the answer

Answer: D

  • A One line loses the early warning. The objective sits tighter than the promise on purpose.
  • B An average draws neither line. The mix-up is about which line is the promise.
  • C An owner does not say which line is the customer's promise, so the two leads still disagree.
  • D Correct. The agreement is the promise to the customer. The objective is the internal goal inside it.

8. Finance reviews a value case for a support agent that cost 40,000 dollars to build and runs at a few cents a ticket on the main support desk. The review is in 30 days, and the sponsor wants numbers, not adjectives. What must the case hold to be credible? Select two.

  1. AA baseline and a measured target for the metric.
  2. BA claim that the agent will be faster at some point in the year.
  3. CAn objective that matches the promise exactly, so there is no margin to argue about.
  4. DThe spend against the saving, in the same period.
Show the answer

Answer: A, D

  • A Correct. A baseline and a measured target show how far the metric moved.
  • B Faster at some point has no number and no date, so finance has nothing to check.
  • C An objective equal to the promise gives no margin, and it says nothing about value.
  • D Correct. A saving means little without the cost of making it, in the same period.

9. A manager asks the team to make support faster, but no metric exists and no one has timed tickets. The team is about to choose a model and design the flow. The first release is due in 30 days, and the sponsor asks for a status every week. What comes before any design?

  1. AThe fastest model on the list, to get a head start on speed while the rest is agreed.
  2. BAn average wait, measured on a good day and used as the starting figure.
  3. CThe outcome, the baseline and the percentile that will measure it.
  4. DOne speed number for every kind of ticket, so the target is simple to state.
Show the answer

Answer: C

  • A Choosing a model first builds for a goal nobody has written down.
  • B An average on a good day hides the tail and gives a baseline that is too kind.
  • C Correct. A design has no target until the outcome, the baseline and the percentile are named.
  • D Kinds of ticket differ in how long they take, so one number fits none of them well.

10. A support director wants fewer angry customers by next quarter. Tickets are logged with reply time and cost, but nothing records how a customer felt. Last quarter's 2,000 tickets are on file, and the plan is due in two weeks. What is the first step toward a target?

  1. APick a measurable stand-in, such as p95 reply time, and set its baseline from last quarter's tickets.
  2. BSet the target on the mean reply time, since an average is easy to explain.
  3. CMake replies as fast as possible, on the view that quicker means calmer.
  4. DStart building, and name an owner for any complaint spike once one appears.
Show the answer

Answer: A

  • A Correct. An outcome nobody measures needs a stand-in with a threshold and a baseline before design starts.
  • B A mean on reply time hides the slow tail where angry customers wait.
  • C Faster is not a target. Without a number there is no point at which the work is done.
  • D An owner named after the spike is late. The owner and the action belong on the row before launch.

11. A launch metric reports accuracy only, and accuracy rose 4 points after a prompt change. Cost per ticket doubled at the same time, and nobody noticed until month end. The team wants to stop this happening again before the next prompt change next week. What should they add?

  1. AA monthly spend report, read at the end of each month by the finance lead.
  2. BA paired metric, such as cost per ticket, reported beside accuracy.
  3. CAn average of accuracy, reported with more decimals so small moves show up.
  4. DOne score that blends accuracy and cost for every slice of the traffic.
Show the answer

Answer: B

  • A A monthly report finds the change a month late. A row with a cap finds it on the day.
  • B Correct. A paired metric beside the main one catches a gain that is paid for somewhere else.
  • C More decimals on the same metric cannot show a cost that the metric never reads.
  • D A blend hides which side moved, and each metric needs its own threshold and action.

12. The latency row of a service level table has a threshold and a place where it is measured, but no owner. A breach was found in the month-end review, three weeks after it began. The team wants the next one found and acted on early, and the review cadence stays monthly. What changes?

  1. AThe objective is set equal to the agreement, so the table is simpler to read for the whole team.
  2. BA spend row is added with its own cap, and the latency row is left as it is.
  3. CThe target moves from 5 seconds to 3 so the desk has some margin.
  4. DThe row gets an owner and a breach action, so a breach is acted on.
Show the answer

Answer: D

  • A An objective equal to the agreement removes the early warning, so the customer sees the miss first.
  • B A spend row is useful, but the latency row still has no owner for its breach.
  • C A tighter number with no owner is still a number nobody acts on. Faster is not the fix here.
  • D Correct. An owner and an action turn a number crossing a line into a task with a name on it.

13. A weekly report for a support agent shows a mean reply time of 2.8 seconds over 1,500 tickets, against a promise of 5 seconds at the 95th percentile. A customer with a long order history complains of 30 second waits, and the ops lead wants the report changed before the next review in 7 days. Which change makes the report catch that customer's wait?

  1. AReport the mean for each day, so a bad day stands out from the rest of the week.
  2. BReport one 95th percentile across all tickets, since one number is easier to read.
  3. CReport the 95th percentile per slice, with long-history tickets on their own row.
  4. DMake the whole desk answer faster, so that every slice moves under the line together.
Show the answer

Answer: C

  • A A daily mean still averages the slow tickets away, so the 30 second waits stay hidden.
  • B One percentile over every ticket can pass while one slice breaches, because fast slices pull it down.
  • C Correct. The promise names the 95th percentile, and a row of its own shows where the slow tail sits.
  • D Faster is not a measure. Without a per-slice number the report still cannot show which slice waits.

14. A launch review must decide whether a new support agent meets its latency goal before it goes to 3 more teams. A test run of 800 tickets gives a mean of 2.1 seconds, a median of 1.4 seconds and a 95th percentile of 9 seconds. The agreement promises 5 seconds at the 95th percentile, and the review has one hour. What should the review conclude?

  1. AIt passes, since both the mean and the median are well under 5 seconds.
  2. BIt fails, since the 95th percentile the agreement names is 9 seconds.
  3. CIt passes on latency, and the review turns to the spend report for the month.
  4. DIt is fine to launch, and the percentile can go in a report after go-live.
Show the answer

Answer: B

  • A The agreement names the 95th percentile, and the mean and median say nothing about it.
  • B Correct. The number the agreement names is 9 seconds against a promise of 5, so the launch fails.
  • C Spend is its own row. It cannot stand in for a latency row that is over its threshold.
  • D A breach known before launch needs an owner and an action now, not a report later.

Play: predict the output

This row promises seconds, and the gateway logs milliseconds.

python
row = SlaRow(
    "latency", "seconds", 5, 4, "gateway p95", "page on-call", "platform lead"
)
breach(row, p95(gateway_samples))  # p95 is 3,100 ms

Predict: The p95 is 3.1 seconds, inside the promise. What does breach return?

  1. A"page on-call", because 3,100 is more than 5.
  2. BNone, because 3.1 seconds is inside 5.
Show the answer

A. The row names its unit, but nothing converts the value to it. Convert where the value is read, and test the row with a sample in the source's own unit.

References

Conclusions

  • Report the percentile the promise names, not the average.
  • Give every row a threshold, a place it is measured, an action and an owner.
  • Set the objective tighter than the promise, and give spend a row.

You can now turn "faster and better" into a measured row that flags a breach.

Service level agreement

The promise made to the customer, with a threshold

Service level objective

The tighter internal goal, set inside the promise

Percentile

The value that a set share of calls meet or beat; p95 is the value 95% of calls meet or beat

Breach action

What happens, and who acts, when a value is over the promise