Model tiering

One pipeline, three steps, three tiers

A tier is a model size: a small one, a middle one and the largest. A bigger tier costs more and takes longer. It does not always answer better, so the right tier depends on the step.

Each step has a golden set of thirty items with known answers, a quality bar and a latency budget. The sweep runs the step on each tier and scores it. The registry holds the bar and the budget, and pick chooses the tier.

Three things go wrong, in this order. One model id is hard-coded in every step, so cost and speed follow the biggest model. No step has a latency budget, so nothing says when a tier is too slow. And the sweep is read as one number for the whole set, so the trade-off inside a step is never seen.

A sweep table for one step

The classify step labels thirty tickets. All three tiers get all thirty right. The small tier takes 3.0 seconds and costs $1.19 for 10,000 tickets. The middle tier takes 2.9 seconds and costs $2.38, and the largest takes 5.9 seconds and costs $4.76.

Open a card to see the calls behind a cell: each keeps its tokens in and out, and its seconds. Cost is those tokens times the tier's price per million tokens, so the small tier's 1,118 in and 493 out come to $1.19 for 10,000 tickets. The prices are this lab's figures from recording day, so check the pricing page.

Each cell is one recorded run of the whole golden set. Its time stands in for the p95 you would read on live traffic, so treat a small gap as noise.

The pick rule, written as code

pick takes a step and the scores of each tier, and returns the lowest tier that meets the step's quality bar and fits its latency budget. If no tier passes, it returns nothing, so a failing step is seen and not hidden.

registry holds the bar and the budget for each step, and MODELS is the only place a tier becomes a model id. A tier is a cost and speed setting. It is not a guarantee of quality, so the bar decides what is good enough.

Does the largest model win every step?

The largest model is the most capable one in the family. It seems the safe choice for every step of the pipeline.

The largest model on every step

This is a real run on Claude Haiku 5.5 and two larger tiers, recorded once and replayed here. The largest tier gets all thirty classify tickets right in 5.9 seconds for $4.76. The small tier gets the same thirty right in 3.0 seconds for $1.19.

The starting pick returns the largest tier whatever the scores, and no step has a budget. Across the three steps that is $32.00 for 10,000 tickets. Nothing in the code says that classify is over time, because nothing gives it a limit.

A registry and a pick per step

registry now gives every step a bar and a latency budget. Classify needs 90% inside 4 seconds. Draft needs every item right inside 12 seconds, because the customer reads it. Review needs 95% inside 10. pick walks the tiers from the smallest and stops at the first that passes both.

The same sweep now chooses the small tier for classify, the middle tier for draft and the middle tier for review. Across the three steps the bill is $15.48 for 10,000 tickets, against $32.00. The id lives in one file, so a new model is one line in MODELS and a new sweep.

Cost is tokens times price. The draft step writes the most, about 2,100 tokens for thirty replies on the middle tier, so it is the largest line in the bill. A saving on a step is worth most where the step writes the most.

The same input now ends well

Switch between the two registries with the same step. The three tiers score the same in both runs. The card marked "This is where the runs split" is the registry's choice: the largest tier without the fix, the small tier with it.

The data did not change. The registry did. With a budget of 4 seconds, the largest tier is too slow for classify, and with a bar of 90% the small tier is good enough.

Try it: tier and step

Change one setting at a time. Every combination is a recorded run, so this works with no internet.

Draft needs the middle tier. The small tier gets 29 of 30 replies right, and the bar is every one. It said an SSD return window was open when its last day had passed. Review needs the middle tier too, because the small tier finds 27 of 30 defects against a bar of 95%.

In this recording no step needs the largest tier. It passes where the middle tier passes, and it is slower and dearer. A harder review set could change that, so the sweep runs again when the work changes.

The sweep as a table

In the starting registry every budget is None. The fix sets 4, 12 and 10 seconds.

Read the table across a row, never down a column. The small tier is the cheapest on every step, and it passes only on classify. The middle tier passes on all three and is chosen for two. The largest passes on all three too, and is over budget on every one.

Quality, time and cost are three columns, and no one decides. Set the budget on p95, since a few slow calls vanish in a mean: 1.6 seconds can sit beside a p95 of 3.4. Extended thinking adds time and is not a tier.

You do: pick the lowest tier that passes

Copy the starting code and open work/model-tiering/tiers.py. pick() returns the largest tier whatever the scores say, and registry() gives no step a latency budget.

./workshop start model-tiering
./workshop check model-tiering
./workshop watch model-tiering

The checks want a budget on every step and the lowest tier that meets both the bar and the budget. When no tier passes, pick() returns None.

Check your understanding

Answer each question, then read why every option is right or wrong. The wrong options are the mistakes engineers make on the job.

Play: predict the output

This pick reads the tiers in the order the sweep file lists them.

def pick(step: str, scores: Json) -> str | None:
    rule = registry()[step]
    for tier, got in scores.items():
        if got["quality"] >= rule["bar"] and got["seconds"] <= rule["budget"]:
            return tier
    return None

References

• Models overview: the tiers, their context windows and their prices.
• Choosing a model: start small and move up only when a test fails.
• Extended thinking: when more thinking helps before a larger tier does.
• Create strong empirical evaluations: the golden set every tier is swept on.

Conclusions

• Choose the tier for each step, from a sweep of tier by step on a golden set.
• Give every step a quality bar and a latency budget, and pick the lowest tier that meets both.
• Read quality, time and cost together, and keep the model id in one place.

You can now sweep a pipeline over tiers, write a registry with a bar and a budget for each step, and pick the lowest passing tier.