Professional, lesson 7
Prompt techniques
Add a technique only for the failure it fixes
A Lumen Supply team adds chain-of-thought to a prompt whose real fault is its output format. The replies get longer and cost nearly five times as much, and none of them parses. This lesson matches each of four failures to the one technique that fixes it, and shows what the wrong ones cost. The Lumen support assistant makes four kinds of mistake. Its triage reply does not parse, its refund sums are wrong, its tone breaks the house rules, and it sends a ticket to the wrong queue. You will run four techniques on each, read the score and the cost, and write the table that picks.
One prompt, four different failures
A technique is one block added to a prompt: a step-by-step instruction, worked examples, an output schema, or a rubric. The model follows the prompt, and your code scores the reply, so a fix is real only when the score moves.
Three things go wrong, in this order. Chain-of-thought is used for a format fault, so cost rises and the format still breaks. Examples are missing where the format is the issue. And no table maps a failure to its technique, so each fix is a guess.
A schema fixes a format fault
Three tickets are triaged with a schema in the prompt. All three replies parse, with the exact keys and allowed values, and the three replies use 135 tokens in all. Ten thousand tickets like these cost $3.22 on Haiku.
Open a card to see the prompt and the reply for one ticket, with its tokens in and out. The score is code and not opinion: the reply must parse as one JSON object with the right keys.
- PromptSchema added to a format prompt
Show the result
{ "prompt": "Triage the ticket. Return the triage as JSON.\n\nReply with one raw JSON object, with no code fence and no other text. Keys: queue (billing, shipping, returns, fraud or tech), priority (low, normal or high) and summary (one sentence).\n\nTicket: The keyboard I ordered shows delivered, but nothing came." } - ScoreTicket 1: ok
Show the result
{ "ticket": "The keyboard I ordered shows delivered, but nothing came.", "reply": "{\"queue\": \"shipping\", \"priority\": \"normal\", \"summary\": \"Customer reports a keyboard order marked as delivered but not received, indicating a possible lost or misdelivered package that needs carrier investigation.\"}", "tokens in": 143, "tokens out": 72 } - ScoreTicket 2: ok
Show the result
{ "ticket": "Can you resend my invoice for order 5521? It is not urgent.", "reply": "{\"queue\": \"billing\", \"priority\": \"low\", \"summary\": \"Customer requests a resend of the invoice for order 5521; not urgent.\"}", "tokens in": 144, "tokens out": 48 } - ScoreTicket 3: ok
Show the result
{ "ticket": "My webcam turns green and then the picture freezes.", "reply": "{\"queue\": \"tech\", \"priority\": \"normal\", \"summary\": \"Customer reports that their webcam displays a green screen and then the picture freezes, indicating a possible hardware or driver fault.\"}", "tokens in": 138, "tokens out": 65 } - pickpick chose Schema for format
FIXED: Schema on format, 3 of 3 tickets pass, 185 tokens out, $0.45 for 10,000 tickets.
The failure-to-technique table, as code
pick takes a failure and returns the technique that fixes it. A schema fixes format, step-by-step reasoning fixes logic, a rubric fixes tone, and worked examples fix routing. build_prompt adds that one block to the task and nothing else.
Thinking before the answer is a lever for reasoning faults. It costs output tokens on every call, so it is not free, and it is the wrong tool for a fault that is not about reasoning.
Does chain-of-thought fix a broken output format?
Chain-of-thought is the best known technique. It seems safe to add whenever a reply is wrong.
Predict: Does chain-of-thought fix a broken output format?
- ANo. A schema or examples fix a format.
- BYes. Thinking first helps everything.
Show the answer
A. Reasoning helps logic faults. A format fault needs a specification the model can copy.
Chain-of-thought on a format fault
This is a real Haiku 5.5 run, recorded once and replayed here. The starting pick returns chain-of-thought for every failure. On the format fault, none of the three replies parses, because the model writes its working before the JSON.
The three replies use 902 tokens, against 135 for the schema. Ten thousand tickets cost $15.77, nearly five times the $3.22 a schema costs, and the format is still broken.
- PromptChain-of-thought added to a format prompt
Show the result
{ "prompt": "Triage the ticket. Return the triage as JSON.\n\nWork through the problem step by step first. Then give your final reply in the form asked above.\n\nTicket: The keyboard I ordered shows delivered, but nothing came." } - ScoreTicket 1: not JSON
Show the result
{ "ticket": "The keyboard I ordered shows delivered, but nothing came.", "reply": "Reasoning\n\n1. Issue: The tracking page shows the keyboard as delivered, but the customer says they never received it. This is a \"delivered but not received\" claim, which is usually handled as a possible carrier error or theft.\n2. Urgency: One lost item, no safety or payment-fraud signals. Medium priority.\n3. Missing information: No order number, delivery address, or delivery date. We need these to check the carrier scan and address before escalating.\n4. Routing: Logistics/shipping claims, with a first step of verifying the delivery scan and asking the customer to check with neighbors and their building's mail area.\n\njson\n{\n \"ticket_type\": \"shipping_issue\",\n \"subcategory\": \"delivered_not_received\",\n \"priority\": \"medium\",\n \"summary\": \"Customer reports a keyboard marked as delivered was not received.\",\n \"missing_information\": [\"order_number\", \"shipping_address\", \"delivery_date_shown_on_tracking\"],\n \"recommended_actions\": [\n \"Request order number and confirm the shipping address on file\",\n \"Pull carrier tracking and proof-of-delivery (signature, photo, or drop-off location)\",\n \"If the address or drop-off location is wrong, flag for carrier claim\",\n \"If the scan is valid, ask customer to check with neighbors, building management, or mail room for 24 to 48 ", "tokens in": 103, "tokens out": 600 } - ScoreTicket 2: not JSON
Show the result
{ "ticket": "Can you resend my invoice for order 5521? It is not urgent.", "reply": "Triage summary\n\n- Request type: Invoice resend\n- Order: 5521\n- Urgency: Low (customer stated it is not urgent)\n- Action needed: Locate order 5521, confirm the requester is the account holder or the order's contact, then resend the invoice to the email on file.\n- Check before sending: Invoices contain order and payment details, so verify the requester's identity against the order before sending anything.\n\njson\n{\n \"ticket_type\": \"invoice_resend\",\n \"order_id\": \"5521\",\n \"priority\": \"low\",\n \"customer_urgency_stated\": \"not_urgent\",\n \"requested_action\": \"resend_invoice\",\n \"required_steps\": [\n \"Locate order 5521 in the order system\",\n \"Verify requester matches the account holder or order contact\",\n \"Resend invoice to the email address on file for the order\",\n \"Reply to customer confirming the invoice was sent\"\n ],\n \"escalation_needed\": false,\n \"notes\": \"Standard low-priority request. Do not send to an address that differs from the order record without verification.\"\n}\n", "tokens in": 104, "tokens out": 512 } - ScoreTicket 3: not JSON
Show the result
{ "ticket": "My webcam turns green and then the picture freezes.", "reply": "Reasoning\n\n- The symptoms are a color cast (green tint) followed by a frozen frame. Together they suggest a video pipeline problem rather than a dead camera.\n- Common causes: a USB bandwidth or power problem (especially with hubs or extension cables), a driver or firmware fault, or a sensor or hardware fault if the failure is persistent across machines.\n- The customer has not said whether this is a new or old unit, the connection type, the OS, or whether it happens in every app. Those details would narrow the cause, so the ticket should ask for them.\n- Severity is moderate. The device is partly usable, and a freeze may indicate a defect, so it should be checked under warranty or return policy.\n\nTriage\n\njson\n{\n \"ticket_summary\": \"Webcam displays a green tint, then the video feed freezes.\",\n \"category\": \"Hardware - Webcam / Video Output\",\n \"likely_causes\": [\n \"USB bandwidth or power insufficient (hub, extension cable, or shared port)\",\n \"Outdated or corrupted camera driver or firmware\",\n \"Application or codec conflict\",\n \"Hardware fault in the sensor or camera board (possible defect)\"\n ],\n \"priority\": \"Medium\",\n ", "tokens in": 98, "tokens out": 600 } - pickpick chose Chain-of-thought for format
STILL_BROKEN: Chain-of-thought on format, 0 of 3 tickets pass, 1712 tokens out, $2.95 for 10,000 tickets.
Match the technique to the failure
pick reads a table from each failure to one technique. It raises an error for a failure it does not know. build_prompt is the same in both files, so only the choice of technique changed.
A task with several steps is split into separate prompts, one technique each. A checked contract sits between them, so a failure belongs to one step. An order that must hold is enforced in code, and the prompt only asks for each step.
The cost of a wrong choice is paid on every call. Chain-of-thought adds about 250 output tokens per ticket for a fault it cannot fix, and 10,000 tickets a month make that $12.55 more than the schema.
The same input now ends well
Switch between the two techniques on the same format fault. The task, the tickets and the model are the same in both runs. Choose Schema. The note "This is where the runs split." appears and Ticket 1 is outlined: its reply parses with the schema and does not with chain-of-thought.
Only the technique changed. The schema run scores 3 of 3, and the chain-of-thought run scores 0 of 3. The last card names what pick returns, and it is the schema on both tabs, because the table is the same.
This part is interactive in the lab: run ./workshop watch prompt-techniques.
Try it: technique and failure
Change one setting at a time. All sixteen combinations are recorded runs, so this works with no internet.
Only five of the sixteen fix their failure. A schema fixes format, a rubric fixes tone, and examples fix routing.
Reasoning is fixed by chain-of-thought, and in these recordings by a rubric too: its replies show the model writing out its working before the answer line. Worked examples got 2 of 3 on format, where one reply used a queue name that does not exist.
This part is interactive in the lab: run ./workshop watch prompt-techniques.
Reasoning is the one failure thinking fixes
Here the same two techniques swap places. Chain-of-thought scores 3 of 3 on the refund sums and costs $14.41 for 10,000 tickets. The schema scores 0 of 3 at $1.91, because it asks for the answer line and nothing before it.
That is the whole lesson in one pair of runs. The cheap technique is wrong for sums, and the costly one is wrong for format. A metric is a dataset plus a rule measures a fix on a dataset and not on one run.
This part is interactive in the lab: run ./workshop watch prompt-techniques.
You do: match the technique to the failure
Copy the starting code and open work/prompt-techniques/technique.py. pick() returns chain-of-thought for every failure, and there is no table from a failure to its technique.
./workshop start prompt-techniques
./workshop check prompt-techniques
./workshop watch prompt-techniquesThe checks want a schema for a format fault, chain-of-thought for reasoning, a rubric for tone and examples for routing. A failure with no technique raises an error.
Check your understanding
Answer each question, then read why every option is right or wrong. The wrong options are the mistakes engineers make on the job.
1. A triage step must return JSON that a parser reads, and about 1 call in 6 returns text the parser rejects. The team added chain-of-thought last week, replies grew 40 percent in tokens, and the failure rate did not change. The cost cap is fixed, and the parser cannot change. What should the team do?
- AAsk for even more careful step-by-step reasoning, since the model is not thinking enough yet.
- BAdd three more sentences to the instruction that tell the model to be careful about the JSON.
- CRaise max_tokens so the long replies are no longer cut off before the JSON closes.
- DAdd a schema with the exact keys and values, and drop the chain-of-thought.
Show the answer
Answer: D
- A More thinking adds tokens and cost, and the reply still has to be parsed. The fault is the form, not the logic.
- B Longer instructions are still prose. They do not give the model a form to copy, so the rate stays near 1 in 6.
- C A cut-off reply is one cause, but the replies grew because of the added reasoning, and the cap is fixed.
- D Correct. A format fault needs a form the model can copy. Without the reasoning, the replies shrink and the cost falls.
2. A refund step returns the answer in the right form every time. But 1 call in 4 gets the arithmetic wrong on orders with a discount and a restocking fee. The cost cap is tight, and the other four steps of the pipeline already pass their scores. Which change fits?
- AAdd chain-of-thought to every step of the pipeline, so none of them can fail later on.
- BAdd stepwise reasoning to the refund step only, then re-score it on the same orders.
- CAdd a schema for the answer line, since a wrong number is a problem with the form.
- DAdd two worked examples of a refund that show the final amounts and no working.
Show the answer
Answer: B
- A The other steps already pass. Reasoning tokens on them are cost with no score change.
- B Correct. Logic faults need working before the answer, and only the failing step pays for it.
- C The form is already right. A schema cannot make the arithmetic correct.
- D Examples with bare amounts show the form, not how to get the number, so the sums still go wrong.
3. A prompt review finds that a team added a new technique to a support prompt last sprint. The review must decide whether the technique was chosen for a named failure or only because it is popular. It has the golden-set scores and the cost per call from before and after. Select two signs that it was added without a matching failure.
- AThe failing cases went from 4 of 6 to 6 of 6 after the change, on the failure it named.
- BCost per call rose and the score on the golden set did not change.
- CThe reply got longer, and the review reads that as the prompt being more thorough.
- DNobody wrote down which failure the technique was meant to fix.
Show the answer
Answer: B, D
- A This is the opposite sign. A named failure and a score that moves are what a matching technique has.
- B Correct. A technique that adds tokens and moves no score is paying for nothing.
- C Length is not quality. Longer replies cost more and say nothing about whether the fault is fixed.
- D Correct. Without a named failure there is nothing to match, so the choice is a guess.
4. A parser reads each reply as JSON and rejects about 1 in 6. One teammate wants chain-of-thought added and another wants a schema, and replies must parse on the first try while the token cost per call does not grow. Select two reasons that favor the schema over chain-of-thought.
- AThe schema gives the reply a form to copy, with named keys and allowed values.
- BA schema makes the model think harder before it answers, so the JSON comes out cleaner.
- CChain-of-thought puts working before the JSON, and that text costs output tokens on every call.
- DA schema works only when worked examples are added to it, so the two have to travel together.
Show the answer
Answer: A, C
- A Correct. A schema is a form the reply can copy, and it adds few tokens.
- B A schema adds no thinking. It fixes the form, which is a different fault from the logic.
- C Correct. Reasoning before the reply puts prose ahead of the JSON, which breaks the parse and adds tokens.
- D A schema is one block that works alone. In the recorded runs it scored 3 of 3 with no examples.
5. A team splits a refund task into steps by writing one prompt with a numbered list of six steps. The steps depend on each other, a late step fails about 1 call in 5, and nobody can tell which step produced the bad output. Each step needs its own check. What is the better design?
- AMake the list longer, with more detail for each of the six steps in the same prompt.
- BAsk the model to repeat each step number before it answers, so that the order is visible.
- CSeparate steps, each with its own prompt and a contract for its input and output.
- DRun the same prompt three times and take the most common reply as the answer.
Show the answer
Answer: C
- A More detail in one prompt makes the single call harder to check, and a failure is still not tied to a step.
- B Repeated numbers show an order but do not check any step's output.
- C Correct. Each step gets the technique its failure needs, and its output is checked before the next step runs.
- D Voting costs three times as much and hides which step is wrong.
6. A numbered list in a prompt says to check the order, then the refund limit, then send the reply. In production the model sometimes sends the reply before checking the limit, 1 call in 30. The order of the steps is a hard requirement, and a wrong send cannot be undone. What fixes this?
- ACode enforces the order, and the prompt only asks for each step.
- BWrite the list in capital letters so that the order is stressed to the model.
- CAdd chain-of-thought so the model thinks about the order of the steps first.
- DAdd two worked examples that follow the list in the right order every time.
Show the answer
Answer: A
- A Correct. A rule that must hold is enforced in code, so the send cannot run before the check passes.
- B Emphasis is still a request. A prompt cannot guarantee that a step ran first.
- C Thinking about the order makes the model more likely to follow it, but not certain, and it adds cost.
- D Examples raise the odds and still leave 1 call in 30 that does not follow them.
Play: predict the output
A step that returns JSON also gets the arithmetic wrong, so a teammate adds both techniques.
system = f"{task['ask']}\n\n{task['schema']}\n\n{STEP_BY_STEP}"
reply = json.loads(response.content[0].text)
Predict: What does json.loads read?
- AThe worked steps first, then the JSON, so it raises an error.
- BOnly the JSON, since the prompt asks for the final reply in the form asked.
Show the answer
A. Thinking written in the reply lands in the text the parser reads. Use extended thinking, which returns its reasoning in its own block, or ask for the answer inside tags and parse only those.
References
- Prompt engineering overview: test first, then pick the technique that fits the failure.
- Prompting best practices: examples, formats, rubrics and thinking in one guide.
- Extended thinking: reasoning in its own block, apart from the reply.
- Structured outputs: a schema the reply must fit, so a format fault cannot ship.
Conclusions
- Name the failure first, then add the one technique that fixes it.
- A schema fixes a format, steps fix reasoning, a rubric fixes tone and worked examples fix routing.
- Read the score and the cost together: a technique that adds tokens and moves no score is a bad bet.
You can now map a failure to its technique, build the prompt for it, and prove with a score that the fix worked.
Technique
One block added to a prompt, such as steps, examples, a schema or a rubric
Chain-of-thought
An instruction to work through the problem before the final answer
Schema
The exact form of the reply: its keys and allowed values
Worked examples
Sample inputs with the reply you want, shown in the prompt
Rubric
A checklist of house rules the reply must meet
Failure
The way a reply goes wrong, named before a fix is chosen