Claude Architect WorkshopPDFEPUBlearnwithparam.comkaas.team

Professional, lesson 10

The retrieval pipeline

Chunk by document shape, retrieve by query shape

The Lumen policy assistant answers from 40 pages. It cuts every page into 500-character pieces, ignores the version and the access list, and finds order numbers with vector search. A larger model will not fix that. This is the assistant at Lumen Supply, an online electronics shop. You will watch three kinds of question go wrong at the index, then fix the index in one file. The model only ever reads what retrieval hands it.

I doWatch the failure and its fix.

One assistant, three kinds of question

The index holds 40 pages: 8 policies in 3 versions each, 6 staff-only pages and 10 order pages. A set of 60 labelled queries says which page should come back for each. The model answers from the passages and cites them.

Three things go wrong, in this order. A cut every 500 characters splits a table, so the answer loses its context. With no version or access filter, a retired page or a private page comes back. And an order number goes through vector search, where every five-digit number looks alike.

Retrieval decides what Claude may read.Questionand who asksFilterversion and accessExact lookuporder idsKeyword + vectorthe restClaudeanswers and citesthe model only sees what retrieval returnsRetrieval decides what Claude may read.Questionand who asksFilterversion and accessExact lookuporder idsKeyword + vectorthe restClaudeanswers and citesthe model only sees what retrieval returns

A cited answer from the current policy

A reader asks for the refund window in the refund policy. Retrieval returns three passages, all from the current version, and the answer says 30 days and cites Refund policy, v3.

Open the index card to read the passages that were sent. Open the Claude card to read the answer. The citation names the title and the version, so a reader can check it against the page.

  1. ReaderWhat is the refund window in the refund policy?
    Show the result
    What is the refund window in the refund policy?
  2. Index3 passages: Refund policy, v3 (Limits), Refund policy, v3 (Overview), Refund policy, v3 (Exceptions)
    Show the result
    1. ## Limits
| Item | Limit |
|---|---|
| Refund window | 30 days |
| Refund cap | $500 |

2. ## Overview
This page sets out the rules that support staff and customers follow for this topic. It applies to every order placed on the Lumen Supply website and in its shops. Read the limits below before you promise anything to a customer, because the numbers change between versions and an older page may still be in a saved link or a printed copy. Check the page number before you quote it, and again when a customer pushes back.

3. ## Exceptions
Orders for custom builds and opened software are handled by the specialist team. Ask a lead when a case is not covered here.
  3. ClaudeThe refund window is 30 days (Refund policy, v3).
    Show the result
    The refund window is 30 days (Refund policy, v3).

The answer cites the current page.

Chunk and search, written as code

Two functions hold the design. The file shown is the starting one. A page has id, title, text, topic, version and access. chunk should split at the headings and keep version (as doc_version), access and topic on every piece.

search should route an order id to an exact lookup and drop what the caller may not read. Then it should blend keyword and vector scores.

Retrieval is measured on its own before any answer is judged. The test asks whether the right page is among the top 3 for each of the 60 queries. When answers drop after a model upgrade and a re-index in the same week, read that number first: if it fell, the index changed what the model saw.

starter/retrieve.py
1"""EDIT THIS FILE. Retrieval has three planted defects.21. chunk(): it cuts every 500 characters, so a table is split in half.32. search(): it never checks the version or who is asking.4 A document has id, title, text, topic, version and access; the caller5 has a role. Keep topic, access and doc_version (the version) on a chunk.6 The newest version is per topic.73. search(): an order number goes through vector search, so it ties.89Run `./workshop check rag-pipeline` until it is green, then10`./workshop watch rag-pipeline`.11"""1213from app.data import DOCS14from app.index import Scorer15from kit.api import Json1617SIZE = 500181920def chunk(doc: Json) -> list[Json]:21 """Split one document into pieces."""22 text = doc["text"]2324 return [25 {"doc": doc["id"], "title": doc["title"], "text": text[i : i + SIZE]}26 for i in range(0, len(text), SIZE)27 ]282930def search(query: str, caller: Json, k: int = 3) -> list[Json]:31 """The k best chunks for a query, by vector similarity alone."""32 chunks = [c for d in DOCS for c in chunk(d)]33 scores = Scorer(chunks).vector(query)34 best = sorted(range(len(chunks)), key=lambda i: -scores[i])3536 return [chunks[i] for i in best[:k]]

Does a larger model fix a wrong retrieval result?

An assistant returns the wrong order page for an exact id. The team raises the answer step to a larger model tier and leaves the index as it is.

Predict: Does the larger model fix the wrong result?

  1. ANo, the fault is in what was retrieved.
  2. BYes, it reasons better about what it is given.
Show the answer

A. A model cannot use a passage that was never retrieved. It reasons better about what it is given, and it was given the wrong pages. Fix the index before the model.

500-character chunks and vectors only

This is a real Claude Haiku 5.5 run, recorded once and replayed here. The reader asks where order O-48214 is. Vector search returned orders O-48216, O-48218 and O-48211.

The model read them, saw the wrong numbers and said it could not help. It did the right thing with bad input. The page that answers the question was never in the prompt.

The refund question fails another way. Two retired versions and a staff page come back, the current version does not, and the model says it cannot help. The staff-only question fails on this index too, and the last step shows it.

  1. ReaderWhere is order O-48214?
    Show the result
    Where is order O-48214?
  2. Index3 passages: Order O-48216, Order O-48218, Order O-48211
    Show the result
    1. # Order O-48216

Item: laptop stand. Status: returned. Carrier: Northline.


2. # Order O-48218

Item: USB hub. Status: delivered. Carrier: Northline.


3. # Order O-48211

Item: USB keyboard. Status: delivered. Carrier: Royal Post.
  3. ClaudeI cannot help with that. Order O-48214 isn't in the passages I have. The passages cover orders O-48216, O-48218, and…
    Show the result
    I cannot help with that. Order O-48214 isn't in the passages I have. The passages cover orders O-48216, O-48218, and O-48211.

Retrieval returned the wrong pages for this question.

  1. ReaderWhat is the refund window in the refund policy?
    Show the result
    What is the refund window in the refund policy?
  2. Index3 passages: Refund override codes, Refund policy, Refund policy
    Show the result
    1. # Refund override codes

Rule
Override code RFD-7714 lets a lead approve a refund past the cap.


2. # Refund policy

Overview
This page sets out the rules that support staff and customers follow for this topic. It applies to every order placed on the Lumen Supply website and in its shops. Read the limits below before you promise anything to a customer, because the numbers change between versions and an older page may still be in a saved link or a printed copy. Check the page number before you quote it, and again when a customer pushes back.

Limits
| Item | Limit |
|---|---|
| Refund win

3. # Refund policy

Overview
This page sets out the rules that support staff and customers follow for this topic. It applies to every order placed on the Lumen Supply website and in its shops. Read the limits below before you promise anything to a customer, because the numbers change between versions and an older page may still be in a saved link or a printed copy. Check the page number before you quote it, and again when a customer pushes back.

Limits
| Item | Limit |
|---|---|
| Refund win
  3. ClaudeThe Refund policy passages I have are cut off at the start of the limits table, before the refund window is stated, so…
    Show the result
    The Refund policy passages I have are cut off at the start of the limits table, before the refund window is stated, so I can't tell you the window from them. Passages [2] and [3] don't show a version number either. Please check the full Refund policy page for the current refund window.

Retrieval returned the wrong pages for this question.

Shape the chunks, filter, then blend

chunk splits at the headings, so the limits table stays whole. Each chunk keeps doc_version and access, which is what the filters read.

search sends an order id to an exact lookup, and that finds O-48214 alone. For everything else it drops the pages the caller may not read and the retired versions. Then it blends a keyword score and a vector score and keeps only the strong matches.

Keywords catch exact codes such as RFD-7714 that vectors blur. A real embedding also catches paraphrases that keywords miss. This lab's vector is a bag of words, so search keeps a chunk only when it shares at least one keyword with the question. Overlapping the 500-character pieces does not help either, because a label can still part from its value.

The index fix is written once and runs on every question. A hosted vector database is warranted only when measured recall at 3 falls short at your size; a small index that meets the goal is enough.

The question, the passages returned and the answer.ReaderIndexClaudequestion1 passageanswer
retrieve.pygreen lines are what the fix adds. file: 66 lines
1"""Retrieval shaped to the documents and to the query."""···1 line not shown3import re···5 lines not shown9ORDER = re.compile(r"O-(\d{5})")10MIN_SCORE = 0.3···3 lines not shown14 """One chunk per heading, each keeping its version and access list."""15 parts = re.split(r"(?m)^(?=## )", doc["text"])16 parts = [p for p in parts if p.strip() and not p.startswith("# ")]17 parts = parts or [doc["text"]]···2 lines not shown20 {21 "doc": doc["id"],22 "title": doc["title"],23 "heading": part.split("\n")[0].lstrip("# "),24 "doc_version": doc["version"],25 "access": doc["access"],26 "topic": doc["topic"],27 "text": part.strip(),28 }29 for part in parts···4 lines not shown34 """Exact lookup for an order id; otherwise filter, then blend scores."""···1 line not shown36 seen = {"public", "staff"} if caller["role"] == "staff" else {"public"}···1 line not shown38 match = ORDER.search(query)39 if match:40 wanted = f"order-{match.group(1)}"41 return [42 c for c in chunks if c["doc"] == wanted and c["access"] in seen43 ][:k]4445 newest: dict[str, int] = {}46 for c in chunks:47 newest[c["topic"]] = max(newest.get(c["topic"], 0), c["doc_version"])4849 scorer = Scorer(chunks)50 key, vec = scorer.keyword(query), scorer.vector(query)51 blend = [52 (0.5 * a + 0.5 * b, i)53 for i, (a, b) in enumerate(zip(key, vec, strict=True))54 ]5556 ranked = sorted(blend, key=lambda s: -s[0])57 kept = [58 chunks[i]59 for score, i in ranked60 if score >= MIN_SCORE61 and key[i] > 062 and chunks[i]["access"] in seen63 and chunks[i]["doc_version"] == newest[chunks[i]["topic"]]64 ]6566 return kept[:k]
We doChange one setting at a time and predict.

The same input now ends well

Use the switch to run the same question on the 500-character index and then on the shaped one. On the second tab the note "This is where the runs split" appears and the Index card is outlined, because that is where the passages differ.

The 500-character index returned 3 wrong orders. The shaped index returned 1 page, the right one, and the answer was out for delivery.

This part is interactive in the lab: run ./workshop watch rag-pipeline.

Try it: index and question

Change one setting at a time. Pick a question, then switch between the two indexes. Every combination is a recorded run, so this works with no internet.

With the shaped index, the staff-only question returns no passage for an outside caller. The model is told so, and it says it cannot help. The access filter ran before the model saw a thing.

This part is interactive in the lab: run ./workshop watch rag-pipeline.

Cite the source

The prompt asks the model to cite the title and version it used. The passages arrive numbered and named, so the citation comes from what retrieval kept.

A citation is only as good as the filter behind it. On the 500-character index an outside caller asks for the fraud extension. The Fraud review playbook comes back, and the model cites it and gives extension 4471. Ask what changed first asks what changed when an answer flips between two runs.

app/prompt.py
1"""The answer prompt: use the passages, cite them, or say you cannot help.23Retrieval decides what the model may see. The prompt only asks it to cite.4"""56SYSTEM = (7 "You answer questions for Lumen Supply, an online electronics shop. "8 "Answer only from the numbered passages you are given. "9 "Cite the passage title and version you used, like (Refund policy, v3). "10 "If no passage answers the question, say you cannot help. "11 "Be brief."12)13EMPTY = "No page this caller may read matches the question."
  1. ReaderWhich extension do I call to escalate a flagged refund to fraud-ops?
    Show the result
    Which extension do I call to escalate a flagged refund to fraud-ops?
  2. Index3 passages: Fraud review playbook, Refund override codes, Refund policy
    Show the result
    1. # Fraud review playbook

Rule
Escalate any flagged refund over $500 to fraud-ops, extension 4471.


2. # Refund override codes

Rule
Override code RFD-7714 lets a lead approve a refund past the cap.


3. # Refund policy

Overview
This page sets out the rules that support staff and customers follow for this topic. It applies to every order placed on the Lumen Supply website and in its shops. Read the limits below before you promise anything to a customer, because the numbers change between versions and an older page may still be in a saved link or a printed copy. Check the page number before you quote it, and again when a customer pushes back.

Limits
| Item | Limit |
|---|---|
| Refund win
  3. ClaudeCall extension 4471 to escalate a flagged refund over $500 to fraud-ops (Fraud review playbook). The passage doesn't…
    Show the result
    Call extension 4471 to escalate a flagged refund over $500 to fraud-ops (Fraud review playbook). The passage doesn't list a version number, so I can't cite one.

Retrieval returned the wrong pages for this question.

You doFix the starting code, then test yourself.

You do: fix the chunks and the filters

Copy the starting code and open work/rag-pipeline/retrieve.py. chunk() cuts every 500 characters, search() ignores the version and the caller's role, and an order number goes through vector search.

Terminal
./workshop start rag-pipeline
./workshop check rag-pipeline
./workshop watch rag-pipeline

The checks keep a table in one chunk, with its version and access on it. Only the current version is returned, and a staff page is hidden from an outsider. Recall at three is measured over sixty queries.

Check your understanding

Answer each question, then read why every option is right or wrong. The wrong options are the mistakes engineers make on the job.

1. A policy assistant indexes 24 policy pages, and each policy exists in 3 versions. A customer was told the refund window is 14 days, which is the text of the retired version 1, while the current page says 30 days. The team wants the answer to come from the current page only, with no change to the prompt. What should retrieval do?

  1. AAsk the model in the prompt to prefer whichever page shows the latest date in its own text.
  2. BRe-embed every page with a larger embedding model so the versions separate better.
  3. CKeep only the newest page by its file name in the answer text, and cite that file.
  4. DKeep doc_version on each chunk and return only the newest version.
Show the answer

Answer: D

  • A The model cannot tell which page is current, and a date in the text is not a version. Retrieval has to decide.
  • B A better embedding ranks the three versions as near twins, because they read almost the same. The version is a fact about the page, not about its meaning.
  • C A name in the answer text is read after the model has already used the page. The filter has to run before it.
  • D Correct. The version travels on each chunk, so search can drop the retired ones before the model reads anything.

2. A refund page is split into 500-character pieces, and its limits table starts at character 461. The row that holds the 30 day window is cut in two, so the label lands apart from the value. Answers about limits are wrong in 9 of 20 test questions. Which change fixes the cause?

  1. ARaise the piece size to 800 characters and test again on the same twenty questions.
  2. BSplit each page at its headings, so the limits table stays whole.
  3. COverlap the pieces by 100 characters so that both halves appear together in a result.
  4. DAsk the model to rebuild the table from the two halves it is given in the prompt.
Show the answer

Answer: B

  • A A different size moves the cut to a different row. Some other page is now cut in some other place.
  • B Correct. The page has a shape, and a heading is where its meaning breaks. A chunk per heading keeps the table whole.
  • C An overlap repeats part of the text but can still leave a label apart from its value, and it costs extra tokens.
  • D The model can only work with the halves it is given, and it may fill the gap with a guess.

3. A customer asked which extension reaches fraud operations, and the assistant returned it from a staff-only page. The system prompt already says to share nothing marked staff. There are 6 staff pages among 40, and the team wants a guarantee that an outside caller never reads one. What provides it?

  1. AAdd the word staff to a list of banned words that the reply is scanned for.
  2. BTell the model twice in the prompt that the staff pages are private and must stay hidden.
  3. CFilter the chunks by the caller's access before the model sees any.
  4. DAsk the model to reply only with the public facts from the page it is given.
Show the answer

Answer: C

  • A A word list on the reply runs after the model has read the page, and the extension number is not the word staff.
  • B The page is already in the model's context. A request not to share it is a request, and the model can lose it.
  • C Correct. A chunk the caller may not read is dropped before the prompt is built, so the model has nothing to leak.
  • D The model would have to decide what is private. The decision belongs to the access list, in code.

4. An audit of a retrieval index with 40 documents finds that a search result carries only the page title and its text. The team must add fields so that a search can drop retired pages and private pages before the model reads them. Select two fields to keep on every chunk.

  1. AHow often the page was opened last month, so that popular pages win.
  2. BThe document's version number.
  3. CThe chunk's position within its page, counted from the top.
  4. DThe access list of the document.
Show the answer

Answer: B, D

  • A Popularity says nothing about which page is current or who may read it. A retired page can be the most opened.
  • B Correct. With the version on the chunk, a filter can keep the newest page and drop the retired ones.
  • C A position says where the text sat in its page. It says nothing about which page is current or who may read it.
  • D Correct. With the access list on the chunk, a filter can drop a private page for an outside caller.

5. A customer asks where order O-48214 is. Vector search returns order pages O-48216, O-48218 and O-48211, because every five-digit number looks alike to it, and the assistant says it cannot find the order. The team is considering a larger model for the answer step. What fixes the fault?

  1. ADetect the order pattern and look the page up by its exact id.
  2. BSend the top 10 pages instead of the top 3 so the right one is likely among them.
  3. CUpgrade the answer step to a larger model that reads the passages more carefully.
  4. DRe-embed the order pages with more vector dimensions so that the ids separate.
Show the answer

Answer: A

  • A Correct. An identifier is an exact string, so an exact lookup finds the page that the vectors could not rank first.
  • B Ten near matches still may not hold the right page, and the model has more noise to read.
  • C A larger model reasons better about what it is given, and it was given the wrong order pages. The fault is upstream.
  • D More dimensions still count a digit string by its shape, so neighbouring ids stay close and the same pages win.

6. In a test of 60 labelled queries, keyword search finds the right page for 38 and misses paraphrases. Vector search finds 41 and misses exact codes such as RFD-7714, and neither passes 80 percent. The team wants one route for the ordinary questions. Which design fits?

  1. ARun keyword search first and use vectors only when it finds nothing at all.
  2. BBlend the keyword score and the vector score and rank by the sum.
  3. CTrain a new embedding model on the 60 queries and use it alone for every question.
  4. DLet the model choose between the two result lists on each question it answers.
Show the answer

Answer: B

  • A A keyword hit that is wrong blocks the vector route, so the paraphrase case is still lost.
  • B Correct. A page that both routes like rises above one that only one route likes, and the blend covers both misses.
  • C Training on the 60 test queries proves nothing, because the score is measured on the same data.
  • D The model would need both lists in its prompt, and it cannot tell which list holds the current page.

7. A team proposes a hosted vector database for 3,000 support pages, and the quote adds 120 ms to each search and a new service to operate. A keyword index and a small in-process vector index already reach 95 percent recall at 3 on the 60 labelled queries. What is the sound decision?

  1. ABuy the database now, because a corpus only grows and moving later costs more.
  2. BDrop the keyword index, because vectors cover the same ground at a lower cost.
  3. CMove to the database and keep the old index as a second copy for safety.
  4. DKeep the small index, and measure again if recall falls.
Show the answer

Answer: D

  • A A plan for growth is not a measured fault. Recall is 95 percent today, and the service adds cost and a failure point.
  • B Vectors miss exact codes and ids, which is the reason the keyword route was added.
  • C Two copies need two syncs, and either can go stale, which is a new fault for no measured gain.
  • D Correct. The measured recall meets the goal at this size, so the simpler index wins until a number says otherwise.

8. A quality review of the policy assistant asks for checks that judge retrieval on its own, before anyone reads an answer. The index holds 40 documents and the team has 60 labelled queries with the page each should return. Select two checks that need no model call and judge only what retrieval returns.

  1. ARecall at 3 over the 60 labelled queries.
  2. BThe average length of the model's replies across the same sixty questions.
  3. CA check that the top results are the current version.
  4. DA count of how many customers liked the tone of the replies this week.
Show the answer

Answer: A, C

  • A Correct. Recall at 3 says whether the right page is among the top three, and it needs no model call.
  • B The length of a reply says nothing about whether the right page was found.
  • C Correct. A result that is the retired version is a retrieval fault, and the check sees it before any answer.
  • D Tone is a property of the answer. A friendly answer can still rest on the wrong page.

9. After a model upgrade, the share of correct answers fell from 91 percent to 78 percent. The same week a re-index changed the chunk size from headings to 500 characters, and recall at 3 fell from 95 percent to 70 percent. The team must pick where to look first. What is the most likely cause to check?

  1. AThe new model, since a model change shipped in the same week as the drop.
  2. BThe wording of the system prompt for the answer step, which shapes every answer.
  3. CThe re-index, since recall at 3 fell before any answer was read.
  4. DThe customers, who may now be asking harder questions than before the upgrade.
Show the answer

Answer: C

  • A The model can only use the pages it is given. A drop in recall is not something a model change explains.
  • B The prompt did not change, and a prompt cannot bring back a passage that was never retrieved.
  • C Correct. Recall fell before any answer was read, so the fault is upstream of the model, in what was retrieved.
  • D The 60 labelled queries are the same, and they show the drop, so the questions did not get harder.

10. A wrong answer is traced to a refund window of 14 days instead of 30. The prompt says to use the current page, and the engineer finds that the page with 30 days was never among the 3 passages sent to the model. The team debates rewriting the prompt. What should the fix target?

  1. AThe index, so the current page is retrieved and the retired one is not.
  2. BThe prompt, with a stronger line telling the model to use the current page.
  3. CThe model, by moving to a larger tier for this step of the pipeline.
  4. DThe customer's question, by asking which policy version the customer means.
Show the answer

Answer: A

  • A Correct. The page was missing from the hits, so the fault is in the index, and a version filter fixes it once.
  • B A prompt cannot name a page that was not sent. The model had no 30 day text to prefer.
  • C A larger model reads the same 3 passages. It cannot use a page that was never retrieved.
  • D A customer does not know which version is current, and that is the assistant's job to know.

11. A retrieved chunk holds a customer's address from a staff-only page, and the reader is a signed-in guest whose role does not cover it. The team can filter at retrieval or ask the answer step to leave the address out. The page holds 6 such fields, and the guarantee must hold on every call. Where should the control sit?

  1. AIn the answer prompt, as a line that lists the 6 fields the model must omit.
  2. BIn the reply, with a scan that blanks anything that looks like an address.
  3. CAt retrieval, so the chunk is dropped before the prompt is built.
  4. DIn the page, with a banner that says the content is private and is for staff.
Show the answer

Answer: C

  • A The address is already in the prompt, and a model can be asked to print it. An instruction is a request.
  • B A scan of the reply runs after the model has read the address, and it misses forms the pattern does not know.
  • C Correct. A chunk that never reaches the model cannot appear in its answer, on any call.
  • D A banner on a page is text. It does not stop retrieval from returning the chunk.

12. An answer cites the Fraud review playbook, but the reader is an outside caller who cannot open that page. Trust in the answers falls after a support lead notices 4 such citations in a week. The team wants a citation to help a reader check the answer. What rule fits?

  1. ADrop the citation and let the answer stand without any source for the reader.
  2. BCite only pages that this caller may read, because retrieval filtered the rest.
  3. CCite the page and add a sentence that says access to it is restricted.
  4. DCite a public page that is close in topic, whatever the source of the answer was.
Show the answer

Answer: B

  • A A citation is how a reader checks the answer. Dropping it leaves a claim nobody can verify.
  • B Correct. If retrieval filters by access first, every page the model cites is one the reader can open.
  • C The warning still shows that a private page exists and what it held. The page should not have been retrieved.
  • D A citation to a page that did not back the answer is false, and it misleads the reader.

Play: predict the output

The content system exports each page's version as text.

python
pages = [
    {"topic": "refunds", "version": "9", "text": "Refunds within 14 days."},
    {"topic": "refunds", "version": "10", "text": "Refunds within 30 days."},
]
newest = max(pages, key=lambda p: p["version"])

Predict: Which refund window does the assistant quote?

  1. A14 days, because the text "9" sorts after "10".
  2. B30 days, because version 10 is the newest.
Show the answer

A. Text compares one character at a time, so "9" beats "10". Convert the version to a number when you index it, and test with a two-digit version.

References

Conclusions

  • Chunk by the shape of the page, and keep the version and access list on each chunk.
  • Route an exact identifier to an exact lookup, and blend keyword and vector scores for the rest.
  • Filter by version and access before the model sees a passage, then check retrieval on its own.

You can now shape an index so the current, permitted page is what the model reads and cites.

Chunk

One piece of a page that is indexed and returned

Recall at k

The share of queries whose right page is in the top k

Exact lookup

Finding a page by its identifier, with no ranking

Hybrid search

A blend of keyword and vector scores

Access filter

A check that drops pages the caller may not read