You will be able to explain when you need retrieval and how a retrieval system answers a question.
Many workplaces want an assistant that can answer questions straight from their own documents, such as a staff handbook, a set of policies or a product manual. The obvious first move is to copy the entire document set into a chat assistant and type each question underneath it. With a handful of pages, that can produce an answer. As the pages pile up, the method stops working, and it has costs and risks even while everything still fits.
Take Daniel (an invented example, along with every figure about his firm). He is the only HR executive at a 200-person engineering firm in Paya Lebar, and half of his week goes on questions he has answered many times before. Staff ask how many days of childcare leave they get, whether they can claim a taxi home after working late, and how to arrange a work-from-home day. The staff handbook and a policy folder hold every answer, about 300 pages between them. Staff don't read them and message Daniel instead. He wants a tool that reads the documents for them, gives the answer and names the page it found the answer on. His first plan is to paste the full handbook into a chat assistant with each question.
You met the context window in AI fundamentals, lesson 5.1, "The context window is the model's only working memory". It is the limit on how much text a model can take in at once. Some current models accept very long inputs, so Daniel's 300 pages may well fit. Even when they fit, sending everything is still a poor design, and Daniel's plan has problems with cost, speed and quality.
On cost, every input token is billed on every request. Lesson 1.3, "Tokens, pricing and limits you have to plan around", walks through the sums. When each question arrives with the whole document set attached, most of what you pay for that question goes on text unrelated to it. Someone asking about childcare leave is also paying for the pages on every other policy.
On speed, a longer input takes the model longer to process, which slows down every answer.
On quality, a model does not give equal attention to every part of a very long input, and details in the middle of a long document can slip past it. AI fundamentals, lesson 5.2, "Why long chats drift and big files get skimmed", covers the reasons. A short input with only the relevant pages usually gets a more accurate answer than a long input with everything in it.
Document sets also get bigger over time. Daniel might later add the employee benefits guides, the safety manual and three years of HR circulars. Sooner or later a set reaches a size that no model's context window can hold.
Think about what you do yourself with one question and a long document. To check the rule on taxis, you would search for "taxi", look over the matches and read closely only the section that fits, without rereading the whole document.
Software can follow the same routine. When a question arrives, code searches the documents and pulls out the few passages most likely to contain the answer. Only those passages go to the model, together with the question and an instruction to answer from them. The model writes the answer without seeing the rest of the documents.
In Daniel's case, the taxi question returns three items: the paragraph on late-night transport in the claims policy, the handbook's overtime section, and last year's circular that changed the claims limit. That is roughly a page of text going to the model in place of 300, and the model never sees the other 295 pages.
This method is called retrieval-augmented generation (RAG). You first retrieve relevant text from your own sources, then have the model generate an answer from the text you retrieved. Retrieval is the search, and generation is the model writing the answer.
Every RAG system has two halves. The preparation half runs once at the start and again whenever the documents change. It splits the documents into passages and stores them in a form that can be searched by meaning. Lesson 3.2 explains how search by meaning works, and lesson 3.3 covers splitting and storing documents. The answering half runs on every question. It searches the stored passages, chooses the best few and passes them to the model with the question.
Daniel gains most from easier updates, because his firm revises its policies every year. With retrieval, he replaces the old document with the new version and reruns the preparation half on that one file. The next question is answered from the new policy.
Someone may suggest training or fine-tuning a model on your documents instead. Fine-tuning takes longer and costs more. It is poor at holding exact facts such as claim limits and leave entitlements, and it cannot tell you which page an answer came from. That rules it out for Daniel, because he specifically wants page references. With retrieval, the facts stay in documents you control, so you can read them, correct them and point to them.
Retrieval still leaves room for error. When the search returns the wrong passage, the model answers from that wrong passage. Lesson 3.4, "Make answers cite their sources and admit gaps", shows how to handle this.
Before you build, settle two things: which documents go in, and what the tool should be able to answer from them. Daniel's documents are the handbook and the policy folder. His questions are the ones staff send him each week.
The activity below asks you to choose both, in this order: first a document set, then the questions it should be able to answer.
Choose a set of documents you would like to ask questions of, such as policies or notes. Write five questions those documents should be able to answer.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).