You will build a small tool that answers questions from your own documents, gives its sources, and says when it does not know.
Daniel is building a small question-answering tool over his firm's HR handbook and policies, so staff can look things up without asking HR each time. Yours works the same way over the documents you picked in lesson 3.1, and it pulls together everything you have made in the module so far.
You type a question and the tool searches your documents for the passages closest to it. It gives back one of two things: an answer with its sources cited, or a plain message saying your documents don't cover that question. It runs in a terminal, so there is no web page to build. Set aside about an hour.
Your coding assistant can write both scripts for you. You need to tell it what each one must do.
The preparation script reads in your documents and cuts them into chunks a few paragraphs long, with each chunk overlapping the next by a little. It turns every chunk into a vector using your provider's embedding model, the kind you tried out in lesson 3.2. The text, the vector and the metadata for each chunk then go into the storage you settled on in lesson 3.3. A local file or a free-tier vector database will do for a practice project. Have the script note down the name of the embedding model, since questions later have to be embedded with that same model.
The answering script works through every question in the same sequence. First it embeds the question using the embedding model the chunks were made with. It pulls back the five chunks that score highest. It puts those into the prompt you wrote in lesson 3.4, the one that cites sources and admits gaps, and numbers and labels each chunk. Last, it sends the prompt to the model and prints the reply, converting the citations into sources a person can read.
Neither script should have the API key typed into it. Both take it from the environment.
Get the assistant to give the answering script a debug option: a flag which, when switched on, prints the retrieved chunks as well as the answer. With it you can see exactly what the model had in front of it when it wrote the reply, and you will use it all the time.
The test set is ten questions. Five come from lesson 3.1. Two are unanswerable questions: things your documents say nothing about, there to check that the tool admits it. Three are everyday questions phrased loosely, in words your documents don't use. Five, two and three make ten (5 + 2 + 3 = 10). Daniel's three loose ones are "my kid is sick can I stay home", "what's the deal with claiming a cab after OT" and "do we get anything for our birthday".
For every answerable question, note the correct answer and the passage it comes from, and do it before the tool runs even once. If you leave it until afterwards, the tool's reply will shape your idea of what the right answer was.
Switch debug on and put all ten questions through the tool. Record the results in a table with a row for each question and five columns in this order:
the question did retrieval find the right chunk: yes, partly or no was the answer correct: yes, partly or no did it cite the right source notes
The second column is retrieval quality: whether search brought back the right passage for the question. You can tell from the debug printout. The third column is answer quality: whether the generated answer used that passage correctly. Keeping these two marks apart is the most useful habit in the module, because it tells you which half of the system needs the fix.
Mark the two unanswerable rows another way. Search hands back the nearest chunks whether or not any of them are relevant, so retrieval always comes back with something for these questions. Judge those rows only on whether the reply correctly said the information could not be found.
The tool doesn't have to score ten out of ten. Expect a mix of right and wrong answers. A tool that says it doesn't know, and explains why, is in better shape than one that answers every question confidently and gets some of them badly wrong. For an illustration, picture one tool that gets 7 right and says it does not know the other 3, giving its reasons, and a second that answers all 10 confidently and gets 2 badly wrong. The first tool is the better one.
On his first run Daniel had 8 answerable questions and 2 unanswerable ones. For 6 of the 8, retrieval brought back the right chunk. Of those 6 answers, 5 were correct. The sixth confused two leave types that happened to share a chunk. Both unanswerable questions received the fixed "could not find this" reply.
That means retrieval missed twice (8 - 6 = 2). One miss was "what's the deal with claiming a cab after OT". The casual phrasing sat closer to the overtime pay policy than to the transport claims section, so the model never received the passage it needed. With two retrieval misses, Daniel decided chunking and search needed work before the prompt did. He broke the long claims section into smaller chunks and added keyword search, and the cab question then worked.
Apply the same rule to your own table. When retrieval misses, change chunking and search first, for example with smaller chunks or added keyword search, and leave the prompt until after that.
Keep your ten questions. Lesson 7.2, "Build a test set from real cases", uses them again as the start of a proper eval set.
You are done when three things are true. Both scripts run without errors and take the API key from the environment. All ten questions are written down, with an expected answer and source for each answerable one. Every row of the table is filled in, with retrieval and answer quality marked separately. Once the table is complete, read down the retrieval column first and the answer column second. That order tells you where to spend your next hour.
Build the tool and run ten test questions through it. Save a table showing retrieval quality and answer quality for each question.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).