Build a small agent that uses tools to answer a question and stops within the limits you set.
For the module 5 capstone, you build a small research agent. It answers a question with two or three tools, stops within limits you set, and writes a readable log of every step. You then run it on three test questions and read each log to see where it chose well and where it wasted steps. Allow about 40 minutes.
The exercise uses four earlier lessons: the loop from lesson 5.1, limits from lesson 5.2, failure checks from lesson 5.3, and the read, draft and act rules from lesson 5.4. The running example is Daniel. He indexed a set of HR documents in module 3 and built his agent to research questions about them.
Keep the toolset small. Daniel's agent has three tools:
search_documents runs the retrieval you built in lesson 3.5 and returns the top few chunks with their sources. calculate is the calculator from lesson 4.4, so all the arithmetic comes from code. write_note adds a line to a notes file that belongs to the current run. The agent can then keep track of what it has found without relying only on the growing conversation.
In lesson 5.4's terms, write_note is a draft action, because it changes nothing outside the run. None of the three tools sends, pays for or deletes anything, so Daniel needs no approval step. If you add a tool that does any of those things, put it behind approval first. Write each tool definition the way lesson 4.2 describes, with a clear description of when to use the tool and when not to.
Have your coding assistant write the loop with three run limits: a step cap, a cost budget and a time limit. Also add a timeout on every individual call. When any limit fires, the loop should stop and write a clear message saying which limit fired, what was done and what is left. Daniel set his step cap at 8 steps (an example figure). He worked out his cost budget from the token prices he had measured, and he gave each run 2 minutes in total.
Add the loop check from lesson 5.3. If the agent tries a tool call with the same name and arguments as one it has already made, don't run it. Return a message telling the agent the call is a repeat.
Log every step with the step number, the tool and its arguments, a short version of the result, the tokens used and the running cost. Write the log to a separate file for each run. Include the goal in every request, which lesson 5.3 recommends to stop the agent drifting. In the system prompt, tell the agent to answer only from what the tools return and to say plainly when it cannot find an answer.
Each question tests a different behaviour. The first should be answerable, and ideally it should need more than one step. Daniel asked for annual leave plus the maximum childcare leave, counted in working days. His handbook is made up. It gives 14 days of annual leave and up to 6 days of childcare leave, so the expected answer is 14 + 6 = 20 working days off. Getting there takes 2 searches and 1 calculation.
The second question should be one the documents cannot answer. Daniel asked whether pets are allowed in the office, and the handbook says nothing about pets. In a good run, the agent searches once or twice, finds nothing relevant and says so. In a bad run, it invents a policy or keeps searching.
The third question should be designed to make the agent loop. Daniel asked it to find form HR-12 and list the form's approval steps. HR-12 is invented and doesn't exist, but the question is worded as if it certainly does. When a model is told that something exists, it may keep searching with different wording. This run should hit the loop check or the step cap.
Run each question and read the full log before you look at the final answer. For each step, decide whether the choice was sensible: whether a search was needed, whether the arguments were good, and whether the agent used what came back. Note any wasted steps. These include repeated searches, detours away from the goal, and arithmetic the model did in its head instead of with calculate.
Daniel's leave run took 4 steps: 2 searches, 1 calculation and 1 note. It answered 20 and gave its sources. The pets run made 2 searches and then answered honestly that the handbook doesn't cover pets. The HR-12 run made 4 searches, each worded differently. Then the loop check caught a repeat, and the step cap stopped the run at 8 with a clear message.
Daniel concluded that the limits worked, but the agent had wasted 3 searches before giving up on HR-12. He added an instruction to the system prompt: if two searches find nothing relevant, say you could not find it. When a log shows wasted steps, adjust your own setup in the same way, for example by changing the system prompt.
You're done when you have a working agent with two or three tools, a step cap, a cost budget and a time limit. You should also have a log file for each of your three runs and a short note on each run saying where the agent chose well and where it wasted steps. At least one of the three runs should have stopped early because of a limit or the loop check. If none did, your limits are probably too loose.
These three logs show you more about agents than any amount of reading. Go and generate them.
Build the agent and run your three test questions. Save the step logs, with your notes on each run.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).