Set limits on steps, time and spend

Add hard limits that stop an agent before it runs out of control.

Open the logs from your agent's recent runs and find the longest one that did its job properly. Count its tool calls and write the number down. Every limit on your agent loop should come from measured runs like this one, and this number sets the first limit.

Marcus sells products and built his first agent to handle customer emails. In this example, which is invented, he left it running overnight on a batch of emails. One email asked about a product he had stopped selling, the Aeropress Go. The agent searched for it and found nothing. It tried a different spelling and found nothing, then searched by category and found other products. It searched for the original again and kept going. By morning it had made hundreds of calls on that one email. His usage page showed the spend. The monthly limit he set in lesson 1.1 meant he noticed it much sooner than he would have without it.

Nothing had malfunctioned. A model can keep trying to finish a task that cannot be finished, and it cannot decide on its own that a run has gone on long enough. Limits in your code make that decision. You need limits on steps, cost, time and the size of the history.

Capping the number of steps

A step cap is a counter on the loop. Add one to it on every iteration. When it reaches the maximum, stop the loop no matter what the model wants to do next.

Base the maximum on runs you have actually seen and logged. Set it at a margin above your longest legitimate run. About double leaves room for a retry or two without allowing a runaway. Marcus's longest legitimate run was the combined delivery question from lesson 5.1, at five tool calls, so he set his cap at ten.

When the cap is hit, stop cleanly. Return a message that says the task was not completed, what was done so far and what is still open. A stop that explains itself gives you something to act on, while a stop that just ends leaves you one more mystery to investigate. Marcus's stop message goes to him, and the customer does not see it: "Stopped after 10 steps. Could not find product 'Aeropress Go' in the catalogue. Customer email moved to manual review."

Tracking what each run costs

Steps are only a rough measure of cost because some steps are much bigger than others. A step that pulls in a long document costs far more than a calculator call, so track cost directly as well.

You saw in lesson 1.4 that every model response includes token usage. A cost budget, also called a run budget, works from those figures. Read the input and output tokens from every call in the run, convert them to money at current prices and keep a running total. Before each new step, check the total against the run budget. Stop if the next step would take you over it.

Set the budget from measured runs, the same way you set the step cap. If a typical run costs a few cents, a budget of a few times that catches runaways without cutting off normal work. Measure whole runs when you do this. Each step resends the growing history, so later steps cost more than earlier ones. If you estimate the cost of a run from its first step alone, you will set the budget too low.

The run budget is your own control inside your code. Keep the provider spending limit from lesson 1.1 as well, because it protects your account if your code has a bug.

Limiting how long a run takes

Users wait for agents, so set two time limits. The first is a timeout on each model call and each tool call, the per-call timeout from lesson 2.3, Handle a bad response without crashing. It stops one slow request from hanging the run. The second is a run time limit on the whole run. Record the start time, and before each step check whether the elapsed time has passed the limit.

Choose the run time limit from what the user can tolerate. A chat support agent gets a short limit, perhaps under a minute. A research agent producing a report that someone reads the next day can have a much longer one.

Keeping the conversation history short

Every step adds two things to the conversation history: the model's tool call, then the tool's result. Some results are long, such as a page of search hits or a full document. After enough steps the history can approach the model's context window, which lesson 1.3, Tokens, pricing and limits you have to plan around, described. When the history reaches that limit the call fails, and well before then each step is slow and expensive.

Shortening the history on purpose is called compaction. Before you add a long tool result, trim it to the part the model needs. Replace the full results of older steps with a short summary. In Marcus's history, an earlier search becomes "Searched catalogue for Aeropress Go: no match." Keep the original goal and the most recent results in full. Some frameworks and providers offer tools that do this automatically, so check what yours offers before you write your own.

Deciding what happens when each limit fires

For every limit, write down in advance what the user sees and what gets logged. These are Marcus's rules:

Step cap: stop, mark the email for manual review and log the steps. Cost budget: stop, mark the email for manual review and alert him if this happens more than twice a day. Time limit: tell the customer someone will reply within a working day.

The activity below asks you to write the same set of limits for an agent you might build, with what happens when each one fires.

Write the limits for an agent you might build: maximum steps, cost budget, time limit and what happens at each.

Course

Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).