Tokens, pricing and limits you have to plan around

Estimate the token count and monthly cost of a feature, and name the limits that can stop a request.

Before you build an AI feature that other people depend on, work out three things: how many tokens a task uses, what that costs, and which limits can stop a request. Priya built a feedback summariser for her workplace, and her manager liked it. The manager then asked what it would cost if the whole customer service team used it every day, and Priya could not answer. She knew the provider charged per token. She did not know how many tokens a piece of feedback came to, how long the summaries ran, or whether a busy morning would hit a limit and make requests fail.

Counting tokens with real inputs

A token is a chunk of text. Each model has its own tokenizer, which decides how text is split. Common English words are often one token each, while long or unusual words break into several. Chinese, Tamil and other scripts often need more tokens than English to say the same thing, and code, URLs and long strings of numbers use more as well. A customer message with order references and Singlish abbreviations will count differently from a tidy textbook paragraph. For these reasons, estimates based on word counts go wrong.

Measure with real samples. Collect ten typical inputs from the job the feature will do. Send them to the model, or put them through the token counting tool or endpoint that most providers offer, which counts tokens without running the model. Each response has a usage field giving the exact input and output token counts. Read those counts, take the average, and plan from it. Also add a few of the longest inputs you can find, since they set the size of the worst case.

When you count, include everything the model receives. The input count covers the system prompt, any examples, any retrieved documents and, in a chat, the whole history that is resent with each message. In many apps the fixed instructions are bigger than the user's message.

Pricing one summary and 1,000 of them

The cost of one request is (input tokens x input price) + (output tokens x output price). Providers quote prices per million tokens, and output tokens usually cost more than input tokens. Prices change and differ by model. Take them from the provider's current pricing page on the day you plan, never from memory or an old blog post. Real prices are usually in US dollars, which you convert at the current rate.

Priya's samples averaged 1,200 input tokens per feedback summary, most of it her system prompt, and 300 output tokens. The prices below are made up, purely to show the method: S$3 per million input tokens and S$15 per million output tokens. The input cost is 1,200 x 3 / 1,000,000 = S$0.0036 and the output cost is 300 x 15 / 1,000,000 = S$0.0045. Each summary costs S$0.0036 + S$0.0045 = S$0.0081, and 1,000 summaries cost S$8.10.

Every estimate needs two more adjustments. First, price the whole chain. If a feature makes three model calls for each user action, each action costs three calls, so you multiply the cost per request by three. Second, add a margin for retries and for the long inputs your samples turned up. With both in place, scale the figure to the volume you expect, for example 1,000 runs.

Limits on input and reply length

Every model has a context window, the most tokens it will handle in one request, and the input and the reply both count towards it. Reply length has its own separate cap, which you can set lower with the maximum output setting. Sending more input than the model accepts makes the call fail with an error. Asking for a reply longer than the output cap, or setting the maximum too low, makes the reply stop mid-sentence, and a cut-off reply is easy to mistake for a finished one. Lesson 2.3, "Handle a bad response without crashing", shows how to detect a reply that was cut off.

Both limits vary widely between models and change with new releases. Look up the context window and output cap for your chosen model on the provider's model documentation page. Check that your longest input plus the reply fits inside the context window, and set the maximum output high enough that replies are not cut off.

Retrying after a 429 error

Providers also limit how much you can send per minute. These rate limits usually count both requests and tokens. They apply to your account or project as a whole, so everyone using your app shares one allowance. New accounts usually start on lower limits that rise over time, but providers decide the move up in different ways. Some, such as OpenAI, raise your tier as your total spend grows, while Anthropic moves organisations up based on their usage history and account standing. Each provider's rate limit documentation explains how its tiers work and lists the current limits. When you go over a limit, the provider rejects the request, usually with an HTTP 429 "too many requests" error.

When your code gets a 429, it should wait and try again. After the first one, pause briefly and retry. If that fails, double the pause and retry again. Stop after a small number of attempts, and if all of them fail, show the user a clear message about what happened. Many official libraries already handle retries automatically, so check yours before writing your own.

Priya needs no special code for a busy Monday morning. A retry with a wait is enough, plus a clear message to the user if every attempt fails.

In the activity below you will do Priya's homework for your own idea: real inputs, real token counts, and a cost estimate for 1,000 runs.

Pick an idea of yours and collect five real inputs for it. Count their tokens with the provider's tool or the usage field, then estimate what 1,000 runs would cost.

Course

Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).