Apply common ways to make an AI feature cheaper and faster, and measure the effect of each.
Every change that cuts the cost or response time of an LLM (large language model) app gets the same check: measure before and after, and use your eval to confirm that quality held. Before you change anything, find out where the money and the time actually go.
Priya built a feedback summariser. After 3 months, customer service teams in 2 other regions were pasting in their own feedback, and the monthly bill grew with usage. Users complained that a batch of 50 items took so long they switched tabs and forgot to come back. Her manager asked whether it could be cheaper and faster. Her first instinct was to switch everything to the cheapest model she could find. That might have worked, or it might have quietly made every summary worse.
Run the eval set from Module 7 and record each case's input tokens, output tokens, cost and response time. Token counts come from the usage field in the API response, as in Lesson 1.4. Prices come from your provider's current pricing page. Without these numbers you are guessing where to optimise.
Priya measured first. Her system prompt, with its examples and schema, made up most of the input tokens on every call. The feedback was usually short and the output small. Most of the time went on running 50 calls one after another, so she knew where to look.
From there the method repeats. Apply one change, rerun the eval and compare case by case. Keep the change if quality holds. If it doesn't, revert it or limit it to the cases where quality held. The levers available are:
use the smallest model that does each step well enough trim the system prompt and examples send less context cap output length and ask for concise formats use prompt caching use batch processing use streaming run calls in parallel
Providers offer models in different sizes. Smaller ones cost less per token and usually respond faster. Larger ones handle harder reasoning, longer documents and subtle judgement better. Simple steps, such as classifying a ticket into one of 5 labels, extracting a date or checking whether a message is in English, often run as well on a small model. Summarising a long, messy complaint or answering a tricky policy question may need the larger one. Assign models per step, not per app.
Never switch models on faith. Change the model for one step only, rerun the eval and compare case by case, as in Lesson 7.4, "Run an eval before and after a prompt change". If quality holds, keep the cheaper model. If some case types get worse, keep the larger model for those types or for the whole step.
Priya tried a smaller model for summaries. Normal cases were unchanged, but 2 of her 6 non-English cases slipped. She kept the smaller model and added a rule that routes non-English feedback to the larger one.
Every token you send costs money and processing time. Shorten the system prompt by removing instructions the eval shows make no difference, and keep only the examples that earn their place. Send only the context a step needs. In the retrieval step from Module 3, you might send the top 3 chunks (passages cut from your documents) instead of the top 10, if the eval shows the answers still hold.
Cap output length at what the task needs plus a margin, and ask for concise formats, such as a 25-word summary instead of a paragraph. Output tokens usually cost more than input tokens and take longer to generate, so shorter outputs save on both. Keep checking the stop reason, the response field that says why the model stopped, as in Lesson 2.3. That way a cap set too tight shows up as a failure and not as a reply that was silently cut off.
Your provider's documentation tells you whether and how it supports these two features, and what they currently cost.
Prompt caching means the provider stores the processed form of a repeated prompt beginning. Later requests that start the same way are charged a lower rate for that part and often process faster. It helps when many requests open with the same long text, such as a system prompt plus examples, which is what Priya's app sends.
Some providers, such as OpenAI, cache by default once a prompt passes a minimum length. Others, such as Anthropic, need you to opt in. With Anthropic you can either mark the exact part to cache or add a single setting to the request that places the cache point for you automatically, so marking the part yourself is optional. Check each provider's own documentation for the current minimum length and how its opt-in works. Put the unchanging content (system prompt, examples, schema) first and the varying content, such as the user's feedback, last. Then turn caching on, or mark the cached part, if your provider requires it.
Batch processing means submitting many requests together. The provider processes them within a stated window instead of immediately, and charges less. It suits work that doesn't need an immediate answer, such as Priya's nightly summary of the day's website feedback. Her team's live paste-and-summarise screen doesn't fit.
Some delay can't be removed, but it can be made easier to sit through. Streaming sends the reply to the user piece by piece as the model generates it, instead of all at once at the end. Total time is the same, but text appears almost immediately, which feels much faster than watching a spinner. It suits text a person reads as it arrives and is less useful for JSON that your code must parse as a whole.
Running calls in parallel, at the same time instead of one after another, cuts the total time for a batch, as long as you stay within your rate limits. Showing each result row as soon as it is ready also makes a batch easier to wait for. For Priya's 50-item batches, these two changes were the bigger win.
The activity below has you measure your own app, apply two of these changes and check with your eval that quality held.
Run your test set and log your app's cost, response time and quality as the first row of a table. Apply two changes one at a time, rerunning the eval after each and adding a row for each change.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).