You will be able to deal with malformed, refused, cut-off and slow responses in a predictable way.
Before your app goes live, list every way a model call can go wrong and decide in advance how the app handles each one. Truncated output, refusals and slow responses are not bugs in your code. They are normal behaviour for a model API under real traffic, and what matters is whether your app already has a plan for each.
Priya built a feedback summariser for her team. It turns each piece of feedback into JSON with an issues list and a sentiment field, writes the summary into a sheet, and shows it on a page her team uses. It ran for a week on real feedback with no problems. Then, on one afternoon, three things happened (the scenario is an invented example).
A long complaint from a corporate client produced JSON that stopped partway through the issues list, because the reply hit the maximum output length. A message containing a rude word came back as a polite refusal instead of a summary. The provider was slow for ten minutes, so her team's page showed a loading spinner until people gave up and refreshed.
Start by reducing how often failures happen. Most models have a randomness setting, usually called temperature. A low temperature makes the model pick its most likely wording, so the same input gives very similar output each time. A high temperature lets the model pick less likely words, so output varies more.
Extraction and classification tasks, such as pulling a delivery date out of an email or labelling a complaint, need low values. You want the same answer on every run, and variety is just noise. Brainstorming tasks, such as suggesting ten names for a product, need higher values, because variety is what you're asking for. Some models fix the temperature or ignore it, so check the model's documentation.
Priya set her summariser's temperature low and immediately saw fewer odd labels. Low temperature reduces variation but does not remove it, so you still need the safeguards below.
In lesson 2.2 you set up validation for the model's output. When that validation fails, use one retry, then a fallback.
The retry must include the specific validation error. Models are good at fixing a named problem, and a blind retry with the same request may repeat the mistake. Priya's retry message tells the model its previous reply failed validation, names the error (sentiment must be one of positive, negative or mixed), and asks for valid JSON again.
If the second attempt also fails, stop and do not loop. A request that fails twice often keeps failing, and every attempt costs money. Send the case to a fallback, which is what your app does when a call can't produce a valid result after the allowed attempts. You can mark the item for human review, use a simpler default value, apologise to the user, or queue the item for later. Priya's fallback marks the feedback "needs review" in the sheet so a person can handle it.
Log every failure and every retry, together with the input that caused it. Patterns in the log show whether the prompt, the schema or the validation needs fixing.
Every response includes a field saying why the model stopped generating. Providers name it differently, often "finish reason" or stop reason. A normal value means the model finished. A different value means it hit the maximum output length.
Output that hits the limit can be cut off mid-object, which leaves invalid JSON, as happened with the corporate complaint. In plain text mode a cut-off reply can look finished, for example when it ends on a full stop that happens to fall just before the limit. Check the stop reason on every call, before you parse anything.
If the output hit the limit, you can raise the maximum output length if the task really needs more room, ask for less output (for example a cap of five issues), or shorten the input. Do not retry a truncated request unchanged, because it will be cut off at the same place.
Refusals need a check too. Some providers return a refusal as a separate field or a distinct stop reason, and others return it as ordinary text. Find out how your provider signals a refusal and send refusals to the fallback. Never treat a refusal as a valid result, such as a summary.
A model call can take a long time when the provider is busy or the output is long. If your code waits forever, so does the user, so set a timeout on every call. Most official libraries accept a timeout setting, and many default to one far longer than any user will wait.
Decide in advance what the user sees when the timeout fires. The worst option is an endless loading spinner, because the user can't tell whether to wait or give up. Priya's timeout message tells the user the summary is taking longer than usual, that their feedback has been saved, and that it will be summarised shortly. A background job then tries again later.
Rate limit errors, which you met in lesson 1.3, belong to the same family as timeouts. Retry after a pause, give up after a small number of attempts, and show a clear message.
In your code, choose the temperature and set the timeout before the call. When the response arrives, check the stop reason, then check for a refusal, then validate. Retry once with the error if validation fails, then fall back, and log each failure and retry with its input.
The activity below asks you to do what Priya did after that afternoon: list every way a call can go wrong for your app, before it happens.
List every way your feedback app could fail, and next to each one write what your code will do.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).