Explain why AI features need systematic evaluation.
Priya built an app that summarises customer feedback for team leads. A week after it went live, one team lead complained that summaries of Malay feedback were coming out in Malay, even though the system prompt said English.
She added a stronger line to the prompt: "always reply in English, even for very short or non-English feedback". To test it, she pasted in three Malay examples, and all three came back in English, so she shipped the change. Two days later, another team lead noticed that short complaints such as "late again" were now summarised as "The customer did not state a problem", something that had never happened before. Her fix for one case had broken another, and she had no way of knowing until someone complained.
The new line had changed how the model treated short feedback in general, and some of those outputs got worse. Priya had tested her change, and the test passed. Testing AI features by hand feels like testing, but it misses most of what matters, for three reasons.
When you test by hand, you pick the inputs. They tend to be the ones you thought of, which means they are typical, tidy and similar to each other. Real users send whatever they actually have: half-sentences, three languages in one message, pasted email signatures, typos, sarcasm, a single emoji, a 2,000-word rant. Failures live in that variety, and hand-picked tries can't reach input types you didn't think of.
A few successful tries only show that the feature can work, which you already knew. They don't show how often it fails on real traffic, or on which kinds of input.
The second problem is that models produce varied output even for identical input. AI fundamentals lesson 3.3, "Why the same question gets a different answer", explains why this happens. Lowering the randomness setting, as suggested in lesson 2.3, reduces the variation but rarely removes it completely. Providers also sometimes update models behind the same model name, and this can shift behaviour without any change on your side.
So one good result doesn't guarantee the next. An input that gives a correct summary today may give a slightly wrong one on the fifth run, which makes a single pass on each test case weak evidence. Run your important cases a few times and look for any that flip between pass and fail. A case that does this is a fragile case, and knowing which cases are fragile tells you where your app is least dependable.
The third problem is the one that caught Priya: every fix can break something else. A prompt is not a set of separate rules that each affect one case. Every line influences every output, so a line written for Malay feedback also changed what happened to "late again".
Ordinary software has the same problem, and programmers solved it long ago with automated tests. They keep a set of tests that check the behaviour that matters and run all of them after every change. If a change breaks something that used to work, a test fails before users see it. That habit, more than any clever technique, is what lets software teams change code confidently.
The AI equivalent of automated tests is an eval, short for evaluation. An eval is a fixed set of test cases, each with an input and a description of a good output, run against your app every time you change something. Each of these changes should get the same eval run before you ship:
a prompt edit a new model a different temperature a change to retrieval a new tool
The key property of an eval is that it is fixed. The cases stay the same between runs, so any difference in results comes from your change and not from different inputs. The set does grow over time: every bug a user reports becomes a new case. You never quietly swap cases out. Run the important cases a few times each so the fragile ones show up.
An eval gives you a number, for example 18 of 20 cases passing. More usefully, it gives you a list of which cases pass and which fail. After a change, you compare that list with the previous run and see exactly what improved and what broke. If Priya had had an eval, her English fix would have shown the new failure on short complaints the moment she ran it, before any team lead saw it.
You have already built the start of an eval. The ten hand-checked emails in lesson 2.4, "Build an extractor that turns emails into JSON", were a small eval, and so were the ten test questions in lesson 3.5. Module 7 makes that habit systematic. Lesson 7.2 builds the test set, lesson 7.3 scores it automatically, and lesson 7.4 uses it to make a real decision.
Before building an eval, it helps to know what you're protecting against. The activity below asks you to think through the changes you are likely to make to your app and what each could break.
List three changes you might make to your app, such as a new prompt or a new model. For each one, write down what could break.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).