Run your test set on two versions of your prompt and use the results to decide which one to ship.
Set aside about 40 minutes and have two things ready: the test set you built in lesson 7.2 and the scoring method you chose for each case in lesson 7.3. You'll use the eval to make a real decision about whether a prompt change should ship, which is where it proves its value.
The worked example follows Priya. Her app summarises customer feedback for her team leads, and she's testing the "always reply in English" change from lesson 7.1.
Ask your coding assistant to write a script that does four things. It reads your test cases from a file, runs each input through the app's real code path, scores each output, and saves a results table. The real code path means the same prompt, the same schema and the same model as the live app, so the results describe what your users actually get.
Scoring can use exact checks, code rules or a model grader with a rubric, whichever you set for each case in lesson 7.3. The results table has one row per case, with columns for case ID, tag, the result of each check, and overall pass or fail.
Put the date and a prompt version label, such as v3 or v4, in the name of each results file so that no run overwrites another. Keep the prompt itself in version control, as lesson 2.1 recommended, so each version label points to an exact text.
Before the first run, check the cost with the arithmetic from lesson 1.3. The eval calls the model once per case, and a model grader adds more calls. For a few dozen cases the cost is usually small, but you'll run the eval often, so work the figure out once at the start and keep it in mind for later runs.
Run the full set on the current prompt with nothing changed and save the results table. This first run is your baseline, the record of how the current version behaves.
Priya's test set has 24 cases (an example figure). On prompt version 3, 20 of 24 passed. Four cases failed: two Malay messages answered in Malay, one Chinese message answered in Chinese, and one pasted email signature summarised as if it were a complaint.
If a baseline case fails when you expected it to pass, look at it now. Sometimes the expected output in the test case is wrong, and you should fix the test case before you continue. Don't fix the app at this stage, because the baseline needs to show the current version as it is.
Make only one change per run, so any difference in the results can be traced to that change. Save the change as a new version, commit it, and rerun the full set. Run every case, including the ones you weren't trying to fix, and save the new results table under the new version label.
Priya changed one line of her system prompt to "Always reply in English, even when the feedback is very short or not in English". She saved it as version 4, committed it, and ran all 24 cases, of which 22 passed.
Put the two tables side by side and sort every case into one of four groups:
passed both times failed both times fixed: failed before the change and passes now broken: passed before the change and fails now
Then check the arithmetic. Previous passes plus fixed minus broken must equal the new number of passes, and if the totals don't match, there's an error somewhere in your comparison.
For Priya, 19 cases passed both times and the email signature case failed both times. All three non-English cases were fixed, and one case was broken: a message that read "late again" had passed on version 3, and the new output was "The customer did not state a problem." Her check works out as 20 passing before + 3 fixed - 1 broken = 22.
The headline moved from 20 to 22, but a higher pass count alone isn't enough to decide on. Read every broken case in full, with its output, before you decide anything. Priya judged that "late again" is exactly the kind of feedback her team leads see every day. A short angry message summarised as "no problem stated" is worse than a summary in the wrong language, because a team lead might skip it entirely, so she did not ship version 4.
Look at the cases that failed both times as well. They show what the change did not touch, so note them for a later round. If you ran important cases several times, as lesson 7.1 suggested, check whether a broken case is just flipping between runs. A case that fails one run in three is fragile and needs attention whichever version you ship.
If the comparison shows a problem, make another single change and run the full set again. Priya wrote version 5, which kept the English rule and added "Very short feedback still describes a problem; summarise what it implies". On the full set it passed 23 of 24, with nothing broken compared with version 3. She shipped version 5 and added the email signature case to her list for the next round.
When you finish, you should have an eval script, two saved results tables (one for each prompt version), and a short decision note. The note says how many cases passed each time, which cases were fixed and which were broken, what you read in each broken case, and which version you will ship and why.
Keep both tables in your repository next to the prompt versions they came from. The next time someone asks why the prompt says what it says, the answer will be in those files.
Run your eval on two prompt versions and save both result tables. Then write a short note saying which version you will ship.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).