Choose a scoring method for each test, and use a model as a grader carefully.
Open your test set and look at a single output, field by field. For each field, note whether it has exactly one right answer, whether it follows a rule you could write down, or whether judging it depends on meaning. That split decides how each part of the output gets scored.
Priya builds an app that summarises customer feedback. She has a test set of 24 cases in a spreadsheet. The first time she ran them, she checked each output by eye. It took 40 minutes, and afterwards she was unsure she had judged the last ten as carefully as the first ten. Checking by eye is slow and inconsistent, and care tends to drop over a long run. She wanted to run the set after every change, so it had to score itself.
Different parts of one output need different kinds of checking. If you choose the right method for each part, your eval (the automated run that scores every case in your test set) is fast enough to run often and trustworthy enough to act on. There are three methods to choose from.
Use an exact check when a field has exactly one right answer: labels, categories, dates, amounts, yes/no flags, IDs. It is a direct comparison. If the expected sentiment is negative, the output says negative or it fails. If the expected date is 2025-03-14, anything else fails.
Exact checks are the cheapest and most reliable scoring you have. They run in milliseconds, cost nothing and never disagree with themselves. They are one reason Module 2 pushed you toward structured output with enums, fields that accept only a value from a fixed list. A field limited to positive, negative or mixed is easy to check exactly, and a free-text description of sentiment is not.
List fields need a decision about how to compare them. Priya's output includes a list of issues. She checks that all expected issues are present and allows extra ones, because an extra minor issue is not wrong. Whatever you decide, write down the comparison rule for each list field so the check stays consistent from run to run.
Many requirements can be checked by code that does not know the right answer. A code rule checks format or content, and a coding assistant can write each one in a few lines. Code rules cost nothing to run and give the same verdict every time, so use one wherever a requirement can be written down as a rule.
Priya wrote these:
the summary is 25 words or fewer all three required fields are present the output is in English only none of a short list of banned words appear any reply that repeats a line from the system prompt is flagged the JSON validates against the schema from Lesson 2.2
Rules cannot check meaning. Whether a summary captures the main problem, or whether a reply is accurate and polite, needs a model to do the grading. This is a model grader, often called LLM-as-a-judge, where LLM stands for large language model. It is a second model call that receives the input, the output and a written rubric.
A rubric is a list of criteria a good output must meet. Word each criterion clearly enough that two careful people would apply it the same way. Narrow yes/no questions work far better than asking for a rating from 1 to 10. Scores out of ten drift between runs and are hard to interpret, while a pass/fail question such as "did it add anything not in the feedback?" has an answer you can check yourself. Ask the grader to give a one-line reason with each answer.
Priya's first rubric for summaries only asked about the main complaint. Her final rubric has three pass/fail questions, each answered with a one-line reason: does the summary state the customer's main complaint; does it avoid any claim not in the original feedback; and could a team lead act on the summary without opening the feedback. The second question came from testing the grader.
Graders make mistakes. They tend to be lenient, they can prefer longer answers, and they may prefer text that reads like their own writing. Test a grader before you rely on it: grade a sample by hand without looking at the grader's verdicts, then compare the two sets and fix the rubric where they disagree.
Priya graded 20 outputs herself this way. In the comparison she found a summary that invented a refund request, which the grader had passed. The grader had no reason to object, because her rubric did not ask about it. That case is why she added the question about unsupported claims.
Repeat a small hand check now and then, and always after you change the grader's model or its rubric.
A single headline pass rate hides patterns. Suppose 20 of 24 cases pass. If all 4 failures are among the 6 non-English cases, two in three non-English users get a bad result, and the total alone would not show it. Lesson 7.2, "Build a test set from real cases", asked you to tag every case. Those tags let you report a pass rate for each tag alongside the total.
Priya's first scored run showed normal cases at 15 of 15 and other-language cases at 3 of 6, which pointed her straight at the problem.
Every scoring run should produce a table with one row per case: the case ID, its tag, the result of every check, and a final pass or fail. Save each table with the date and the prompt version it tested. Lesson 7.4 puts two saved tables side by side.
With a scoring method fixed for each of your own cases, your set scores itself in minutes, so you can rerun it after every prompt change and see a drop in one tag before your users run into it.
Pick a scoring method for each of your 20 cases, and write the comparison rule for any list field. For any case a model will grade, write the rubric as pass/fail questions with a one-line reason each.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).