You will be able to test an assistant on image, audio and tool tasks and record where it helped and where it failed.
Grace is a sales coordinator for an events company in Ubi. During one working week she gave her AI assistant a whiteboard photo, a spreadsheet screenshot and a voice note, then checked every answer against the original.
You can run the same test on your own assistant. The last four lessons covered how assistants read images, handle audio and call tools. The exercise shows how well yours does those jobs on material from your own work. You count its mistakes, fill in the short results table below, and give each task one of three verdicts: use, check or avoid. Allow about 25 minutes. Your assistant has to accept images. If it can accept audio too, use that as well.
Keep everything non-confidential. Brainstorm notes for the office party are fine. A whiteboard with client names on it is not, and Lesson 8.3 explains why. Each of your three inputs also has to be a different type.
For handwriting, use whatever you already have, such as the whiteboard after a meeting, a page from your notebook or a sticky note with a list on it. Take the photo on your phone the way you usually would. Leave the mess as it is and don't arrange anything for the camera.
Your second input is a screenshot of a spreadsheet with numbers in it, at least 10 rows long. A staff roster, a sales tracker or your household budget will do. A sheet that has totals on it gives you more to test.
Your third input depends on your assistant. If it takes audio, record a clip of one to two minutes. A voice note to yourself listing three or four tasks works best. A short recording of a conversation is also fine, as long as everyone in it agreed to be recorded. If your assistant doesn't take audio, photograph a printed document with small text instead, such as a receipt or a page of terms and conditions.
You can only mark a reply if there is a correct answer to compare it with. A question like "what do you think of this?" gives you nothing to mark. Word each request so the original decides whether the reply is right:
Whiteboard or other handwriting: type out everything on it word for word, in the order it appears. Spreadsheet: list every value in one column and give their total. Audio: give the action items, with who does what and by when. Printed document: type out one paragraph exactly.
Start a new chat for every input, so the assistant's work on one input doesn't affect how it handles the next. As each reply comes back, paste it into your results table.
Put the original next to the reply and go through it one line at a time. It takes a while, and the slow comparison is the point of the exercise. Look hardest at smudged corners and small print. Lesson 7.1 showed that when an image reader can't make something out, it doesn't leave a gap. It puts in something plausible, and you can easily miss that if you only skim.
Write down two numbers for each input. The first is the total number of items, where an item is one word, one number or one action. The second is how many of those items the assistant got wrong. Divide the wrong count by the total to get the error rate, which is only a rough measure. If 2 out of 20 numbers are wrong (an invented example), the error rate is one in ten.
Next to the count, note what kind of error each one was and what it would cost you. Errors differ in cost. A misspelt name in brainstorm notes does no harm, but a wrong figure in a total does.
Give every task one verdict.
Use means the assistant made few or no errors, and any it did make are cheap and quick to spot. A light check is enough.
Check means the output is worth having, but errors turned up in places that matter. You treat it as a first draft and compare each important item with the original.
Avoid means the errors are so many, or so costly, that fixing the output takes longer than doing the job yourself.
Grace's whiteboard had 34 words on it, and the assistant read 31 correctly (all her figures are invented). It got one handwritten name wrong and made up a word where the marker had faded. She could spot and fix both at a glance, so she marked the whiteboard use.
From her spreadsheet she picked a column of 15 values. The assistant read 14 of them correctly but turned S$1,280 into S$1,230, so its total was S$50 short. She marked it check, because a total she can't trust means going back over every line.
Her voice note was two minutes long. It was mostly in English with a few Malay and Hokkien words, and it named three colleagues. The assistant got the tasks broadly right, but it gave one task to the wrong person and changed "Thursday" to "Tuesday". Two errors in four action items is too many for something she would forward to her team, so she marked it avoid for now.
When she finished the table, one result surprised her: the assistant did best on the task she had expected to be hardest. Her results are one possible outcome and are not a benchmark. Your numbers will vary with your assistant and your material. Once your verdicts are in, you will know which jobs you can hand over with a light check and which you should keep doing by hand.
Fill in your own table the same way, one input at a time, and keep a note of the errors that would have cost you most.
Fill in a results table for at least three inputs: what you asked, what it returned, the errors you found and a verdict of use, check or avoid.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).