You will be able to compare two or three assistants on your own tasks using the same prompts and a simple scoring sheet.
Make a list of the jobs you've handed to an AI assistant over the past couple of weeks, and note roughly how often each one comes up. You'll use it to choose the tasks for a test that takes about thirty-five minutes. When you finish, you'll have a written choice of main assistant and a note of any task you'd rather send to a different one.
People tend to pick an assistant much as they pick a phone. They try one, settle into it, and assume the rest are similar or not as good. Reviews can't tell you which one suits you best (Lesson 8.1, What actually differs between assistants, explained the reasons), so this test uses your own work instead: identical prompts in each assistant, marked on a simple scoring sheet.
Before starting, make sure you can use at least two assistants, and three if possible. A free plan covers most tasks. Where a task depends on a feature such as file upload, confirm that every assistant offers it on the plan you're on. If the test is for work, stick to the assistants your employer permits for the kind of data involved, as set out in Lesson 8.2, Data settings and the rules at your workplace.
Choose five tasks from your list, weighted the way your own use is weighted. Someone who mostly drafts emails with an assistant should make two of the five emails. At least one task should involve a document, for example summarising a report or answering questions about a PDF. At least one should contain a fact you can verify, such as a question about a rule that an official website can confirm.
Grace, the recruitment team lead you met earlier in this module, settled on these five. The first was a job ad for an accounts executive role, and the second a rejection email to a candidate who had made it to the final round. The third was a summary of three CVs measured against a job description, with the candidates' details taken out as her policy required. The fourth was a question about the current rules for a particular work pass, which she could verify on the Ministry of Manpower website, and the fifth was a set of interview questions for a sales role.
Draft each prompt a single time and put in everything the course has taught you: context, task, constraints and format. Paste that same text, word for word, into every assistant. You may notice a flaw after the first run. Leave the prompt as it is, because fixing it part way through would hand the assistants you test later a stronger prompt than the earlier ones received.
Open a new chat for each task so nothing said earlier shapes the reply. Match the modes as far as the products allow, standard with standard and reasoning with reasoning. Check custom instructions and memory as well. If one assistant has them switched on and another doesn't, turn them off for the test, or give every assistant equivalent instructions using the setup record from Lesson 6.4, Set up your assistant for your working week.
Collect the replies in a single document and label each one with a letter only: A, B or C. Write down which assistant each letter stands for in a separate note.
Give every output a mark from 1 to 5 on each of these:
Accuracy: everything in the output is correct, checked against the source for the document and fact tasks. Usefulness: the output does the job you needed it for. Edits needed: 5 if you could use it unchanged, 1 if you would rewrite it.
A task can score at most 15.
If you can, mark the outputs without knowing which assistant wrote each one. This is called blind scoring. A colleague can shuffle the outputs and cover the letters, or you can work from the lettered document and keep your note closed until the marking is done. Most people already favour one assistant before the test starts, and knowing who wrote an answer usually pulls the mark towards the favourite.
Take the time to check the fact task against the official source. When Grace looked up the work pass rules on the Ministry of Manpower website, assistant A's answer agreed with the current rules. Assistant B gave a figure that had been right in an earlier year and presented it as the current one. No review had told her as much as that one result about how far she could rely on each assistant for regulatory questions.
Once everything is marked, add up the scores for each assistant and study the pattern across the tasks as well as the totals. Taking Grace's tasks in the order listed above, A scored 12, 13, 9, 12 and 14, a total of 60, and B scored 11, 12, 13, 6 and 10, a total of 52 (invented example figures).
A led on the job ad, the rejection email and the interview questions, the three writing tasks her team does most often. On the work pass task, A scored 12 and B scored 6, because B's out-of-date figure lost it heavily on accuracy. On the CV summary, B scored 13 against A's 9, four points higher.
Treat small gaps with caution. A difference of one point on a single task could easily be chance, while a gap of four points is big enough to take seriously, so Grace counted the CV result. She chose A as her team's main assistant, on condition that it was the one her agency already had a business plan for. She also noted that document-heavy summaries might be worth sending to B if the agency ever approved it.
Lay out the scoring sheet with one row per task. Under each assistant, give three columns for the scores and a fourth for the total. Beneath the sheet, write your decision in three sentences. The first names your main assistant, the second names the task, if any, that you'll send elsewhere, and the third says what would make you rerun the test, for example a new version of either assistant.
Keep the five prompts. They are a ready-made test you can run again in a few months, when the products have changed, as they will. Start by listing the five tasks, and write the prompt for the first one.
Score five tasks across at least two assistants on a scoring sheet, then write three sentences on which assistant you will use for what.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).