You will measure how often an assistant is right on money questions where you can check the answer yourself.
Darren now knows the theory: assistants are good at explaining and unreliable on figures. But he wants to know how unreliable, on the kind of questions he actually asks, with the assistant he actually uses. Reading about failure rates in general won't tell him that. Testing it on ten questions he can check will.
This exercise builds that test. It takes about twenty-five minutes, and at the end you'll know where your assistant earns your trust and where it doesn't.
You need questions where you can find the right answer yourself from a source you trust. Split them evenly.
Five are explanation questions. These ask how something works: what a deductible is on a hospital plan, how a flat rate loan differs from a reducing-balance one, what the free-look period on an insurance policy is for, what an expense ratio measures, how dollar-cost averaging works. You can check these against a course lesson, a MoneySense guide or the product documents themselves.
Five are fact or figure questions with an official source. These ask for a specific current number or rule: the current CPF contribution rate for your age group, the interest rate on one of the CPF accounts, the latest Singapore Savings Bonds rates, the income tax rate on one band of income, a fee on a fund or card you hold. You'll check these on the CPF Board, IRAS or MAS website, or in your own product document.
Write all ten down before you ask any of them. If you choose questions after seeing answers, you'll drift towards the ones it got right.
Open a fresh chat for each question. A long conversation carries earlier answers forward, and you want each answer to stand on its own.
Use the same opening for every question, for example: "I live in Singapore. Please answer briefly." Then the question. Keeping the wording fixed means the differences you see come from the questions, not from how you happened to phrase them that day.
If your assistant searches the web on its own, note whether it did for each question. That turns out to matter when you compare results.
Now take each answer to its source and give it a verdict. Use three: right, wrong, or out of date. Out of date means the figure was correct at some point but isn't now, which you'll often see on the official page as an older rate in a table of past values.
For explanations, a few judgement calls come up. If the answer is right in general but leaves out something that would matter to you, mark it right and add a note. If it gets the idea backwards, mark it wrong.
For figures, be strict. A rate that's close isn't right. If the answer gave a range instead of a number, mark it against whether the true figure falls inside the range, and note that it dodged.
Record the source you used for each verdict: the site and the page, or the document and the page number. A verdict without a source is just your memory against the assistant's.
Here are two of Darren's rows, with the details as he recorded them.
His explanation question was "What does co-insurance mean on a hospital plan?" The answer said it's the share of a claim you pay yourself after the deductible, and gave an example with made-up figures. He checked it against the glossary in his own plan's policy document. Verdict: right. Source: his policy document, the definitions page.
One of his figure questions was a calculation, which he added out of curiosity: "If I put S$10,000 in an account paying 3% a year compounded monthly, what do I have after ten years?" The assistant said about S$13,439. Darren put the same inputs into a spreadsheet with the FV function, the way How money works teaches in lesson 2.4, Build a compounding table in a spreadsheet, and got S$13,493.54. When he tried yearly compounding in the same sheet, he got S$13,439.16, so the assistant had quietly answered a slightly different question from the one he asked. These figures are an example, but that kind of slip is common. He marked it wrong and wrote "own spreadsheet" as the source.
The gap is about S$54. On a question this size it would have slipped past him if he hadn't checked, which is why the check is the habit and not the exception.
Count your verdicts in two groups: right answers out of the five explanations, and right answers out of the five figures.
Keep the two counts apart instead of adding them into one score out of ten, because a combined score hides what you're looking for. Most people who run this test see a clear gap between explanations and figures, and the size of that gap, for your assistant and your questions, is the useful result. Strong explanations and weak figures would be evidence for the rule from lesson 1.1. If the figures did better than you expected, look at whether the assistant searched the web on those questions and whether the pages it found were the official ones.
Darren's result was four out of five on explanations and two out of five on figures, with one of the two wrong figures out of date rather than invented. That told him something general advice could not.
Your own ten questions come next. Choose them before you open the assistant, and keep the source column honest.
Fill in a ten-row test sheet with question, answer, verdict and source, and write two sentences on where the assistant was reliable and where it was not.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).