You will be able to measure how often an assistant gets money questions right when you can check the answers yourself.
Darren already knew the rule from lesson 1.1: AI assistants are good at explaining things and unreliable on figures. What they wanted to find out was how unreliable their own assistant was on the money questions they really ask.
General reading about failure rates won't answer that for you. A personal ten-question test will. It takes about twenty-five minutes, and you score explanations and figures separately so you can see the gap between them for your own assistant.
Each of the ten questions needs an answer you can find on your own, from a source you trust. Take five of one kind and five of the other.
Five should be explanation questions, which ask how something works. You can check these against a course lesson, a MoneySense guide or the product documents themselves. Possible ones are what a deductible is on a hospital plan, how a flat rate loan differs from a reducing-balance loan, what the free-look period on most insurance policies is for (travel, car and maid insurance are exceptions, as MoneySense explains), what an expense ratio measures, and how dollar-cost averaging works.
The other five should be fact or figure questions, which ask for a specific current number or rule that has an official source. You check these on the CPF Board, IRAS or MAS website, or in your own product document. Some suggestions:
the current CPF contribution rate for your age group the interest rate on one of the CPF accounts the latest Singapore Savings Bonds rates the income tax rate on one band of income a fee on a fund or card you hold
Put all ten on paper before you ask the first one. If you pick questions after seeing answers, you tend to drift towards the ones the assistant handled well.
Use a new chat for every question. In a long conversation the assistant carries earlier answers forward, and you want each answer to stand alone.
Begin every question with the same opening words, so any difference between answers comes from the questions themselves and has nothing to do with how you phrased things that day. One fixed opening you could use is "I live in Singapore. Please answer briefly." followed by the question.
Some assistants search the web without being asked. Next to each question, note whether yours did. You'll need that note when you compare results at the end.
Every answer gets one of three verdicts: right, wrong, or out of date. A figure is out of date if it was correct at some point and isn't now. The official page often shows this, because the assistant's number turns up there as an older rate in a table of past values.
Explanations call for judgement. If an explanation is correct in general but leaves out something that matters to you, mark it right and add a note about what's missing. If it gets the idea backwards, mark it wrong. Figures are marked strictly, and a rate that is close does not count as right. When the assistant gives a range instead of a number, decide based on whether the true figure sits inside that range, and note that it avoided giving a number.
Next to every verdict, write the source: the site and page, or the document and page number. A verdict with no source is only your memory set against the assistant's.
Darren asked, "What does co-insurance mean on a hospital plan?" The assistant replied that co-insurance is the share of a claim you pay yourself after the deductible, and illustrated it with made-up figures. Darren looked up the glossary in the policy document for their own plan, marked the answer right, and wrote the source down as the policy document, definitions page.
Out of curiosity, Darren made one of the figure questions a calculation: S$10,000 at 3% a year, compounded monthly, after ten years (invented example figures). The assistant said about S$13,439. Darren put the same inputs into a spreadsheet using the FV function, which How money works, lesson 2.4, Build a compounding table in a spreadsheet, teaches, and got S$13,493.54. That comes from 10,000 x (1 + 0.03/12)^120. Changing the same sheet to yearly compounding, 10,000 x 1.03^10, gave S$13,439.16, which matched the assistant's figure. The assistant had used yearly compounding when the question asked for monthly. Darren marked the answer wrong and recorded the source as "own spreadsheet".
The difference was about S$54, small enough that Darren would have missed it without checking. An assistant can answer a slightly different question from the one you asked without telling you, as this one did. Slips like this are common and small errors are easy to overlook, so check every answer as a matter of habit, including the ones where the number looks fine.
Count right answers in two groups: explanations out of 5 and figures out of 5. Keep them as two separate scores. If you add them into one score out of ten, the gap between them disappears from view.
Darren ended with 4 out of 5 on explanations and 2 out of 5 on figures. One of the figure answers that missed was an older rate that has since changed, so its verdict was out of date. The two scores told Darren something about this assistant on these questions that general advice could not.
Most people who run the test find a clear gap between the two scores. What you're after is the size of that gap for your assistant and your questions. Strong explanations and weak figures support the rule from lesson 1.1. If your figures scored higher than you expected, go back to your notes and check whether the assistant searched the web on those questions, and whether the pages it found were official.
Your own ten questions come next. Choose them before you open the assistant, and keep the source column honest.
Fill in a ten-row test sheet with the question, the answer, your verdict and your source. Then write two sentences on where the assistant was reliable and where it was not.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).