Recognise automation bias in yourself and in your team.
Open the last piece of work an AI assistant drafted for you. Choose one number, fact or claim in it that you relied on or passed on without comparing it to a source. If you use an assistant often, you'll probably find one quickly, and most regular users do. A known habit lets items like this slip through, and once you understand how it works you can put a guard in place.
Siew Ling is an accountant for a trading company in Paya Lebar (a fictional person; her story and every figure in it are invented). She is qualified, and checking things is her job. For three months she has used an AI assistant to produce a first draft of the monthly variance commentary her managers receive. Every draft so far has been clear, well organised and accurate.
This month's commentary is due during month-end close, when she also has four other deadlines. She skims the draft, as she has done for weeks, and forwards it. Two days later the finance director asks her a question. The commentary says freight costs fell, so why does the table directly above it show them rising?
The mistake slipped past her because the draft read as though a finance professional had written it, so nothing in it prompted a closer look.
Automation bias is the tendency to accept what an automated system tells you and stop checking it, even when it is wrong. Careful people are as prone to it as careless ones. The term comes from research on people who work with automated systems in fields like aviation and healthcare. That research came before today's AI assistants existed. The same pattern turns up wherever people work next to machines that are reliable most of the time.
The first form is acting on a wrong output because the system put it in front of you, as Siew Ling did when she forwarded the freight commentary. The second form is failing to notice a problem because the system raised no flag, and you had come to depend on it to raise one. Trusting the spelling of a word because the spell checker left it without an underline is an example. You are counting the tool's silence as a check.
Neither form says anything about how clever you are. A check costs effort, and after a system has been right many times your brain reasonably stops spending that effort. When a tool has been correct a hundred times, checking it on the hundred-and-first seems like a waste. Careful people fall into automation bias for exactly this reason: the tool is usually right.
Older automated systems did not write prose. AI assistants do, and they write it well. A typical answer is neatly structured, free of grammar slips and confident, often with headings, bullet points and a reassuring closing summary.
Readers tend to rate smooth text as more trustworthy, whether or not it is accurate. A message full of typos and hesitations makes people suspicious. A polished one does not, even when its errors are just as serious. The fluency that makes an assistant pleasant to work with also makes its mistakes harder to notice. Lesson 6.1, "Why a fluent answer can still be false", in the course "AI fundamentals: what it is, how it works, where it fails", explains why a model writes confident prose whether or not the facts beneath it are right.
The strength of automation bias varies. Three conditions push it up, and they often arrive at the same time.
The first is being busy. Under time pressure the check is the first thing people cut, which is where Siew Ling was during month-end close.
The second is a tool with a good record. The more often it has been right, the less you check it. If a tool gets things wrong half the time, you stay alert. If it is right ninety-nine times in a hundred, you relax and trust it on the hundredth.
The third is a task you couldn't easily do yourself. If you don't know the correct answer, you have no way to spot a wrong one. Asking for an explanation of a contract clause in a language you can't read, or of a technical subject you've never studied, puts you in this position. You are then just hoping the output is correct.
When the moment comes, a check will always seem unnecessary. The draft looks fine, you have other work, and the tool hasn't let you down before. Each single choice to skip looks sensible.
The method has three parts. First, accept that skipping will always feel reasonable at the time. Second, settle on a fixed check rule ahead of time, matched to the type of work, rather than weighing up each case when it arrives. Third, apply that rule every time without deciding again. Many pilots run through checklists for steps they have performed thousands of times, because that much experience tempts them to skip those steps. In the US, the Federal Aviation Administration (FAA) requires airline crews to follow approved checklist procedures. For other flying, the FAA recommends checklists as a guard against complacency but does not require them.
The rule depends on the work:
For a report, compare every number with its source. For an email to a client, read the whole thing aloud before you send it. For a summary of a long document, check three claims against the original.
After the freight query, Siew Ling made her own rule. Before any AI-drafted commentary goes out, she compares every direction word (rose, fell, increased, decreased) with the table. It takes her about three minutes. She no longer decides each month whether to do it, since the rule has already made that decision. In the second month it caught another reversed figure.
To work out which of your tasks need a rule of this kind, use Lesson 7.3, "Decide what to hand over, what to assist, what to keep".
Now think about the last time you used something an assistant produced without really checking it. It might have been a work email, a summary, a calculation or a fact you passed on. Picture, honestly, what would have happened if that one had been wrong.
Think of the last AI output you used without checking. Write down what would have happened if it had been wrong.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).