You will be able to recognise automation bias in yourself and your team.
Siew Ling is an accountant at a trading company in Paya Lebar. For three months she's been using an assistant to draft the monthly variance commentary for her managers. It's been excellent: clear, well organised, accurate. This month she reads it quickly, as she has for weeks now, and sends it on. Two days later her finance director asks why the commentary says freight costs fell, when the table right above it shows they rose.
Siew Ling isn't careless. She's a qualified accountant who checks things for a living. What happened to her has a name, and it happens to careful people precisely because the tool is usually right.
Automation bias is the tendency to accept what an automated system tells you and stop checking it, even when it's wrong. The term comes from research on people working with automated systems in settings like aviation and healthcare, long before today's AI assistants, and the pattern turns up wherever people work alongside machines that are usually reliable.
It shows up in two ways. Sometimes you act on a wrong output because the system suggested it. Siew Ling sent the wrong freight commentary. Sometimes you miss a problem because the system didn't flag it, and you'd come to rely on it to tell you. If your spell checker doesn't underline a word, you assume it's spelled right.
Neither is a failure of intelligence. Checking takes effort, and when a system has been right a hundred times, the brain sensibly stops spending effort on it. The trouble is that the hundred-and-first time may be the one that matters.
AI assistants add something that older automated systems didn't have: they write beautifully. Their answers are well structured, grammatically clean and confident in tone, with headings, bullet points and a reassuring summary at the end.
People tend to judge text that reads smoothly as more trustworthy, whether or not it's correct. An answer full of typos and hesitations would make you suspicious. A polished one doesn't, even when the content is just as wrong. AI fundamentals: what it is, how it works, where it fails explains in lesson 6.1, Why a fluent answer can still be false, why a model produces confident prose whether or not the facts underneath are right.
So the very quality that makes assistants pleasant to use also makes their mistakes harder to see. Siew Ling's commentary read like a finance professional had written it. That's exactly why the wrong direction on freight costs slipped through.
Automation bias isn't equally strong all the time. Three conditions make it worse, and they often come together.
The first is being busy, because a rushed person drops the check before anything else. Siew Ling's slip happened during month-end close, when she had four other deadlines.
The second is a tool that's usually right. The better the track record, the less you check. An assistant that gets things wrong half the time keeps you alert. One that's right ninety-nine times in a hundred lulls you into trusting the hundredth.
The third is a task you couldn't easily do yourself. If you don't know the right answer, you can't tell when the output is wrong. Someone asking an assistant to explain a contract clause in a language they don't read, or a technical topic they've never studied, has no way to catch an error, so whatever they think they're doing, they're really hoping.
Here's the uncomfortable part. At the moment you decide, a check always feels unnecessary, since the output looks fine, you're busy, and it's been right before. Every individual decision to skip the check seems reasonable.
That's why checking has to be decided in advance, as a rule, rather than judged case by case. Pilots use checklists for steps they've done thousands of times, precisely because experience makes skipping them tempting. You can do the same with AI output.
After her mistake, Siew Ling set herself a rule. Before sending any AI-drafted commentary, she checks every direction word, such as rose, fell, increased and decreased, against the table. It takes about three minutes, and she doesn't decide each month whether to do it, because the rule already decided. In the second month, it caught another reversed figure.
Your rule will depend on your work. For a report, it might be checking every number against its source. For an email to a client, it might be a read-through out loud before you press send. For a summary of a long document, spot-checking three claims against the original. Lesson 7.3, Decide what to hand over, what to assist, what to keep, helps you decide which tasks need a rule like this.
Think about the last time you used something an assistant produced without really checking it. It might have been a work email, a summary, a calculation or a fact you passed on. Picture, honestly, what would have happened if that one had been wrong.
Recall the last AI output you used without checking and write what would have happened if it had been wrong.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).