Automation Bias

At noon on 9 June 1995, the cruise ship Royal Majesty left Bermuda for Boston with more than a thousand passengers and crew aboard. It had a modern integrated bridge: a GPS receiver fed the ship's position to an autopilot, which steered the course the officers had set. Not long after departure, the cable from the GPS antenna came loose. The receiver could no longer hear the satellites, so it did what it was designed to do. It switched to dead reckoning, estimating the ship's position from its speed and heading alone, without allowing for wind or current. The change was signalled by a short alarm and small letters on a display in the chart room, which was rarely visited at sea. Nobody noticed.

For the next day and a half, the autopilot kept steering by a position that drifted further from the truth every hour. The screens looked normal. The officers on watch checked the display, saw the ship on its planned track and moved on. There were clues that didn't fit, including a buoy that should have been seen and wasn't, and lights reported by lookouts that didn't match the chart. Each was explained away. At about 10.25 that night, the ship ran aground on the Rose and Crown Shoal near Nantucket Island, miles from where its instruments said it was. Nobody was hurt, but the ship was badly damaged.

The US National Transportation Safety Board concluded that the main cause was the watch officers' over-reliance on the automated features of the bridge system, and it recommended that officers routinely check their primary navigation system against independent sources, such as other radio aids, visual bearings and the depth sounder.

Nobody on that bridge was careless in the ordinary sense. They trusted a system that was usually right. That's automation bias, the tendency to rely on an automated system's output instead of our own checking and judgement, so that we miss problems the system doesn't flag and follow its advice even when other information says it's wrong.

The term was popularised in the 1990s by the psychologists Kathleen Mosier and Linda Skitka, working with colleagues on research funded by NASA into how pilots used increasingly automated cockpits. In a 1999 study, Skitka, Mosier and Mark Burdick asked eighty students to "fly" a simplified flight simulation. Half had an automated monitoring aid, described to them as highly reliable but not perfect. The other half relied on the ordinary instruments alone. The ordinary instruments were always accurate. Only the aid was occasionally wrong.

The researchers named two kinds of mistake. Omission errors happened when the aid failed to flag an event and people missed it too, because they had stopped watching for things the aid didn't announce. Commission errors happened when people followed the aid's recommendation even though the other indicators, all correct and all in front of them, said it was wrong. The group with the aid performed worse on monitoring than the group without it. Mosier and colleagues found similar errors among experienced professional pilots in a more realistic simulator, so experience was no protection.

This line of work has held up well. In 1997, Raja Parasuraman and Victor Riley described the ways people use, misuse, disuse and abuse automation, and in 2010 Parasuraman and Dietrich Manzey reviewed the evidence on complacency and automation bias. They found it in novices and experts, in single operators and in teams, and concluded that simple training or instructions didn't reliably remove it. In medicine, a 2012 systematic review by Kate Goddard, Abdul Roudsari and Jeremy Wyatt found the same pattern with clinical decision support: when the software gave wrong advice, clinicians sometimes changed correct decisions to match it. The size of the effect varies a lot with how reliable the system is, how busy the person is and how the advice is shown, but the direction is consistent.

Automation bias grows out of a sensible habit. Our attention is limited, and handing work to a reliable tool is the whole point of having the tool. If a calculator has been right ten thousand times, checking its next answer by hand is a waste of effort. We do the same with dependable colleagues. Trust saves effort, and effort is scarce.

The trouble is that the better a system is, the less practice we get at catching its failures. In 1983, the engineering psychologist Lisanne Bainbridge called this one of the "ironies of automation". Automate the routine work, and the human is left with two jobs people are bad at: watching for rare failures for hours, and taking over suddenly when something unusual happens, often with skills that have faded from lack of use. A machine that's right 99% of the time is almost designed to make us stop looking for the 1%.

Machines also come with an air of objectivity. A number on a screen doesn't hesitate or look unsure, so it can feel more trustworthy than our own memory or a vague sense that something's off. That's close to the appeal to authority I wrote about earlier in this series, except the authority has no face.

Automation bias doesn't only happen in a single person's head. It can be built into institutions. From 1999, the UK Post Office rolled out an accounting system called Horizon, built by Fujitsu, to its branches. Over the following years, many sub-postmasters, people running local post offices, found that Horizon showed shortfalls they couldn't explain. The Post Office insisted the system was robust, required them to make up the missing money and prosecuted hundreds of them for theft, fraud and false accounting between 1999 and 2015. People lost savings, homes and reputations; some went to prison.

Part of what made this possible was a legal assumption. In England and Wales, the law had once required evidence that a computer was working properly before its records could be relied on in court. That requirement was repealed in 1999, and courts went back to presuming that a computer was working correctly unless someone could show otherwise, which is very hard for an individual to do against an organisation that controls the system. In 2019, a High Court judgment in a group case brought by sub-postmasters found that Horizon had contained bugs, errors and defects capable of causing the shortfalls. In 2024, Parliament passed a law quashing most of the convictions.

There are Indian versions at the scale of a single journey. On the night of 23 November 2024, three men driving near Faridpur in Bareilly district, Uttar Pradesh, died when their car went off the end of a bridge over the Ramganga river, part of which had collapsed in floods earlier. News reports and police statements said they were following a navigation app that still showed the route, and that there were no barriers or warning signs. Police investigated local officials, and the app maker said it was cooperating. It isn't fair to blame the drivers: it was night, the road was unmarked and the map was wrong. But it shows how completely we hand our attention to a voice that has usually been right.

Picture a team building an AI assistant for finance staff at mid-sized companies. It reads incoming invoices, matches them to purchase orders and suggests "approve" or "query". In testing, it agrees with a careful human reviewer about 97% of the time. The launch design is clean: a list of invoices, each with the AI's suggestion in a green or amber badge, and an "Approve all suggested" button at the top.

Three weeks into the pilot, the numbers look great. Average review time has fallen from four minutes per invoice to ten seconds. Then an auditor finds a duplicate invoice that was paid twice, and a vendor whose bank details had changed the week before. The AI suggested approval for both. So did the reviewers, who, it turns out, had mostly stopped opening the invoices at all. Nobody was lazy. The design told them the checking had been done.

The better version starts from the people, not the model. The team stops showing every suggestion with the same confident badge. Instead, the assistant shows what it checked and what it couldn't: "matched to purchase order; vendor bank details changed 6 days ago." Effort is matched to stakes. Routine, low-value invoices from long-standing vendors can be approved in bulk; new vendors, changed bank details and large amounts need the reviewer to look at the specific field that matters before the button works. For a small random sample, the reviewer decides before seeing the AI's suggestion, a technique that Zana Buçinca, Krzysztof Gajos and colleagues called a cognitive forcing function in a 2021 study, where it reduced over-reliance on AI advice. And the team watches one number closely: how often reviewers disagree with the AI. If that falls to almost zero, it doesn't mean the AI is perfect. It probably means nobody is looking.

When I worked on an agentic AI system at Rocketium, one question we kept coming back to was where a person should be asked to look before the agent acts, and where asking would only train them to click yes. There's no universal answer, but it's a design question, not just a model question.

The goal isn't to make people distrust machines. Spellcheckers, autopilots, fraud alerts and maps are right far more often than we are, and relying on them is rational. The human factors researchers John Lee and Katrina See called the aim "appropriate reliance": trust that matches how reliable the system actually is, in that situation. Honest design helps people calibrate. It shows confidence where the system has it, uncertainty where it doesn't, sources they can check, and a clear way to override.

It becomes manipulation when a product deliberately inflates trust beyond what it has earned, by presenting guesses as certainties, hiding that no human reviews the output, or claiming abilities it hasn't shown. In 2024, the US Federal Trade Commission brought a case against DoNotPay, which had marketed its service as "the world's first robot lawyer". The FTC said the company hadn't tested whether its AI performed at the level of a human lawyer. Under the final order in early 2025, DoNotPay agreed to pay $193,000 and stop making such claims without evidence.

Regulators are starting to name the bias itself. The European Union's AI Act, adopted in 2024, requires high-risk AI systems to be designed so that the people overseeing them can remain aware of the tendency to automatically rely or over-rely on the output, and it calls this automation bias by name. For anyone designing AI products, that's a rare thing: a cognitive bias written directly into law as a design requirement.

The first habit is to ask, of any automated suggestion I design, "What would a person need to see to notice if this were wrong?" If the answer is "nothing on this screen", the design is relying on luck.

The second is to treat agreement rates as a warning sign as well as a success metric. When people accept nearly every suggestion, I want to know whether that's because the system is excellent or because they've stopped looking. Usually it's some of both.

The third is personal. When a tool gives me an answer that surprises me, I try to treat the surprise as information rather than brushing it aside. That's the moment the officers on the Royal Majesty had, more than once, when the buoy didn't appear and the lights didn't match. The screen said everything was fine, and it was easier to believe the screen. In the next post, on the illusion of control, I'll look at a bias that runs almost the other way: believing our own actions steer outcomes that are really out of our hands.

Further reading: Linda Skitka, Kathleen Mosier and Mark Burdick, "Does automation bias decision-making?" (1999) · Lisanne Bainbridge, "Ironies of Automation" (1983) · Raja Parasuraman and Dietrich Manzey, "Complacency and Bias in Human Use of Automation: An Attentional Integration" (2010) · John D. Lee and Katrina A. See, "Trust in Automation: Designing for Appropriate Reliance" (2004) · National Transportation Safety Board, Grounding of the Panamanian Passenger Ship Royal Majesty on Rose and Crown Shoal near Nantucket, Massachusetts, June 10, 1995 (1997)