Confirmation Bias

In 1960 the psychologist Peter Wason, at University College London, gave people a sequence of three numbers, 2, 4, 6, and told them it followed a rule he had in mind. Their job was to find the rule. They could write down any three numbers they liked, along with their reason for choosing them, and Wason would say whether the new sequence fitted. When they felt sure, they could announce the rule.

Most people quickly formed a theory, such as "even numbers going up by two", and then tested sequences that fitted it: 8, 10, 12, then 20, 22, 24, then 100, 102, 104. Every answer came back yes, so they felt more and more certain, and they announced their rule with confidence. It was wrong. The real rule was simply "any three numbers in increasing order". In Wason's original study, only six of the 29 students found it at their first announcement. Few people tried a sequence that could have proved their theory wrong, such as 1, 2, 3 or 6, 4, 2.

What I love about this experiment is how fair it is. Every answer people received was true. They simply asked only the questions that could give them good news.

That's confirmation bias: we look for evidence that agrees with what we already believe, we read unclear evidence as support, and we remember the hits better than the misses. What we rarely do is go looking for the evidence that wouldn't agree.

In the last post, on the false consensus effect, I wrote about assuming other people think the way we do. This one is about what happens after we've formed an idea, any idea, about a user, a feature or a colleague. From that moment, the idea starts quietly choosing what we notice.

Wason followed the 2-4-6 task in 1966 with an even more famous puzzle, now called the Wason selection task. People choose which of four cards to turn over to check a rule, and most pick the cards that could confirm it while missing the one that could break it. When the same logic is dressed up as a familiar social rule, such as checking who is drinking in a bar, people do far better.

The finding that matters most for designers came in 1978. The psychologists Mark Snyder and William Swann asked people to interview a stranger and find out whether that person was an extravert. Another group was asked to find out whether the person was an introvert. Given a list of possible questions, the first group chose ones like "What would you do if you wanted to liven things up at a party?", and the second chose ones like "What factors make it hard for you to really open up to people?" Each question assumed the answer it was looking for. Even a shy person can describe livening up a party if you ask them how they would do it. So the interviews tended to produce the evidence the interviewers had been asked to find.

In 1998, the psychologist Raymond Nickerson pulled decades of this work together in a long review, calling confirmation bias "a ubiquitous phenomenon in many guises". The guises include how we search for evidence, how we interpret it and how we remember it. I touched on the interpreting part in the post on cherry picking, where students on both sides of the death penalty debate rated the same mixed studies in opposite ways.

How well does all this hold up? Better than a lot of famous psychology. The 2-4-6 task and the selection task have been repeated many times with similar results, and the broad pattern, now often called myside bias, shows up across many kinds of study. What researchers still argue about is what the results mean. Confirmation bias is an umbrella term for several different habits, and some of them, as we'll see, are not always mistakes.

Part of the answer is effort. Checking whether something is true by trying the cases where it should work is quick and natural. Imagining all the ways it could fail takes more work, and our minds are thrifty. Part of it is feeling. Being right is pleasant. Evidence that disagrees feels like a threat, so we look at it harder and find reasons to dismiss it.

There's also a more generous explanation. In 2011, the cognitive scientists Hugo Mercier and Dan Sperber argued that human reasoning may not have evolved mainly to help a lone person find the truth. It may have evolved to help us argue: to produce reasons that persuade others and to evaluate the reasons others give us. If that's right, a bias towards finding support for our own side makes sense, because in a group, someone else is busy finding the flaws. The trouble starts when everyone in the room is on the same side.

A team that has spent three months on a concept is, by definition, a room full of people on the same side.

On 11 March 2004, bombs on commuter trains in Madrid killed 191 people. Spanish police found a fingerprint on a bag of detonators and shared images of it abroad. At the FBI, a computer search produced a list of possible matches, and an experienced examiner identified one of them as Brandon Mayfield, a lawyer in Portland, Oregon. Two more FBI examiners checked the work and agreed. So did an independent examiner later appointed by the court. The Spanish police had doubts and said so, but Mayfield was arrested in May 2004 and held for about two weeks. Then the Spanish authorities matched the print to an Algerian man, and Mayfield was released. The FBI apologised, and in 2006 the US government settled his lawsuit for two million dollars.

The US Justice Department's Inspector General later reviewed what went wrong. Part of it was bad luck: the two prints were unusually similar. But the report also described a kind of circular reasoning. Once the first examiner had decided on a match, features in the blurry crime-scene print began to be "seen" because they were present in Mayfield's clear one. The verifiers knew a colleague had already called it a match. And the report suggested that once investigators learned Mayfield had converted to Islam, they may have been less inclined to look again.

In 2006, the cognitive psychologist Itiel Dror and his colleagues tested this directly. They gave five experienced fingerprint experts a pair of prints that each of them had, years earlier, judged to be a match. This time they were told, casually, that these were the prints from the Mayfield case, the famous mistake. Four of the five changed their minds, three of them now calling it a clear non-match. The prints hadn't changed. The expectation had.

India has a lovely example of the opposite habit: designing a test that could fail. Astrology is widely trusted in India, and believers can always point to predictions that came true. The astrophysicist Jayant Narlikar and his colleagues in Pune, working with the rationalist Narendra Dabholkar, collected the birth details of two hundred children, half of them described by their teachers as academically bright and half from special schools for children with intellectual disabilities. Astrologers were invited to take a random set of forty horoscopes and say which child belonged to which group, and the test was run so that nobody handling the charts knew the answers. Twenty-seven astrologers sent in their judgements. The results, published in Current Science in 2009, found that none of them did better than chance. Whatever you believe about astrology, that's what a fair test looks like: it gives the idea a real chance to be wrong.

Picture a team in Bengaluru that has built an AI assistant inside a note-taking app. It suggests the next sentence as you type. Everyone on the team uses it every day and loves it. The product manager has already told leadership it's the headline feature for the next release, and the team schedules six usability sessions to "validate" it.

The discussion guide gets written the night before. It asks, "How helpful did you find the smart suggestions?" and "What did you like most about writing with the assistant?" During the sessions, the note-taker types quickly whenever a participant smiles or says "oh, that's clever". When a participant ignores the suggestions for ten minutes, nothing gets written down. When one says the grey text is distracting, the team decides she's "not the target user".

The synthesis deck says, "Users found the assistant helpful (5 of 6)." Every quote in it is real. But the questions assumed helpfulness, just as Snyder and Swann's questions assumed extraversion, and the notes captured only reactions, not silences. The test simply had no way to fail.

The better version starts before the first session. The team writes down, in a sentence, what result would make them doubt the feature: say, if fewer than half the participants accept a suggestion without being prompted, or if more than two find it distracting. The tasks become neutral, "Write a note to your landlord about the leaking tap", so the team can see whether people use the assistant on their own. Questions turn open: "What did you notice while you were writing?" The note-taker logs what people do, including the things they don't do. And one person is given the job of building the case against the feature in the synthesis, out loud, before the team agrees on anything. If the assistant survives that, it has earned its place in the release.

Not every search for confirming evidence is a mistake. In 1987, the psychologists Joshua Klayman and Young-Won Ha pointed out that what people do in the 2-4-6 task, testing cases they expect to work, is often a very reasonable strategy. It only fails so badly in Wason's task because the guessed rule, "up by two", sits entirely inside the real rule, "any increasing numbers", so no positive test can ever reveal the difference. In many real situations, especially when the thing you're looking for is rare, checking the cases where you expect a yes is the quickest way to find out you're wrong.

Holding on to a good idea in the face of a few awkward results can also be wise. Designers shouldn't kill a concept because one participant in eight disliked it. And people arguing hard for their own ideas is healthy, as long as someone else is allowed to argue just as hard against them.

So the bias isn't in looking for support. It's in looking only for support, and in never giving the idea a fair chance to fail. Letting go of a belief after it has been shown to be wrong is a different problem, which I'll come back to in a later post on belief perseverance.

Before a test, I write down what result would change my mind. It has to be specific enough that I could actually fail it. If I can't write that line, I'm not testing an idea, I'm decorating one.

After a test, I read the notes looking for the moments that went against the idea first, before I let myself enjoy the ones that supported it. I also count, so that "users loved it" becomes "four of six used it without being asked, two never touched it".

And whenever I can, I ask someone who disagrees with me to review the findings. It's uncomfortable. It's also the closest a design team gets to the Spanish police in the Mayfield case: someone outside the room who has no reason to see what we're hoping to see.

I still think about Wason's students, writing down 20, 22, 24 and hearing yes, yes, yes. They weren't foolish. They just never asked a question that could have told them no. In the next post I'll look at a bias that tilts our expectations about the future rather than our reading of the past: optimism bias.

Further reading: P. C. Wason, "On the Failure to Eliminate Hypotheses in a Conceptual Task" (1960) · Raymond Nickerson, "Confirmation Bias: A Ubiquitous Phenomenon in Many Guises" (1998) · Mark Snyder and William Swann, "Hypothesis-Testing Processes in Social Interaction" (1978) · Joshua Klayman and Young-Won Ha, "Confirmation, Disconfirmation, and Information in Hypothesis Testing" (1987) · Hugo Mercier and Dan Sperber, The Enigma of Reason (2017)