Optimism Bias

In the early 1990s, the psychologist Robert Emery of the University of Virginia and the law professor Lynn Baker asked a group of people a simple question at one of the happiest moments of their lives. They had just applied for a marriage licence. What, the researchers asked, is the chance that an American marriage today will end in divorce? The answers were impressively accurate. On average, people guessed around half, which was close to the figures of the time.

Then the researchers asked a second question. What is the chance that your own marriage will end in divorce? The typical answer was zero. Not low. Zero. The same people who knew the odds for everyone else were sure they had escaped them. Baker and Emery also asked law students, some of whom had just taken a course in family law and knew exactly how divorces tend to go. The extra knowledge made almost no difference to how they saw their own future. The study was published in 1993 with a title I've always liked: "When Every Relationship Is Above Average".

I don't find this sad. If anything, it's touching. Nobody should get married expecting a fifty-fifty outcome. But it shows something that runs through almost every plan we make. We can know the statistics perfectly well and still feel that they describe other people.

That's optimism bias: the tendency to believe that good things are more likely to happen to us, and bad things less likely, than they really are, or than they are for people like us. It's sometimes called unrealistic optimism or comparative optimism. It's close to two other biases I'll write about later in this series, overconfidence, which is about how sure we are of what we know, and the planning fallacy, which is about how long our tasks will take. Optimism bias is the broader feeling underneath: that the future is on our side.

The classic study came in 1980. The psychologist Neil Weinstein at Rutgers University gave students a list of forty-two events that might happen to them one day. Some were good, such as owning their own home, liking their first job or living past eighty. Some were bad, such as having a heart attack, developing a drinking problem or being fired. For each one, students rated their own chances compared with the chances of other students at the same university. Logically, a group can't be above average on average. But most students rated themselves as more likely than their classmates to experience the good events, and less likely to experience the bad ones.

Thirty years later, the neuroscientist Tali Sharot and her colleagues at University College London looked at how this optimism survives contact with facts. In a 2011 study, people estimated their chances of eighty bad events, such as being burgled or getting cancer, and were then told the real average rate. When the real figure was better than they'd feared, they revised their estimate a good deal. When it was worse than they'd hoped, they barely moved. Good news got in. Bad news bounced off.

I want to be honest about how firm this ground is, because the research has been argued over. In 2011, the psychologists Adam Harris and Ulrike Hahn showed that some of the classic comparative studies could produce an appearance of optimism even from people reasoning perfectly well, because of how rare events and rating scales behave statistically. In 2016, Hahn, Harris and others argued that much of the good news and bad news difference in updating could come from similar artefacts. Sharot and her colleagues have replied with newer designs, and the debate continues.

The effect also varies by culture. In 1995, Steven Heine and Darrin Lehman found that Japanese students showed much less unrealistic optimism than Canadian students. So I'd put it this way: the everyday pattern, people seeing their own future through a slightly rosier lens than the evidence supports, is well documented. The exact size of it, and some of the lab methods used to measure it, are less certain than popular books suggest.

Optimism feels good, and feeling good is useful. In a well-known 1988 review, the psychologists Shelley Taylor and Jonathon Brown argued that mildly positive illusions about ourselves, our control over events and our future are common in mentally healthy people, and seem to help them stay motivated, work harder and cope with setbacks. Someone who expected the worst at every turn might never start anything.

There's also an attention story. When we imagine our own future, we picture our plans, our intentions and our efforts. When we think about other people, we think about base rates. I know I'll be careful, so the statistics about careless people don't feel like they're about me. And the bad possibilities are vague, while the good one, the one we're actually working towards, is vivid and detailed.

Workplaces add their own pressure. Optimism sounds like commitment. In a planning meeting, the person who says "this might not work" can sound like they don't believe in the team, even when they're simply describing what usually happens.

In 2002, the UK Treasury asked the engineering consultancy Mott MacDonald to look back at fifty large public projects completed over the previous two decades, from buildings and civil engineering works to equipment and IT, and compare what they were expected to cost and how long they were expected to take with what actually happened. The answer was consistent. Projects took longer and cost more than their business cases had said, and some kinds of project were much worse than others. The causes the consultants found were rarely bad luck alone. An inadequate business case, one that hadn't properly worked out what was needed, was among the biggest contributors.

What the Treasury did next is the part I find remarkable. Rather than simply telling people to be more realistic, it wrote a correction into its official appraisal rules. Supplementary guidance to the Green Book, the Treasury's handbook for appraising public spending, published in 2003, tells officials to adjust their estimates for "optimism bias" by adding a percentage based on how similar projects had overrun in the past. For standard buildings, the suggested upper bound for capital costs is 24%. For non-standard civil engineering, such as an unusual bridge or tunnel, it's 66%. The adjustment can then be reduced, but only as the project shows evidence that its specific risks are being managed.

It isn't a perfect system, and people argue about the right percentages. But it treats optimism the way an engineer treats a known load: as a predictable force that will be there whether or not anyone believes in it. I find that a useful model for design teams. We don't need to stop hoping. We need a habit that adds back what hope leaves out.

Picture a team at a payments company in Pune building a voice feature that lets people pay bills by speaking in Marathi, Hindi or Tamil. The prototype is lovely. In the planning meeting, the forecast slide says 30% of active users will try it in the first month. Nobody can say where the 30% came from, but it feels conservative, because everyone in the room uses the prototype daily and can't imagine going back.

The launch plan has one ending. It covers the announcement, the in-app banner and a celebration for the team. It doesn't say what happens if the speech model struggles with mixed-language sentences, which is how many people actually talk. It doesn't say what happens if people feel awkward speaking a payment aloud on a crowded bus. When a researcher raises the second point, the room agrees it's interesting and moves on, because "our users are different".

Every person in that meeting is competent. The problem is that the plan was written from the inside, by people imagining their own effort, rather than from the outside, by people looking at how features like this usually go.

The better version uses a technique the psychologist Gary Klein called the premortem. Before launch, the team imagines it is three months later and the feature has clearly failed. Everyone spends a few minutes writing down, privately, why it failed. Then they read the reasons aloud. Klein drew on research by Deborah Mitchell, Edward Russo and Nancy Pennington from 1989, who found that imagining an outcome had already happened helped people come up with more reasons for it. In the premortem, the mixed-language problem and the crowded-bus problem appear in the first five minutes, from people who were too polite to raise them earlier. The forecast gets a range instead of a single number, the launch plan gets a second ending, and the team builds a quick way to measure trust before it measures adoption.

I don't want to argue anyone out of optimism. Starting a company, writing a book, designing something new or getting married all require a willingness to act as though the odds don't quite apply to you. Optimism keeps people trying after failures, and in many studies more optimistic people cope better with illness and stress. A team that expected every project to fail would never ship anything worth making.

The useful distinction, I think, is between the decision to try and the plan for trying. Optimism is a good reason to start. It's a poor basis for a budget, a deadline or a safety margin. The newlyweds in Baker and Emery's study were right to marry with hope. They might also have been wise to talk about money and families before the wedding, as if the statistics could include them.

There's also a difference between optimism about effort and optimism about luck. Believing that hard work will help is often accurate. Believing that the bad outcomes simply won't come to us is where the bias does its damage.

The first habit is to ask what usually happens. Before I trust my own forecast for a feature, I look for how similar features did, in our product or in others. My plan is special to me. To the world it's one of many.

The second is to run a small premortem whenever something matters. Ten minutes, private notes, then reasons read aloud. It's the cheapest risk review I know, and it gives quieter people permission to say what they were already thinking.

The third is to give every hopeful number a partner. If I write "30% adoption", I also write what I'll do if it's 5%. That one line turns a hope into a plan.

I still think about those couples, accurate about everyone else and certain about themselves. I hope most of them were right. In the next post I'll look at a bias that makes the things we already have seem worth more than they are: the endowment effect.

Further reading: Neil Weinstein, "Unrealistic Optimism About Future Life Events" (1980) · Lynn Baker and Robert Emery, "When Every Relationship Is Above Average" (1993) · Tali Sharot, The Optimism Bias (2011) · Shelley Taylor and Jonathon Brown, "Illusion and Well-Being" (1988) · Gary Klein, "Performing a Project Premortem" (2007)