The Overconfidence Effect

On 28 January 1986, the space shuttle Challenger broke apart seventy-three seconds after lifting off from Florida, and all seven people on board were killed. It had been an unusually cold morning. The investigation found that the cause was a failed seal in one of the solid rocket boosters: rubber O-rings that had stiffened in the cold and let burning gas escape. Engineers at Morton Thiokol, the company that made the boosters, had argued the night before that it was too cold to launch. They were overruled.

The physicist Richard Feynman was appointed to the presidential commission that investigated the disaster. Alongside the work on the seals, he kept asking a simpler question: how likely did NASA think a disaster was in the first place? He put what he found into an appendix to the commission's report, titled "Personal Observations on the Reliability of the Shuttle". Estimates of the chance of losing a shuttle and its crew, he wrote, ranged from roughly 1 in 100 to 1 in 100,000. The higher figures came from the working engineers. The very low ones came from management.

Feynman worked out what the management figure meant. At 1 in 100,000, you could launch a shuttle every day for about three hundred years and expect to lose only one. The shuttle's own record of worrying near misses suggested nothing of the kind. In the end the shuttle flew 135 missions, and two of them, Challenger in 1986 and Columbia in 2003, were lost with their crews. That's about one in seventy, far closer to the engineers' figure than to the official one.

This is the overconfidence effect: being more certain of our judgements than their accuracy justifies. It isn't the same as optimism bias, which I wrote about earlier in this series. Optimism is about expecting good things. Overconfidence is about certainty, and you can be overconfident about a bad outcome too. It's also different from the planning fallacy, our habit of underestimating time and cost, which gets its own post shortly. Overconfidence is simply the gap between how sure we feel and how often we turn out to be right.

One of the cleanest demonstrations came from a classroom. In 1969, Marc Alpert and Howard Raiffa at Harvard asked students, many of them on the MBA programme, to estimate quantities they couldn't know exactly, the kind of figures you'd find in an almanac or a table of economic statistics. Instead of a single guess, each student gave a range: a low number they were 99 per cent sure the true value was above, and a high number they were 99 per cent sure it was below. If the students had been well calibrated, the true answer would have fallen outside their ranges only about 2 per cent of the time. It fell outside roughly 40 per cent of the time. The paper was finally published in 1982, in the collection Judgment under Uncertainty.

Through the 1970s, Sarah Lichtenstein, Baruch Fischhoff and Lawrence Phillips studied the same question, which they called calibration. A person is well calibrated if, of all the things they say they're 80 per cent sure about, about 80 per cent turn out to be true. On general-knowledge questions, people were reliably overconfident, and the gap was largest on hard questions. On very easy questions, interestingly, people were often slightly underconfident. Psychologists call this the hard-easy effect.

In 2008, Don Moore and Paul Healy pointed out that researchers had been using "overconfidence" for three different things. Overestimation is thinking you're better than you are. Overplacement is thinking you're better than other people. Overprecision is being too sure that you know the truth, the narrow ranges of Alpert and Raiffa's students. The first two change direction with the task, as the hard-easy effect suggests. Overprecision is the most stubborn of the three. It turns up across tasks, countries and levels of expertise.

You may be expecting the Dunning-Kruger effect here, the 1999 finding that the weakest performers overestimated themselves most. Its data were real, but several researchers have since argued that much of the famous pattern can appear from statistics alone, through regression to the mean, so I treat it more cautiously than its reputation suggests.

Part of the answer, as Kahneman argued in Thinking, Fast and Slow, is that confidence is a feeling about the story, not about the evidence. When the facts we have fit together neatly, we feel sure, and we rarely ask how many facts we don't have. A small, consistent set of clues often feels more convincing than a large, messy one, even when the messy one is closer to the truth.

Feedback is the second part. Most of our judgements are never scored. We estimate, decide and move on, and by the time the outcome arrives we've forgotten what we predicted, or we remember predicting what actually happened. Without a record, there's nothing to correct the feeling.

The third part is social. Confidence persuades. In a 2012 study, Cameron Anderson and his colleagues at Berkeley found that people who were overconfident about their abilities were seen by others as more competent and gained more status in groups, even when they weren't more competent at all. If confidence wins arguments and promotions, there's little pressure to tone it down.

None of this is foolish. Feeling sure lets us commit, move quickly and reassure the people who depend on us, and most everyday judgements don't need a carefully measured range. The trouble starts when that feeling is applied to rare, expensive, irreversible decisions, exactly where feedback is scarcest.

Long-Term Capital Management was founded in 1994 by John Meriwether, a celebrated bond trader from Salomon Brothers. Its partners included Myron Scholes and Robert Merton, two economists who shared the 1997 Nobel Prize in economics for their work on pricing options. The fund's strategy was to find tiny price differences between closely related bonds, bet that the gaps would close, and borrow heavily to make those tiny gains worthwhile. For its first few years it was spectacularly successful, returning around 40 per cent a year after fees in 1995 and 1996.

Its risk models were built on years of market data, and by their reckoning a really large loss was extraordinarily unlikely. Then, in August 1998, Russia defaulted on its debt. Investors everywhere rushed into the safest assets, and the gaps the fund had bet would narrow grew wider instead. Because it had borrowed so much, roughly twenty-five dollars for every dollar of its own capital, small moves became enormous losses. It lost around $4.6 billion in a few months. In September 1998, the Federal Reserve Bank of New York brought together fourteen banks, which put in $3.6 billion to take over the fund and wind it down slowly, for fear that a sudden collapse would spread through the financial system.

The people involved weren't careless. They were among the most able people in finance. What failed was the precision of their certainty: a model of the past was treated as a narrow, reliable range for the future, and the borrowing left no room for being wrong. Roger Lowenstein's When Genius Failed tells the story in detail. Intelligence didn't protect anyone. If anything, it made the narrow range easier to believe.

Picture a team at a software company building an AI feature that reads incoming support tickets and sends each one to the right team: billing, delivery, technical faults and so on. The data scientist tests it on three thousand past tickets and reports 92 per cent accuracy. In the roadmap review, the product lead tells the leadership group that the feature will route "about 92 per cent" of tickets correctly, and that the support team can be planned around that. Nobody asks how sure the number is. It sounds precise, so it sounds known.

Now look at what sits underneath it. The test tickets came from the last two months, all written in English, before a new product line launched. Real customers write in Hindi, Tamil and Hinglish, paste screenshots, and complain about two things at once. The 92 per cent was a fair measurement of one sample. Treating it as a forecast for every future ticket turned a point on a chart into a promise. In the first week after launch, the feature routes about 78 per cent of tickets correctly and the support queue backs up.

The honest version takes about ten more minutes. The data scientist reports a range instead of a point: on tickets like the test set, somewhere between 89 and 94 per cent; on the mixed, messier tickets the company actually receives, unknown, possibly much lower. The team lists what could push it outside that range. They launch to 5 per cent of tickets first, measure, and narrow the range based on what they see. Nobody had to be less ambitious. They just had to stop treating a measurement as a promise.

Some people are remarkably well calibrated, and they share one thing: fast, frequent, honest feedback. Weather forecasters are the classic example. In the 1970s, Allan Murphy and Robert Winkler studied forecasters at the United States National Weather Service and found that when they said there was a 30 per cent chance of rain, it rained on close to 30 per cent of those days. They make the same kind of prediction every day and find out within hours whether they were right.

The India Meteorological Department does something I admire every April, when it issues its first forecast for the south-west monsoon. Rather than announce a single number, it gives the season's expected rainfall as a percentage of the long-period average, with a model error of a few percentage points either way, and probabilities for each category from deficient to excess. Farmers, ministries and markets plan around that forecast, and the range tells them how much weight it can bear. That's what calibrated confidence looks like in public.

Confidence has uses beyond accuracy, too. A design lead who hedges every sentence of a pitch may never get the project funded. A little overconfidence may help people persist and persuade, with one condition: the confidence should be in the decision to try, not in a narrow prediction of how it will turn out. "I'm confident this is worth doing" and "I'm confident this will work" are different sentences, and usually only the first is defensible.

The first habit is to replace points with ranges. When I estimate anything that matters, I give a low and a high number I'd be genuinely surprised to fall outside. Then I widen it, because Alpert and Raiffa's students taught me that a first range is almost always too narrow.

The second is a prediction log. Before a usability test or a launch, I write down what I expect and how sure I am, in a dated note. Reading it back a month later is uncomfortable and useful. It's the closest I've come to the kind of daily feedback forecasters get.

The third is to separate the speaker's confidence from the evidence. When someone, including me, says "I'm sure", I ask what it's based on and how often that kind of evidence has been right before. A confident voice in a meeting tells me something about the person. It doesn't yet tell me much about the world.

Feynman ended his appendix by saying that reality must take precedence over public relations, "for nature cannot be fooled". The engineers' 1 in 100 wasn't a gloomy number. It was a calibrated one, and it was nearer the truth. In the next post I'll look at what happens after we've committed to something and the evidence starts to turn against it: sunk cost bias.

Further reading: Richard Feynman, "Personal Observations on the Reliability of the Shuttle", Appendix F of the Report of the Presidential Commission on the Space Shuttle Challenger Accident (1986) · Don Moore and Paul Healy, "The Trouble with Overconfidence" (2008) · Don Moore, Perfectly Confident (2020) · Roger Lowenstein, When Genius Failed (2000) · Marc Alpert and Howard Raiffa, "A Progress Report on the Training of Probability Assessors" (1982)