Hindsight Bias

In February 1972, Richard Nixon became the first American president to visit the People's Republic of China. A few months later he went to Moscow. Both trips were enormous news, and nobody knew in advance exactly how they would go. Would Nixon meet Mao Zedong in person? Would the two countries announce some kind of agreement? Would the trip be judged a success?

Just before the trips, the psychologists Baruch Fischhoff and Ruth Beyth, in Jerusalem, asked a group of students to estimate the probability of a list of possible outcomes like these. Then they waited. Some time after Nixon came home, from a couple of weeks to several months later, they went back to the same students and asked two things: what they thought had actually happened, and what probabilities they remembered giving before the trip.

The students' memories had moved. For the events they believed had happened, they remembered giving higher probabilities than they really had. For the events that hadn't happened, they remembered giving lower ones. Nobody had lied. They simply remembered their past selves as better forecasters than those selves had been. The paper, published in 1975, had a title that says it all: "I knew it would happen".

That's hindsight bias: once we know an outcome, we tend to believe it was more predictable than it really was, and we misremember what we expected beforehand. People often call it the knew-it-all-along effect. Fischhoff called it creeping determinism, the way a known outcome slowly makes everything that led to it look inevitable.

In the last post, on the endowment effect, I wrote about how owning something changes what it's worth to us. Hindsight bias does something similar to the past. Once we own the ending, the story before it rearranges itself to fit.

In a companion study, also published in 1975, Fischhoff gave people short accounts of historical events they were unlikely to know much about, such as a war in the early nineteenth century between the British and the Gurkhas of Nepal. Each account ended with one of four possible outcomes, or with no outcome at all. People who were told a particular outcome rated it as much more likely than people who weren't told anything. They also judged the details in the passage that pointed towards that outcome as the most relevant ones. And when Fischhoff asked them to answer as if they didn't know the outcome, they couldn't do it. The knowledge had already been absorbed.

Hindsight bias is one of the sturdier findings in psychology. A 1991 meta-analysis by Jay Christensen-Szalanski and Cynthia Willham, combining more than a hundred studies, found a reliable effect, though a modest one in size, and later reviews have agreed. It appears in children and older adults, in experts and novices, and in many countries. In 2012, the psychologists Neal Roese and Kathleen Vohs suggested it has three layers: memory distortion ("I said it would happen"), inevitability ("it had to happen") and foreseeability ("I could have seen it coming").

The third layer is the one that does the most damage, because it turns into blame. In a 1991 study, Robert Caplan, Karen Posner and Frederick Cheney asked more than a hundred anaesthesiologists to judge whether the care given in a series of real cases had been appropriate. The cases were identical except for the outcome, which had been altered for some readers. When the patient was described as having suffered permanent harm, the reviewers were markedly less likely to judge the very same care as appropriate.

Courts face the same problem. In 1995, Kim Kamin and Jeffrey Rachlinski asked people to decide whether a city should have paid for a flood precaution. Asked beforehand, about a quarter thought the precaution was needed. Told that a flood had happened, more than half said the city had been negligent not to take it, and a warning about hindsight bias barely changed their answers.

Part of the reason is that our memory isn't a recording. When we learn something new, we update what we know, and the old version is usually not kept. In 2000, the psychologists Ulrich Hoffrage, Ralph Hertwig and Gerd Gigerenzer argued that hindsight bias is largely a side effect of this updating. A mind that quietly replaces old beliefs with better ones is, most of the time, a good mind to have. You want to remember the correct answer, not every wrong guess you made along the way. The cost is that we lose track of how uncertain we once were.

The other part is our appetite for stories. A good story has causes that lead to an effect. Once we know the effect, we go back and pick out the causes, and a messy, uncertain moment becomes a clean narrative.

Indian politics gave a vivid example in 2004. The governing National Democratic Alliance called an early general election and campaigned on the slogan "India Shining", celebrating economic growth. Most opinion polls and exit polls predicted it would return to power. Instead, the Congress-led alliance won more seats and formed the government, with Manmohan Singh as prime minister. Within days the explanation seemed obvious to many commentators: of course a slogan about shining prosperity had alienated rural and poorer voters who hadn't felt it. That may be part of the truth, and analysts have also pointed to state-level alliances and local factors. But very few of the people who found it obvious in May had predicted it in March.

On 7 December 1941, Japanese aircraft attacked the US Pacific Fleet at Pearl Harbor, Hawaii, killing more than two thousand Americans. In the years that followed, a series of official inquiries asked how it could have happened, and afterwards the warning signs seemed everywhere. American codebreakers had been reading Japanese diplomatic messages. Among them was a request from Tokyo to its consulate in Honolulu, asking for reports on the exact positions of ships in the harbour. Senior commanders in Hawaii had received a "war warning" in late November. Both commanders there were relieved of their posts, and debates about their responsibility went on for decades.

In 1962, the historian and analyst Roberta Wohlstetter published Pearl Harbor: Warning and Decision, which won the Bancroft Prize. Her argument was that the problem was not a lack of signals. It was that the relevant signals were buried in a much larger volume of what she called noise: other alarms, other likely targets, other messages pointing in different directions. Japan's consulates in many ports were sending similar reports. The most expected danger in Hawaii was sabotage, not an air attack. After the attack, investigators knew which signals mattered and could line them up into an obvious pattern. Before it, the same signals were a few threads in a heap.

Wohlstetter wasn't saying nobody made mistakes. She was saying that judging the people involved by what we know now, rather than what they could reasonably have known then, gives us the wrong lessons. If we think the signals were obvious, we'll conclude the fix is to stop being careless. If we see how they were buried, we'll work on how organisations sort signals from noise.

Picture a team that launched an AI feature which suggests replies to customer messages for small online sellers. Six weeks later, usage is a fraction of what they expected. The retrospective starts with a timeline on the wall, and very quickly someone says, "Honestly, we should have known sellers wouldn't trust auto-replies with their customers." Everyone nods. A researcher's comment from an early interview, one participant saying she'd "never let a robot talk to my buyers", is pulled up and read aloud. It now looks like a prophecy.

By the end of the hour, the lesson is that the product manager ignored the research. Nobody mentions that the same interview round produced nine other concerns, most of which turned out not to matter, or that two sellers in the pilot used the feature constantly. Nobody remembers that the researcher herself had rated trust as a medium risk at the time. The story has a cause, a culprit and a moral, and it's mostly built after the fact.

The danger isn't only unfairness to the product manager. It's that the team learns the wrong lesson. "Listen harder to the research" sounds wise, but they did listen, to ten concerns at once, and had no way to know which one mattered most.

The better version starts before launch. The team keeps a short decision log: what they decided, what they expected, how confident they were and which risks they knew about. In the retrospective, they read the log first, before anyone looks at the outcome data. The questions change. Instead of "how did we miss this?", they ask "what did we know, how sure were we, and was the decision reasonable given that?" Sometimes the answer is that it was a good decision with an unlucky result. Sometimes it's that a real warning was ignored. The log lets them tell the difference, and the lesson becomes concrete: test the riskiest assumption, such as trust, with a cheap experiment before building the whole feature.

Hindsight itself isn't the problem. Looking back is how we learn. After a failure, we know things we didn't know before, and we should use them. Updating our beliefs in the light of outcomes is what good learners do, which is why Hoffrage and his colleagues saw the bias as the price of a useful habit.

And sometimes the outcome really was foreseeable. When a warning was clear, specific and repeated, and someone chose to ignore it, calling that hindsight bias would be an excuse. The test is whether the signal stood out at the time, or only stands out now that we know the ending.

There's a related trap called outcome bias: judging a decision by its result rather than by its quality. A risky bet that happened to pay off isn't proof of wisdom, and a sensible plan that met bad luck isn't proof of foolishness. Hindsight bias makes outcome bias easier, because once we know the result, the decision seems to have been obviously right or obviously wrong.

The first habit is writing predictions down. Before a test or launch, I note what I expect and how sure I am, even in one line. Later, my memory can't quietly edit it.

The second is a small ritual before research readouts. I ask the room to write down what they expect the results to show before I share them. Afterwards, nobody can say "we already knew that" without checking their own note, and it's often humbling, for me most of all.

The third is a question I bring to every retrospective: what did we actually know at the time we decided? I try to separate the signals we had from the ones we only recognise now, and to judge the decision by the first set.

I often think of those students in Jerusalem, sure they'd seen Nixon's trips coming, their old forecasts quietly rewritten in their heads. In the next post I'll look at another way we misjudge other people's decisions, by blaming their character and forgetting their situation: the fundamental attribution error.

Further reading: Baruch Fischhoff, "Hindsight ≠ Foresight: The Effect of Outcome Knowledge on Judgment under Uncertainty" (1975) · Baruch Fischhoff and Ruth Beyth, "'I Knew It Would Happen': Remembered Probabilities of Once-Future Things" (1975) · Roberta Wohlstetter, Pearl Harbor: Warning and Decision (1962) · Neal Roese and Kathleen Vohs, "Hindsight Bias" (2012) · Annie Duke, Thinking in Bets (2018)