The Planning Fallacy

In January 1957, a thirty-eight-year-old Danish architect called Jørn Utzon won an international competition to design an opera house on Bennelong Point, a spit of land in Sydney Harbour. His entry was little more than a set of sketches: a cluster of white shells, like sails, rising from a great stone platform. The government of New South Wales put the cost at £3.5 million, about 7 million Australian dollars in the decimal currency that came later, and hoped to finish in early 1963.

Construction began in 1959, before anyone knew how the shells could actually be built. The state premier, Joseph Cahill, was keen to start quickly, reportedly so that the project would be too far along for a future government to cancel. Working out the geometry of the shells took Utzon and the engineers at Ove Arup years; the answer, cutting every shell from the surface of one imaginary sphere, came only in the early 1960s. Costs climbed, politics turned, and in 1966, after a dispute with a new state government over payments and design, Utzon resigned and left Australia. He never returned to see the building finished. A team of Australian architects completed the interiors, and Queen Elizabeth II opened the Sydney Opera House in October 1973, ten years late. It had cost 102 million dollars, roughly fourteen times the original estimate.

Most of us will never design anything so ambitious, but we make the same kind of forecast every week. The report will be done by Friday. The redesign will take one sprint. Daniel Kahneman and Amos Tversky gave the pattern a name in 1979: the planning fallacy, our tendency to underestimate how long a task will take, and often what it will cost, even when we know that similar tasks in the past ran over.

That last part is what makes it a fallacy rather than simple ignorance. It's not that we lack information about how things usually go. It's that we don't use it. I wrote earlier in this series about optimism bias and about overconfidence. The planning fallacy is their practical cousin, specific to our own plans.

In 1994, Roger Buehler, Dale Griffin and Michael Ross asked final-year psychology students in Canada when they expected to finish their honours theses, the long research project at the end of their degree. On average, the students predicted 34 days. They took about 55. Fewer than a third finished by the date they'd predicted. The researchers also asked for a pessimistic estimate, assuming everything went as badly as it possibly could. The average was 49 days, and even that wasn't pessimistic enough. Most students took longer than their own worst case.

The researchers also listened to how people predicted. Asked to think aloud, students talked almost entirely about the future: the steps of the plan, the chapters still to write. They hardly mentioned their own past projects, though most admitted those had run late. When they did remember past delays, they explained them away as one-off events, an illness or a broken laptop, that surely wouldn't happen again. People predicting how long someone else would take were more pessimistic, which suggests the problem lies partly in being the one with the plan.

Kahneman had learned this himself. In the 1970s he worked with a team in Israel writing a curriculum and textbook on judgement and decision-making for high schools. About a year in, he asked everyone to estimate how long the project would take to finish, and the guesses clustered around two more years. Then he asked the team's curriculum expert, Seymour Fox, how long similar projects had taken other teams. Fox admitted that about 40 per cent of such teams never finished at all, and those that did took seven to ten years. They carried on anyway. The book took eight more years, and by then the ministry had lost interest. Kahneman called the first kind of forecast the inside view, built from the details of your own case, and the second the outside view, built from what happened to cases like yours. He and Dan Lovallo set out the distinction in 1993.

The finding has been reproduced many times, for tasks from tax returns to software, though for very short, simple tasks people sometimes overestimate instead, and we may also misremember how long past tasks took.

When we picture a project, we picture it going right. We imagine the steps in order, each taking a sensible amount of time, and we add them up. That's the inside view, and it comes naturally, because the plan is what's in front of us. What we can't picture are the things that will go wrong, because we don't yet know what they are. In a project of twenty steps where each step has a small chance of delay, some delay somewhere is almost certain, yet no single delay ever seems likely enough to plan for. The cognitive scientist Douglas Hofstadter turned this into a joke in 1979, a law named after himself: a task always takes longer than you expect, even when you allow for the fact that it always takes longer than you expect.

Bent Flyvbjerg, who has spent decades studying large projects, adds a less innocent reason. Low estimates win approval. A project that admits it will cost twice as much may never be funded, so the plans that survive the competition are often the most optimistic ones. He calls this strategic misrepresentation.

The inside view isn't useless. Breaking a task into steps is how we find out what it involves, and a little optimism is what gets anyone to start. It works for short, familiar tasks. It fails on large, new, one-off projects, where the steps we can't see are the ones that matter.

Berlin Brandenburg Airport was meant to replace the city's ageing airports, and construction began in 2006. Its opening was planned for 2011, then moved to June 2012. Then, less than a month before the date, the opening was called off. The fire protection system, including complicated machinery for extracting smoke from the terminal, didn't work and couldn't be approved. Years of redesign, faulty wiring and missed target dates followed. The airport finally opened on 31 October 2020, in the middle of the pandemic, to almost empty halls. Its cost had risen to roughly three times the early budget, around six billion euros by most estimates.

Flyvbjerg's database of more than sixteen thousand projects suggests Berlin is unusual only in degree. In How Big Things Get Done, written with Dan Gardner, he reports that only about 8.5 per cent of projects came in on budget and on time, and about half a per cent delivered on budget, on time and with the benefits promised. His remedy is reference class forecasting, a formal version of Kahneman's outside view. Find a set of similar finished projects, look at how far their actual costs and times strayed from their original estimates, and use that as your starting point before adjusting for anything genuinely special about your case. Britain's Department for Transport adopted the method for large transport schemes in the 2000s.

India has no shortage of overrunning projects. The Ministry of Statistics and Programme Implementation publishes regular reports on central government projects costing ₹150 crore or more, and at any time hundreds of them are running late. The exceptions are worth studying. The first phase of the Delhi Metro was finished in 2005 under its managing director E. Sreedharan, well ahead of its deadline and within its budget. Accounts of how it was done describe an organisation that treated time as a hard limit rather than a hope: small teams with authority to decide on the spot, contractors held strictly to dates, and, reportedly, clocks in the offices counting down the days to each deadline. Later phases haven't all gone as smoothly, but the first one showed that the fallacy isn't a law of nature.

Picture a product team at a payments company about to redesign its onboarding, the screens that take a new user from download to a verified account. The designer, the engineering lead and the product manager sit together and break the work into tasks: research, wireframes, visual design, building eight screens, connecting to the identity-verification service, testing. Each person estimates their own part. Added up, it comes to six weeks, and they commit to launching before Diwali.

Everything in that plan is reasonable, and almost everything is missing. The legal team needs to review every line of consent text, and they're busy. The identity-verification provider changes its interface halfway through. Two designers take leave for family weddings. Accessibility testing finds problems with the form on older Android phones. The festival freeze on releases starts a week earlier than anyone remembered. None of these was in the plan, because nobody could have named them on day one. The redesign launches after fourteen weeks, well after Diwali.

The better version starts outside the project. Before anyone breaks down tasks, someone pulls up the last five comparable projects at the company: how long each was estimated to take, and how long it really took. Say they ran, on average, about twice their estimates. That ratio becomes the starting point: twelve weeks, not six. Then the team breaks the work down, and here some research helps. In 2004, Justin Kruger and Matt Evans found that asking people to list the smaller parts of a task, the meetings, approvals and checks as well as the main work, made their estimates longer and more accurate. Finally, the team shares the estimate as a range, ten to fourteen weeks, with a checkpoint at week four to compare the plan with reality.

In 1967, the economist Albert Hirschman made a provocative argument he called the principle of the Hiding Hand. Studying development projects, he noticed that many succeeded only because their planners had underestimated the difficulties. Had they known the true cost, they would never have started. Having started, they found creative ways around problems they hadn't foreseen. Ignorance of the obstacles, he suggested, can be a gift. The Sydney Opera House is the example everyone reaches for: a financial and personal ordeal on the way, and now a World Heritage Site and the image of a city.

In 2016, Flyvbjerg and the legal scholar Cass Sunstein replied that this is survivorship in disguise. For every Opera House there are many projects where the hiding hand hid nothing good, and people were simply left with the bill. They called it the malevolent hiding hand. I find both arguments useful. Sometimes a bold, underestimated project is worth doing anyway, but that should be a choice made with open eyes, not an accident of bad forecasting.

The first thing I do is look backwards before I look forwards. I keep a simple log of my estimates and how long things actually took. My personal multiplier for anything involving other teams is close to two, and knowing that number is more useful than any amount of careful planning.

The second is to unpack the boring parts. When I estimate, I write down every review, approval, handover, holiday and dependency I can think of, not just the design work I enjoy imagining.

The third is to give dates as ranges, and to name a checkpoint where we'll compare the plan with reality while there's still time to change course.

Utzon's building was finished without him, a decade late and many times over budget, and it's one of the most loved buildings of the twentieth century. The lesson isn't that the estimate didn't matter. Everyone involved, the people who paid and the architect who left, would have been better served by a forecast that started from how hard such buildings really are. In the next post I'll look at what happens to a belief after the evidence for it has been taken away: belief perseverance.

Further reading: Bent Flyvbjerg and Dan Gardner, How Big Things Get Done (2023) · Roger Buehler, Dale Griffin and Michael Ross, "Exploring the 'Planning Fallacy'" (1994) · Daniel Kahneman and Dan Lovallo, "Timid Choices and Bold Forecasts" (1993) · Daniel Kahneman, Thinking, Fast and Slow (2011) · Albert O. Hirschman, "The Principle of the Hiding Hand" (1967)