Anecdotal Evidence

In 1978 a television crew filmed a group of teenagers on a visit to Rahway State Prison in New Jersey. The visit was run by a group of inmates serving life sentences, who had started what they called a juvenile awareness project. The teenagers, all of whom had been in some kind of trouble, were brought inside, sat down in front of the men and shouted at, mocked and threatened for hours. The inmates described in graphic detail what prison life would do to them. Some of the teenagers walked in smirking and walked out shaken.

The documentary, Scared Straight!, narrated by the actor Peter Falk, was shown on television across the United States and won the Academy Award for best documentary feature. It ended by suggesting that the visit had turned most of these young people around. The faces were unforgettable. Programmes modelled on Rahway spread to many states and other countries, sold on the strength of exactly those faces.

Around the same time, the criminologist James Finckenauer studied the Rahway programme the slow way, comparing young people who went through it with similar young people who didn't. He found no sign that it reduced offending. If anything, the group that attended did worse. Over the following decades, Anthony Petrosino and his colleagues gathered the randomised trials of programmes like it into systematic reviews. Their conclusion was blunt. On average, taking young people to be frightened by prisoners made them more likely to offend, not less. In 2011, when a television series revived the idea, the US Department of Justice publicly warned against such programmes.

The stories in the film were real. The teenagers really were frightened. What the stories could not tell anyone was what would have happened to those same young people without the visit, or to the hundreds who went through it off camera. That is the anecdotal fallacy: treating a vivid story, a testimonial or a single striking case as proof of a general claim, especially when better evidence exists. Logic textbooks also call it misleading vividness, because the problem is not that the story is false. It's that it feels like far more evidence than it is.

Arguing from a single case is one of the oldest moves in persuasion. Aristotle's Rhetoric treats the example as one of the two basic tools of a speaker, the public speaker's version of induction. You tell the assembly what happened when a neighbouring city trusted a tyrant, and you let them draw the lesson for themselves. Aristotle was clear that examples persuade, and just as clear that a rhetorical argument works on probabilities, not proof.

Two thousand years later, David Hume made the problem of testimony sharper in his essay "Of Miracles" (1748). A wise person, he argued, proportions belief to the evidence. When someone tells you a remarkable story, you have to weigh their testimony against everything else you know about how the world usually works, and ask which is more likely: that the remarkable thing happened, or that the witness is mistaken. Hume was writing about miracles, but his question applies to every testimonial on a product page.

It's worth separating this from its close relative, which I wrote about in the post on hasty generalisation. A hasty generalisation draws a big conclusion from too few cases. The anecdotal fallacy is subtler. It often happens when the good evidence is already on the table, and a single story still wins the argument. The trouble is not only the size of the sample. It's the weight that a story carries in our minds.

In 1977 the psychologists Eugene Borgida and Richard Nisbett ran a simple study with students choosing which psychology courses to take next. Some students were given the average ratings of each course, collected from dozens of students who had taken it. Others heard just a handful of students talk about the courses in person. The few face-to-face comments shaped choices more strongly than the summary of many ratings, even though the summary was plainly better evidence.

Part of the reason is memory. Amos Tversky and Daniel Kahneman showed in 1973 that we judge how common something is by how easily examples come to mind, a shortcut they called the availability heuristic. A story with a face, a place and a feeling is easy to recall. An average is not. So when we later ask ourselves "how often does this happen?", the story answers first and loudest.

The other part is that stories switch off our arguing. Melanie Green and Timothy Brock found in 2000 that when people are absorbed in a narrative, what they called transported into it, they become more accepting of the beliefs it carries and less likely to pick holes in it. A spreadsheet invites you to check it. A story invites you to feel it. In meetings this has a predictable effect. The person with the most vivid example often wins, and a single angry customer quoted word for word can outweigh a dashboard everyone has stopped looking at.

India has given me the clearest case I know of stories and numbers pulling in opposite directions. Over the last few years millions of people have started trading futures and options, contracts that let you bet on the price of shares or an index with a small amount of money. A great deal of the encouragement has come through social media: screenshots of a single day's profit, videos of young traders who quit their jobs, Telegram channels full of followers thanking a tipster for a winning call. Each one is a testimonial.

In January 2023 the Securities and Exchange Board of India, the market regulator, published a study of individual traders in equity futures and options. It found that in the financial year 2021 to 2022, about 89% of them lost money, with an average loss of around one lakh rupees. A larger study in September 2024, covering the three years to March 2024, found that about 93% of more than a crore of individual traders had made losses, adding up to well over a lakh crore rupees. SEBI has since tightened its rules on the influencers who promote trading, barring brokers and advisers from working with unregistered people who give advice or advertise returns.

Every one of those winning screenshots may have been genuine. That's what makes testimonials so persuasive and so misleading. A true story about one winner tells you almost nothing about your own chances, because the losers rarely post. The question that matters is not "did this happen to someone?" but "how often does it happen, out of how many?"

Picture a small team building an invoicing app for shop owners. On Monday the founder comes into the roadmap meeting with a story. Over the weekend she helped a friend who runs a café set up the app, and he couldn't work out how to add GST to an invoice. He got so frustrated that he went back to his old spreadsheet. She describes the moment in detail: his face, the screen, the words he used.

The room is moved. A sales colleague adds that a prospect last month also said something about taxes being confusing. Within twenty minutes, a redesign of the tax settings has jumped to the top of the quarter's roadmap, above the payment reminders feature that weeks of research had pointed to.

Look at what actually came into the room: one detailed story, and one half-remembered remark. Neither says how many users struggle with tax settings, whether the café owner was a typical user, or whether he would have found the field on his own without someone helping. The story is evidence that the problem can happen. It is not yet evidence of how often it happens or how much it matters.

The honest version treats the story as a lead. Someone searches the support inbox for "GST" and "tax". Someone checks how many new users reach the invoice screen and abandon it there. A designer runs five quick sessions with shop owners who have never seen the app, and watches whether they find the field. If the problem is common, the team now has numbers to go with the story, and the story becomes the perfect way to explain them. If it isn't, the founder's friend still deserves a better tooltip, and the payment reminders stay where the research put them. Either way, the story did its job. It pointed somewhere worth looking.

Stories are not the enemy of evidence. Sometimes a single case is decisive. If someone claims that something can never happen, one well-documented instance proves them wrong. If a payment screen crashes for one participant in a usability test, you don't need a hundred more to know it needs fixing.

Single cases are also how many important discoveries begin. In 1961 the Australian obstetrician William McBride wrote a short letter to The Lancet describing babies born with severe malformations to mothers who had taken a new sedative, thalidomide. In Germany, the paediatrician Widukind Lenz had independently become suspicious. Those case reports were anecdotes, and they helped get the drug withdrawn. Systems that collect individual reports of side effects, such as the Pharmacovigilance Programme of India, exist precisely because rare dangers often show up first as stories.

Qualitative research is built on stories too. A good interview tells you why something happens in a way no chart can. The political scientist Raymond Wolfinger is often credited with saying "the plural of anecdote is data", the reverse of the more famous version. Collected carefully, many stories can become real evidence. The fallacy lies in letting one story, chosen because it was striking, stand in for the many you haven't heard.

The first thing I ask for is the denominator. One user out of how many? One success out of how many attempts? A story with no denominator is a question, not an answer.

The second is to ask what the opposite story would look like, and whether anyone would have told it. People who didn't struggle with a feature rarely tell us about it. Traders who lost money rarely post screenshots. If the stories I hear only come from one side, their number tells me very little.

The third is to notice when I'm the one telling the story. Vividness works on the person telling it as much as on the listener. When I catch myself describing one user's frustration in great detail in a meeting, I try to follow it with what I know about everyone else, or to say plainly that I don't know yet.

The faces in Scared Straight! were real, and so was the fear in them. What the film couldn't show was the comparison group, the young people who weren't filmed and whose lives went on without a visit to Rahway. In the next post, on the false dilemma, I'll look at another way an argument hides what isn't in the picture: by offering only two choices when there are more.

Further reading: Anthony Petrosino and colleagues, "Scared Straight and Other Juvenile Awareness Programs for Preventing Juvenile Delinquency" (Campbell Systematic Review, 2013) · Richard Nisbett and Lee Ross, Human Inference (1980) · David Hume, "Of Miracles", in An Enquiry Concerning Human Understanding (1748) · Amos Tversky and Daniel Kahneman, "Availability: A Heuristic for Judging Frequency and Probability" (1973) · James Finckenauer, Scared Straight! and the Panacea Phenomenon (1982)