Stereotyping Bias

There's an old riddle that goes roughly like this. A father and his son are in a car accident. The father dies at the scene, and the boy is rushed to hospital. The surgeon on duty looks at him and says, "I can't operate on this boy. He's my son." How can that be?

In 2014, the psychologist Deborah Belle and her student Mikaela Wapman at Boston University put the riddle to 197 psychology students and 103 children aged seven to seventeen. The answer, of course, is that the surgeon is the boy's mother. Only about 14 per cent of the students and 15 per cent of the children thought of it. Many came up with something far more elaborate: a stepfather, a second father, or a dream. Even among students who described themselves as feminists, more than three-quarters missed it. The team later published the work, with more studies, in the journal Sex Roles in 2021.

Nobody in that study believed women can't be surgeons. But when a sentence said "surgeon", their minds supplied a man before they had time to think, and the picture was so quick and so confident that it crowded out the obvious answer.

That's stereotyping bias: judging an individual by beliefs about a group we've placed them in, and letting those beliefs fill in what we don't actually know. A stereotype can be about gender, age, caste, religion, region, accent or profession. It's a close neighbour of the fundamental attribution error, which I wrote about earlier in this series, but the two are different. The attribution error explains someone's behaviour by their character instead of their situation. Stereotyping fills in the person from their category, often before they've done anything at all.

The word comes from printing. By the end of the eighteenth century, printers were casting a solid metal plate from a mould of set type, so that a whole page could be printed again and again without resetting it. The French called the plate a stéréotype, from the Greek for "solid" and "impression". In 1922, the American journalist Walter Lippmann borrowed the word in his book Public Opinion for the "pictures in our heads" we carry about people we've never met. He meant it partly as a description: we can't experience the whole world first-hand, so we lean on impressions printed once and reused.

Psychologists have long tested how those pictures work. In 1989, Patricia Devine showed that people who score low on prejudice still know the common cultural stereotypes as well as anyone else, and that the stereotypes come to mind automatically. Where people differ is in what happens next: whether they notice the picture and correct it. In the post on the division fallacy I mentioned Gordon Allport's argument that thinking in categories is unavoidable. Devine's work suggested that the category arrives first, and fairness, if it comes, takes effort.

The clearest evidence that this matters outside a laboratory comes from field experiments. In a study published in 2004, the economists Marianne Bertrand and Sendhil Mullainathan sent about five thousand fictitious CVs in response to job advertisements in Boston and Chicago. The CVs were randomly given names that sounded typically white, such as Emily or Greg, or typically African American, such as Lakisha or Jamal. The white-sounding names received about 50 per cent more calls for interviews. In 2017, a meta-analysis led by Lincoln Quillian pooled every comparable American hiring experiment since 1989 and found no sign that the gap had shrunk in twenty-five years.

I should be honest that not every famous finding in this area is as solid. The Implicit Association Test, introduced by Anthony Greenwald and colleagues in 1998 and taken by millions online, measures how quickly people pair groups with good or bad words. But a person's score can change a lot between sittings, and meta-analyses have found that it predicts real behaviour only weakly. Stereotype threat, the idea that reminding people of a negative stereotype about their group harms their performance, began with a striking 1995 study by Claude Steele and Joshua Aronson. Later work found signs of publication bias, and a large preregistered study in Dutch schools in 2018 found no effect. The callback experiments, by contrast, have been repeated in many countries.

In 1994, Neil Macrae, Alan Milne and Galen Bodenhausen asked people to form impressions of several imaginary individuals from lists of personality traits while listening to a talk about the geography and economy of Indonesia. When the trait lists came with a group label attached, people remembered more of the traits and also more of the talk. The label let them sort the information quickly, which freed attention for everything else. The paper's title called stereotypes "energy-saving devices".

That's the shortcut underneath. Categories are how we cope with a world full of strangers. If I see someone in a uniform at a railway station, it's useful to assume they can tell me which platform my train leaves from. Most of the time, treating a new thing as an example of a familiar kind is exactly right.

The cost shows up when we're tired, rushed or overloaded, which describes most teams the week before a launch. Bodenhausen found in 1990 that people who described themselves as morning types leaned more on stereotypes when tested in the evening, and night owls did so in the morning. The same shortcut that saves effort spends fairness, and it spends more of it when we have less attention to give.

India has its own field evidence. In a study published in the Economic and Political Weekly in 2007, the economist Sukhadeo Thorat and the sociologist Paul Attewell responded to job advertisements from private companies with applications from well-qualified graduates that were matched in every way except the name. Each name signalled a high-caste Hindu, a Dalit or a Muslim applicant. The odds of the Dalit applicant being invited to the next stage were about two-thirds of the odds for the high-caste applicant with the same qualifications. For the Muslim applicant, they were about one-third.

A second study in Delhi, by Abhijit Banerjee, Marianne Bertrand, Saugato Datta and Sendhil Mullainathan, published in 2009, sent 3,160 fictitious CVs to software and call-centre openings. For software jobs, they found no significant gap by caste. For call-centre jobs, they did, mainly for applicants from Other Backward Classes, and they found no gap for Muslim names in either sector. I find the two more useful together than apart. They used different jobs and methods and don't fully agree, which is why one study shouldn't settle the question either way. Both measure only the first step of hiring, and both show that a name alone can change what happens to a person before anyone has met them.

Stereotypes don't only live in people. They can be learned by the systems we build. In 2018, Reuters reported that Amazon had spent several years building an experimental model to rate job applicants. It was trained on CVs the company had received over the previous ten years, most of them from men, because the tech industry was mostly men. The model learned to mark down CVs containing the word "women's", as in "women's chess club captain", and to downgrade graduates of two women's colleges. Engineers edited it to ignore those terms but couldn't be sure it wouldn't find other proxies, and the project was dropped. Amazon said its recruiters never used the tool to evaluate candidates. Nobody typed a stereotype into that model. It found the pattern in past decisions and printed it again, which is almost exactly what a printer's plate does.

Picture a team designing bill payments for a bank's app. They've interviewed fourteen customers, and now they're in a room turning sticky notes into personas. One persona takes shape quickly: a retired man of sixty-eight who finds apps confusing and asks his son for help. Someone finds a stock photo of a kind-looking grandfather.

Now look at the notes the persona came from. Three of the fourteen participants were over sixty. One of them did struggle and had his son set up payees. The other two paid electricity, phone and gas bills for several relatives every month and used UPI more fluently than the junior designer taking notes. Meanwhile, two participants in their twenties said they found adding a new payee frightening, because they'd heard about scams and didn't trust themselves to spot one. None of that made it onto the persona. The grandfather was built from one participant and everyone's assumptions.

The persona then shapes the product. The team proposes a "simple mode" switched on automatically for customers over sixty, using the age from their bank records. It hides scheduled payments. The two fluent older customers lose the features they use most, and the nervous young customers get no reassurance at all, because the design has decided that age is what makes people need help.

The better version builds the persona from behaviours and needs rather than demographics: "pays bills for other people", "unsure about new payees". It records how many participants showed each pattern and keeps age as a fact about people, not the explanation for them. Simple mode becomes something anyone can choose, with a gentle offer when the app sees someone hesitating on a new payee. And before any persona goes on the wall, someone asks which of its details came from the notes and which came from the room.

Not every generalisation about a group is a bias. Some describe real differences in averages. On average, people's eyes lose contrast sensitivity as they age, so choosing a readable default text size and stronger contrast for a pension app is good design, not stereotyping. The psychologist Lee Jussim has argued that many stereotypes match real group averages better than psychologists once assumed, a point worth taking seriously.

The trouble is the step from the group to the person, which is the division fallacy in another form. A true average can still be a terrible guess about any individual, because the spread within a group is usually much wider than the gap between groups. In 1980, Anne Locksley and her colleagues found that once people had even a little specific information about an individual's behaviour, the group label stopped affecting their judgement almost entirely. That suggests a practical rule. Design for the whole range of a group, set defaults from real data, let each person adjust, and the moment you have information about the actual person, let it win.

The first habit is a swap test. When I catch myself making an assumption about a user, a colleague or a candidate, I swap the group and ask whether I'd make the same assumption about a different kind of person. If a sixty-eight-year-old needs a simple mode, does a nervous twenty-four-year-old? If the answer changes, the assumption came from the label.

The second is to separate what I saw from what I filled in. In research notes I keep two columns, one for what a participant actually said or did, and one for my interpretation. Personas only get built from the first column. When I review portfolios, I look at the work before the name and the college.

The third is to slow down at the moments that matter. Stereotypes thrive when I'm tired, so I try not to make decisions about people, such as hiring or who a feature is "for", at the end of a long day. And when a model makes decisions about people, I check its outputs group by group, because a stamp can be inherited without anyone meaning to press it.

I think about the surgeon riddle often. The answer isn't hard. It's just slower than the picture that arrives first. In the next post I'll look at a shortcut that judges how common something is by how easily an example comes to mind: the availability heuristic.

Further reading: Walter Lippmann, Public Opinion (1922) · Marianne Bertrand and Sendhil Mullainathan, "Are Emily and Greg More Employable than Lakisha and Jamal?" (2004) · Sukhadeo Thorat and Paul Attewell, "The Legacy of Social Exclusion: A Correspondence Study of Job Discrimination in India" (2007) · C. Neil Macrae, Alan Milne and Galen Bodenhausen, "Stereotypes as Energy-Saving Devices" (1994) · Deborah Belle and colleagues, "I can't operate, that boy is my son!: Gender schemas and a classic riddle" (2021)