Explain three well-documented bias cases and what each one teaches.
Before reading on, write the four routes of bias from lesson 3.1, "Where bias in AI comes from", down one side of a sheet of paper. Keep it beside you, and tick the route or routes you see at work in each of the three cases below. Claims that an AI system is biased are easy to brush off as theory or opinion. These three were investigated, documented and reported, and each failed in its own way. Each also leaves you with a check you can use.
Reuters reported in 2018 that Amazon had built an experimental tool for rating job applicants' CVs, and that the company later dropped it. As reported, the tool learned from CVs sent to Amazon over about ten years. Men had written most of them, since men made up most of the technology industry's workforce in that period, and the system picked up that pattern.
Reuters said the tool gave lower ratings to CVs with the word "women's" in them, such as "women's chess club captain", and to people who had graduated from two all-women's colleges. Amazon's engineers changed the system to ignore those particular terms, but they had no way to be certain it wouldn't sort candidates with the same effect by some other means. According to people familiar with the project who spoke to Reuters, Amazon's recruiters looked at the tool's recommendations when searching for new hires but never relied on those rankings alone. Amazon's own statement, that the tool was never used by its recruiters to evaluate candidates, was reported by other outlets such as Quartz.
Two routes from lesson 3.1 are working together here: route one, learning from the past, and route two, proxies. No one instructed the system to favour men. It drew that preference out of the historical CVs, then latched onto words that did the work of gender for it. A proxy is a word or feature standing in for a protected characteristic like gender; here "women's" and the names of the two colleges played that role.
For Farah, the HR executive from lesson 3.1 (an invented example), the point applies directly. If her company trains a CV ranking tool on the people it hired before, the tool takes on the choices the company made before.
Researchers Joy Buolamwini and Timnit Gebru published Gender Shades in 2018. They ran commercial facial analysis systems from several large technology companies on a task the companies sold them for: deciding the gender of a face in a photo.
Test sets in common use then were not balanced by gender and skin tone, so the two researchers assembled one that was. On lighter-skinned men the systems did very well. On darker-skinned women their accuracy dropped sharply, with error rates far higher than for every other group. The accuracy numbers the companies advertised never showed this, because the standard test sets held mostly lighter-skinned faces.
The case is an instance of route three, gaps in the data: training or test data with too few examples of some groups. It also exposes a problem with testing. When test data shares the gaps of the training data, a system can earn a high score while failing badly for an entire group. The companies went on to release updated systems, and the study shifted how many researchers and companies approach measuring accuracy.
ProPublica, a US news organisation, published its analysis of COMPAS in 2016. COMPAS is a commercial tool that parts of the American justice system use to estimate how likely a defendant is to reoffend. ProPublica took the scores given to people arrested in one Florida county and checked them against who actually reoffended in the years that followed.
It found that black defendants who did not reoffend were almost twice as likely as white defendants to have been labelled higher risk. White defendants who did reoffend, it reported, were more often labelled lower risk. The company that makes COMPAS rejected the analysis, saying that by its own measure the scores were equally accurate for both groups.
Each side had a point, and that is the deeper lesson here. Later research showed that when groups have different underlying rates, several common definitions of fairness cannot all be satisfied at the same time. Deciding which one counts is a judgement about values that people have to make; a tool cannot make it for them. So when a vendor calls a system "fair", find out which definition of fairness they used before giving the word any weight.
Hiring, faces and courts have little in common, yet in each case the problem came to light the same way, through outcomes broken down by group: results compared for each group separately. Someone looked at whose CVs were marked down, whose faces were misclassified and who was wrongly labelled high risk. Overall accuracy did not reveal the problem in any of the three.
Before an AI tool is used to make decisions about people, someone needs to test how it behaves for different groups and set those results side by side, looking past the overall accuracy figure. This has to happen before the tool is in use. Once complaints arrive, the decisions have been made and people have already been affected.
You will rarely run studies on the scale of these three. You can put two questions to the vendor: have you tested results across groups, and can we see them? Farah put exactly those questions to her vendor, and they are open to you too. Where a vendor describes the system as fair, ask which definition of fairness sits behind the claim. For your everyday use of AI assistants, the next lesson sets out a small version of the same test.
With that check in mind, go back through the three cases and ask: at what point, before any harm was done, could a simple check have shown the problem?
For each case, write one sentence on where the bias came from and one check that could have caught it sooner.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).