14.8. Case Studies and Examples

Amazon developed a hiring AI that showed bias against women. It had learned from past hiring data dominated by male candidates, which led to the system penalizing resumes with the word "women" (e.g., “women’s chess club”).

The Amazon recruitment AI tool was created to automate the hiring process, but it ended up being biased against female applicants. This happened because the AI learned from historical hiring data, which primarily featured male candidates. As a result, it began penalizing resumes that included terms associated with women, such as involvement in women-focused groups. This case highlights how AI can unintentionally reinforce existing biases present in training data, leading to unfair treatment based on gender.

Imagine a high school where a principal, when selecting students for a leadership program, only considers applicants from the football team. If most football players are boys, it would unfairly disadvantage girls. This is similar to how the Amazon AI worked: it inadvertently favored resumes from men because it learned from data that did not adequately represent women.

COMPAS is a software used to predict criminal reoffending in the U.S. It was found to predict higher risk scores for Black defendants than White ones, even when actual reoffending rates were similar.

The COMPAS algorithm is designed to assess the likelihood of a person reoffending, which is crucial in judicial settings for decisions about bail or sentencing. However, investigations revealed that the algorithm disproportionately assigned higher risk scores to Black defendants compared to White defendants, even when their reoffending rates did not differ significantly. This discrepancy indicates that the algorithm may have inherited biases from the data it was trained on, leading to unfair judicial outcomes based on race.

Think of a teacher who grades students based only on past performance of students from similar backgrounds. If historically, students from one background have better grades because of supportive resources, the teacher might wrongly predict that the next student from that group will continue to excel, overlooking their individual circumstances. Similarly, COMPAS failed to assess individual risks fairly by relying on biased historical data.

Studies have shown that many facial recognition systems are less accurate for darker-skinned individuals, raising concerns about racial bias in law enforcement tools.

Facial recognition technology is increasingly deployed in law enforcement for identifying suspects. However, research has shown that these systems often misidentify individuals with darker skin tones more frequently than those with lighter skin. This inaccuracy can lead to wrongful accusations or arrests, perpetuating racial biases within the criminal justice system. The findings reveal a significant flaw in how these technologies are developed and trained, underscoring the need for diverse datasets to avoid such disparities.

Consider a scenario where a camera designed to recognize people at a theme park works well for light-skinned guests but struggles to accurately identify dark-skinned guests. If a dark-skinned visitor is mistakenly identified as a troublemaker and removed from the park, it leads to embarrassment and loss of trust in the technology. This analogy illustrates the real-world consequences of biased facial recognition systems.

--