10.7. Case Studies

COMPAS is a software used in the US to predict re-offending risks. Studies showed racial bias, predicting black individuals as more likely to reoffend than white individuals, even when untrue. Lesson: Biased data can lead to unethical outcomes in justice.

COMPAS stands for Correctional Offender Management Profiling for Alternative Sanctions. It's a tool that some courts use to assess the likelihood of a person committing a crime again. However, research has found that this software often incorrectly predicts that black individuals would be more likely to reoffend, even if they haven’t. This highlights that if the data fed into an AI system carries biases, the system's outcomes can also be unjust. This is a critical issue because it can lead to harsher sentences or unjust treatment based on flawed predictions, reinforcing existing racial inequalities in the justice system.

Imagine a teacher who grades students based on historical test scores without considering the individual efforts of current students. If the teacher notices that a specific group of students historically struggles, they might unfairly lower expectations for them, perpetuating the cycle of underachievement. Similarly, COMPAS fails to account for individual circumstances and instead relies on biased historical data.

Amazon developed an AI to automate hiring but it downgraded resumes with the word “women’s” in them (like "women's college"). Lesson: AI can reflect historical biases and discriminate unfairly.

Amazon created an AI recruitment tool to streamline the hiring process. However, it was discovered that the tool was biased against resumes that contained the term 'women's,' which penalized applicants from women's colleges or programs. This occurred because the AI learned from historical hiring data that favored male candidates. As a result, the algorithm reflected these historical biases, leading to unfair discrimination against women applying for jobs. This case illustrates the importance of scrutinizing AI systems to ensure they promote equality rather than perpetuate past biases.

Consider a scenario where a club only accepts members who have degrees from prestigious universities. Over time, it might inadvertently favor candidates from those schools who historically have more men than women, resulting in a lack of diversity. This is analogous to how the Amazon AI tool functioned by favoring certain language and backgrounds while ignoring qualified individuals.

DeepMind (a Google company) used NHS patient data for a health app without fully informing users. Lesson: Even well-intentioned AI can raise privacy concerns if not handled ethically.

DeepMind partnered with NHS to develop an app that uses patient data to monitor health issues. However, it was criticized for not fully informing patients about how their sensitive health information would be used. This raised significant privacy concerns because patients were unaware that their data was being utilized in this way, even if the intent was to improve healthcare. This situation emphasizes the necessity of transparency and consent in AI applications, ensuring that users have a clear understanding of how their data is being utilized.

Think about a restaurant that uses customer feedback to improve its menu but never informs diners that their opinions are being recorded. If customers found out later, they might feel betrayed or uncomfortable. Similarly, patients should be aware and agree to how their health information is used, highlighting the importance of ethical practices in AI involving sensitive data.

--