Key AI Ethics Case Studies

Why This Matters

These case studies aren't just cautionary tales—they're the foundation for understanding how AI systems can go wrong and what principles should guide responsible development. You're being tested on your ability to identify bias, privacy violations, accountability gaps, dual-use concerns, and the limits of automation in real-world contexts. Each case illustrates specific failure modes that appear repeatedly across the field, from training data problems to inadequate human oversight.

Don't just memorize what happened with each company. Know why each case matters: What ethical principle was violated? What systemic issue does it reveal? When an FRQ asks you to analyze an AI ethics scenario, these cases become your evidence. The strongest answers connect specific incidents to broader concepts like algorithmic fairness, informed consent, transparency, and accountability structures.

---

Bias and Fairness Failures

When AI systems learn from historical data, they often encode and amplify existing societal biases. The training data reflects past human decisions—including discriminatory ones—and the algorithm treats these patterns as features to optimize.

Microsoft's Tay Chatbot

• Learned toxic behavior within hours—Tay was designed to mimic conversational patterns from Twitter users, but trolls deliberately fed it racist and offensive content
• Unfiltered training data created a system with no guardrails against harmful outputs, demonstrating that real-time learning from public input is inherently risky
• Shutdown within 24 hours raised foundational questions about content moderation and whether AI systems can safely learn from uncurated human behavior

Amazon's Recruitment Tool

• Systematically penalized female candidates because the system was trained on a decade of hiring data that reflected male-dominated tech industry patterns
• Downgraded resumes containing "women's" (as in "women's chess club"), revealing how proxy discrimination can emerge even without explicit gender variables
• Scrapped entirely rather than fixed, illustrating that biased training data can make a system unsalvageable without complete redesign

OpenAI's GPT-3 Concerns

• Perpetuates biases from internet text—the model reflects stereotypes, misinformation, and harmful associations present in its massive training corpus
• Generates convincing misinformation at scale, raising concerns about AI-generated content being weaponized for propaganda or fraud
• No reliable detection method exists for AI-written text, emphasizing the need for responsible release practices and access controls

---

Privacy and Consent Violations

AI systems often require massive datasets, creating pressure to collect information without meaningful user consent. The ethical line between data that's technically accessible and data that's ethically usable is frequently crossed.

Facebook's Cambridge Analytica Scandal

• Harvested data from 87 million users through a personality quiz that exploited Facebook's API to access not just quiz-takers but all their friends' data
• Used for political microtargeting without consent, demonstrating how personal data can be weaponized to manipulate democratic processes
• Triggered GDPR enforcement and ongoing regulatory scrutiny, making this the defining case for data protection and informed consent requirements

Clearview AI's Facial Recognition Database

• Scraped billions of images from social media platforms without user knowledge, creating a searchable database sold primarily to law enforcement
• No opt-out mechanism existed for individuals whose faces were indexed, violating basic principles of consent and data subject rights
• Banned in multiple countries including Canada and Australia, illustrating how surveillance technology can exceed legal and ethical boundaries even when technically legal in some jurisdictions

Apple's CSAM Detection Debate

• On-device scanning proposed to detect child sexual abuse material before upload to iCloud, blurring the line between local privacy and cloud monitoring
• Critics warned of mission creep—the same technology could be adapted for political censorship or expanded surveillance under government pressure
• Indefinitely delayed after backlash, highlighting the tension between child safety and device privacy with no clear resolution

---

Accountability in High-Stakes Decisions

When AI systems make decisions affecting health, safety, or freedom, the question of who bears responsibility for errors becomes critical. Automation can obscure accountability, making it unclear whether humans, companies, or algorithms are to blame.

IBM Watson's Cancer Treatment Failures

• Provided unsafe recommendations including suggesting treatments that could cause severe bleeding in patients already at risk
• Trained on hypothetical cases rather than real patient data at some institutions, meaning the system learned from oncologists' theoretical preferences rather than actual outcomes
• Revealed limits of medical AI—complex diseases require contextual judgment that current systems cannot reliably provide, reinforcing the need for human oversight in life-or-death decisions

Uber's Self-Driving Car Fatality

• First autonomous vehicle pedestrian death occurred when the system detected Elaine Herzberg but classified her as a false positive and didn't brake
• Safety driver was distracted watching video on her phone, raising questions about whether human backup provides meaningful oversight during monotonous monitoring tasks
• Criminal charges filed against the safety driver but not Uber, illustrating the accountability gap when human and machine responsibilities are poorly defined

---

Dual-Use and Military Applications

AI technologies developed for beneficial purposes can often be repurposed for surveillance, warfare, or social control. The same capabilities that enable useful applications create potential for misuse.

Google's Project Maven

• Applied AI to drone footage analysis to identify objects and people of interest, potentially streamlining military targeting decisions
• 4,000+ employees signed protest letter arguing that Google should not be in the business of war, demonstrating that corporate ethics can be shaped by workforce pressure
• Contract not renewed and Google published AI principles excluding weapons development, establishing a precedent for ethical red lines in corporate AI policy

China's Social Credit System

• Aggregates behavioral data from financial records, social media, and surveillance cameras to generate scores affecting citizens' access to travel, loans, and employment
• Creates chilling effects on dissent and nonconformity, as citizens modify behavior to avoid score penalties—a form of algorithmic social control
• No meaningful appeal process for many scoring decisions, raising concerns about due process and the opacity of automated governance systems

---

Quick Reference Table

---

Self-Check Questions

1. Which two case studies best illustrate how biased training data can produce discriminatory outcomes, and what distinguishes the type of bias in each?

2. If an FRQ asks you to discuss informed consent in AI data collection, which cases would you compare, and what different consent violations does each represent?

3. Both IBM Watson and Uber's self-driving car involved AI in safety-critical applications. What does each case reveal about the appropriate level of human oversight for high-stakes AI decisions?

4. How does Google's response to Project Maven demonstrate the role of employee activism in shaping corporate AI ethics, and why might this model not work in other contexts?

5. Compare Clearview AI and China's social credit system: both involve surveillance, but what ethical principles does each primarily violate, and how do the accountability structures differ?