Algorithmic Bias: When Code Creates Unfair Outcomes

Why it exists

Algorithms are used to automate decisions at scale. The problem of algorithmic bias exists because the data used to train these systems, and the rules they operate on, often contain hidden, systematic unfairness from the real world. When an algorithm learns from biased history, it automates that bias.

The mental model

An algorithm is like a diligent but uncritical student. If you give it a library of books reflecting historical prejudices and ask it to define a 'good candidate,' its answer will be systematically skewed. The algorithm isn't malicious; it's just a mirror reflecting the bias in its training material, often amplifying it at scale.

How it works

Algorithmic bias creates a systematic and repeatable tendency to produce unfair outcomes. This happens when a model learns patterns from historical data. If that data reflects past discrimination (e.g., a company historically hired more men), the model learns to associate male-coded traits with success. It then applies this learned pattern to new decisions, perpetuating the original bias by privileging one category over another.

When to use it

The concept of algorithmic bias is relevant in any automated system making decisions about people, especially when trained on historical data. This includes credit scoring systems, resume screeners for hiring, risk assessments in the justice system, and even content recommendation engines. It's a critical lens for evaluating the fairness of any sociotechnical system.

When not to use it

There is no situation where creating algorithmic bias is acceptable. Instead, you must be most wary of it in high-stakes domains affecting people's lives, opportunities, or freedoms. The risk is highest when the training data is known to reflect historical societal inequities, as the algorithm will likely learn and perpetuate those same unfair patterns as its intended function.

One canonical example

Consider a hiring tool trained on 10 years of a company's resume data to find the best software engineers. If, historically, the company predominantly hired men, the algorithm might learn to penalize resumes that include words like 'women's' (e.g., 'women's coding club') or favor resumes with stereotypically male-associated activities. The system creates an unfair, biased outcome by privileging one group based on correlations in the data, even if 'gender' is not an explicit input.