Explainer: What Is Machine Learning Bias?

Machine learning bias is not a bug. It is a feature of systems trained on human decisions - and the first step to fixing it is understanding exactly what kind of bias you are dealing with.

The One-Sentence Definition

Machine learning bias occurs when a model produces systematically different outcomes for different demographic groups - not because of legitimate differences in the underlying signal, but because of how the training data was collected, labelled, or used.

Why It Matters

Every audit in this repository starts from the same premise: the bias is real, it is measurable, and it is fixable. But before you can measure or fix it, you need a precise vocabulary for what you are looking at.

ML bias is not one thing. It enters a model at multiple stages - in the data before training begins, in the labels that define what the model is optimised for, in the features chosen to represent the world, and in the feedback loops that form when a deployed model's outputs become the next round of training data. A practitioner who treats all of these as one undifferentiated problem will keep patching the wrong stage.

The real-world stakes are not abstract. COMPAS-style recidivism tools flag Black defendants as high-risk at nearly twice the rate of white defendants with equivalent profiles. Hiring AI screens out women from technical roles before a human sees the resume. Credit scoring penalises young borrowers for not having lived long enough to accumulate tenure. These outcomes are not accidents - they are the predictable result of training on historical data that encoded historical inequality, and then deploying the result as if it were neutral.

The Four Entry Points

ML bias enters the pipeline at four distinct stages. Each one has a different cause, a different detection method, and a different fix.

1. Training Data Bias

The model learns from data. If the data misrepresents the world - because certain groups were undersampled, oversampled, or not collected at all - the model inherits that misrepresentation.

This is sampling bias: when the population in the training set does not match the population the model will be deployed on. A facial recognition model trained predominantly on lighter-skinned faces will underperform on darker-skinned faces - not because the architecture is wrong, but because the distribution of training examples was wrong. See Sampling Bias for the full treatment.

2. Label Bias

Labels define what the model is trying to predict. If those labels were generated by human decisions - hiring outcomes, loan approvals, recidivism flags - they inherit the prejudice of the humans who made them.

A hiring model trained on past hiring decisions learns to replicate what past hiring managers preferred. If past managers systematically downgraded women, the model learns to downgrade women. The features may look clean. The label is the problem. See Label Bias for the full treatment.

3. Proxy Variables

Removing a protected attribute - race, gender, age - from the feature set is the most common first response to documented bias. It is almost never sufficient.

A proxy variable is a feature that is not the protected attribute but correlates strongly with it. Zip code correlates with race due to historical redlining. Employment tenure correlates with age because young people cannot have long tenures. Relationship status correlates with sex because caregiving roles are gendered. When a protected attribute is removed, the model reconstructs it from the proxies that remain.

Every audit in this repository demonstrates this. COMPAS reconstructs race from custody status. The hiring model reconstructs gender from age. German credit scoring reconstructs age from employment tenure. See Proxy Variables for the full treatment.

4. Feedback Loop Bias

A deployed model produces outputs. Those outputs influence real-world decisions. Those decisions generate new data. That new data is used to retrain the model. If the original model was biased, the outputs were biased, the new data encodes that bias, and the retrained model is more biased than the original - not less.

This is feedback loop bias, and it compounds across retraining cycles. A predictive policing model that over-flags certain neighbourhoods increases arrests in those neighbourhoods, which increases the training signal for flagging those neighbourhoods. The model did not introduce the disparity - but it amplified it and laundered it as objective prediction. See Feedback Loop Bias for the full treatment.

The Two Kinds of Bias By Intent

Beyond the four entry points, bias in AI systems is also classified by whether the protected attribute is used intentionally or structurally.

Disparate treatment is direct: the protected attribute, or a known proxy for it, is explicitly included as a model input. COMPAS including race as a feature is disparate treatment. It is the easier case to detect and - under US employment and lending law - the more clearly unlawful one. See Disparate Treatment.

Disparate impact is structural: the model does not use the protected attribute directly, but produces outcomes that fall disproportionately on one group - typically measured by the Four-Fifths Rule. A test that disqualifies 50% of white applicants and 80% of Black applicants creates disparate impact even if race never appears in the model. See Disparate Impact.

Most production bias is disparate impact, not disparate treatment. The protected attribute was removed. The proxies were not.

Concrete Example: COMPAS - Audit 01

The COMPAS audit is the sharpest illustration of how all four entry points converge in a single system.

Training data: 70,000+ records from Broward County, Florida. Predominantly from a jurisdiction with documented history of racially unequal policing - meaning the training distribution reflects over-policing of Black communities, not actual recidivism rates.

Labels: The label is is_recid - whether a defendant reoffended within two years. Reoffending is measured by re-arrest, not by actual criminal behaviour. In a jurisdiction where Black defendants are more likely to be stopped, searched, and arrested for equivalent behaviour, re-arrest is a biased label. The model trains to predict re-arrest and produces a proxy for policing intensity, not criminal propensity.

Proxy variable: Custody status correlates with race because pretrial detention patterns reflect unequal bail access, which reflects income inequality, which is racially structured. Removing race while leaving custody status in the model leaves most of the racial signal intact.

Feedback loop: COMPAS scores influence bail and sentencing decisions. A defendant flagged as high-risk is more likely to be detained pretrial. Pretrial detention increases the probability of conviction and reoffending (due to job loss, housing instability, and network effects). The model produces the outcome it predicted, and the outcome enters the next training set as evidence the prediction was correct.

The biased model flags Black defendants as high-risk at 86.77%. The fair model - with race and custody status removed - reduces that gap to 15.69%. A 71% reduction from one targeted intervention on one proxy variable.

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

df = pd.read_csv("COMPAS/compas-scores-raw.csv")

# Biased model: race and proxy both present
X_biased  = df[['age', 'priors_count', 'juv_fel_count', 'juv_misd_count',
                 'sex', 'c_charge_degree', 'race', 'CustodyStatus']]
y         = df['is_recid']

X_train, X_test, y_train, y_test = train_test_split(
    X_biased, y, test_size=0.2, random_state=42
)

model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

# Measure demographic parity gap
preds   = model.predict(X_test)
results = X_test.copy()
results['prediction'] = preds

black_rate = results[results['race'] == 'African-American']['prediction'].mean()
white_rate = results[results['race'] == 'Caucasian']['prediction'].mean()

print(f"Black defendants flagged high-risk: {black_rate:.2%}")
print(f"White defendants flagged high-risk: {white_rate:.2%}")
print(f"Fairness gap: {abs(black_rate - white_rate):.2%}")

Detection Code

The standard fairness check for any binary classifier is demographic parity - the difference in positive prediction rates across groups. It requires only the model's predictions and the demographic column.

import pandas as pd

def demographic_parity_report(df_test, prediction_col, group_col):
    """
    Print positive prediction rates by group and compute the fairness gap.

    df_test:        test-set DataFrame with predictions and group labels
    prediction_col: name of the binary prediction column (0 or 1)
    group_col:      name of the demographic group column
    """
    rates = (
        df_test.groupby(group_col)[prediction_col]
        .mean()
        .rename("positive_rate")
        .reset_index()
        .sort_values("positive_rate", ascending=False)
    )

    print(f"\n--- Demographic Parity Report: {group_col} ---\n")
    for _, row in rates.iterrows():
        print(f"  {row[group_col]:30s}  {row['positive_rate']:.2%}")

    gap = rates['positive_rate'].max() - rates['positive_rate'].min()
    print(f"\n  Fairness Gap: {gap:.2%}")

    if gap > 0.20:
        print("  ⚠  Gap exceeds 20% - likely to breach the Four-Fifths Rule.")
    elif gap > 0.05:
        print("  ⚠  Gap is material - proxy analysis recommended.")
    else:
        print("  ✓  Gap is below 5%.")

    return rates, gap


# Usage example (COMPAS audit)
# results['prediction'] = model.predict(X_test)
# demographic_parity_report(results, prediction_col='prediction', group_col='race')

To check for proxy variables once a gap is detected:

from scipy.stats import chi2_contingency

def check_proxy(df, feature_col, protected_col, p_threshold=0.05):
    """
    Chi-squared test for association between a candidate feature and a protected attribute.
    Returns True if the feature is likely a proxy.
    """
    ct              = pd.crosstab(df[feature_col], df[protected_col])
    chi2, p, dof, _ = chi2_contingency(ct)
    is_proxy        = p < p_threshold

    label = "⚠  PROXY DETECTED" if is_proxy else "✓  no significant association"
    print(f"  {label}  |  {feature_col} ↔ {protected_col}  |  p={p:.4f}")
    return is_proxy


# Check all non-protected features
# for col in feature_cols:
#     check_proxy(df, feature_col=col, protected_col='race')

The Fairness Metric Question

Detecting a gap requires choosing a metric. Fair Code uses demographic parity as the default: equal positive prediction rates across groups. But demographic parity is not the only option, and it is not always the right one. There are at least three fairness metrics that practitioners commonly reach for, and it has been mathematically proven that you cannot satisfy all three simultaneously except in degenerate cases.

Metric	What it requires
Demographic Parity	Equal positive prediction rates across groups
Equalized Odds	Equal true positive and false positive rates across groups
Predictive Parity	Equal precision (positive predictive value) across groups

Choosing the wrong metric for a domain produces a model that looks fair on the metric you chose and is biased on the metrics you ignored. See Why Fairness Metrics Conflict for the full mathematical proof and guidance on which metric fits which domain.

Limitations

Bias Reduction Is Not Bias Elimination

Every audit in this repository demonstrates reduction, not elimination. Removing proxies reduces the fairness gap because it cuts off the most direct channels through which the protected signal flows into the model. It does not remove structural inequality from the world, and it does not remove all correlations between the remaining features and the protected attribute. A 70% reduction is a meaningful improvement. It is not a certification of fairness.

The Protected Attribute Must Be Measured to Be Audited

You cannot measure a fairness gap without demographic labels in your test set. Many production systems do not collect this data - either because the system was not designed for auditing, or because collection is legally complex in certain jurisdictions. A model that has never been audited by race is not a model that is unbiased by race. It is a model whose bias has not been measured.

Demographic Parity May Not Be the Appropriate Metric for Your Domain

A recidivism tool and a medical diagnostic tool have different fairness requirements. Equalising positive prediction rates in a medical context might mean under-diagnosing a disease in one group while over-diagnosing it in another. The correct metric for clinical AI is typically closer to equalized odds - equal sensitivity and specificity across groups - not demographic parity. Applying the wrong metric produces a false sense of fairness.

Proxy Variables - The single most common mechanism by which protected signal survives removal of the protected attribute.
Sampling Bias - How a misrepresentative training distribution creates bias before the first line of model code is written.
Label Bias - How historical human decisions pollute the labels the model is trained to replicate.
Feedback Loop Bias - How a biased deployment compounds bias across retraining cycles.
Disparate Impact - The legal standard and the Four-Fifths Rule.
Disparate Treatment - When the protected attribute is used directly.
Why Fairness Metrics Conflict - The mathematical impossibility result that constrains all fairness work.
Proxy Entanglement - What happens when proxy variables form a correlated cluster rather than independent channels.

COMPAS/ - All four bias entry points in a single production system: biased training data, biased labels, proxy variables, and feedback loop risk.
AI Fair Recruitment/ - Label bias and proxy variables in a hiring context; gender reconstructed from age after sex is removed.
German Credit Lending/ - Age discrimination via employment tenure as a proxy.
Benefits Denial/ - Multiple protected attributes, multiple proxy clusters, welfare eligibility decisions.

What Is Machine Learning Bias?