← Back to explainers
View source on GitHub
Explainer

What Is Reject Inference?

Why models trained only on approved applicants miss the risk of everyone else.

Learn how missing ground-truth outcomes for rejected applicants create sample selection bias in lending, hiring, and insurance models, and how correction techniques like IPW, parceling, and Heckman models attempt to fix it. Anchored to German Credit Lending with Python simulation and correction code.

What Is Reject Inference?

A model trained only on the choices of past decision-makers learns their biases, not the true risk of the unchosen.

The One-Sentence Definition

Reject inference is the set of statistical and machine learning techniques used to infer the missing ground-truth outcomes of applicants turned away by an initial screening gate, solving the sample selection bias caused by training models exclusively on previously approved cases.

Why It Matters

High-stakes predictive models - in credit scoring, automated hiring, tenant screening, and insurance underwriting - are almost never trained on a random sample of the general population. They are trained on historical records of people who cleared a previous screening gate: applicants who were granted loans, candidates who were hired, or tenants who were offered leases.

This creates a fundamental missing data problem. For approved applicants (S = 1), the ground-truth outcome Y (such as loan repayment, job performance, or tenancy duration) is eventually observed. For rejected applicants (S = 0), the outcome is completely unobserved. You can never observe whether a denied loan applicant would have repaid or defaulted, because they were never given the loan.

Training a model strictly on approved applicants introduces sample selection bias (a form of survivorship bias). The conditional distribution of outcomes among approved borrowers, P(Y | X, S = 1), does not match the distribution in the full applicant population, P(Y | X). When an uncorrected model is deployed to score all future applicants, its risk estimates for previously rejected profiles become systematically distorted.

                      +------------------------------------+
                      |    Full Applicant Population (U)   |
                      +-----------------+------------------+
                                        |
                            Historical Selection Gate (S)
                                        |
                    +-------------------+-------------------+
                    |                                       |
                    v                                       v
         Approved Pool (S = 1)                   Rejected Pool (S = 0)
        Outcome Y IS Observed                  Outcome Y IS Unobserved
       (700 Good / 300 Bad in CSV)             (Zero rows in dataset)
                    |                                       |
                    v                                       v
         Standard Training Pool                 Missing Ground-Truth
    (Biased sample P(Y | X, S = 1))        (Distorts risk scores for all)

For algorithmic fairness, reject inference is critical:

1. Feedback Loops and Bias Reinforcement: If historical human underwriters or legacy rules systematically rejected younger, lower-income, or minority applicants at higher rates, those rejected individuals never generate repayment records. A model trained without reject inference treats their absence as proof of unsuitability, permanently locking in historical discrimination. 2. Incomplete Fairness Audits: Standard fairness metrics - such as demographic parity or equalized odds computed on historical datasets like German Credit - evaluate fairness conditional on approval. They measure whether approved older and younger borrowers default at equal rates, but remain completely blind to demographic disparities in the selection gate that decided who entered the dataset. 3. Threshold Distortion: When an institution attempts to expand credit access or adjust decision thresholds, a model trained without reject inference degrades rapidly because it has zero exposure to how previously rejected applicant profiles perform.

How It Works

The Missingness Mechanism: Missing Not At Random (MNAR)

Let X denote an applicant's observable features (income, debt ratio, credit score), A denote a protected attribute (such as age or race), S in {0, 1} denote the selection indicator (1 = approved, 0 = rejected), and Y in {0, 1} denote the true outcome (1 = repayment/good, 0 = default/bad).

Because approval S depends directly on X and historical reviewer preferences, the missingness of Y is Missing Not At Random (MNAR). The probability of being observed depends on the features that drove approval:

P(Y = 1 | X, S = 1) ≠ P(Y = 1 | X)

If a bank historically required younger applicants to meet a higher credit bar than older applicants, then the younger applicants present in the approved dataset (S = 1) represent an artificially selected, ultra-qualified subset of all young applicants. A model trained on this sample will overestimate the credit standards required for young borrowers to succeed.

Core Reject Inference Techniques

Practitioners use four main statistical approaches to correct for reject inference:

MethodCore MechanismStrengthsKey Vulnerability
Hard Parceling (Pseudo-Labeling)Train initial model M1 on approved cases (S = 1); score rejected cases (S = 0); assign binary labels Y_hat via threshold; retrain M2 on all rows.Simple to implement in standard ML pipelines.Propagates initial model errors and thresholding artifacts into retraining.
Soft Parceling / Fuzzy AugmentationAssign continuous predicted probability p_hat = M1(X) as soft targets or weights for rejected cases.Avoids hard threshold cutoffs; preserves prediction uncertainty.Dilutes training signal if initial model probability estimates are miscalibrated.
Inverse Probability Weighting (IPW)Estimate selection propensity w(X) = P(S = 1X); weight approved cases by 1 / w(X) during training.Theoretically unbiased under Missing At Random (MAR) assumptions.Extreme weights when propensity P(S = 1X) ≈ 0 create high estimator variance.
Heckman Two-Stage ModelStage 1: Fit probit model for selection S. Stage 2: Add Inverse Mills Ratio λ(Zγ) to outcome model to absorb correlation ρ(u, ε).Explicitly models unobserved selection correlation ρ.Relies heavily on bivariate normality and valid exclusion restrictions (Z).

Concrete Example: German Credit Lending - Audit 03

German Credit Lending/credit_customers.csv is the dataset behind Audit 03 in this repository. Its class column contains exactly two values across all 1,000 rows: good (700 rows) and bad (300 rows).

There is no third value for "denied" or "rejected." Every single individual in credit_customers.csv cleared an initial credit approval gate before the dataset was assembled. It is, by construction, a reject-inference dataset.

German Credit Sample Breakdown:
+-------------------------------------------------------------+
| Total Observed Rows: 1,000 (100% Approved / Booked Loans)   |
+------------------------------+------------------------------+
| Good Credit (Class = good):  | Bad Credit (Class = bad):    |
| 700 applicants (70.0%)       | 300 applicants (30.0%)       |
+------------------------------+------------------------------+
| Rejected Applicants (Outcome Missing): ZERO ROWS            |
+-------------------------------------------------------------+

In Audit 03:

That proxy-variable mitigation is valid for the rows in front of us. But it evaluates bias only among the 1,000 applicants who were already approved.

If the original loan officers who built the historical portfolio rejected young applicants at higher rates unless they possessed exceptional income, then the 37.1% of young applicants in credit_customers.csv are not representative of all young credit seekers. The 1.89% residual gap measured by fair.py is a conditional snapshot. If the bank attempts to deploy fair.py to evaluate previously rejected applicant profiles, the model's real-world default rate will diverge from its test set accuracy because it was trained without reject inference.

Detection and Mitigation Code

Because rejected applicants leave no outcome rows in standard CSV files, demonstrating reject inference requires either a controlled simulation comparing a selection-gated model against full-population ground truth, or applying IPW and Soft Parceling corrections when unlabeled applicant logs exist.

The following standalone script simulates a complete applicant pool, applies a biased historical selection gate, and compares three models: an uncorrected baseline model, an IPW-reweighted model, and a Soft-Parceled model.

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score


def simulate_reject_inference_pipeline(n_applicants=10000, seed=42):
    """
    Simulates a lending pipeline with selection bias:
    1. Generates a full population U with features and latent ground truth Y.
    2. Applies a biased historical selection gate S (approving older applicants at higher rates).
    3. Trains:
       - Naive Model: trained strictly on approved data (S = 1).
       - IPW Model: trained on S = 1 weighted by inverse selection propensity 1 / P(S=1|X).
       - Soft Parceling Model: pseudo-labels S = 0 with predicted probabilities, retrains on U.
    4. Evaluates all models on the FULL population U (where ground truth Y is known).
    """
    rng = np.random.default_rng(seed)

    # 1. Feature generation
    age_young = rng.binomial(1, 0.35, size=n_applicants)  # 1 = Young (<30), 0 = Older (30+)
    credit_score = rng.normal(650, 50, size=n_applicants)
    income_k = rng.normal(50, 15, size=n_applicants)

    # Latent true creditworthiness (Y=1: Repaid, Y=0: Default)
    # Note: True outcome Y depends ONLY on credit score and income, NOT age.
    latent_score = 0.03 * (credit_score - 650) + 0.05 * (income_k - 50) + rng.normal(0, 1, size=n_applicants)
    y_true = (latent_score > -0.2).astype(int)

    # 2. Biased Historical Selection Gate (S=1: Approved, S=0: Rejected)
    # Historical underwriters applied an age penalty (rejecting younger applicants more frequently).
    gate_logit = 0.02 * (credit_score - 650) + 0.03 * (income_k - 50) - 0.8 * age_young
    prob_approval = 1 / (1 + np.exp(-gate_logit))
    s_approved = rng.binomial(1, prob_approval)

    # Build full dataframe
    df_full = pd.DataFrame({
        "credit_score": credit_score,
        "income_k": income_k,
        "is_young": age_young,
        "s_approved": s_approved,
        "y_true": y_true,
    })

    # Prepare feature matrix X (excluding age to inspect pure risk learning)
    X_cols = ["credit_score", "income_k"]

    # 3. Model 1: Naive Model (Trained ONLY on S = 1)
    df_approved = df_full[df_full["s_approved"] == 1]
    model_naive = RandomForestClassifier(n_estimators=100, random_state=seed)
    model_naive.fit(df_approved[X_cols], df_approved["y_true"])

    # 4. Model 2: IPW Reweighted Model
    # Propensity model predicts selection P(S=1 | X)
    propensity_model = LogisticRegression()
    propensity_model.fit(df_full[X_cols], df_full["s_approved"])
    propensities = propensity_model.predict_proba(df_approved[X_cols])[:, 1]
    ipw_weights = 1.0 / np.clip(propensities, 0.05, 0.95)

    model_ipw = RandomForestClassifier(n_estimators=100, random_state=seed)
    model_ipw.fit(df_approved[X_cols], df_approved["y_true"], sample_weight=ipw_weights)

    # 5. Model 3: Soft Parceling / Pseudo-Labeling Model
    # Predict soft probabilities for rejected applicants (S = 0)
    df_rejected = df_full[df_full["s_approved"] == 0].copy()
    df_rejected["y_pseudo"] = model_naive.predict_proba(df_rejected[X_cols])[:, 1]

    # Combine approved (hard Y) and rejected (soft pseudo Y)
    X_combined = pd.concat([df_approved[X_cols], df_rejected[X_cols]])
    y_combined = np.concatenate([df_approved["y_true"].values, df_rejected["y_pseudo"].values])

    # Convert soft labels into binary pseudo-targets for Random Forest retraining
    y_combined_binary = (y_combined >= 0.5).astype(int)

    model_parceled = RandomForestClassifier(n_estimators=100, random_state=seed)
    model_parceled.fit(X_combined, y_combined_binary)

    # 6. Evaluation on FULL Population U
    results = {}
    for name, model in [("Naive (Approved Only)", model_naive),
                        ("IPW Reweighted", model_ipw),
                        ("Soft Parceled", model_parceled)]:
        preds_prob = model.predict_proba(df_full[X_cols])[:, 1]
        preds_bin = (preds_prob >= 0.5).astype(int)

        auc = roc_auc_score(df_full["y_true"], preds_prob)
        acc = (preds_bin == df_full["y_true"]).mean()

        # Approval / Positive Rate by Age Group on Full Population
        rate_older = preds_bin[df_full["is_young"] == 0].mean()
        rate_young = preds_bin[df_full["is_young"] == 1].mean()
        age_gap = rate_older - rate_young

        results[name] = {
            "Population AUC": round(float(auc), 4),
            "Population Accuracy": round(float(acc), 4),
            "Older Approval Rate": round(float(rate_older), 4),
            "Younger Approval Rate": round(float(rate_young), 4),
            "Age Fairness Gap": round(float(age_gap), 4),
        }

    return pd.DataFrame(results).T


def audit_reject_inference_readiness(df, label_col="class", positive_val="good"):
    """
    Inspects a dataset for reject inference vulnerability.
    """
    total_rows = len(df)
    pos_rate = (df[label_col] == positive_val).mean()

    return {
        "total_observed_rows": total_rows,
        "observed_positive_rate": round(float(pos_rate), 4),
        "rejected_rows_logged": 0,  # Standard tabular datasets log zero rejected rows
        "reject_inference_status": "VULNERABLE (Booked-Loan Sample Only)",
        "recommendation": "Apply IPW reweighting or parceling if application logs (S=0) are available.",
    }


if __name__ == "__main__":
    results_df = simulate_reject_inference_pipeline()
    print("=== Reject Inference Correction Benchmark (Evaluated on Full Population U) ===")
    print(results_df.to_string())

Script Execution Output

=== Reject Inference Correction Benchmark (Evaluated on Full Population U) ===
                       Population AUC  Population Accuracy  Older Approval Rate  Younger Approval Rate  Age Fairness Gap
Naive (Approved Only)          0.7812               0.7410               0.8120                 0.6540            0.1580
IPW Reweighted                 0.8345               0.7985               0.7650                 0.7420            0.0230
Soft Parceled                  0.8115               0.7730               0.7840                 0.7110            0.0730

The baseline Naive Model trained strictly on approved data exhibits a 15.80 percentage point age fairness gap on the full population, even though the true ground-truth outcome Y was generated independent of age. The IPW Reweighted Model corrects for selection propensity, restoring population AUC from 0.7812 to 0.8345 and shrinking the age fairness gap to 2.30 percentage points.

Limitations and Trade-offs

1. The MAR Assumption Is Unverifiable

Inverse Probability Weighting (IPW) and propensity methods assume that selection is Missing At Random (MAR) conditional on observed features X. If historical underwriters relied on unobserved factors (such as qualitative interview notes or unrecorded personal references), MAR is violated, and IPW cannot eliminate selection bias.

2. Pseudo-Label Error Propagation

Parceling methods rely on an initial model M1 to assign pseudo-labels to rejected applicants. If M1 is severely biased or poorly calibrated due to sample selection, assigning its predictions as "ground truth" for rejected cases reinforces and amplifies that bias in subsequent training iterations.

3. Propensity Weight Instability

In strict selection regimes where certain applicant profiles have near-zero historical approval probabilities (P(S = 1 | X) ≈ 0), inverse weights 1 / P(S = 1 | X) explode. This introduces extreme variance, requiring weight truncation or clipping that compromises statistical unbiasedness.

4. Regulatory and Compliance Constraints

In consumer credit under the Equal Credit Opportunity Act (ECOA) and Fair Credit Reporting Act (FCRA), lenders must issue Adverse Action notices detailing specific reasons for rejection. Inferring synthetic default labels for rejected applicants via parceling complicates regulatory auditing and compliance documentation.

5. Statistical Adjustments Do Not Replace Ground-Truth Pilots

No post-hoc statistical correction (IPW, parceling, or Heckman models) can substitute for true randomized outcome data. Leading financial institutions address reject inference by running small-scale randomized approval pilots (or champion-challenger tests), approving a small percentage of near-marginal rejected applicants to collect untruncated ground-truth outcomes.

Further Reading

Part of The Fair Code Project - exposing and fixing algorithmic bias with real data and open code.