Fairness in AI decision-making: what it means and how to reduce bias

What fairness means when AI makes decisions

Fairness is not a property you can read off a model's accuracy. The Fairlearn project, which maintains one of the main open-source fairness toolkits, defines unfairness by its effect on people rather than by its cause or anyone's intent. Two kinds of harm matter most when a model feeds decisions:

• Allocation harms: the system extends or withholds an opportunity, a resource or information, as in hiring, admissions and lending.
• Quality-of-service harms: the system works worse for some people than for others even when nothing is withheld, as with face recognition or speech-to-text that fails more often for some groups.

Most practical work uses group fairness. You name the groups at risk, using sensitive features such as sex, age, ethnicity or disability, and compare how the model treats each one. The alternative, individual fairness, was set out by Dwork and colleagues in 2011: similar people should receive similar outcomes. It is appealing, but it depends on a task-specific measure of which people are similar, which is hard to agree on in practice.

The NIST AI Risk Management Framework counts fairness, with harmful bias managed, among the characteristics of trustworthy AI, and adds two warnings. Perceptions of fairness differ between cultures and change with the application. And a system whose harmful biases have been reduced is not necessarily fair: predictions balanced across groups can still be inaccessible to people with disabilities, or deepen disparities that already exist. Fairlearn's shorthand for this is that fairness is sociotechnical. The code is only part of it.

The fairness definitions teams actually use

Group fairness is made precise with parity constraints: some statistic of the model's behaviour must be comparable across groups. Choosing the statistic is the real decision. These are the ones you will meet in papers, toolkits and audits:

Each one protects something different, and each has a known weakness. Hardt, Price and Srebro, who defined equalized odds and equal opportunity in 2016, point out that demographic parity can be met by accepting qualified applicants in one group and unqualified ones in another, as long as the percentages match, and that it rules out even a perfect predictor when the true outcome differs between groups. Equalized odds always allows the perfectly accurate model, and it penalizes a model that works well only for the majority. Equal opportunity relaxes it to the people whose true outcome is the favourable one: borrowers who would repay should have the same chance of getting the loan, whatever their group.

Calibration asks that scores mean what they claim in every group. As Kleinberg, Mullainathan and Raghavan note, if a score is not calibrated within groups, a decision-maker has a reason to treat two people with the same score differently because of their group, which defeats the point of a score.

Toolkits report each definition as a gap. In Fairlearn, the demographic parity difference is the highest group selection rate minus the lowest, and the demographic parity ratio is the lowest divided by the highest, so a ratio of 1.0 means parity.

Why a model cannot satisfy every fairness definition at once

In 2016, Kleinberg, Mullainathan and Raghavan formalized three conditions for a risk score: calibration within groups, balance for the positive class (people who turn out positive get the same average score in each group) and balance for the negative class (the same for people who turn out negative). They proved that no score can meet all three except in two special cases: perfect prediction, or groups with equal base rates. The result holds however the score is produced, by an algorithm or by people.

Chouldechova reached the same wall from the other side in 2017. Error rates, the positive predictive value (PPV) and the prevalence p of the outcome in a group are tied together:

FPR = p / (1 - p) × (1 - PPV) / PPV × (1 - FNR)

If two groups have different prevalence and the tool is equally precise for both (predictive parity), their false positive rate and false negative rate cannot both be equal. In the recidivism data she analyzed, the reoffending rate was 51% for Black defendants and 39% for White defendants, so a score with equal PPV had to produce unequal error rates. That is the arithmetic behind the COMPAS debate: critics pointed to unbalanced error rates, the tool's defenders showed it was calibrated, and both were right about what they measured.

The practical consequence is a choice, not a defect you can engineer away. Ask which mistake hurts more in this decision (a qualified person rejected, or an unqualified one accepted), who bears it, and whether the base rates themselves reflect past unfairness.

Where bias in AI decision-making comes from

NIST groups the sources of AI bias into three categories: systemic bias in data, institutions and society; computational and statistical bias, often from samples that do not represent the population; and human-cognitive bias in how people build, read and act on AI output. None of them needs prejudice or intent. In practice they reach a decision system through a handful of routes.

Data that leaves people out

A model learns less about groups it rarely sees. In the Gender Shades study (2018), Buolamwini and Gebru found that two facial analysis benchmarks, IJB-A and Adience, were 79.6% and 86.2% lighter-skinned subjects. Testing three commercial gender classifiers on a balanced set, they measured error rates of up to 34.7% for darker-skinned women, against at most 0.8% for lighter-skinned men.

Labels that measure the wrong thing

The target a model predicts is often a stand-in for what you care about. A 2019 study in Science by Obermeyer and colleagues examined an algorithm that health systems used to pick patients for extra care. It predicted health-care costs as a proxy for health needs, but less money is spent on Black patients because of unequal access to care. At the same risk score, Black patients were considerably sicker than White patients, and the authors estimated that removing the disparity would raise the percentage of Black patients receiving additional help from 17.7% to 46.5%. By some measures of predictive accuracy, cost looked like an effective proxy.

Proxies for protected traits

Deleting the sensitive column does not remove the information. Hardt, Price and Srebro call this redundant encoding: other features can predict the protected attribute, so "fairness through unawareness" does not work. A postal code, a school, a first name or a gap in employment history can carry the same signal. It also means you usually need the sensitive attribute, stored and handled lawfully, to measure whether the model treats groups differently at all.

Feedback loops

When a model's decisions decide what data gets collected next, an early bias can grow. Ensign and colleagues proved this for predictive policing (FAT* 2018): arrests found where police were sent feed the next prediction, so patrols keep returning to the same neighbourhoods regardless of the true crime rate. The same risk exists wherever only approved applicants generate repayment data, or only flagged transactions get investigated.

People and scale

People also over-trust automated output. The EU AI Act calls this automation bias and requires high-risk systems to help the people overseeing them stay aware of it. Scale adds a newer problem: when many employers or agencies use the same vendor's model, the same person can be rejected everywhere at once. Researchers call this systemic exclusion, and it is one of the reasons for the randomization proposal below.

How to measure fairness in an AI model

Fairlearn's assessment guide breaks a fairness assessment into four steps. They work with any toolkit:

1. Identify the harms. Is the risk allocation (someone denied an interview) or quality of service (a transcription that fails more often for some speakers)?
2. Identify the groups that might be harmed, including intersections such as age by sex, and groups specific to the use case, such as non-native speakers for speech-to-text.
3. Quantify the harm with a metric that matches it. For job screening, the false negative rate (qualified candidates screened out) says more than overall accuracy.
4. Compare across groups: the gap between the best and worst group, the ratio, and the worst group's value on its own.

Fairlearn's MetricFrame does the bookkeeping. It evaluates ordinary scikit-learn metrics overall and per group; pass it a DataFrame with several sensitive columns and it reports every intersection:

from fairlearn.metrics import (
    MetricFrame, count, false_positive_rate, selection_rate,
    equalized_odds_difference,
)
from sklearn.metrics import recall_score

mf = MetricFrame(
    metrics={
        "selection_rate": selection_rate,
        "tpr": recall_score,
        "fpr": false_positive_rate,
        "count": count,
    },
    y_true=y_test,
    y_pred=y_pred,
    sensitive_features=X_test[["sex", "age_band"]],
)
print(mf.by_group)      # every metric, for every intersection of sex and age band
print(mf.difference())  # the largest gap between groups, per metric
print(equalized_odds_difference(y_test, y_pred, sensitive_features=X_test["sex"]))

Always include the count. Small groups and thin intersections produce noisy rates, and a gap measured on a handful of people is a reason to collect more data, not a finding. Measure on data the model has not seen, rerun the assessment for every model change, and keep the results: an assessment nobody can find later does not count as evidence.

How to reduce bias before, during and after training

Mitigation methods are grouped by where they act. Both main toolkits cover all three stages:

Some practical differences follow from the stage:

• Pre-processing changes only the data, so the training code stays as it is. It cannot rescue a target label that measures the wrong thing.
• In-processing builds the fairness constraint into training, so accuracy is traded against it directly. You have to retrain, and the constraint is only as good as the definition you chose.
• Post-processing works on top of any model that outputs scores, including one bought from a vendor. It usually needs the sensitive attribute at decision time, because it sets group-specific thresholds. Whether that is allowed depends on the jurisdiction and the domain, so ask counsel before you ship it.

Hardt, Price and Srebro showed that meeting equalized odds exactly can require group-specific thresholds that are randomized for some scores, while equal opportunity can be met without randomizing. AIF360's EqOddsPostprocessing likewise solves a linear program for the probabilities with which to change output labels. Randomness is already part of standard fairness tooling, which puts the MIT proposal below in context.

Two cautions from the toolkits' own documentation. Fairlearn's mitigation guide says its algorithms guarantee how close they drive an unfairness metric to zero, and that this does not mean the results are fair. And AIF360's documentation lists more than a dozen algorithms, so picking one is easy, while picking the right definition is not. Fix the cause where you can: in the health-care case the bias sat in the choice of target, so that is where the fix belongs, not in a fairness algorithm.

Where structured randomization fits, and where it does not

In July 2024, MIT News reported on a paper by Shomik Jain and Ashia Wilson of MIT and Kathleen Creel of Northeastern University. Its title states the claim: Scarce Resource Allocations That Rely On Machine Learning Should Be Randomized, a position paper presented at ICML 2024 and published in PMLR volume 235. It argues against a habit most fairness methods share: once bias has been reduced, the top-ranked candidates get the job interviews, the program places or the organs, deterministically, every time.

The authors give two reasons to randomize when there are fewer goods than people with a claim to them:

• Claims. Following philosopher John Broome, everyone with a claim deserves a chance, and a stronger claim should raise that chance without wiping out a slightly weaker one. A lottery weighted by the strength of claims does both; a strict top-k cut-off does neither.
• Uncertainty. Model scores carry error, and a deterministic rule repeats the same mistake on the same person. Across many decision-makers using similar models, that becomes systemic exclusion. Over time, when one allocation raises the chance of the next, it becomes patterned inequality.

They propose several procedures and test them in simulations and on real data. A partial lottery hands part of the places to the clear leaders and draws the rest by weighted lottery among the candidates nearest the cut-off. A variance-based version trains several bootstrapped models, gives a place to anyone every model ranks in the top k, and weights everyone else by the share of models that rank them there. A third uses conformal prediction to flag outliers the model has little data about, and holds an unweighted lottery among all of them for the places outliers would have received.

The cost was small in their experiments. On 2003 Swiss unemployment data, allocating job-search and training programs to people likely to stay unemployed for a year, a plain top-25% selection achieved an observed utility of 63.3% on average. Randomizing half the places across the candidates nearest the cut-off lowered utility by 0.8%. With a random forest, the variance-based method cost 0.2% of utility while randomizing 32% of the places among 15% of the people. In simulations with normally distributed claims, a partial lottery that gave up about 2% of expected utility cut the systemic exclusion rate by more than 20% once more than two decision-makers were involved.

If you consider a lottery for a real allocation, treat it as policy, not a model setting. Wilson told MIT News that how much to randomize is for all the stakeholders in the allocation to decide. Record each candidate's weight, the random seed and the draw, so every outcome can be reproduced and audited, and tell applicants that part of the process is a lottery. And do not count on ordinary human inconsistency to do the same job: the paper argues it cannot, because human variation is not distributed according to people's claims, and social biases concentrate bad outcomes on marginalized groups.

Fair AI rules and standards as of September 2026

Most fairness obligations now come from a mix of voluntary frameworks, standards and, for some uses, law.

NIST AI RMF and ISO/IEC standards

The NIST AI Risk Management Framework (AI RMF 1.0), released on January 26, 2023, is voluntary and organizes the work into four functions: Govern, Map, Measure and Manage. Its subcategory MEASURE 2.11 asks that fairness and bias, as identified in the Map function, be evaluated and the results documented. ISO/IEC 42001:2023, published in December 2023, specifies requirements for establishing, running and continually improving an AI management system. For the technical side, ISO/IEC TR 24027:2021 describes techniques for measuring and assessing bias in AI systems and AI-aided decisions across the whole lifecycle, from data collection to use.

The EU AI Act's high-risk obligations

The EU AI Act (Regulation (EU) 2024/1689) is where fairness becomes a legal duty for many decision systems. Its Annex III lists high-risk uses that include recruitment and decisions about workers, eligibility for public assistance benefits, creditworthiness and credit scores, risk assessment and pricing in life and health insurance, and emergency call triage. For those systems:

• Data governance (Article 10): training, validation and testing data must be examined for possible biases that could affect health, safety or fundamental rights or lead to prohibited discrimination, especially where outputs feed future inputs, with measures to detect, prevent and mitigate them.
• Human oversight (Article 14): the system must let the people overseeing it understand its limits, stay aware of automation bias, and disregard or override its output.
• Fundamental rights impact assessment (Article 27): required before use by public bodies, private entities providing public services, and deployers of credit-scoring and life or health insurance systems.
• Explanation (Article 86): a person affected by a decision based on an Annex III system, with legal or similarly significant effects, has the right to a clear and meaningful explanation of the AI system's role and the main elements of the decision.

The dates moved in 2026. The Act entered into force on August 1, 2024 and has applied in general since August 2, 2026. The Digital Omnibus on AI (Regulation (EU) 2026/1744 of July 8, 2026), in force since July 27, 2026 according to the European Commission, set the high-risk requirements to apply from December 2, 2027 for Annex III uses and from August 2, 2028 for AI in regulated products. It also moved the permission to process special categories of personal data for bias detection and correction into a new Article 4a, with strict conditions: only when other data, including synthetic or anonymized data, will not do; with pseudonymization and access controls; with no transfer to other parties; and with deletion once the bias is corrected. The article extends that permission to deployers of high-risk systems and to other AI systems, without making bias testing mandatory for them. How these rules sit alongside AI laws in the US and Canada is summarized in how AI is changing society. For generative models, see our guide to generative AI risks and rules.

Canada: no federal AI law yet

As of September 2026, Canada has no federal statute regulating AI. Bill C-27, which contained the Artificial Intelligence and Data Act, was still in committee in the House of Commons when the parliamentary session ended on January 6, 2025, so it never became law. A new privacy bill, Bill C-36, was introduced on June 15, 2026 by the Minister of Artificial Intelligence and Digital Innovation and is at second reading. It would replace the privacy part of PIPEDA with a Protecting Privacy and Consumer Data Act, and it contains no AI-specific regime like the one AIDA proposed. As its first-reading text stands, it would require an organization that used an automated decision system for a prediction, recommendation or decision with a legal or similarly significant effect to explain it on request (the personal information used, its source and the principal factors) and to let the person make written representations to an employee who can review it.

Federal institutions already work under the Treasury Board's Directive on Automated Decision-Making. It requires a published algorithmic impact assessment before a system goes into production, testing of the data and model for unintended biases before production, and scheduled monitoring of outcomes. The latest version dates from June 24, 2025, and systems built or bought before then had until June 24, 2026 to meet the updated requirements.

A fairness checklist before AI decides about people

Before a model's output affects who gets something, work through these in order:

1. Write down the decision: who it affects, what they gain or lose, and which mistake costs them most.
2. Check the target. Confirm the label measures what you mean, not a convenient proxy such as cost for need.
3. Choose the fairness definition with its reasons, and have an accountable owner sign it off.
4. Get the data to measure it: the sensitive attributes you need, collected and stored lawfully, with access limited to the people doing the assessment.
5. Measure per group and intersection, with counts, on held-out data, and agree in advance what gap is acceptable.
6. Mitigate where the problem starts, then measure fairness and accuracy again together.
7. Design the human role: who can overrule the model, how people ask for an explanation or appeal, and how reviewers avoid rubber-stamping.
8. For scarce allocations with uncertain scores, consider a weighted lottery near the cut-off, logged and disclosed.
9. Monitor outcomes after launch, watch for feedback loops, and rerun the assessment on every model change.

If you are putting a model in front of decisions like these, our AI and automation service builds an evaluation set from your real cases, runs it before any model, prompt or source change goes live, and maps findings to the NIST AI RMF. It does not build systems that make decisions about people, such as hiring or credit, without human review. For the wider questions of building AI into a product, from data to regulation, see our guide to intelligent apps.