Research Note

When Fairness Metrics Disagree

Fairness evaluation in machine learning is often treated as a straightforward measurement problem. In practice, however, different fairness metrics can produce conflicting conclusions about the same model.

A system may appear fair according to one metric while showing substantial disparity according to another. This creates a major challenge for deployment decision-making in high-stakes AI systems.

Why Metric Disagreement Matters

Fairness metrics measure different forms of model behaviour. Some focus on false positive rates, others on false negative rates, demographic parity, or predictive consistency.

Because these metrics capture different properties, disagreement between them is not unusual. However, deployment decisions are often made without considering the uncertainty introduced by conflicting fairness evaluations.

Operational Risk

In high-stakes settings, fairness disagreement is not simply a reporting issue. It can become a deployment-risk issue.

Systems that appear acceptable under one metric may still show disparities or other assurance concerns when assessed under alternative fairness criteria.

Beyond Single-Metric Evaluation

Depending on the system, evidence, and deployment context, relevant assurance activity may include:

• Multi-metric fairness evaluation
• Subgroup disparity analysis
• Threshold sensitivity assessment
• Context-aware deployment review
• Decision-risk analysis

Evaluating fairness through a single metric alone may provide an incomplete view of real deployment suitability.