AI fairness cannot be audited with average accuracy and a single score

NIST SP 1270 categorizes AI bias into three main categories: systemic, statistical and computational, and human. Bias arises not only from training data and algorithms but also through institutions, organizational practices, problem framing, operator cognition, and judgments made during use; thus, auditing is not complete by reading model cards alone. [1]

First, map which decisions—such as hiring, lending, identity verification, or content display—AI outputs enter, and whether humans can override them. Even with the same error rate, harm differs between uses where candidates are additionally reviewed and those where services are automatically rejected. Fairness goals are not abstract equality but are derived backwards from harms to avoid and unacceptable errors. [1]

NISTIR 8280 collected 189 face recognition algorithms from 99 developers and evaluated demographic disparities by separating 1:1 matching and 1:N search, as well as false positives and false negatives. The report confirms disparities in many algorithms while also showing significant differences between algorithms, including some with high accuracy and small disparities. [2]

Therefore, audit forms should record the target population, task (matching, classification, ranking, etc.), thresholds, false positives and false negatives, comparison groups, sample sizes, and confidence intervals. If overall accuracy is the only KPI, sample insufficiency in minority populations and high-harm errors become buried in the average. If operational data differs from training/validation data, re-evaluate as a separate population. [2]

Pass/fail lines are determined separately from the difference in overall accuracy, based on the disadvantage each type of false positive and false negative imposes on each population. Groups with small samples and high uncertainty are not rounded to pass; instead, measurement is deemed impossible, and applicability is suspended. The condition for transition is not to exploit the magnitude of differences seen in face recognition, but to re-measure using the same task, threshold, and operational population. [2]

Gender Shades evaluated three commercial gender classification systems across intersecting groups of skin tone and gender. While misclassification rates reached up to 34.7% for women with dark skin, the maximum misclassification rate for men with light skin was 0.8%, revealing disparities that are hard to see in single-attribute or overall averages. [3]

These results cannot be directly applied to all current face recognition or generative AI systems. Nevertheless, audit designs that examine intersecting populations, within-group variability, and extremely small groups, in addition to one-dimensional aggregation by gender or race alone, can be generalized. If samples are small, do not conclude that “there is no difference”; instead, choose to deem it unmeasurable and add data or narrow the scope of use. [1] [3]

The source of between-group disparities is not limited to the model. Stratify by shooting conditions, input quality, subject composition, thresholds, and the rate at which operators re-confirmed to verify at which stage disparities widen. [1] [3] If disparities shrink when input conditions are standardized, correct the collection process; if they remain under the same conditions, correct the model or threshold; if they widen after output, prioritize correcting operational procedures.

The disparities in misclassification rates observed in NIST’s socio-technical taxonomy, face recognition evaluations, and Gender Shades show that visibility changes depending on overall values, error types, populations, and intersecting groups. [1] [2] [3] These are not causal effect sizes. In auditing, establish priority rules in advance for cases where different aggregations yield different conclusions. Also fix the role of each metric.

The adjudication form should list, line by line, the harm to avoid by use, corresponding error types, populations to evaluate, comparison criteria, and unacceptable outcomes. Do not average into a composite score when overall accuracy and false negatives for specific populations move in opposite directions; instead, record which one is being protected and who approved the trade-off. If the trade-off cannot be approved, narrow the scope of automated use.

As a hypothetical example, if overall errors decrease but false positives for a group leading to significant harm increase, do not use average improvement as a reason for approval. Lay out options such as narrowing applicability, returning decisions for that group to humans, or splitting thresholds for re-testing, and adjudicate conditional approval or stoppage based on the harm definition. Any remaining harm under any option is left separately as an unresolved item.

Populations with insufficient samples are neither passing nor failing, but in a state where evidence necessary for judgment is lacking. To prevent them from entering permanent operation, record the target population, missing error types, current uncertainty, temporarily permitted uses, prohibited uses, and the deadline for supplementing evidence as “evidence debt” in a separate ledger. Also define the upper limit of judgments executable while the debt remains.

Resolution of evidence debt is not determined solely by increasing case counts. Reconfirm whether evaluation can be done with the same population, threshold, and error definition as actual operation, whether minority samples are absorbed into the overall average, and whether acquiring additional data does not create new harms. If conditions are not met, maintain or narrow the applicability. Bias in the acquisition method itself is also a subject of review.

Upon expiration, the debt must be updated to resolved, extended, or usage stopped, and extensions without cause are not permitted. If failures for that group are found through complaints or reviews, reopen the adjudication form without waiting for the regular date. This transforms the blank of “could not measure” into an operational target with an owner and next decision authority. The number of extensions is also recorded in the history.

• CONFLICT: List competing metrics and the harms they represent
• PRIORITY: Define the error type to protect, target population, and adjudication responsible party
• DEBT: Record reasons for unmeasurability, provisional uses, prohibited uses, and deadlines
• RESOLVE: Update to resolved, extended, or stopped based on additional evidence

Do not let measurement results end in a report; link them to deployment decisions, applicability scope, human verification, and stop criteria. The more a use involves automatic decisions causing significant harm, the stricter the allowable disparity by population, and the earlier prepare a window for appealing misjudgments, personnel for review, and deadlines for correction. Also record changes in fairness improvement and overall performance simultaneously.

After deployment, input distributions, thresholds, upstream data, and user behavior change. Use not only fixed cycles like monthly or quarterly, but also model version changes, expansion of target populations, increased complaints, and misjudgment incidents as triggers for re-auditing. Make the audit responsible party independent of the development team, and treat observations from affected users and field operators in meetings with the same weight as quantitative metrics. [1] [2]

The effectiveness of remedies is tracked not by the existence of a window, but from appeal through review, correction, and impact recovery. The audit responsible party reports on misjudgments by population, whether appeals were accepted, reasons for overturned decisions, and whether corrections were extended to similar cases. When complaints or changes in the operational population exceed preconditions, narrow the target use without waiting for the regular date. [1] [2]

• Define harms and error types by use before averages
• Check differences within groups and by intersecting attributes, not just between groups
• Do not treat unmeasurable small populations as zero-risk
• Determine responsible parties and conditions for stopping, reviewing, correcting, and resuming
• Audit Responsible Party: Record error types and uncertainty by population side-by-side
• Business Responsible Party: Decide applicability to unmeasurable populations
• Remedy Responsible Party: Connect review results to corrections in similar cases