Why KPIs Break: Operationalizing Goodhart’s Law

Holmström and Milgrom’s multitasking theory demonstrates that when only some tasks are measurable, linking strong rewards to those metrics causes effort to skew toward the measured targets.[1] For instance, aggressively pursuing the number of customer service cases may delay hard-to-record consultations and mentoring of junior staff, as local rationality undermines organizational objectives.

In metric design, list not only desired outcomes but also hard-to-measure tasks that are critical to avoid losing; if these cannot be directly measured, options include weakening rewards for the primary metric, preserving professional discretion, incorporating qualitative reviews, or separating roles. The starting point is avoiding the conflation of areas lacking precise numbers with areas lacking value.

Before linking strong rewards, shadow-run the primary metric and observe how frontline effort allocation changes; if hard-to-record consultations, mentoring, or preventive work decrease despite improvements in the measured target, effort substitution is a plausible alternative explanation.[1] For work where counter-metrics cannot be observed, decide to avoid increasing the intensity of numerical targets and retain discretionary reviews.

Bevan and Hood analyzed governance in UK public healthcare using strong goals and sanctions, examining the assumption that measured metrics represent the entire service and considering gaming such as reclassification of waiters.[2] Distortions include not only manipulation of numbers but also diverting resources to the target and rendering out-of-scope quality invisible.

When the frontline adapts to metrics, labeling it immediately as fraud hides information; instead, distinguish which actions improve customer value, which exploit definitional loopholes, and which shift burdens to other groups. Audit exceptions, reclassifications, last-minute changes before deadlines, and stagnation at the tails of distributions, treating rapid numerical improvements as a starting point for investigation as well as success.

In anomaly detection, do not immediately judge rapid improvements as fraud, but decompose them into actual improvement, definitional changes, reclassification, and displacement to out-of-scope groups. Compare public values with internal records, pre- and post-deadlines, and target versus non-target groups; if improvements lacking customer value concentrate, suspect systemic factors.[2] Protecting frontline staff who raise objections from evaluation targets is also a measurement condition for identifying loopholes early.

Propper et al. compared England, which introduced strong waiting time targets, with Scotland, which did not impose similar sanctions. Analyzing the proportion of long-term waits exceeding six, nine, and twelve months, they reported a relative decrease in England.[3] Goodhart’s Law does not mean goals are necessarily ineffective.

The key is not to equate improvements in the primary metric with success of the overall objective; even if waiting times decrease, separately verify whether case selection, clinical quality, staff workload, or short-term waiters have worsened. As with policy evaluation, examine pre-introduction trends, comparators, concurrent changes, and differences between public and internal values to separate outcomes from substitution.

When judging the effect of goal introduction, fix pre-introduction trends and comparators, and retain concurrent personnel, demand, and institutional changes as alternative explanations. Define success only within the range where the primary metric improved and counter-metrics were maintained. When transferring results observed in healthcare waiting times to other industries, verify whether enforceability and measurability are similar, without assuming the effect size carries over.[3]

While the UK public healthcare case analysis considered gaming such as reclassification, one cannot conclude from observing a specific system that denominator manipulation occurs in all other KPIs.[2] Here, we treat it as a limited and general audit hypothesis to verify whether changes in achievement rates can be explained by subject replacement. Denominator definition changes are also audited independently.

In the recommended denominator bridge table, connect opening subjects, new inflows during the period, exclusions, completions, and closing balances using the same definitions. Even if the achievement rate appears to rise, if difficult cases are moved out of scope, display this separately from processing capacity improvements. Link exclusion reasons not just to free-text descriptions but to the exact version of the applied rule.

We do not treat changes in the denominator itself as fraud, as demand composition or systems may legitimately change; instead, verify the reason for change, the approver, and the outcomes of cases moved out of scope. Show whether primary metric improvements are visible in a fixed group and whether reversing the swap changes the judgment, sending only unexplained differences to additional audits. Place changes on the inflow side in the same table.

Objections do not constitute a random sample representing all cases. Nevertheless, they can be limitedly used for hypothesis generation to find where definitional edges, recording omissions, and burdens on out-of-scope groups concentrate. As an operational proposal below, pair objection cases with non-objection cases under identical conditions to reconfirm metric calculations and actual value. Also sample a layer unaware of the objections.

In case audits, do not pre-classify objection reasons as justified or unjustified; instead, pass original records, applied definitions, calculated values, frontline judgments, and customer or operational outcomes to independent verifiers blinded to the metric values. If the same malfunction is found in non-objection controls, treat it not as an exception of those who spoke up, but as a candidate problem with the definition or operation.

Do not decide on changes solely based on the number of objections. Verify whether recalculations change rankings or rewards, whether the same malfunction spreads to identical conditions, and whether new loopholes emerge after correction. If definitions are revised, allow a sufficient parallel run period with the old version to avoid unconditionally linking it to a series of improvements. During the parallel run, recalculate the same cases in both versions and explicitly state the switch date.

• DENOMINATOR BRIDGE: Connect opening balances, inflows, exclusions, completions, and closing balances along with rule versions
• APPEAL PAIR: Extract non-objection cases under identical conditions to objection cases
• BLIND RECALCULATION: Blind the metric value and recalculate definition application and results from original records
• CHANGE DECISION: Record disposition by distinguishing local exceptions, widespread malfunctions, definition revisions, and reward suspensions

Do not burden a single KPI with the objective; instead, form a small set of outcome metrics, process metrics, quality guardrails, distribution metrics, and long-term metrics. It is not necessary to link all to rewards. Separate diagnostic metrics from evaluation metrics, and strictly define procedures for definition changes, manipulability, and objections for those used in individual assessments.

In operational meetings, continuously ask not only for explanations of achievement rates but also whether the cheapest actions to raise metrics align with those that increase customer value. Loopholes found by the frontline are design tests, not materials for punishment. If metrics have expiration dates, abolition conditions, counter-metrics, and unannounced audits in advance, they become learning circuits for KPIs to return to objectives.

Assign objective owners, definition managers, auditors, and abolition approvers to each KPI. If counter-metric deterioration, concentration of reclassification, or loss of correlation with the objective persists, disconnect them from rewards first and revert to diagnostic metrics. When revising definitions, separate versions without pretending continuity with the past, and do not reconnect to individual assessments until improvement is confirmed.

• Objective: Write separately the outcomes substituted by metrics and the values that cannot be substituted
• Counter: Combine quality with speed, distribution with averages, and long-term with short-term
• Audit: Investigate sudden changes, reclassifications, displacement to out-of-scope groups, and differences with internal values
• Revision/Abolition: Determine expiration dates, change responsibility, objections, and abolition conditions