{ "schemaVersion": "1.0", "language": "en", "canonicalUrl": "https://catalysphere.co.jp/en/articles/kpi-goodharts-law/", "machineReadableUrl": "https://catalysphere.co.jp/en/articles/kpi-goodharts-law/index.json", "publisher": { "name": "Catalysphere", "url": "https://catalysphere.co.jp/", "description": "A research and consulting studio connecting AI and big-data research with practice." }, "article": { "slug": "kpi-goodharts-law", "title": "Why KPIs Break: Operationalizing Goodhart’s Law", "summary": "We do not end the discussion with the notion that “metrics become targets and the frontline deteriorates”; instead, we distinguish bias toward easily measurable work, gaming, and actual improvement to design robust operations that resist breakdown.", "takeaways": [ "The problem with KPIs lies not in quantification itself, but in concentrating rewards and attention on a measurable aspect, thereby substituting it for the overall objective.", "This is because strong goals can drive outcomes but also induce neglect, manipulation, and displacement of burden onto unmeasured work.", "Combine the primary metric with counter-metrics for quality, distribution, and long-term impact, and regularly revise or abolish them through audits and frontline objections." ], "publishedAt": "2026-07-13", "updatedAt": "2026-10-01", "readingMinutes": 8, "topics": [ "KPI", "Goodhart’s Law", "Incentives", "Organizational Design" ], "domains": [ "organization-adoption", "governance-risk", "data-analytics" ], "evidenceKind": "research-review", "status": "published", "image": { "prompt": "Abstract scientific editorial illustration for the research question “KPIはなぜ壊れるのか：Goodhartの法則を運用に落とす”, interpreted symbolically rather than typographically: separate knowledge clusters connected through a shared operating network, with a few gold checkpoints making coordination and handoffs visible; deep Catalysphere navy and indigo field, muted scientific blue, restrained antique-gold structure, translucent geometry, fine particles, subtle paper grain, generous negative space, one clear centered metaphor that survives a narrow mobile cover crop, calm premium research-journal art direction, no readable text, no logo, no literal chart, no photorealistic people.", "negativePrompt": "text, letters, numerals, watermark, logo, dashboard UI, humanoid robot, server rack, binary rain, stock-photo office, literal quantitative chart, red alarm graphics, neon cyberpunk, purple gradient, clutter", "alt": "Abstract diagram showing multiple knowledge clusters connecting to a single operational network via golden verification points", "aspectRatio": "16:9", "assetPath": "article-art/kpi-goodharts-law.png", "status": "approved" }, "lead": "Even when KPIs improve, customer experience and frontline capabilities may deteriorate; this is not because people dislike numbers, but because they rationally adapt to metrics tied to rewards and sanctions. Observe the effects and side effects of goals on the same screen, and treat metrics as revisable hypotheses rather than fixed entities.", "visual": { "eyebrow": "INCENTIVES / KPI DESIGN", "title": "Separate measuring, moving, and protecting.", "caption": "Arranging theory, healthcare system case analyses, and policy evaluations reveals that goals are not ineffective but can simultaneously generate outcomes and distortions.", "metrics": [ { "value": "MULTITASK", "label": "Theory explaining the concentration of effort on easily measurable work", "source": "Holmström & Milgrom, 1991", "referenceId": 1 }, { "value": "GAMING", "label": "Manipulation and narrow vision observed in goal governance", "source": "Bevan & Hood, 2006", "referenceId": 2 }, { "value": "6 / 9 / 12M", "label": "Thresholds used in policy evaluation of long-term waits", "source": "Propper et al., 2008", "referenceId": 3 } ] }, "sections": [ { "heading": "Easily measurable work is over-optimized", "paragraphs": [ "Holmström and Milgrom’s multitasking theory demonstrates that when only some tasks are measurable, linking strong rewards to those metrics causes effort to skew toward the measured targets.[1] For instance, aggressively pursuing the number of customer service cases may delay hard-to-record consultations and mentoring of junior staff, as local rationality undermines organizational objectives.", "In metric design, list not only desired outcomes but also hard-to-measure tasks that are critical to avoid losing; if these cannot be directly measured, options include weakening rewards for the primary metric, preserving professional discretion, incorporating qualitative reviews, or separating roles. The starting point is avoiding the conflation of areas lacking precise numbers with areas lacking value.", "Before linking strong rewards, shadow-run the primary metric and observe how frontline effort allocation changes; if hard-to-record consultations, mentoring, or preventive work decrease despite improvements in the measured target, effort substitution is a plausible alternative explanation.[1] For work where counter-metrics cannot be observed, decide to avoid increasing the intensity of numerical targets and retain discretionary reviews." ] }, { "heading": "Gaming is created by systems, not villains", "paragraphs": [ "Bevan and Hood analyzed governance in UK public healthcare using strong goals and sanctions, examining the assumption that measured metrics represent the entire service and considering gaming such as reclassification of waiters.[2] Distortions include not only manipulation of numbers but also diverting resources to the target and rendering out-of-scope quality invisible.", "When the frontline adapts to metrics, labeling it immediately as fraud hides information; instead, distinguish which actions improve customer value, which exploit definitional loopholes, and which shift burdens to other groups. Audit exceptions, reclassifications, last-minute changes before deadlines, and stagnation at the tails of distributions, treating rapid numerical improvements as a starting point for investigation as well as success.", "In anomaly detection, do not immediately judge rapid improvements as fraud, but decompose them into actual improvement, definitional changes, reclassification, and displacement to out-of-scope groups. Compare public values with internal records, pre- and post-deadlines, and target versus non-target groups; if improvements lacking customer value concentrate, suspect systemic factors.[2] Protecting frontline staff who raise objections from evaluation targets is also a measurement condition for identifying loopholes early." ] }, { "heading": "Goals can also drive actual outcomes", "paragraphs": [ "Propper et al. compared England, which introduced strong waiting time targets, with Scotland, which did not impose similar sanctions. Analyzing the proportion of long-term waits exceeding six, nine, and twelve months, they reported a relative decrease in England.[3] Goodhart’s Law does not mean goals are necessarily ineffective.", "The key is not to equate improvements in the primary metric with success of the overall objective; even if waiting times decrease, separately verify whether case selection, clinical quality, staff workload, or short-term waiters have worsened. As with policy evaluation, examine pre-introduction trends, comparators, concurrent changes, and differences between public and internal values to separate outcomes from substitution.", "When judging the effect of goal introduction, fix pre-introduction trends and comparators, and retain concurrent personnel, demand, and institutional changes as alternative explanations. Define success only within the range where the primary metric improved and counter-metrics were maintained. When transferring results observed in healthcare waiting times to other industries, verify whether enforceability and measurability are similar, without assuming the effect size carries over.[3]" ] }, { "heading": "Separate denominator swaps from improvement", "paragraphs": [ "While the UK public healthcare case analysis considered gaming such as reclassification, one cannot conclude from observing a specific system that denominator manipulation occurs in all other KPIs.[2] Here, we treat it as a limited and general audit hypothesis to verify whether changes in achievement rates can be explained by subject replacement. Denominator definition changes are also audited independently.", "In the recommended denominator bridge table, connect opening subjects, new inflows during the period, exclusions, completions, and closing balances using the same definitions. Even if the achievement rate appears to rise, if difficult cases are moved out of scope, display this separately from processing capacity improvements. Link exclusion reasons not just to free-text descriptions but to the exact version of the applied rule.", "We do not treat changes in the denominator itself as fraud, as demand composition or systems may legitimately change; instead, verify the reason for change, the approver, and the outcomes of cases moved out of scope. Show whether primary metric improvements are visible in a fixed group and whether reversing the swap changes the judgment, sending only unexplained differences to additional audits. Place changes on the inflow side in the same table." ], "visual": { "kind": "record-table", "eyebrow": "AUTHORED DESIGN PROPOSAL", "title": "Bridge denominator movements separately from completion", "caption": "Untested Catalysphere audit proposal with no sample values. Use mutually exclusive states and common dates and rule versions so opening plus inflow equals completion plus other exits plus closing stock. Use consistent case IDs; record reopening and transfers back as re-entry inflows. If a case has multiple exits, count exit events separately and do not confuse them with unique-case completion rates.", "provenance": "catalysphere-authored", "columns": [ { "id": "c0", "label": "State" }, { "id": "c1", "label": "Evidence to record" }, { "id": "c2", "label": "What to follow" } ], "rows": [ { "id": "r0", "label": "Opening", "cells": [ "Eligibility and rule version at opening", "Starting comparison set" ] }, { "id": "r1", "label": "Inflow", "cells": [ "Arrival time, difficulty and reason for eligibility; include re-entry from reopening or transfer back", "Changes in demand mix" ] }, { "id": "r2", "label": "Completion", "cells": [ "Completion criteria and confirmed time", "Actual outcome" ] }, { "id": "r3", "label": "Other exit", "cells": [ "Mutually exclusive exclusion or transfer reason", "Outcome and burden outside the metric" ] }, { "id": "r4", "label": "Closing stock", "cells": [ "Unfinished cases at the same closing time", "Backlog and next-period connection" ] } ] } }, { "heading": "Turn objections into samples of metric malfunction", "paragraphs": [ "Objections do not constitute a random sample representing all cases. Nevertheless, they can be limitedly used for hypothesis generation to find where definitional edges, recording omissions, and burdens on out-of-scope groups concentrate. As an operational proposal below, pair objection cases with non-objection cases under identical conditions to reconfirm metric calculations and actual value. Also sample a layer unaware of the objections.", "In case audits, do not pre-classify objection reasons as justified or unjustified; instead, pass original records, applied definitions, calculated values, frontline judgments, and customer or operational outcomes to independent verifiers blinded to the metric values. If the same malfunction is found in non-objection controls, treat it not as an exception of those who spoke up, but as a candidate problem with the definition or operation.", "Do not decide on changes solely based on the number of objections. Verify whether recalculations change rankings or rewards, whether the same malfunction spreads to identical conditions, and whether new loopholes emerge after correction. If definitions are revised, allow a sufficient parallel run period with the old version to avoid unconditionally linking it to a series of improvements. During the parallel run, recalculate the same cases in both versions and explicitly state the switch date." ], "bullets": [ "DENOMINATOR BRIDGE: Connect opening balances, inflows, exclusions, completions, and closing balances along with rule versions", "APPEAL PAIR: Extract non-objection cases under identical conditions to objection cases", "BLIND RECALCULATION: Blind the metric value and recalculate definition application and results from original records", "CHANGE DECISION: Record disposition by distinguishing local exceptions, widespread malfunctions, definition revisions, and reward suspensions" ] }, { "heading": "Make KPIs a falsifiable set of metrics", "paragraphs": [ "Do not burden a single KPI with the objective; instead, form a small set of outcome metrics, process metrics, quality guardrails, distribution metrics, and long-term metrics. It is not necessary to link all to rewards. Separate diagnostic metrics from evaluation metrics, and strictly define procedures for definition changes, manipulability, and objections for those used in individual assessments.", "In operational meetings, continuously ask not only for explanations of achievement rates but also whether the cheapest actions to raise metrics align with those that increase customer value. Loopholes found by the frontline are design tests, not materials for punishment. If metrics have expiration dates, abolition conditions, counter-metrics, and unannounced audits in advance, they become learning circuits for KPIs to return to objectives.", "Assign objective owners, definition managers, auditors, and abolition approvers to each KPI. If counter-metric deterioration, concentration of reclassification, or loss of correlation with the objective persists, disconnect them from rewards first and revert to diagnostic metrics. When revising definitions, separate versions without pretending continuity with the past, and do not reconnect to individual assessments until improvement is confirmed." ], "bullets": [ "Objective: Write separately the outcomes substituted by metrics and the values that cannot be substituted", "Counter: Combine quality with speed, distribution with averages, and long-term with short-term", "Audit: Investigate sudden changes, reclassifications, displacement to out-of-scope groups, and differences with internal values", "Revision/Abolition: Determine expiration dates, change responsibility, objections, and abolition conditions" ] } ], "note": "Holmström and Milgrom formalized theory and did not conduct experiments estimating the effect size of specific KPIs.[1] Bevan and Hood conducted case analyses of specific periods in the UK healthcare system, so distortions in other industries may not occur with the same intensity.[2] Propper et al.’s difference-in-differences analysis also could not fully exclude concurrent changes across regions, and improvements in waiting times alone cannot evaluate overall medical quality.[3]", "sources": [ { "id": 1, "authors": "Bengt Holmström, Paul Milgrom", "year": 1991, "title": "Multitask Principal-Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design", "venue": "Journal of Law, Economics, & Organization, 7(Special Issue), 24–52", "url": "https://doi.org/10.1093/jleo/7.special_issue.24", "finding": "Theorized conditions under which strong incentives for easily measurable activities distort effort allocation when measurability differs across multiple tasks.", "limitation": "Did not directly estimate the average effect size of specific KPIs or counter-metrics in actual organizations using assumption-based formal models." }, { "id": 2, "authors": "Gwyn Bevan, Christopher Hood", "year": 2006, "title": "What’s Measured Is What Matters: Targets and Gaming in the English Public Health Care System", "venue": "Public Administration, 84(3), 517–538", "url": "https://doi.org/10.1111/j.1467-9299.2006.00600.x", "finding": "Analyzed goal governance in UK public healthcare, organizing problems governance faces due to metric representation, gaming, and information asymmetry.", "limitation": "Centered on administrative cases of specific periods and systems, this analysis is not a randomized experiment and does not determine the frequency of observed manipulations or pure causal effects." }, { "id": 3, "authors": "Carol Propper, Matt Sutton, Carolyn Whitnall, Frank Windmeijer", "year": 2008, "title": "Did ‘targets and terror’ reduce waiting times in England for hospital care?", "venue": "The B.E. Journal of Economic Analysis & Policy, 8(2), Article 5", "url": "https://doi.org/10.2202/1935-1682.1863", "finding": "Evaluated the relationship between strong goal systems and relative decreases in long-term hospitalization waits using a difference-in-differences comparison between England and Scotland.", "limitation": "As a quasi-experiment using a single healthcare system reform, concurrent changes across regions may remain, and results are also influenced by the choice of public versus internal values." } ], "url": "https://catalysphere.co.jp/en/articles/kpi-goodharts-law/", "machineReadableUrl": "https://catalysphere.co.jp/en/articles/kpi-goodharts-law/index.json", "topicFacets": [ { "id": "evaluation-measurement", "name": "Evaluation & Measurement", "url": "https://catalysphere.co.jp/en/topics/evaluation-measurement/" }, { "id": "organization-adoption", "name": "Organization & Adoption", "url": "https://catalysphere.co.jp/en/topics/organization-adoption/" } ] } }