Where do data quality defects become decision-making errors?

Wang and Strong extracted 179 quality attributes from surveys of data users and organized them into 15 dimensions across four categories: intrinsic, contextual, representational, and accessibility. [1] This framework posits that even if values match facts, quality for decision-making is low if the data is outdated, difficult to interpret, lacks necessary granularity, or is inaccessible due to permissions.

Therefore, quality criteria should be defined at the decision-making level, not merely as pass/fail for datasets. By first specifying who compares which options and when, we assign required levels of freshness, completeness, consistency, and traceability. Rather than polishing all items equally, it is easier to link quality investment to decision value by inspecting variables that could reverse the conclusion.

The priority of critical data is determined by decision sensitivity, not the magnitude of missing rates. We shift freshness, values, and granularity within realistic ranges and register as critical data elements those items where recommended choices or target populations change. Since quality dimensions change meaning depending on the usage context, we do not carry over the same passing stamp when repurposing for another decision but instead re-run sensitivity tests. [1]

Obermeyer et al. audited an algorithm selecting patients for a healthcare management program. Black patients had a higher disease burden at the same risk score. In a counterfactual simulation removing the racial gap in disease burden, the proportion of Black patients automatically selected above the 97th percentile of risk scores was estimated to rise from 17.7% to 46.5%. [2] Even accurate cost data can introduce healthcare access disparities when used as a proxy for need.

The place to correct here is not only model accuracy. We map business objectives, prediction labels, input data, thresholds, on-site actions, and final outcomes as a single causal chain, placing counterexamples at each arrow. By explicitly stating the conditions under which proxy indicators represent the objective and the groups, periods, or systems where they no longer do, we can quickly identify the risk that accuracy improvements may exacerbate unfairness or deviation from the objective.

When group differences are found, we do not fix the cause solely on input value errors. We list competing hypotheses such as the possibility that proxy indicators missed the objective, disparities in service access, and differences in on-site responses after thresholds, comparing which changes move the target selection. [2] If disparities remain after label correction, we return the problem to the redesign of the objective variable or business process rather than model improvement.

Sambasivan et al. interviewed 53 AI practitioners in India, East and West Africa, and the US, confirming data cascades in 92% of the surveyed cases. [3] The problem is not simple missing data but the process by which upstream compromises—such as ambiguous collection purposes, distance from experts, and undervalued data work—amplify into technical and ethical issues downstream.

Treating quality defects as a lack of individual attention discourages reporting. We must evaluate those who halted inputs, challenged label definitions, or recorded exceptions during operation as those who prevented losses, rather than those who delayed deadlines. Quality meetings must address not only average missing rates but also who was unable to correct and had to detour, and which exceptions fell silent.

In one retrospective audit using medical costs as a proxy for need, it was shown that even with the same risk scores, there were differences in disease burden, and replacing the objective changed the selection composition. [2] This is not evidence that proxy indicators in other domains fail for the same reason; thus, we limit cost or group differences to insights for finding discordant cases rather than generalizing them.

The following are diagnostic recommendations. We extract pairs of cases where the proxy value is the same but the objective state differs, and cases where the objective state is similar but the proxy value differs. For each pair, we list individual explanatory candidates such as recording errors, usage opportunities, institutional pricing, timing differences, and target definitions, removing only those candidates rejected by additional data. We also fix the rules for selecting pairs.

Even if we find conditions where inconsistencies concentrate, we do not immediately call them causes. We perform recalculation excluding the conditions, verification with alternative measurements closer to the objective, and case reviews with blinded decision-makers, comparing which changes move the target selection. If explanations remain competing, we stop automatic determination by proxy values and switch to a category requiring additional information.

The number of defects corrected does not reveal the impact on decisions. In recommended case audits, we save the pre-correction input, the conversion at that time, the model output, and the operator's judgment, then substitute only the corrected value to replay the same process. We extract cases where the result changed as the range potentially affected. We also retain reasons for cases excluded from replay.

There are other explanations for differences after replay. If the current model or rules differ from those at the time, we cannot create the same processing conditions, and we cannot attribute the change solely to the corrected value. Therefore, we distinguish between cases where the version at the time can be reproduced and those that can only be tested with the current version. The latter are treated as candidates for investigation rather than confirmed impact, and we do not automate the withdrawal of past judgments.

For affected cases, we do not close with re-evaluation alone but record whether the original action was executed, whether it can be canceled, and whether corrections reached the stakeholders. Retain unchanged cases and record how their conditions differ from changed cases. If using a negative control, specify its selection and expected invariance in advance. Recovery completion is determined by the status of the target decision's measures, not the data correction date.

• DISCORDANT PAIR: Save two-way case pairs where the correspondence between proxy value and objective state breaks down.
• EXPLANATION SET: Retain candidates for recording, usage opportunity, institution, timing, and target definition with rejection results.
• REPLAY: Use the version at the time and corrected values to extract only cases where output and judgment changed.
• REMEDY STATUS: Track unexecuted, executed, cancellable, and correction-notify status by decision unit.

In practice, we create a quality ledger for each critical decision. We place the objective, target population, indicator definition, data owner, generation time, known missing data, proxy indicator assumptions, groups susceptible to impact, and stop conditions in the same place. Upon changes, we re-approve not only the model but also the combination of definition and usage scenario, notifying users that comparison with the past has been severed.

Furthermore, we link predictions, operator judgments, executed actions, outcomes, and later-discovered corrections. This allows us to learn separately from input anomalies, proxy indicator failures, on-site non-compliance, and environmental changes. The goal of quality metrics is not to show zero defects but to maintain a state where critical defects are discovered before reaching decision-making and can be recovered even after reaching it.

We do not place ownership of the quality ledger solely with the data department but also list the business owner who defines the criteria and the system owner who executes corrections. Monitoring includes not only the number of defects but also whether corrections were reflected downstream from discovery to decision stop, and whether erroneous judgments were withdrawn. Given the insight that upstream compromises chain, we do not treat cases with uncorrectable detours as complete. [3]

• WHAT: Prioritize definitions, freshness, missing data, and proxy indicators that reverse decisions.
• WHO: Determine the owner, correction authority, and objection channel for each data item.
• WHEN: Re-inspect at collection, model update, institutional change, and outcome revelation.
• LEARN: Link from prediction to outcome and record points where errors amplified.
• Business Owner: Explicitly state conditions where the objective and proxy indicator align.
• Data Owner: Prioritize correction of quality defects that reverse decisions.
• Operations Owner: Close the loop on stop, correction, and reflection to affected parties.