Does Big Data Really Improve Decision-Making?

Ghasemaghaei and Turel surveyed 149 data analysts at companies utilizing big data. They found a positive relationship between data volume and speed and the perception of knowledge hiding behaviors such as obscuring information or feigning ignorance, and a negative relationship between avoidant knowledge hiding and self-reported decision quality. [1] Data volume reaches decisions through accountability and sharing behaviors within the organization.

However, this is a cross-sectional questionnaire survey and does not prove that data volume causes knowledge hiding. The key point is not to evaluate the data foundation solely on technology. Even with the same sales or customer data, if definitions, aggregation timing, missing data handling, and viewing permissions differ by department, increasing the numbers may lead to more disputes and selective use.

When measuring foundation value, in addition to holding volume and processing speed, track the time it takes to resolve definition conflicts for the same metric, decisions held up without evidence, and history of updating decisions based on results. Since the association with knowledge hiding does not establish causality, do not conclude that higher numbers are bad; instead, investigate whether power disparities and weak accountability are occurring simultaneously. [1]

Shmueli organized that models explaining phenomena and models predicting unknown outcomes have different purposes, and high explanatory power does not imply high predictive power. [2] Explaining sales declines, predicting customers likely to churn, and estimating the effect of discount strategies are not the same analyses, even if they use similar data. Mixing questions leads to mistaking variables with strong correlations as policy levers.

For analysis requests, write who, when, and what choice the results are for. For description, consistency of definitions is central; for prediction, reproducibility on unused data; for causality, comparability of counterfactuals. Do not collapse model accuracy into a single common score; instead, decide the type of error acceptable for the decision-making purpose first.

In analysis reviews, place at least competing explanations for why the results are correct. Even if predictive accuracy is high, changing the policy does not necessarily change the outcome. [2] If the decision-maker requires intervention effects, do not approve based solely on the ranking of predictive models; return to the design where the comparison group, start time, and outcomes were defined first. [3]

• Description: Align what happened using the same definitions and periods
• Prediction: Determine which errors to reduce with future unobserved data
• Causality: Verify what changes compared to the case where no intervention occurred
• Operation: Decide who receives the results, deadlines, options, and exception handling

Hernán and Robins showed the framework of a target trial, which explicitly states the hypothetical trial to answer when considering policy effects from observational data where randomized trials are not possible. [3] By placing subjects, the policies to compare, allocation timing, follow-up start, outcomes, and analysis methods first, it becomes easier to spot the danger of changing the question to fit the available data.

Even with large amounts of history, biases such as unrecorded reasons for choosing a policy, different conditions for the comparison group, and mixing pre-start information with post-start data do not disappear. Analysis of observational data begins with explicitly stating the missing comparison conditions. Advanced algorithms do not automatically fix unmeasured confounders or incorrect timelines.

The impact of the same prediction error varies by case. For cases far from the decision boundary, small accuracy improvements do not change behavior, while near the boundary, slight uncertainty can reverse the choice. Therefore, define the band where actual choices change, not just the average accuracy of all cases, and measure errors and holds within that band. This is a proposal for allocating analytical resources, not a new empirical result.

Consider a hypothetical example. In predicting which cases require additional support, assume that top and bottom cases do not change judgment with current information, but only those near the boundary have unstable rankings. In this case, rather than adding new data to all cases, test input verification and human re-judgment for the boundary band. However, since the boundary itself moves due to cost or personnel constraints, keep the number of selectable cases as a version alongside model values.

The recommended evaluation is to record misselections, holds, and changes after verification in the boundary band, without masking them with high accuracy outside the boundary. If specific groups or difficult cases are concentrated near the boundary, do not use average improvement as the basis for application; review the application conditions. Distinguish whether cases where the decision-maker did not use the prediction were those where prediction was unnecessary or those where it was not trusted.

Decisions to increase data sources need an endpoint. Even if new columns or history improve analytical metrics, if they do not change the choice, boundary, or execution action, the decision-making value is limited. Compare the cost of additional acquisition, definition adjustments, delays, explanation burden, and the possibility of avoiding misselection; stop connections once the range where the conclusion does not change is confirmed. This is not capacity reduction, but a determination of sufficiency for the question.

Consider a hypothetical example. Adding certain external data may finely change prediction rankings, but the executable top-tier cases remain the same, and judgments after verification do not change. In that case, you can decline additional connections after confirming reproducibility. Conversely, if there is a possibility of reducing significant misselections in the boundary band, test the information value with a limited sample rather than company-wide unification. Do not calculate benefits assuming the unknown is zero.

The recommended review involves documenting the question to answer, potential changes in choice, current uncertainty, and actions to take after addition for each data source. Do not adopt based solely on the expectation of accuracy improvement if actions cannot be changed after observation, deadlines cannot be met, or comparison conditions are broken. For data not adopted, leave re-review conditions as candidates that do not currently change the judgment, rather than deeming them unnecessary.

• Decision-Making: Fix who can change which choice and when
• Inversion Boundary: Place the value/rank/uncertainty at which the choice changes
• Additional Information: Connect acquisition cost, delay, definition adjustment, and expected changes
• Cutoff: Define conditions where choices do not change with addition and triggers for re-review

A valuable data foundation does not just collect inputs; it returns decisions, actions, results, and reasons for correction. When predictions miss, do not blame only the model; separately record changes in input definitions, workarounds on the ground, policy implementation rates, and changes in the external environment. This allows distinguishing between accuracy decline and operational failure.

The starting point is defining one management choice, the necessary evidence, deadlines, loss of error, and responsible person. Then link data, decide comparison methods, and save reasons for not using results. Tracking the percentage of decisions updated and the time to update, rather than storage volume, prevents integration itself from becoming the purpose.

Also decide on stopping conditions for data integration. If adding new data sources does not change choices, results cannot be verified, or definition conflict adjustments exceed decision deadlines, stop the plan to increase connections and repair the current decision path. Data owners manage foundation operation, and business owners manage decision updates; do not consider value creation until both metrics align.

• Definition: Manage the population, denominator, timing, and exclusion conditions of metrics with an owner
• Comparison: Fix the baseline group, intervention timing, observation period, and confounders before analysis
• Action: Decide who changes what at which threshold and who receives exceptions
• Learning: Record differences between prediction, decision, implementation, and results, and feed back into definitions and procedures
• Analysis Owner: Fix the purpose of description, prediction, and causality at the time of request
• Data Owner: Resolve definition conflicts and inability to track results
• Business Owner: Save reasons for updating choices after analysis