How much can human prediction improve through training, aggregation, and falsification?

Chang et al. examined a training session of less than one hour covering probability, base rates, updating, and cognitive biases in a geopolitical forecasting tournament, reporting a 6–11% improvement in Brier accuracy compared to the control condition.[1] This suggests that even brief interventions can demonstrate the potential to share a format that translates vague intuitions into probabilities and receives feedback from outcomes.

In practice, we fix the prediction target, deadline, success criteria, and information sources used for judgment at the time the question is published. "I think it will succeed" cannot be scored, but "the probability that the published indicator meets the condition by the specified date" can be verified against results. Predictors are asked to briefly record base rates, key falsifications, and observations that would trigger updates, rather than the length of their reasoning.

At the time of question publication, we also determine the resolution authority and the process for handling disputes. If we do not fix how to handle cases where information sources are later revised, conditions are only partially met, or publication is delayed at the deadline, changes in scoring criteria rather than predictive ability will drive the scores. When validating the effects of training, use questions and scoring rules with the same definitions, and record participant selection differences separately. [1]

Baron et al. argued that simply averaging the probabilities of multiple individuals tends to compress them toward the neutral point, and that under certain conditions, correcting the aggregated value to be more extreme can bring it closer to actual frequencies.[2] Two reasons why aggregated probabilities become overly conservative are that only part of the information held by individuals is shared, and that averaging probabilities containing errors occurs.

However, extremization is not a ritual to strengthen confidence. The appropriate correction depends on whether the predictors' information is nearly independent or whether they are reacting en masse to the same news. First, collect initial values without seeing others' predictions, and compare the median, weighted average, and corrections calibrated with past data. Do not change aggregation rules conveniently after seeing results; instead, validate them externally by period.

To distinguish whether improvements in aggregated values are due to extremization or the weighting of a few experts, compare simple averages, pre-determined weights, and calibrated corrections over the same external period. If predictors rely on the same information sources, the number of independent pieces of information does not increase even if the number of people is large. If corrections worsen in groups with strong information overlap, review the diversity of information sources rather than the amount of correction. [2]

Hauenstein et al. re-analyzed the first two years of tournament data and reported that when adjusting for methodological variance such as the timing of answering questions and the selection of items answered, much of the training and team effects shrank, and in the best model, the training effect in the second year could not be confirmed. [3] The same data can lead to different conclusions depending on how one handles who answered what and when.

This is not a final conclusion that training is meaningless, but a warning against equating observed performance differences with intervention effects. If enthusiastic participants answer more questions, answer earlier, and answer questions they are good at, the response behavior itself affects the scores. In policy evaluation, save participation rates, response delays, missing data, and question difficulty, decide on the primary analysis before the intervention, and avoid comparing only convenient participants.

In intervention evaluation, who did not answer which question and when is also outcome data. If the training group selects only questions they are good at, answers many questions at an early stage, or has missing data for difficult questions, an alternative explanation of response behavior remains for the score differences. [3] If all participants cannot be compared on the same questions, report pre-determined missing data handling and sensitivity analyses, and do not make expansion decisions based solely on the performance of complete responders.

Corrections to aggregated probabilities depend on the overlap of information shared by predictors, and extremization does not always improve outcomes.[2] However, this methodology does not provide a procedure to automatically identify information sources within an organization. Below, we operationalize this as a proposal for pre-auditing the independence of reasoning rather than the number of people.

Simultaneously with initial predictions, always leave brief identifiers for the primary information, internal analysis, others' opinions, and speculations referenced. Collect this before sharing content, and do not count summaries or reprints of the same material as separate reasoning. Even if reasoning cannot be disclosed, distinguish whether it was independently obtained or came from a shared channel, and do not assume independence for unknown overlaps. Distinguish undisclosed information sources separately.

As a hypothetical example, if ten people read the same report, we do not treat it as ten independent observations. On the other hand, predictions reaching the same conclusion via different primary information are kept as separate paths. Aggregation rules compare sensitivity by reasoning group, showing whether the conclusion holds even when assuming strong overlap, rather than collapsing the largest group into a single vote.

When predictions are used for business decisions, subsequent actions can change the outcomes. This is not a section on estimating the causal effect of predictions, but an operational proposal to keep scoring targets distinct. Save unconditional predictions before action and conditional predictions for cases where measures are taken versus not taken as separate versions, and do not overwrite the implemented action later.

In a hypothetical example, a high predicted failure probability prompts prevention and no failure occurs. The outcome without prevention is unobserved. The pre-action unconditional forecast is still scored against the actual outcome under its predeclared event and deadline. Do not infer the causal value of acting on the forecast solely from the post-action result; retain the original assumptions, actions, and updates together. Keep the unobserved no-prevention conditional outcome separate from the unconditional forecast.

Do not add conditions retrospectively without a new question version. Fix when decision-makers saw the forecast and retain subsequent actions and updates as branches. Do not remove action-affected cases from the predeclared scoring set after seeing their outcomes. Report intervention status, the scored denominator, and unobserved conditional outcomes separately. Evaluating causal effects requires a separate design.

• EVIDENCE LINEAGE: Preserve the origin and shared path of primary information, internal analysis, others' opinions, and speculations
• FORECAST SNAPSHOT: Fix separate versions for before others view, before decision-making, and after action updates
• ACTION BRANCH: Connect the measures implemented after seeing the prediction with conditional predictions based on whether measures were taken
• SCORING STATUS: Distinguish between normal resolution, conditional resolution, caution in interpretation post-intervention, and unscorable

The unit of operation is not just the ranking of predictors. Track question design, base rate reference, independent initial predictions, aggregation, updating, resolution, and review in a single ledger. In reviews, rather than blaming narratives that missed, classify whether information was missing, known but weighted incorrectly, updated too late, or if the question itself was ambiguous.

Metrics include not just average scores, but calibration showing the match between probabilities and realized frequencies, discrimination separating occurring from non-occurring events, sharpness of predictions, and timeliness of updates. For high-impact decisions, record both aggregated values and minority counterarguments. What organizations need is not always those who hit, but mechanisms that record how they miss and can quickly correct the next judgment.

The owner of the prediction ledger separates question design, aggregation, resolution, and decision-making use. When calibration breaks, re-examine question definitions, information overlap, update delays, and resolution rules before blaming individuals. If aggregation methods are changed, compare them by going back to past periods, and revert to the old method if improvement is not seen in external periods. Do not adopt methods that eliminate minority counterarguments.

• Definition: Fix deadlines, success criteria, and judgment information sources at the time of question publication
• Independence: Record initial probabilities and key counterarguments before seeing others' values
• Aggregation: Pre-set averaging, weighting, and correction rules, and validate them over external periods
• Learning: Review calibration, discrimination, update speed, missing data, and response behavior