• Corpus ID: 19098048

Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments

@article{Chang2016DevelopingEP,
  title={Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments},
  author={Welton Chang and Eva Chen and Barbara A. Mellers and Philip E. Tetlock},
  journal={Judgment and Decision Making},
  year={2016},
  volume={11},
  pages={509-526}
}
The heuristics-and-biases research program highlights reasons for expecting people to be poor intuitive forecasters. This article tests the power of a cognitive-debiasing training module (“CHAMPS KNOW†) to improve probability judgments in a four-year series of geopolitical forecasting tournaments sponsored by the U.S. intelligence community. Although the training lasted less than one hour, it consistently improved accuracy (Brier scores) by 6 to 11% over the control condition. Cognitive… 
Accelerating Learning in Active Management: The Alpha-Brier Process
TLDR
The authors propose the Alpha-Brier process to help firms improve forecasts guiding investment decisions and lets managers test which methods of training, incentivizing, and aggregating judgments deliver the biggest boosts to accuracy.
Forecasting the Accuracy of Forecasters from Properties of Forecasting Rationales
TLDR
Methods from natural language processing (NLP) and computational text analysis are adapted to identify distinctive reasoning strategies in the rationales of top forecasters, including cognitive styles that gauge tolerance of clashing perspectives and efforts to blend them into coherent conclusions.
The Value of Precision in Probability Assessment: Evidence from a Large-Scale Geopolitical Forecasting Tournament
Scholars, practitioners, and pundits often leave their assessments of uncertainty vague when debating foreign policy, arguing that clearer probability estimates would provide arbitrary detail instead
Bias, Information, Noise: The BIN Model of Forecasting
TLDR
A Bayesian BIN model (Bias, Information, Noise) is proposed for disentangling the underlying processes that enable forecasters and forecasting methods to improve – either by tamping down bias and noise in judgment or by ramping up the efficient extraction of valid information from the environment.
From discipline-centered rivalries to solution-centered science: Producing better probability estimates for policy makers.
From 2011 to 2015, the U.S. intelligence community sponsored a series of forecasting tournaments that challenged university-based researchers to invent measurably better methods of forecasting
Rethinking the training of intelligence analysts
TLDR
This work proposes a new approach to analytic training, adopting scientifically validated content and regularly testing training to avoid institutionalizing new dogmas, and incentivizing analysts to view training guidelines as means to the end of improved accuracy, not an end in itself.
A survey of human judgement and quantitative forecasting methods
TLDR
It is argued that future research in the field of human/machine forecasting needs to consider all of them when investigating predictive performance, including quantitative, data-driven forecasting models and hybrid methods that involve both humans and algorithmic approaches.
Improving national strategic foresight with the use of forecasting tournaments and its implications for the study of international relations
  • J. Kleňha
  • Political Science
    Stosunki Międzynarodowe – International Relations
  • 2022
Improving national strategic foresight can help the formation of more robust and informed policies, including foreign policy. Predicated upon the theory behind peer-prediction elicitation methods
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 106 REFERENCES
Intuitive Prediction: Biases and Corrective Procedures
Introduction Any significant activity of forecasting involves a large component of judgment, intuition, and educated guesswork. Indeed, the opinions of experts are the source of many technological,
Forecasting Tournaments
Forecasting tournaments are level-playing-field competitions that reveal which individuals, teams, or algorithms generate more accurate probability estimates on which topics. This article describes a
Psychological Strategies for Winning a Geopolitical Forecasting Tournament
TLDR
Support is found for three psychological drivers of accuracy: training, teaming, and tracking in a 2-year geopolitical forecasting tournament that produced the best forecasts 2 years in a row.
Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions
  • B. Mellers, Eric Stone, P. Tetlock
  • Psychology
    Perspectives on psychological science : a journal of the Association for Psychological Science
  • 2015
TLDR
The findings suggest that superforecasters are partly discovered and partly created—and that the high-performance incentives of tournaments highlight aspects of human judgment that would not come to light in laboratory paradigms focused on typical performance.
Curbing Optimism Bias and Strategic Misrepresentation in Planning: Reference Class Forecasting in Practice
Absract The American Planning Association recently endorsed a new forecasting method called reference class forecasting, which is based on theories of planning and decision-making that won the 2002
Improving Reliability of Judgmental Forecasts
TLDR
Five principles can improve reliability in judgmental forecasting: Organize and present information in a form that clearly emphasizes relevant information, limit the amount of information used, and require justification of forecasts.
Robust analogizing and the outside view: two empirical tests of case‐based decision making
TLDR
This work examines models of analogy and presents findings from two empirical tests, finding that an ‘outside view’—forming a reference class of analogies—performs better than a few analogies familiar to the decision maker.
Judgment under Uncertainty
The thirty-five chapters in this book describe various judgmental heuristics and the biases they produce, not only in laboratory experiments but in important social, medical, and political situations
The role of actively open-minded thinking in information acquisition, accuracy, and calibration
TLDR
This work examines whether attributes associated with persistence in information acquisition can predict performance in an estimation task, and focuses on actively open-minded thinking, need for cognition, grit, and the tendency to maximize or satisfice when making decisions.
Improving Predictive Accuracy with a Combination of Human Intuition and Mechanical Decision Aids.
TLDR
Examination of the intuitive combination of human judgment and mechanical prediction under varied information conditions provides insight into how and when combining mechanical prediction and human intuition is likely to result in improved accuracy.
...
1
2
3
4
5
...