Psychological Strategies for Winning a Geopolitical Forecasting Tournament

@article{Mellers2014PsychologicalSF,
  title={Psychological Strategies for Winning a Geopolitical Forecasting Tournament},
  author={Barbara A. Mellers and Lyle Ungar and Jonathan Baron and Jaime Ramos and Burcu Gurcay and Katrina M. Fincher and Sydney E. Scott and Don A. Moore and Pavel Atanasov and Samuel A. Swift and Terry Murray and Eric Stone and Philip E. Tetlock},
  journal={Psychological Science},
  year={2014},
  volume={25},
  pages={1106 - 1115}
}
Five university-based research groups competed to recruit forecasters, elicit their predictions, and aggregate those predictions to assign the most accurate probabilities to events in a 2-year geopolitical forecasting tournament. Our group tested and found support for three psychological drivers of accuracy: training, teaming, and tracking. Probability training corrected cognitive biases, encouraged forecasters to use reference classes, and provided forecasters with heuristics, such as… 
Confidence Calibration in a Multiyear Geopolitical Forecasting Competition
TLDR
This study employs a unique data set from a geopolitical forecasting tournament spanning three years in which thousands of forecasters predicted the outcomes of hundreds of events to apply insights from research to structure the questions, interactions, and elicitations to improve forecasts.
Forecasting Tournaments
Forecasting tournaments are level-playing-field competitions that reveal which individuals, teams, or algorithms generate more accurate probability estimates on which topics. This article describes a
Geopolitical Forecasting Skill in Strategic Intelligence
Extending research by the authors on intelligence forecasting, the forecasting skill of 3622 geopolitical forecasts extracted from strategic intelligence reports was examined. The codable subset of
Forecasting the Accuracy of Forecasters from Properties of Forecasting Rationales
TLDR
Methods from natural language processing (NLP) and computational text analysis are adapted to identify distinctive reasoning strategies in the rationales of top forecasters, including cognitive styles that gauge tolerance of clashing perspectives and efforts to blend them into coherent conclusions.
Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments
The heuristics-and-biases research program highlights reasons for expecting people to be poor intuitive forecasters. This article tests the power of a cognitive-debiasing training module (“CHAMPS
From discipline-centered rivalries to solution-centered science: Producing better probability estimates for policy makers.
From 2011 to 2015, the U.S. intelligence community sponsored a series of forecasting tournaments that challenged university-based researchers to invent measurably better methods of forecasting
What Makes Foreign Policy Teams Tick: Explaining Variation in Group Performance at Geopolitical Forecasting
When do groups—be they countries, administrations, or other organizations—more or less accurately understand the world around them and assess political choices? Some argue that group decision-making
Forecasting forecaster accuracy: Contributions of past performance and individual differences
A growing body of research indicates that forecasting skill is a unique and stable trait: forecasters with a track record of high accuracy tend to maintain this record. But how does one identify
Item response models of probability judgments: Application to a geopolitical forecasting tournament.
TLDR
This article develops and study methods for evaluating forecasters and forecasting questions in dynamic environments based on item response models and extends traditional models to handle subjective probabilities and applies a specific model to geopolitical forecasts.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 46 REFERENCES
The psychology of intelligence analysis: drivers of prediction accuracy in world politics.
TLDR
A profile of the best forecasters is developed; they were better at inductive reasoning, pattern detection, cognitive flexibility, and open-mindedness; they had greater understanding of geopolitics, training in probabilistic reasoning, and opportunities to succeed in cognitively enriched team environments.
Two Reasons to Make Aggregated Probability Forecasts More Extreme
TLDR
It is shown that the same transformation function can approximately eliminate both distorting effects with different parameters for the mean and the median, and how, in principle, use of the median can help distinguish the two effects.
Probability aggregation in time-series: Dynamic hierarchical modeling of sparse expert beliefs
TLDR
This paper presents a hierarchical model that takes into account the expert's level of self-reported expertise and produces aggregate probabilities that are sharp and well calibrated both in- and out-of-sample.
The effects of feedback and training on the performance of probability forecasters
Calibration of Subjective Probability Judgments in a Naturalistic Setting.
TLDR
A close correlation is observed between the subjective probability judgments of horses' success, implicit in the bettors' wagering activities, and the objective probability of success of the horses concerned.
Robust analogizing and the outside view: two empirical tests of case‐based decision making
TLDR
This work examines models of analogy and presents findings from two empirical tests, finding that an ‘outside view’—forming a reference class of analogies—performs better than a few analogies familiar to the decision maker.
Strategies for revising judgment: how (and how well) people use others' opinions.
TLDR
The authors developed the probability, accuracy, redundancy (PAR) model and found that averaging was the more effective strategy across a wide range of commonly encountered environments and that despite this finding, people tend to favor the choosing strategy.
Pooling of Unshared Information in Group Decision Making: Biased Information Sampling During Discussion
Decision-making groups can potentially benefit from pooling members' information, particularly when members individually have partial and biased information but collectively can compose an unbiased
Probability Forecasting in Meteorology
TLDR
The history of probability forecasting in meteorology is reviewed to acquaint statisticians with this body of literature and recent methodological, experimental, and operational activities in this field are described.
Facing uncertainty in the game of bridge: A calibration study
...
1
2
3
4
5
...