Confidence Calibration in a Multiyear Geopolitical Forecasting Competition

@article{Moore2017ConfidenceCI,
  title={Confidence Calibration in a Multiyear Geopolitical Forecasting Competition},
  author={Don A. Moore and Samuel A. Swift and Angela Minster and Barbara A. Mellers and Lyle H. Ungar and Philip E. Tetlock and Heather H. J. Yang and Elizabeth R. Tenney},
  journal={Manag. Sci.},
  year={2017},
  volume={63},
  pages={3552-3565}
}
This research examines the development of confidence and accuracy over time in the context of forecasting. Although overconfidence has been studied in many contexts, little research examines its progression over long periods of time or in consequential policy domains. This study employs a unique data set from a geopolitical forecasting tournament spanning three years in which thousands of forecasters predicted the outcomes of hundreds of events. We sought to apply insights from research to… 
View via Publisher
escholarship.org
26 Citations
Background Citations
10
Methods Citations
1
Results Citations
1

26 Citations

Citation Type

Citation Type

Forecasting forecaster accuracy: Contributions of past performance and individual differences
A growing body of research indicates that forecasting skill is a unique and stable trait: forecasters with a track record of high accuracy tend to maintain this record. But how does one identify…
Forecasting the Accuracy of Forecasters from Properties of Forecasting Rationales
TLDR
Methods from natural language processing (NLP) and computational text analysis are adapted to identify distinctive reasoning strategies in the rationales of top forecasters, including cognitive styles that gauge tolerance of clashing perspectives and efforts to blend them into coherent conclusions.
Combining Prediction Intervals in the M4 Competition
TLDR
While averaging interval endpoints maintains its practical appeal as simple to implement and performs quite well when data sets are large, the median and the interior trimmed average are found to be robust aggregators for the prediction interval submissions across all 100,000 time series.
Combining prediction intervals in the M4 competition
A strategy to improve expert technology forecasts
TLDR
A hybrid approach to expert elicitation is outlined that iteratively combines the judgments of technical domain experts with those of experts who are knowledgeable about broader issues of technology adoption and public policy to improve forecasts of future technologies.
Worker Overconfidence: Field Evidence and Implications for Employee Turnover and Returns from Training
Combining weekly productivity data with weekly productivity beliefs for a large sample of truckers over two years, we show that workers tend to systematically and persistently over-predict their…
How well did experts and laypeople forecast the size of the COVID-19 pandemic?
TLDR
Experts substantially underestimated the ultimate extent of the pandemic, and the mean number of predictions for which the expert intervals contained the actual outcome was only 1.8, suggesting that experts should consider broadening the range of scenarios they consider plausible.
Distilling the Wisdom of Crowds: Prediction Markets versus Prediction Polls
We report the results of the first large-scale, long-term, experimental test between two crowd sourcing methods – prediction markets and prediction polls. More than 2,400 participants made forecasts…
Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls
TLDR
Team prediction polls outperformed prediction markets when poll forecasts were aggregated with algorithms using temporal decay, performance weighting and recalibration, and the biggest advantage of prediction polls occurred at the start of long-duration questions.
Worker overconfidence: Field evidence and implications for employee turnover and firm profits
Combining weekly productivity data with weekly productivity beliefs for a large sample of truckers over 2 years, we show that workers tend to systematically and persistently overpredict their…
...
1
2
3
...

References

SHOWING 1-10 OF 127 REFERENCES
Psychological Strategies for Winning a Geopolitical Forecasting Tournament
TLDR
Support is found for three psychological drivers of accuracy: training, teaming, and tracking in a 2-year geopolitical forecasting tournament that produced the best forecasts 2 years in a row.
Accuracy of forecasts in strategic intelligence
Significance Forecasting is a vital part of strategic intelligence, offering policy makers indications about probable future conditions and aiding sound decision making. Nevertheless, there has not…
Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions
  • B. Mellers, Eric Stone, P. Tetlock
  • Psychology
    Perspectives on psychological science : a journal of the Association for Psychological Science
  • 2015
TLDR
The findings suggest that superforecasters are partly discovered and partly created—and that the high-performance incentives of tournaments highlight aspects of human judgment that would not come to light in laboratory paradigms focused on typical performance.
Probability aggregation in time-series: Dynamic hierarchical modeling of sparse expert beliefs
TLDR
This paper presents a hierarchical model that takes into account the expert's level of self-reported expertise and produces aggregate probabilities that are sharp and well calibrated both in- and out-of-sample.
Overconfidence in Forecasts of Own Performance: An Experimental Study
Systematic overconfidence by individuals regarding their abilities and prospects could have important economic consequences. But overconfidence has received little direct testing within economics. We…
The effects of feedback and training on the performance of probability forecasters
The psychology of intelligence analysis: drivers of prediction accuracy in world politics.
TLDR
A profile of the best forecasters is developed; they were better at inductive reasoning, pattern detection, cognitive flexibility, and open-mindedness; they had greater understanding of geopolitics, training in probabilistic reasoning, and opportunities to succeed in cognitively enriched team environments.
Overconfidence and trading volume
Theoretical models predict that overconfident investors will trade more than rational investors. We directly test this hypothesis by correlating individual overconfidence scores with several measures…
Coherence, Calibration, and Expertise in Judgmental Probability Forecasting
Abstract This study investigates the relationship between judgmental probability forecasting performance, self-rated expertise, and degree of coherence with the probability laws. Self-rated expertise…
Distilling the Wisdom of Crowds: Prediction Markets versus Prediction Polls
We report the results of the first large-scale, long-term, experimental test between two crowd sourcing methods – prediction markets and prediction polls. More than 2,400 participants made forecasts…
...
1
2
3
4
5
...