• Corpus ID: 2849592

An Empirical Comparison of Algorithms for Aggregating Expert Predictions

  title={An Empirical Comparison of Algorithms for Aggregating Expert Predictions},
  author={Varsha Dani and Omid Madani and David M. Pennock and Sumit K. Sanghai and Brian Galebach},
Predicting the outcomes of future events is a challenging problem for which a variety of solution methods have been explored and attempted. We present an empirical comparison of a variety of online and offline adaptive algorithms for aggregating experts' predictions of the outcomes of five years of US National Football League games (1319 games) using expert probability elicitations obtained from an Internet contest called ProbabilitySports. We find that it is difficult to improve over simple… 

Figures from this paper

The Wisdom of Crowds in Rank Ordering Problems

A Bayesian version of a Thurstonian model that aggregates orderings across individuals is introduced, and it is compared to heuristic aggregation techniques inspired by existing models of social choice and voting theory.

Prediction without markets

It is found that the relative advantage of prediction markets is surprisingly small, as measured by squared error, calibration, and discrimination, and as policy makers consider adoption, costs should be weighed against potentially modest benefits.

Team selection for prediction tasks

It is shown that in case of aggregating experts’ opinions by simple averaging, finding the best team can be modeled with an integer quadratic programming and it is proved its NP-hardness whereas its relaxation is solvable in polynomial time.

Exploiting oddsmaker bias to improve the prediction of NFL outcomes

Accurately predicting the outcome of sporting events has been a goal for many groups who seek to maximize profit. What makes this challenging is that the outcome of an event can be influenced by many

A Psychological Model for Aggregating Judgments of Magnitude

A psychologically-motivated model is developed and illustrated for aggregating judgments of magnitude across experts that provides aggregated estimates that are implicitly based on the application of nonlinear weights to individual judgments.

The Wisdom of Crowds in the Recollection of Order Information

This research reports the performance of individuals in a series of general knowledge tasks, where the goal is to reconstruct from memory the order of historic events, or the orderof items along some physical dimension, using two Bayesian models for aggregating order information.

Real-time ranking with concept drift using expert advice

This work presents an online learning approach to ranking with concept drift, using weighted majority techniques, and demonstrates that it outperforms previously known batch-learning methods and other online methods that do not account for concept drift.

An axiomatic characterization of wagering mechanisms

TwitterPaul: Extracting and Aggregating Twitter Predictions

TwitterPaul is introduced, a system designed to make use of Social Media data to help to predict game outcomes for the 2010 FIFA World Cup tournament and it is found that the quality of extractions is more important than the quantity, suggesting that high precision methods working on a medium-sized dataset are preferable over low precision methods that use a larger amount of data.

Answering the een: Machine Learning and Financial Crises

Financial crises cause economic, social and political havoc. Macroprudential policies are gaining traction but are still severely under-researched compared to monetary and €scal policy. We use the



Information markets vs. opinion pools: an empirical comparison

A unique data source of almost 2000 people's subjective probability judgments on 2003 US National Football League games is used and it is found that arithmetic average is a robust and efficient pooling function and logarithmic aggregation functions offer bolder predictions than linear aggregation functions.

A Generative Bayesian Model for Aggregating Experts' Probabilities

This work develops a generative Bayesian aggregation model for probabilistic classification that results in a weighted logarithmic opinion pool (LogOps) that satisfies consistency criteria such as the external Bayesian property.

Prediction Markets

We analyze the extent to which simple markets can be used to aggregate disperse information into efficient forecasts of uncertain future events. Drawing together data from a range of prediction

How to use expert advice

This work analyzes algorithms that predict a binary value by combining the predictions of several prediction strategies, called `experts', and shows how this leads to certain kinds of pattern recognition/learning algorithms with performance bounds that improve on the best results currently known in this context.

Prediction Markets: Does Money Matter?

A real-world on-line experiment pitting the predictions of TradeSports.com and NewsFutures.com regarding American Football outcomes during the fall-winter 2003-2004 NFL season revealed both types of markets exhibited significant predictive powers, and remarkable performance compared to individual humans.

Representing Aggregate Belief through the Competitive Equilibrium of a Securities Market

It is argued that the market-based approach provides a plausible mechanism for belief aggregation in multiagent systems, as it directly addresses self-motivated agent incentives for participation and for truthfulness, and can provide a decision-theoretic foundation for the "expert weights" often employed in centralized pooling techniques.

Combining Probability Distributions: A Critique and an Annotated Bibliography

A taxonomy of solutions is presented which serves as the framework for a survey of recent theoretical developments in the area and a number of current research directions are mentioned and an extensive, current annotated bibliography is included.

Consensus diagnosis: a simulation study

Comparative and individual analyses indicated that for panels assembled under circumstances similar to those of this study, small, simple methods work best.

“Good” Probability Assessors

Abstract Since a meteorologist's predictions are subjective, a framework for the evaluation of meteorological probability assessors must be consistent with the theory of subjective probability. Such

Exponentiated Gradient Versus Gradient Descent for Linear Predictors

The bounds suggest that the losses of the algorithms are in general incomparable, but EG(+/-) has a much smaller loss if only a few components of the input are relevant for the predictions, which is quite tight already on simple artificial data.