• DOI:10.1093/bjps/axm033
• Corpus ID: 8863978

Bayes not Bust! Why Simplicity is no Problem for Bayesians1

@article{Dowe2007BayesNB,
  title={Bayes not Bust! Why Simplicity is no Problem for Bayesians1},
  author={David L. Dowe and Steve Gardner and Graham Oppy},
  journal={The British Journal for the Philosophy of Science},
  year={2007},
  volume={58},
  pages={709 - 754}
}

• D. Dowe, Steve Gardner, G. Oppy
• Published 1 December 2007
• Computer Science
• The British Journal for the Philosophy of Science

The advent of formal definitions of the simplicity of a theory has important implications for model selection. But what is the best way to define simplicity? Forster and Sober ([1994]) advocate the use of Akaike's Information Criterion (AIC), a non-Bayesian formalisation of the notion of simplicity. This forms an important part of their wider attack on Bayesianism in the philosophy of science. We defend a Bayesian alternative: the simplicity of a theory is to be characterised in terms of Wallace's Minimum Message Length (MML). We show that AIC is inadequate for many statistical problems where MML performs well. Whereas MML is always defined, AIC can be undefined. Whereas MML is not known ever to be statistically inconsistent, AIC can be. Even when defined and consistent, AIC performs worse than MML on small sample sizes. MML is statistically invariant under 1-to-1 re-parametrisation, thus avoiding a common criticism of Bayesian approaches. We also show that MML provides answers to many of Forster's objections to Bayesianism. Hence an important part of the attack on Bayesianism fails. 1. Introduction2. The Curve Fitting Problem 2.1 Curves and families of curves 2.2 Noise 2.3 The method of Maximum Likelihood 2.4 ML and over-fitting3. Akaike's Information Criterion (AIC)4. The Predictive Accuracy Framework5. The Minimum Message Length (MML) Principle 5.1 The Strict MML estimator 5.2 An example: The binomial distribution 5.3 Properties of the SMML estimator 5.3.1  Bayesianism 5.3.2  Language invariance 5.3.3Generality 5.3.4  Consistency and efficiency 5.4 Similarity to false oracles 5.5 Approximations to SMML6. Criticisms of AIC 6.1 Problems with ML 6.1.1  Small sample bias in a Gaussian distribution 6.1.2  The von Mises circular and von Mises—Fisher spherical distributions 6.1.3  The Neyman–Scott problem 6.1.4  Neyman–Scott, predictive accuracy and minimum expected KL distance 6.2 Other problems with AIC 6.2.1  Univariate polynomial regression 6.2.2  Autoregressive econometric time series 6.2.3  Multivariate second-order polynomial model selection 6.2.4  Gap or no gap: a clustering-like problem for AIC 6.3 Conclusions from the comparison of MML and AIC7. Meeting Forster's objections to Bayesianism 7.1 The sub-family problem 7.2 The problem of approximation, or, which framework for statistics?8. ConclusionA. Details of the derivation of the Strict MML estimatorB. MML, AIC and the Gap vs. No Gap Problem B.1 Expected size of the largest gap B.2 Performance of AIC on the gap vs. no gap problem B.3 Performance of MML in the gap vs. no gap problem Introduction The Curve Fitting Problem 2.1 Curves and families of curves 2.2 Noise 2.3 The method of Maximum Likelihood 2.4 ML and over-fitting Akaike's Information Criterion (AIC) The Predictive Accuracy Framework The Minimum Message Length (MML) Principle 5.1 The Strict MML estimator 5.2 An example: The binomial distribution 5.3 Properties of the SMML estimator 5.3.1  Bayesianism 5.3.2  Language invariance 5.3.3Generality 5.3.4  Consistency and efficiency 5.4 Similarity to false oracles 5.5 Approximations to SMML Criticisms of AIC 6.1 Problems with ML 6.1.1  Small sample bias in a Gaussian distribution 6.1.2  The von Mises circular and von Mises—Fisher spherical distributions 6.1.3  The Neyman–Scott problem 6.1.4  Neyman–Scott, predictive accuracy and minimum expected KL distance 6.2 Other problems with AIC 6.2.1  Univariate polynomial regression 6.2.2  Autoregressive econometric time series 6.2.3  Multivariate second-order polynomial model selection 6.2.4  Gap or no gap: a clustering-like problem for AIC 6.3 Conclusions from the comparison of MML and AIC Meeting Forster's objections to Bayesianism 7.1 The sub-family problem 7.2 The problem of approximation, or, which framework for statistics? Conclusion Details of the derivation of the Strict MML estimator MML, AIC and the Gap vs. No Gap Problem B.1 Expected size of the largest gap B.2 Performance of AIC on the gap vs. no gap problem B.3 Performance of MML in the gap vs. no gap problem