On the Jeffreys-Lindley Paradox and the Looming Reproducibility Crisis in Machine Learning

@article{Berrar2017OnTJ,
  title={On the Jeffreys-Lindley Paradox and the Looming Reproducibility Crisis in Machine Learning},
  author={Daniel P. Berrar and Werner Dubitzky},
  journal={2017 IEEE International Conference on Data Science and Advanced Analytics (DSAA)},
  year={2017},
  pages={334-340}
}
Null hypothesis significance testing has become a mainstay in machine learning, with the p-value being firmly embedded in the current research practice. Significance testing is widely believed to lend scientific rigor to the interpretation of empirical findings; however, its serious problems have received scant attention in the machine learning literature so far. Here, we investigate one particular problem: the Jeffreys-Lindley paradox. This paradox describes a statistical conundrum where the… CONTINUE READING