Optional Stopping with Bayes Factors: a categorization and extension of folklore results, with an application to invariant situations

@article{Hendriksen2018OptionalSW,
  title={Optional Stopping with Bayes Factors: a categorization and extension of folklore results, with an application to invariant situations},
  author={Allard A. Hendriksen and Rianne de Heide and Peter Gr{\"u}nwald},
  journal={ArXiv},
  year={2018},
  volume={abs/1807.09077}
}
It is often claimed that Bayesian methods, in particular Bayes factor methods for hypothesis testing, can deal with optional stopping. We first give an overview, using elementary probability theory, of three different mathematical meanings that various authors give to this claim: (1) stopping rule independence, (2) posterior calibration and (3) (semi-) frequentist robustness to optional stopping. We then prove theorems to the effect that these claims do indeed hold in a general measure… Expand
Why optional stopping is a problem for Bayesians
Recently, optional stopping has been a subject of debate in the Bayesian psychology community. Rouder (2014) argues that optional stopping is no problem for Bayesians, and even recommends the use ofExpand
Why optional stopping can be a problem for Bayesians
TLDR
It is illustrated that, as soon as the parameters of interest are equipped with default or pragmatic priors—which means, in most practical applications of Bayes factor hypothesis testing—resilience to optional stopping can break down. Expand
Safe Testing
TLDR
Sharing Fisherian, Neymanian and Jeffreys-Bayesian interpretations, S-values and safe tests may provide a methodology acceptable to adherents of all three schools. Expand
Safe tests for 2 x 2 contingency tables and the Cochran-Mantel-Haenszel test
Rationale In my thesis, I developed implementations of safe statistics: a new framework for collecting evidence for hypotheses, particularly suitable for online and sequential learning [1].Expand
The Bayesian Methodology of Sir Harold Jeffreys as a Practical Alternative to the P Value Hypothesis Test
TLDR
Harold Jeffreys’s Bayes factor methodology as implemented in the open-source software JASP is explained and its practical relevance is showcased with two examples. Expand
The Bayesian Methodology of Sir Harold Jeffreys as a Practical Alternative to the P-value Hypothesis Test
TLDR
Harold Jeffreys’s Bayes factor methodology as implemented in the open-source software JASP is explained and its practical relevance is showcased with two examples. Expand
Bayes Factors for Peri-Null Hypotheses
A perennial objection against Bayes factor point-null hypothesis tests is that the point-null hypothesis is known to be false from the outset. Following Morey and Rouder (2011) we examine theExpand
Bayesian inference in numerical cognition: A tutorial using JASP
Researchers in numerical cognition rely on hypothesis testing and parameter estimation to evaluate the evidential value of data. Though there has been increased interest in Bayesian statistics as anExpand
Bayesian Hodges-Lehmann tests for statistical equivalence in the two-sample setting: Power analysis, type I error rates and equivalence boundary selection in biomedical research
  • Riko Kelter
  • Medicine
  • BMC Medical Research Methodology
  • 2021
TLDR
Bayesian Hodges-Lehmann tests for statistical equivalence differ in their sensitivity to the prior modeling, power, and the associated type I error rates, but results show that the relationship is even reverse for the Bayes factor and ROPE based equivalence tests. Expand
The Perils of Misspecified Priors and Optional Stopping in Multi-Armed Bandits
  • Markus Loecher
  • Computer Science, Medicine
  • Frontiers in Artificial Intelligence
  • 2021
TLDR
The effects of optional stopping in a particular class of multi-armed bandit experiments, which randomly allocates observations to arms proportional to the Bayesian posterior probability that each arm is optimal (Thompson sampling), are investigated. Expand
...
1
2
...

References

SHOWING 1-10 OF 58 REFERENCES
Optional stopping: No problem for Bayesians
  • J. Rouder
  • Psychology, Medicine
  • Psychonomic bulletin & review
  • 2014
TLDR
In this article, it is shown through simulation that the interpretation of Bayesian quantities does not depend on the stopping rule, and the proper interpretation ofBayesian quantities as measures of subjective belief on theoretical positions is emphasized. Expand
Almost the Best of Three Worlds: Risk, Consistency and Optional Stopping for the Switch Criterion in Nested Model Selection
We study the switch distribution, introduced by Van Erven et al. (2012), applied to model selection and subsequent estimation. While switching was known to be strongly consistent, here we show thatExpand
Almost the best of three worlds: Risk, consistency and optional stopping for the switch criterion in nested model selection
We study the switch distribution, introduced by van Erven, Grunwald and De Rooij (2012), applied to model selection and subsequent estimation. While switching was known to be strongly consistent,Expand
Why optional stopping is a problem for Bayesians
Recently, optional stopping has been a subject of debate in the Bayesian psychology community. Rouder (2014) argues that optional stopping is no problem for Bayesians, and even recommends the use ofExpand
BAYES FACTORS AND MARGINAL DISTRIBUTIONS IN INVARIANT SITUATIONS
SUMMARY. In Bayesian analysis with a "minimal" data set and common non informative priors, the (formal) marginal density of the data is surprisingly often independent of the error distribution. ThisExpand
Safe Testing
TLDR
Sharing Fisherian, Neymanian and Jeffreys-Bayesian interpretations, S-values and safe tests may provide a methodology acceptable to adherents of all three schools. Expand
Objective priors for the bivariate normal model
Summary Objective Bayesian inference for the multivariate normal distribution is illustrated, using dieren t types of formal objective priors (Jereys, invariant, reference and matching), dieren tExpand
The frequentist implications of optional stopping on Bayesian hypothesis tests
TLDR
The impact of optional stopping on the resulting Bayes factors in two common situations: when the truth is a combination of the hypotheses, such as in a heterogeneous population, and when a hypothesis is composite—taking multiple parameter values—such as the alternative hypothesis in a t-test. Expand
Sequential Hypothesis Testing With Bayes Factors: Efficiently Testing Mean Differences
TLDR
This contribution investigates the properties of a procedure for Bayesian hypothesis testing that allows optional stopping with unlimited multiple testing, even after each participant, and investigates the long-term rate of misleading evidence, the average expected sample sizes, and the biasedness of effect size estimates when an SBF design is applied to a test of mean differences between 2 groups. Expand
The Likelihood Principle
Publisher Summary The likelihood principle (LP) is a normative principle for evaluating statistical inference procedures. The LP can be proved from arguably self-evident premises; indeed, it can beExpand
...
1
2
3
4
5
...