# Optional Stopping with Bayes Factors: a categorization and extension of folklore results, with an application to invariant situations

@article{Hendriksen2018OptionalSW, title={Optional Stopping with Bayes Factors: a categorization and extension of folklore results, with an application to invariant situations}, author={Allard A. Hendriksen and Rianne de Heide and Peter Gr{\"u}nwald}, journal={ArXiv}, year={2018}, volume={abs/1807.09077} }

It is often claimed that Bayesian methods, in particular Bayes factor methods for hypothesis testing, can deal with optional stopping. We first give an overview, using elementary probability theory, of three different mathematical meanings that various authors give to this claim: (1) stopping rule independence, (2) posterior calibration and (3) (semi-) frequentist robustness to optional stopping. We then prove theorems to the effect that these claims do indeed hold in a general measure… Expand

#### Topics from this paper

#### 15 Citations

Why optional stopping is a problem for Bayesians

- Mathematics
- 2017

Recently, optional stopping has been a subject of debate in the Bayesian psychology community. Rouder (2014) argues that optional stopping is no problem for Bayesians, and even recommends the use of… Expand

Why optional stopping can be a problem for Bayesians

- Medicine, Psychology
- Psychonomic bulletin & review
- 2020

It is illustrated that, as soon as the parameters of interest are equipped with default or pragmatic priors—which means, in most practical applications of Bayes factor hypothesis testing—resilience to optional stopping can break down. Expand

Safe Testing

- Mathematics, Computer Science
- 2020 Information Theory and Applications Workshop (ITA)
- 2020

Sharing Fisherian, Neymanian and Jeffreys-Bayesian interpretations, S-values and safe tests may provide a methodology acceptable to adherents of all three schools. Expand

Safe tests for 2 x 2 contingency tables and the Cochran-Mantel-Haenszel test

- 2019

Rationale In my thesis, I developed implementations of safe statistics: a new framework for collecting evidence for hypotheses, particularly suitable for online and sequential learning [1].… Expand

The Bayesian Methodology of Sir Harold Jeffreys as a Practical Alternative to the P Value Hypothesis Test

- Computational Brain & Behavior
- 2020

Harold Jeffreys’s Bayes factor methodology as implemented in the open-source software JASP is explained and its practical relevance is showcased with two examples. Expand

The Bayesian Methodology of Sir Harold Jeffreys as a Practical Alternative to the P-value Hypothesis Test

- Computer Science
- 2019

Harold Jeffreys’s Bayes factor methodology as implemented in the open-source software JASP is explained and its practical relevance is showcased with two examples. Expand

Bayes Factors for Peri-Null Hypotheses

- Mathematics
- 2021

A perennial objection against Bayes factor point-null hypothesis tests is that the point-null hypothesis is known to be false from the outset. Following Morey and Rouder (2011) we examine the… Expand

Bayesian inference in numerical cognition: A tutorial using JASP

- 2020

Researchers in numerical cognition rely on hypothesis testing and parameter estimation to evaluate the evidential value of data. Though there has been increased interest in Bayesian statistics as an… Expand

Bayesian Hodges-Lehmann tests for statistical equivalence in the two-sample setting: Power analysis, type I error rates and equivalence boundary selection in biomedical research

- Medicine
- BMC Medical Research Methodology
- 2021

Bayesian Hodges-Lehmann tests for statistical equivalence differ in their sensitivity to the prior modeling, power, and the associated type I error rates, but results show that the relationship is even reverse for the Bayes factor and ROPE based equivalence tests. Expand

The Perils of Misspecified Priors and Optional Stopping in Multi-Armed Bandits

- Computer Science, Medicine
- Frontiers in Artificial Intelligence
- 2021

The effects of optional stopping in a particular class of multi-armed bandit experiments, which randomly allocates observations to arms proportional to the Bayesian posterior probability that each arm is optimal (Thompson sampling), are investigated. Expand

#### References

SHOWING 1-10 OF 58 REFERENCES

Optional stopping: No problem for Bayesians

- Psychology, Medicine
- Psychonomic bulletin & review
- 2014

In this article, it is shown through simulation that the interpretation of Bayesian quantities does not depend on the stopping rule, and the proper interpretation ofBayesian quantities as measures of subjective belief on theoretical positions is emphasized. Expand

Almost the Best of Three Worlds: Risk, Consistency and Optional Stopping for the Switch Criterion in Nested Model Selection

- Mathematics
- 2014

We study the switch distribution, introduced by Van Erven et al. (2012), applied to model selection and subsequent estimation. While switching was known to be strongly consistent, here we show that… Expand

Almost the best of three worlds: Risk, consistency and optional stopping for the switch criterion in nested model selection

- Mathematics
- 2018

We study the switch distribution, introduced by van Erven, Grunwald and De Rooij (2012), applied to model selection and subsequent estimation. While switching was known to be strongly consistent,… Expand

Why optional stopping is a problem for Bayesians

- Mathematics
- 2017

Recently, optional stopping has been a subject of debate in the Bayesian psychology community. Rouder (2014) argues that optional stopping is no problem for Bayesians, and even recommends the use of… Expand

BAYES FACTORS AND MARGINAL DISTRIBUTIONS IN INVARIANT SITUATIONS

- Mathematics
- 2016

SUMMARY. In Bayesian analysis with a "minimal" data set and common non informative priors, the (formal) marginal density of the data is surprisingly often independent of the error distribution. This… Expand

Safe Testing

- Mathematics, Computer Science
- 2020 Information Theory and Applications Workshop (ITA)
- 2020

Sharing Fisherian, Neymanian and Jeffreys-Bayesian interpretations, S-values and safe tests may provide a methodology acceptable to adherents of all three schools. Expand

Objective priors for the bivariate normal model

- Mathematics
- 2008

Summary Objective Bayesian inference for the multivariate normal distribution is illustrated, using dieren t types of formal objective priors (Jereys, invariant, reference and matching), dieren t… Expand

The frequentist implications of optional stopping on Bayesian hypothesis tests

- Psychology, Medicine
- Psychonomic bulletin & review
- 2014

The impact of optional stopping on the resulting Bayes factors in two common situations: when the truth is a combination of the hypotheses, such as in a heterogeneous population, and when a hypothesis is composite—taking multiple parameter values—such as the alternative hypothesis in a t-test. Expand

Sequential Hypothesis Testing With Bayes Factors: Efficiently Testing Mean Differences

- Mathematics, Medicine
- Psychological methods
- 2017

This contribution investigates the properties of a procedure for Bayesian hypothesis testing that allows optional stopping with unlimited multiple testing, even after each participant, and investigates the long-term rate of misleading evidence, the average expected sample sizes, and the biasedness of effect size estimates when an SBF design is applied to a test of mean differences between 2 groups. Expand

The Likelihood Principle

- Mathematics
- 2011

Publisher Summary The likelihood principle (LP) is a normative principle for evaluating statistical inference procedures. The LP can be proved from arguably self-evident premises; indeed, it can be… Expand