When to adjust alpha during multiple testing: a consideration of disjunction, conjunction, and individual testing

  title={When to adjust alpha during multiple testing: a consideration of disjunction, conjunction, and individual testing},
  author={Mark Rubin},
  • Mark Rubin
  • Published 6 July 2021
  • Mathematics
  • Synthese
Scientists often adjust their significance threshold (alpha level) during null hypothesis significance testing in order to take into account multiple testing and multiple comparisons. This alpha adjustment has become particularly relevant in the context of the replication crisis in science. The present article considers the conditions in which this alpha adjustment is appropriate and the conditions in which it is inappropriate. A distinction is drawn between three types of multiple testing… Expand

Figures and Tables from this paper

(When) should you adjust inferences for multiple hypothesis testing?
The use of multiple hypothesis testing adjustments varies widely in applied economic research, without consensus on when and how it should be done. We provide a game-theoretic foundation for thisExpand
Frequentist inference for cluster randomised trials with multiple primary outcomes
The use of a single primary outcome is generally either recommended or required by many influential randomised trial guidelines to avoid the problem of “multiple testing”. Without correction, theExpand
There’s No Need to Lower the Significance Threshold when Conducting Single Tests of Multiple Individual Hypotheses
In summary, it is only necessary to lower the significance threshold when undertaking multiple tests of a single joint null hypothesis using an union-intersection approach. It is not necessary toExpand
Testing the effects of the Shamiri Intervention and its components on anxiety, depression, wellbeing, and academic functioning in Kenyan adolescents: study protocol for a five-arm randomized controlled trial
This five-arm randomized controlled trial will shed light on the mechanisms and outcomes targeted by each individual intervention, helping prioritize which mental health interventions are most important to disseminate. Expand
Functional cognitive and language outcomes after cerebral hemispherectomy for hemimegalencephaly
Children with HME have delayed language milestones and continue to require significant language and literacy support long-term after cerebral hemispherectomy, and children who had undergone a right hemisphere resection and those who were older at epilepsy onset were more likely to have better cognitive and language outcomes. Expand
Are all threats equal? Associations of childhood exposure to physical attack versus threatened violence with preadolescent brain structure
Experiences of physical attack and threatened violence may have quantitively different neurodevelopmental effects, and differences between types of threatening experiences may be neurodevelopmentally salient. Expand


Setting an Optimal α That Minimizes Errors in Null Hypothesis Significance Tests
This work cannot identify a rationale for continuing to arbitrarily use α = 0.05 for null hypothesis significance tests in any field, when it is possible to determine an optimal α, which results in stronger scientific inferences. Expand
Analysis goals, error-cost sensitivity, and analysis hacking: Essential considerations in hypothesis testing and multiple comparisons.
  • S. Greenland
  • Medicine
  • Paediatric and perinatal epidemiology
  • 2020
Issues arising in single-parameter inference (such as error costs and loss functions) that are often skipped in basic statistics, yet are crucial to understanding controversies in testing and multiple comparisons are reviewed. Expand
Best (but oft-forgotten) practices: the multiple problems of multiplicity-whether and how to correct for many statistical tests.
  • D. Streiner
  • Computer Science, Medicine
  • The American journal of clinical nutrition
  • 2015
How multiple tests lead to an inflation of the α level is discussed and the following different contexts in which multiplicity arises are explored: testing for baseline differences in various types of studies, having >1 outcome variable, conducting statistical tests that produce >1 P value, taking multiple "peeks" at the data, and unplanned, post hoc analyses. Expand
Familywise Error in Multiple Comparisons: Disentangling a Knot through a Critique of O'Keefe's Arguments against Alpha Adjustment
This essay discusses a statistical problem of multiple comparisons and argues for adjusting familywise alpha under certain conditions. The principle of familywise correction dictates that, uponExpand
Reversing Bonferroni.
It is shown that it is sometimes appropriate to reverse the logic of the classic Bonferroni correction and increase the significance criterion in order to maintain an intended compound Type I error rate across multiple tests. Expand
Optimal Procedures for Multiple Testing Problems
This paper formulation of multiple testing of simple hypotheses as an infinite-dimensional optimization problem, seeking the most powerful rejection policy which guarantees strong control of the selected measure, and characterize maximin rules for complex alternatives. Expand
Do p Values Lose Their Meaning in Exploratory Analyses? It Depends How You Define the Familywise Error Rate
Several researchers have recently argued that p values lose their meaning in exploratory analyses due to an unknown inflation of the alpha level (e.g., Nosek & Lakens, 2014; Wagenmakers, 2016). ForExpand
No Adjustments Are Needed for Multiple Comparisons
  • K. Rothman
  • Computer Science, Medicine
  • Epidemiology
  • 1990
A policy of not making adjustments for multiple comparisons is preferable because it will lead to fewer errors of interpretation when the data under evaluation are not random numbers but actual observations on nature. Expand
When to combine hypotheses and adjust for multiple tests.
  • P. Veazie
  • Mathematics, Medicine
  • Health services research
  • 2006
OBJECTIVE To provide guidelines for identifying composite hypotheses and addressing the probability of false rejection for multiple hypotheses. DATA SOURCES AND STUDY SETTING Examples from theExpand
Adjusting for multiple testing--when and how?
The existing multiple test procedures are summarized for the most important multiplicity situations and it is emphasized that adjustments for multiple testing are required in confirmatory studies whenever results from multiple tests have to be combined in one final conclusion and decision. Expand