No Adjustments Are Needed for Multiple Comparisons

  title={No Adjustments Are Needed for Multiple Comparisons},
  author={Kenneth J. Rothman},
  • K. Rothman
  • Published 1 January 1990
  • Education
  • Epidemiology
Adjustments for making multiple comparisons in large bodies of data are recommended to avoid rejecting the null hypothesis too readily. Unfortunately, reducing the type I error for null associations increases the type II error for those associations that are not null. The theoretical basis for advocating a routine adjustment for multiple comparisons is the “universal null hypothesis” that “chance” serves as the first-order explanation for observed phenomena. This hypothesis undermines the basic… 
Why multiple hypothesis test corrections provide poor control of false positives in the real world
It is argued that a single well-defined false positive rate (FPR) does not even exist and the freedom scientists have to choose the error rate to control, the collection of tests to include in the adjustment, and the method of correction provides too much flexibility for strong error control.
Analysis goals, error-cost sensitivity, and analysis hacking: Essential considerations in hypothesis testing and multiple comparisons.
  • S. Greenland
  • Biology
    Paediatric and perinatal epidemiology
  • 2020
Issues arising in single-parameter inference (such as error costs and loss functions) that are often skipped in basic statistics, yet are crucial to understanding controversies in testing and multiple comparisons are reviewed.
What's wrong with Bonferroni adjustments
This paper advances the view, widely held by epidemiologists, that Bonferroni adjustments are, at best, unnecessary and, at worst, deleterious to sound statistical inference.
Alpha Level Adjustments for Multiple Dependent Variable Analyses and Their Applicability – A Review
Researchers in sport science typically find themselves simultaneously examining many questions and hypothesis through multiple dependant variable analyses. When more than one independent test is
Do p Values Lose Their Meaning in Exploratory Analyses? It Depends How You Define the Familywise Error Rate
Several researchers have recently argued that p values lose their meaning in exploratory analyses due to an unknown inflation of the alpha level (e.g., Nosek & Lakens, 2014; Wagenmakers, 2016). For
More Powerful Multiple Testing in Randomized Experiments with Non-Compliance
An analysis method for experiments involving both features that merges posterior predictive $p$-values for complier causal effects with randomization-based multiple comparisons adjustments is proposed; the results are valid familywise tests that are doubly advantageous.
Semi-Bayes and empirical Bayes adjustment methods for multiple comparisons.
Empirical Bayes and semi-Bayes methods can enable the avoidance of numerous false positive associations, and can produce effect estimates that are, on the average, more valid.
Significance level adjustments for multiple testing in health studies : A case for false discovery rate control
It is demonstrated how false discovery rate adjustments follow this principle, and that researchers may benefit by considering such adjustments for use in health and medical studies.
Multiple comparisons and P values.
An explanation of this apparent contradiction of why an investigator who wishes to compare treatments A and B be required to demonstrate a greater level of statistical significance if he or she studies treatment C at the same time is offered.


Scientific Method
THE subject of the leading article of NATURE of May 28 must commend itself to the earnest consideration of all those who view with consternation the present drift of our civilisation towards chaos.
Using the Environment to Explain and Predict Mortality
Mortality at ages 45-74 has been studied in the larger County Boroughs of England and Wales. In the younger segment of this span of years premature death (and morbidity) from the chronic diseases is
The Oxford English Dictionary
The publication of the final volume of the OED Supplement marks the completion of a "work which will last longer and prove more influential than anything else published this half-century" (The Times
Summing Up: The Science of Reviewing Research.
A Checklist for Evaluating Reviews Reference Index, a guide to organizing a reviewing strategy, and a list of procedures for evaluating reviews.
Medical Uses of Statistics.
This work explains the purpose of statistical methods in medical studies and analyzes the statistical techniques used by clinical investigators, with special emphasis on studies published in "The New
Comparing the means of several groups.
The scientific method. Science 1986;231:659
  • 1986