No Adjustments Are Needed for Multiple Comparisons

  title={No Adjustments Are Needed for Multiple Comparisons},
  author={Kenneth J. Rothman},
  • K. Rothman
  • Published 1 January 1990
  • Education
  • Epidemiology
Adjustments for making multiple comparisons in large bodies of data are recommended to avoid rejecting the null hypothesis too readily. Unfortunately, reducing the type I error for null associations increases the type II error for those associations that are not null. The theoretical basis for advocating a routine adjustment for multiple comparisons is the “universal null hypothesis” that “chance” serves as the first-order explanation for observed phenomena. This hypothesis undermines the basic… 

Why multiple hypothesis test corrections provide poor control of false positives in the real world

It is argued that a single well-defined false positive rate (FPR) does not even exist and the freedom scientists have to choose the error rate to control, the collection of tests to include in the adjustment, and the method of correction provides too much flexibility for strong error control.

Analysis goals, error-cost sensitivity, and analysis hacking: Essential considerations in hypothesis testing and multiple comparisons.

  • S. Greenland
  • Biology
    Paediatric and perinatal epidemiology
  • 2020
Issues arising in single-parameter inference (such as error costs and loss functions) that are often skipped in basic statistics, yet are crucial to understanding controversies in testing and multiple comparisons are reviewed.

What's wrong with Bonferroni adjustments

This paper advances the view, widely held by epidemiologists, that Bonferroni adjustments are, at best, unnecessary and, at worst, deleterious to sound statistical inference.

Do p Values Lose Their Meaning in Exploratory Analyses? It Depends How You Define the Familywise Error Rate

Several researchers have recently argued that p values lose their meaning in exploratory analyses due to an unknown inflation of the alpha level (e.g., Nosek & Lakens, 2014; Wagenmakers, 2016). For

More Powerful Multiple Testing in Randomized Experiments with Non-Compliance

An analysis method for experiments involving both features that merges posterior predictive $p$-values for complier causal effects with randomization-based multiple comparisons adjustments is proposed; the results are valid familywise tests that are doubly advantageous.

Semi-Bayes and empirical Bayes adjustment methods for multiple comparisons.

Empirical Bayes and semi-Bayes methods can enable the avoidance of numerous false positive associations, and can produce effect estimates that are, on the average, more valid.

Significance level adjustments for multiple testing in health studies : A case for false discovery rate control

It is demonstrated how false discovery rate adjustments follow this principle, and that researchers may benefit by considering such adjustments for use in health and medical studies.

Multiple comparisons and P values.

An explanation of this apparent contradiction of why an investigator who wishes to compare treatments A and B be required to demonstrate a greater level of statistical significance if he or she studies treatment C at the same time is offered.

Do multiple outcome measures require p-value adjustment?

  • R. Feise
  • Medicine
    BMC medical research methodology
  • 2002
The primary aim of this study was to estimate the need to make appropriate p-value adjustments in clinical trials to compensate for a possible increased risk in committing Type I errors when multiple outcome measures are used.



Exploring Data Tables, Trends and Shapes.

Edited by three well-known and respected statisticians, this book is another on exploratory data analysis (EDA), and is part of the prestigious Wiley Series on Probability and Mathematical

Scientific Method

THE subject of the leading article of NATURE of May 28 must commend itself to the earnest consideration of all those who view with consternation the present drift of our civilisation towards chaos.

Using the Environment to Explain and Predict Mortality

Mortality at ages 45-74 has been studied in the larger County Boroughs of England and Wales. In the younger segment of this span of years premature death (and morbidity) from the chronic diseases is

The Oxford English Dictionary

The publication of the final volume of the OED Supplement marks the completion of a "work which will last longer and prove more influential than anything else published this half-century" (The Times

Medical Uses of Statistics.

This work explains the purpose of statistical methods in medical studies and analyzes the statistical techniques used by clinical investigators, with special emphasis on studies published in "The New

Summing Up: The Science of Reviewing Research

A Checklist for Evaluating Reviews Reference Index, a guide to organizing a reviewing strategy, and a list of procedures for evaluating reviews.

Comparing the means of several groups.