Many tests of significance: new methods for controlling type I errors.

  title={Many tests of significance: new methods for controlling type I errors.},
  author={H. J. Keselman and Charles W. Miller and Burt S. Holland},
  journal={Psychological methods},
  volume={16 4},
There have been many discussions of how Type I errors should be controlled when many hypotheses are tested (e.g., all possible comparisons of means, correlations, proportions, the coefficients in hierarchical models, etc.). By and large, researchers have adopted familywise (FWER) control, though this practice certainly is not universal. Familywise control is intended to deal with the multiplicity issue of computing many tests of significance, yet such control is conservative--that is, less… 

Tables from this paper

Per Family or Familywise Type I Error Control: "Eether, Eyether, Neether, Nyther, Let's Call the Whole Thing Off!"

Frane (2015) pointed out the difference between per-family and familywise Type I error control and how different multiple comparison procedures control one method but not necessarily the other. He

Multiplicity Control, School Uniforms, and Other Perplexing Debates

Researchers in psychology are frequently confronted with the issue of analysing multiple relationships simultaneously. For example, this could involve multiple outcome variables or multiple

Platform Trials: the Impact of common Controls on Type One Error and Power

The impact of a common control arm in platform trials on the type one error and power in comparison to what would have been obtained with a platform trial with individual control arms in a simulation study is investigated and the need for adjustment in terms of a family definition or hypotheses dependencies is discussed.

Post Hoc Tests

Post Hoc Tests Familywise Error Familywise error (FWE) is also known as alpha inflation or cumulative Type I error. Familywise error represents the probability that any one of a set of comparisons or

The use of meta‐analytic statistical significance testing

A review of 130 meta-analyses revealed a strong reliance on statistical significance testing without consideration of Type I errors or the use of multiplicity corrections, and meta-analysts must consider these issues prior to conducting the review.

A practical guide to the analysis of non-response and attrition in longitudinal research using a real data example

Selective non-participation and attrition pose a ubiquitous threat to the validity of inferences drawn from observational longitudinal studies. We investigate various potential predictors for

The Influence of Daily Coping on Anxiety Under Examination Stress: A Model of Interindividual Differences in Intraindividual Change

Within-person analyses of longitudinal data on anxiety in 321 persons over 35 days provided evidence on coping effectiveness for the typical person, how effectiveness changed across time, and the extent to which individuals differed in their effectiveness.



Stepup procedures for control of generalizations of the familywise error rate

Consider the multiple testing problem of testing null hypotheses H 1 ,...,H s . A classical approach to dealing with the multiplicity problem is to restrict attention to procedures that control the

Generalizations of the familywise error rate

Consider the problem of simultaneously testing null hypotheses H 1, … H s . The usual approach to dealing with the multiplicity problem is to restrict attention to procedures that control the

Controlling the false discovery rate: a practical and powerful approach to multiple testing

SUMMARY The common approach to the multiplicity problem calls for controlling the familywise error rate (FWER). This approach, though, has faults, and we point out a few. A different approach to

Some Results on False Discovery Rate in Stepwise multiple testing procedures

The concept of false discovery rate (FDR) has been receiving increasing attention by researchers in multiple hypotheses testing. This paper produces some theoretical results on the FDR in the context

Generalizing Simes' test and Hochberg's stepup procedure

In a multiple testing problem where one is willing to tolerate a few false rejections, procedure controlling the familywise error rate (FWER) can potentially be improved in terms of its ability to

A note on the monotonicity of the critical values of a step-up test

Multiple Comparisons among Means

Abstract Methods for constructing simultaneous confidence intervals for all possible linear contrasts among several means of normally distributed variables have been given by Scheffe and Tukey. In

A sharper Bonferroni procedure for multiple tests of significance

SUMMARY A simple procedure for multiple tests of significance based on individual p-values is derived. This simple procedure is sharper than Holm's (1979) sequentially rejective procedure. Both

Multiple Comparison Procedures


Designing Experiments and Analyzing Data: A Model Comparison Perspective

This book discusses conceptual Bases of Experimental Design and Analysis, model Comparisons for Between-Subjects Designs, and an Introduction to Multilevel Hierarchical Mixed Models: Nested Designs.