Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies

  title={Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies},
  author={Ang{\'e}lique O. J. Cramer and Don van Ravenzwaaij and Dora Matzke and Helen Steingroever and Ruud Wetzels and Raoul P. P. P. Grasman and Lourens J. Waldorp and Eric-Jan Wagenmakers},
  booktitle={Psychonomic bulletin & review},
Many psychologists do not realize that exploratory use of the popular multiway analysis of variance harbors a multiple-comparison problem. In the case of two factors, three separate null hypotheses are subject to test (i.e., two main effects and one interaction). Consequently, the probability of at least one Type I error (if all null hypotheses are true) is 14 % rather than 5 %, if the three tests are independent. We explain the multiple-comparison problem and demonstrate that researchers… CONTINUE READING
Related Discussions
This paper has been referenced on Twitter 89 times. VIEW TWEETS

From This Paper

Figures, tables, and topics from this paper.

Explore Further: Topics Discussed in This Paper


Publications citing this paper.
Showing 1-10 of 28 extracted citations

Spatial biases in motion extrapolation for manual interception.

Journal of experimental psychology. Human perception and performance • 2018


Publications referenced by this paper.
Showing 1-10 of 40 references

Methodology: Foundations of inference and research in the behavioral sciences

A. D. de Groot
View 5 Excerpts
Highly Influenced

A sequentially rejective test procedure based on a modified Bonferroni inequality

D. M. Rom
Biometrika, 77, 663–665. • 1990
View 4 Excerpts
Highly Influenced

Some generalizations of the t-method in simultaneous inference

Y. Hochberg
Journal of Multivariate Analysis, 4, 224–234. • 1974
View 4 Excerpts
Highly Influenced

Confirmatory methods, or huge samples, are required to obtain power for the evaluation of theories

I. Klugkist, L. Post, F. Haarhuis, F. van Wesel
Open Journal of Statistics, • 2014

Registered reports: A method to increase the credibility of published results

B. A. Nosek, D. Lakens
Social Psychology, • 2014
View 1 Excerpt

Registered reports and replications in Attention, Perception, & Psychophysics [Editorial

J. M. Wolfe
Attention, Perception, & Psychophysics, 75, 781–783. doi:10.3758/s13414-013-05025 • 2013
View 1 Excerpt

Trust in science would be improved by study pre-registration

C. D. Chambers, M Munafo
The Guardian. Retrieved from www.theguardian.com/science/blog/2013/jun/05/ trust-in-science-study-pre-registration • 2013

An Agenda for Purely Confirmatory Research.

Perspectives on psychological science : a journal of the Association for Psychological Science • 2012

Similar Papers

Loading similar papers…