Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature

@article{Szcs2017EmpiricalAO,
  title={Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature},
  author={D{\'e}nes Szűcs and John P. A. Ioannidis},
  journal={PLoS Biology},
  year={2017},
  volume={15}
}
We have empirically assessed the distribution of published effect sizes and estimated power by analyzing 26,841 statistical records from 3,801 cognitive neuroscience and psychology papers published recently. The reported median effect size was D = 0.93 (interquartile range: 0.64–1.46) for nominally statistically significant results and D = 0.24 (0.11–0.42) for nonsignificant results. Median power to detect small, medium, and large effects was 0.12, 0.44, and 0.73, reflecting no improvement… 

Figures and Tables from this paper

Correction: Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature

TLDR
The recently reported low replication success in psychology is realistic and worse performance may be expected for cognitive neuroscience, and false report probability is likely to exceed 50% for the whole literature.

Effect size and statistical power in the rodent fear conditioning literature – A systematic review

TLDR
Effect sizes and statistical power have a wide distribution in the rodent fear conditioning literature, but do not seem to have a large influence on how results are described or cited.

Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

TLDR
It is concluded that intelligence research does show signs of low power and publication bias, but that these problems seem less severe than in many other scientific fields.

Estimating statistical power, posterior probability and publication bias of psychological research using the observed replication rate

TLDR
It is shown how Bayes' theorem can be used to better understand the implications of the 36% reproducibility rate of published psychological findings reported by the Open Science Collaboration, suggesting that even when studied associations are truly NULL, the literature to be dominated by statistically significant findings.

The relation between statistical power and inference in fMRI

TLDR
This work aimed to clarify the power problem by considering and contrasting two simulated scenarios of such possible brain-behavior correlations: weak diffuse effects and strong localized effects.

Limited evidence for the effect of red color on cognitive performance: A meta-analysis

TLDR
A random-effects meta-analysis that compared test performance after viewing red or a control color found no evidential value for an effect of red color on intellectual performance was available, casting doubt on the existence of a robust color-priming effect in achievement situations.

The Prior Odds of Testing a True Effect in Cognitive and Social Psychology

Efforts to increase replication rates in psychology generally consist of recommended improvements to methodology, such as increasing sample sizes to increase power or using a lower alpha level.

Running head : PRIOR ODDS IN COGNITIVE AND SOCIAL PSYCHOLOGY 1 The Prior Odds of Testing a True Effect in Cognitive and Social Psychology

Efforts to increase replication rates in psychology generally consist of recommended improvements to methodology, such as increasing sample sizes to increase power and using less flexible statistical

Using and Understanding Power in Psychological Research: A Survey Study

TLDR
If psychological researchers are expected to compute a priori power analyses to plan their research, clearer educational material and guidelines should be made available.

An empirically based power primer for laboratory aggression research.

TLDR
A "power primer" that laboratory aggression researchers can use as a resource when planning studies using this methodology, using simulation-based power analyses and effect size estimates derived from recent literature reviews, and a series of best practice recommendations.
...

References

SHOWING 1-10 OF 65 REFERENCES

Statistical power of psychological research: what have we gained in 20 years?

  • J. Rossi
  • Psychology
    Journal of consulting and clinical psychology
  • 1990
TLDR
The implications of these results concerning the proliferation of Type I errors in the published literature, the failure of replication studies, and the interpretation of null (negative) results are emphasized.

Do Studies of Statistical Power Have an Effect on the Power of Studies?

The long-term impact of studies of statistical power is investigated using J. Cohen's (1962) pioneering work as an example. We argue that the impact is nil; the power of studies in the same journal

Effect size estimates: current use, calculations, and interpretation.

TLDR
A straightforward guide to understanding, selecting, calculating, and interpreting effect sizes for many types of data and to methods for calculating effect size confidence intervals and power analysis is provided.

Do studies of statistical power have an effect on the power of studies

The long-term impact of studies of statistical power is investigated using J. Cohen's (1962) pioneering work as an example. We argue that the impact is nil; the power of studies in the same journal

On the Reproducibility of Psychological Science

TLDR
The results of this reanalysis provide a compelling argument for both increasing the threshold required for declaring scientific discoveries and for adopting statistical summaries of evidence that account for the high proportion of tested hypotheses that are false.

A survey of the statistical power of research in behavioral ecology and animal behavior

TLDR
There was a significant correlation between power and reported p value for both first and last tests, suggesting that failure to observe significant relationships is partly owing to small sample sizes, as power increases with sample size.

An exploratory test for an excess of significant findings

TLDR
A test to explore biases stemming from the pursuit of nominal statistical significance was developed and demonstrated a clear or possible excess of significant studies in 6 of 8 large meta-analyses and in the wide domain of neuroleptic treatments.

Distributions of p-values smaller than .05 in psychology: what is going on?

TLDR
This paper examined 258,050 test results across 30,710 articles from eight high impact journals to investigate the existence of a peculiar prevalence of p-values just below .05 (i.e., a bump) in the psychological literature, and a potential increase thereof over time.

The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates

Replication is often viewed as the demarcation between science and nonscience. However, contrary to the commonly held view, we show that in the current (selective) publication system replications may

Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition 1

TLDR
It is argued that, in some cases, other analysis problems likely created entirely spurious correlations and the data from these studies could be reanalyzed with unbiased methods to provide accurate estimates of the correlations in question and urge authors to perform such reanalyses.
...