The preregistration revolution

@article{Nosek2018ThePR,
  title={The preregistration revolution},
  author={Brian A. Nosek and Charles R. Ebersole and Alexander Carl DeHaven and David Thomas Mellor},
  journal={Proceedings of the National Academy of Sciences},
  year={2018},
  volume={115},
  pages={2600 - 2606}
}
Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define… Expand

Topics from this paper

Does preregistration improve the credibility of research findings?
Preregistration entails researchers registering their planned research hypotheses, methods, and analyses in a time-stamped document before they undertake their data collection and analyses. ThisExpand
Preregistration and reproducibility
Abstract Many view preregistration as a promising way to improve research credibility. However, scholars have argued that using pre-analysis plans in Experimental Economics has limited benefits. ThisExpand
Preregistering qualitative research
The threat to reproducibility and awareness of current rates of research misbehavior sparked initiatives to better academic science. One initiative is preregistration of quantitative research. WeExpand
Preregistering qualitative research
TLDR
This work contextualizes preregistrations alongside another initiative to enhance credibility in qualitative research: the confirmability audit and suggests modifications to the Open Science Framework preregistration form to tailor it for qualitative studies. Expand
Preregistration in the Context of Expertise Research: Benefits, Challenges, and Recommendations
A number of recent reforms in psychological science have centered around following best practices to improve the robustness and reliability of empirical findings. Among these, preregistration hasExpand
The preregistration revolution needs to distinguish between predictions and analyses
  • A. Ledgerwood
  • Psychology, Medicine
  • Proceedings of the National Academy of Sciences
  • 2018
TLDR
The language used in making this important argument creates unnecessary confusion and masks a crucial distinction between two types of preregistration: preregistering a theoretical, a priori, directional prediction and an analysis plan that serves to clarify how evidence is produced. Expand
Pre-Analysis Plans: An Early Stocktaking
Pre-analysis plans (PAPs) have been championed as a solution to the problem of research credibility, but without any evidence that PAPs actually bolster the credibility of research. We analyze aExpand
Preregistration Is Neither Sufficient nor Necessary for Good Science
To address widespread perceptions of a reproducibility crisis in the social sciences, a growing number of scholars recommend the systematic preregistration of empirical studies. The purpose of thisExpand
PPS_2100093 1..17
Pre-analysis plans (PAPs) have been championed as a solution to the problem of research credibility, but without any evidence that PAPs actually bolster the credibility of research. We analyze aExpand
Easy preregistration will benefit any research
To the Editor — There is shared support from Riley et al.1 and Wolfe and Kanwisher2 for the principles of increasing the transparency, rigour and reproducibility of science and “[fulfilling] anExpand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 74 REFERENCES
Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability
An academic scientist's professional success depends on publishing. Publishing norms emphasize novel, positive results. As such, disciplinary incentives encourage design, analysis, and reportingExpand
Consequences of Prejudice Against the Null Hypothesis
The consequences of prejudice against accepting the null hypothesis were examined through (a) a mathematical model intended to stimulate the research-publication process and (b) case studies ofExpand
Using Split Samples to Improve Inference on Causal Effects
TLDR
Simulations indicate that, under empirically relevant settings, the proposed method delivers more power than a PAP but is better suited for exploratory analyses where there is significant uncertainty about the outcomes of interest. Expand
An Agenda for Purely Confirmatory Research
TLDR
This article proposes that researchers preregister their studies and indicate in advance the analyses they intend to conduct, and proposes that only these analyses deserve the label “confirmatory,” and only for these analyses are the common statistical tests valid. Expand
Why Most Published Research Findings Are False
TLDR
Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Expand
Robustness in the Strategy of Scientific Model Building.
Publisher Summary Robustness may be defined as the property of a procedure which renders the answers it gives insensitive to departures, of a kind which occur in practice, from ideal assumptions.Expand
Detecting and avoiding likely false‐positive findings – a practical guide
TLDR
It is argued that a culture of ‘you can publish if your study is rigorous’ creates a systematic bias against the null hypothesis which renders meta‐analyses questionable and may even lead to a situation where hypotheses become difficult to falsify. Expand
Simultaneous and selective inference: Current successes and future challenges.
  • Y. Benjamini
  • Computer Science, Medicine
  • Biometrical journal. Biometrische Zeitschrift
  • 2010
TLDR
The vitality of the field in the future depends upon the ability to continue and address the real needs of statistical analyses in current problems and two application areas offering new challenges have received less attention in the community to date are discussed. Expand
Hindsight ≠ foresight: the effect of outcome knowledge on judgment under uncertainty*
TLDR
It is argued that a lack of awareness of outcome knowledge can seriously restrict one’s ability to judge or learn from the past. Expand
Publication bias in the social sciences: Unlocking the file drawer
TLDR
Fully half of peer-reviewed and implemented social science experiments are not published, providing direct evidence of publication bias and identifying the stage of research production at which publication bias occurs: Authors do not write up and submit null findings. Expand
...
1
2
3
4
5
...