Metascience on Peer Review: Testing the Effects of a Study’s Originality and Statistical Significance in a Field Experiment

  title={Metascience on Peer Review: Testing the Effects of a Study’s Originality and Statistical Significance in a Field Experiment},
  author={Malte Elson and Markus Huff and Sonja Utz},
  journal={Advances in Methods and Practices in Psychological Science},
  pages={53 - 65}
Peer review has become the gold standard in scientific publishing as a selection method and a refinement scheme for research reports. However, despite its pervasiveness and conferred importance, relatively little empirical research has been conducted to document its effectiveness. Further, there is evidence that factors other than a submission’s merits can substantially influence peer reviewers’ evaluations. We report the results of a metascientific field experiment on the effect of the… 
6 Citations

Figures and Tables from this paper

Do peers share the same criteria for assessing grant applications?
This study examines a basic assumption of peer review, namely, the idea that there is a consensus on evaluation criteria among peers, which is a necessary condition for the reliability of peer judgements, and identifies two consensus classes, two consensus-close classes, and a consensus-far class.
Caution, Preprint! Brief Explanations Allow Nonscientists to Differentiate Between Preprints and Peer-Reviewed Journal Articles
A growing number of psychological research findings are initially published as preprints. Preprints are not peer reviewed and thus did not undergo the established scientific quality-control process.
Individual integrity and public morality in scientific publishing
ABSTRACT. Science and science reporting are under threat. Knowingly or not, researchers and clinicians are part of this debacle. This is not due so much to the notorious replication crisis, as to our
Gender and geographical disparity in editorial boards of journals in psychology and neuroscience.
It is suggested that editorial positions in academic journals-possibly one of the most powerful decision-making roles in academic psychology and neuroscience-are balanced in neither gender nor geographical representation.
Crisis Ahead? Why Human-Robot Interaction User Studies May Have Replicability Problems and Directions for Improvement
There is a confidence crisis in many scientific disciplines, in particular disciplines researching human behavior, as many effects of original experiments have not been replicated successfully in


Publication prejudices: An experimental study of confirmatory bias in the peer review system
  • M. Mahoney
  • Psychology
    Cognitive Therapy and Research
  • 2005
Confirmatory bias is the tendency to emphasize and believe experiences which support one's views and to ignore or discredit those which do not. The effects of this tendency have been repeatedly
Effects of editorial peer review: a systematic review.
CONTEXT Editorial peer review is widely used to select submissions to journals for publication and is presumed to improve their usefulness. Sufficient research on peer review has been published to
Measuring the quality of editorial peer review.
Until the objectives of peer-review are properly defined, it will remain almost impossible to assess or improve its effectiveness, and research needed to understand the broader effects of peer review poses many methodologic problems and would require the cooperation of many parts of the scientific community.
A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants
According to the first meta-analysis for the inter-rater reliability (IRR) of journal peer reviews, the IRR of peer assessments is quite limited and needs improvement (e.g., reader system).
Estimating the reproducibility of psychological science
A large-scale assessment suggests that experimental reproducibility in psychology leaves a lot to be desired, and correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
Reviewer bias: a blinded experimental study.
  • E. Ernst, K. Resch
  • Psychology
    The Journal of laboratory and clinical medicine
  • 1994
Evaluating interrater consistency in reviewing a single manuscript and determining whether a referee's likely predisposition influences his or her attitude toward this manuscript suggest significant impact of reviewer bias on referee's judgment and imply that the peer review system in its present form has room for improvements in fairness and consistency.
Finding the Missing Science: The Fate of Studies Submitted for Review by a Human Subjects Committee
Publication bias, including prejudice against the null hypothesis, and other biasing filters may operate on researchers as well as journal editors and reviewers. A survey asked 33 psychology
Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015
It is found that peer beliefs of replicability are strongly related to replicable, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.
Manuscript Quality before and after Peer Review and Editing at Annals of Internal Medicine
Whether the quality of accepted manuscripts was improved by peer-review and editorial processes and, if it was, which aspects were most improved, is studied.
Peer review: a flawed process at the heart of science and journals.
  • Richard Smith
  • Medicine
    Journal of the Royal Society of Medicine
  • 2006
Peer review is at the heart of the processes of not just medical journals but of all of science, the method by which grants are allocated, papers published, academics promoted, and Nobel prizes won, yet it has until recently been unstudied.