Can Results-Free Review Reduce Publication Bias? The Results and Implications of a Pilot Study

  title={Can Results-Free Review Reduce Publication Bias? The Results and Implications of a Pilot Study},
  author={Michael G. Findley and Nathan M. Jensen and Edmund J. Malesky and Thomas B. Pepinsky},
  journal={Comparative Political Studies},
  pages={1667 - 1703}
In 2015, Comparative Political Studies embarked on a landmark pilot study in research transparency in the social sciences. The editors issued an open call for submissions of manuscripts that contained no mention of their actual results, incentivizing reviewers to evaluate manuscripts based on their theoretical contributions, research designs, and analysis plans. The three papers in this special issue are the result of this process that began with 19 submissions. In this article, we describe the… Expand
Preventing the ends from justifying the means: withholding results to address publication bias in peer-review
BMC Psychology is launching a pilot to trial a new ‘results-free’ peer-review process, whereby editors and reviewers are blinded to the study’s results, initially assessing manuscripts on the scientific merits of the rationale and methods alone. Expand
Pre-acceptance as a method to combat publication bias in area studies: a pilot in the Caucasus
Many social scientists in the readership of Caucasus Survey will have heard of the replication crisis (Open Science Collaboration 2015). This refers to the problem whereby results from seminalExpand
Introduction to the Special Series on Results-Blind Peer Review: An Experimental Analysis on Editorial Recommendations and Manuscript Evaluations
Publication bias occurs when studies with statistically significant results and large effects are more likely to be published than similarly rigorous studies with null and mixed findings.Expand
Results-blind review: a masked crusader for science
ABSTRACT The organizational sciences are entering a confidence crisis where a growing number of scholars recognize our science is imperfect and our knowledge base is potentially flawed. OneExpand
Does the removal of results from a submitted paper reduce publication bias
Abstract A recent paper by Findley et al. (2016) in Comparative Political Studies suggests that the removal of results from a paper may decrease publication bias. However, in the biomedical sciences,Expand
Publication Biases in Replication Studies
One of the strongest findings across the sciences is that publication bias occurs. Of particular note is a “file drawer bias” where statistically significant results are privileged overExpand
Publication Biases in Replication Studies
Abstract One of the strongest findings across the sciences is that publication bias occurs. Of particular note is a “file drawer bias” where statistically significant results are privileged overExpand
Aims: The modern scientific publishing system suffers from many problems, amongst which one of the most important  is the pressure to publish positive results. A potentially simple way to mitigateExpand
If journals embraced conditional equivalence testing, would research be better?
We consider the reliability of published science: the probability that scientific claims put forth are true. Low reliability within many scientific fields is of major concern for researchers,Expand
Transparency, Reproducibility, and the Credibility of Economics Research
There is growing interest in enhancing research transparency and reproducibility in economics and other scientific fields. We survey existing work on these topics within economics, and discuss theExpand


Research Preregistration in Political Science: The Case, Counterarguments, and a Response to Critiques
ABSTRACT This article describes the current debate on the practice of preregistration in political science—that is, publicly releasing a research design before observing outcome data. The case inExpand
Publication bias in the social sciences: Unlocking the file drawer
Fully half of peer-reviewed and implemented social science experiments are not published, providing direct evidence of publication bias and identifying the stage of research production at which publication bias occurs: Authors do not write up and submit null findings. Expand
The existence of publication bias and risk factors for its occurrence.
Publication bias is the tendency on the parts of investigators, reviewers, and editors to submit or accept manuscripts for publication based on the direction or strength of the study findings. MuchExpand
Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration
It is demonstrated here that even with this level of advanced specification, the scope for fishing is considerable when there is latitude over selection of covariates, subgroups, and other elements of an analysis plan. Expand
An Equivalence Approach to Balance and Placebo Tests
This article argues that equivalence tests are better able to incorporate substantive considerations about what constitutes good balance on covariates and placebo outcomes than traditional tests, and shows how to apply these procedures to tests of design. Expand
Increasing the Credibility of Political Science Research: A Proposal for Journal Reforms
  • B. Nyhan
  • Political Science
  • PS: Political Science & Politics
  • 2015
In this white paper, I propose a series of reforms intended to improve the rigor, timeliness, and replicability of published political science research. These changes could be proposed as part of theExpand
Evidence from a Field Experiment on Financial Transparency
Observational studies of compliance with international law have produced mixed findings and been plagued by methodological challenges that might be addressed through random assignment to treatmentExpand
Promoting Transparency in Social Science Research
There is growing appreciation for the advantages of experimentation in the social sciences, and changes have been particularly pronounced in development economics, where hundreds of randomized trials have been carried out over the last decade. Expand
Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa
Abstract There is some evidence that in fields where statistical tests of significance are commonly used, research which yields nonsignificant results is not published. Such research being unknown toExpand
The Insignificance of Null Hypothesis Significance Testing
The current method of hypothesis testing in the social sciences is under intense criticism, yet most political scientists are unaware of the important issues being raised. Criticisms focus on theExpand