Underreporting in Psychology Experiments

@article{Franco2016UnderreportingIP,
  title={Underreporting in Psychology Experiments},
  author={Annie Franco and Neil Malhotra and G{\'a}bor Simonovits},
  journal={Social Psychological and Personality Science},
  year={2016},
  volume={7},
  pages={12 - 8}
}
Many scholars have raised concerns about the credibility of empirical findings in psychology, arguing that the proportion of false positives reported in the published literature dramatically exceeds the rate implied by standard significance levels. A major contributor of false positives is the practice of reporting a subset of the potentially relevant statistical analyses pertaining to a research project. This study is the first to provide direct evidence of selective underreporting in… 

Figures and Tables from this paper

Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Publication bias and questionable research practices in primary research can lead to badly overestimated effects in meta-analysis. Methodologists have proposed a variety of statistical approaches to
What Meta-Analyses Reveal About the Replicability of Psychological Research
TLDR
The low power and high heterogeneity that the survey finds fully explain recent difficulties to replicate highly regarded psychological studies and reveal challenges for scientific progress in psychology.
The perceived role of null results in school psychology research and publication
Researchers appear to assume that published research is limited to significant findings. If that is the case, it may be related to perceived or actual publication bias (i.e., journals publishing only
Scientific Misconduct in Psychology: A Systematic Review of Prevalence Estimates and New Empirical Data
Spectacular cases of scientific misconduct have contributed to concerns about the validity of published results in psychology. In our systematic review, we identified 16 studies reporting prevalence
Publication Biases in Replication Studies
Abstract One of the strongest findings across the sciences is that publication bias occurs. Of particular note is a “file drawer bias” where statistically significant results are privileged over
Black and White discrimination in the United States: Evidence from an archive of survey experiment studies
This study reports results from a new analysis of 17 survey experiment studies that permitted assessment of racial discrimination, drawn from the archives of the Time-sharing Experiments for the
Questionable research practices among italian research psychologists
TLDR
The similarity of results obtained in the United States, this study, and a related study conducted in Germany suggest that adoption of these practices is an international phenomenon and is likely due to systemic features of the international research and publication processes.
Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis
TLDR
Examination of evidence of publication bias in a large-scale data set of primary studies that were included in 83 meta-analyses published in Psychological Bulletin and 499 systematic reviews from the Cochrane Database of Systematic Reviews found evidence of publications bias that appeared to be similar in both psychology and medicine.
Replication and Reproducibility in Cross-Cultural Psychology
Replication is the scientific gold standard that enables the confirmation of research findings. Concerns related to publication bias, flexibility in data analysis, and high-profile cases of academic
Should Psychology Journals Adopt Specialized Statistical Review?
Readers of peer-reviewed research may assume that the reported statistical analyses supporting scientific claims have been closely scrutinized and surpass a high-quality threshold. However,
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 18 REFERENCES
Underreporting in Political Science Survey Experiments: Comparing Questionnaires to Published Results
The accuracy of published findings is compromised when researchers fail to report and adjust for multiple testing. Preregistration of studies and the requirement of preanalysis plans for publication
Finding the Missing Science : The Fate of Studies Submitted for Review by a Human Subjects Committee
Publication bias, including prejudice against the null hypothesis, and other biasing filters may operate on researchers as well as journal editors and reviewers. A survey asked 33 psychology
False-Positive Psychology
TLDR
It is shown that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings, flexibility in data collection, analysis, and reporting dramatically increases actual false- positive rates, and a simple, low-cost, and straightforwardly effective disclosure-based solution is suggested.
An Agenda for Purely Confirmatory Research
TLDR
This article proposes that researchers preregister their studies and indicate in advance the analyses they intend to conduct, and proposes that only these analyses deserve the label “confirmatory,” and only for these analyses are the common statistical tests valid.
Let’s Put Our Money Where Our Mouth Is
  • J. Maner
  • Psychology
    Perspectives on psychological science : a journal of the Association for Psychological Science
  • 2014
TLDR
Recommendations include changing the way reviewers respond to imperfections in empirical data, focusing less on individual tests of statistical significance and more on meta-analyses, and attending carefully to the theoretical contribution of a manuscript in addition to its methodological rigor.
Publication bias in the social sciences: Unlocking the file drawer
TLDR
Fully half of peer-reviewed and implemented social science experiments are not published, providing direct evidence of publication bias and identifying the stage of research production at which publication bias occurs: Authors do not write up and submit null findings.
Registered Reports A Method to Increase the Credibility of Published Results
Ignoring replications and negative results is bad for science. This special issue presents a novel publishing format – Registered Reports – as a partial solution. Peer review occurs prior to data
The New Statistics
TLDR
An eight-step new-statistics strategy for research with integrity is described, which starts with formulation of research questions in estimation terms, has no place for NHST, and is aimed at building a cumulative quantitative discipline.
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
TLDR
It is found that the percentage of respondents who have engaged in questionable practices was surprisingly high, which suggests that some questionable practices may constitute the prevailing research norm.
Promoting Transparency in Social Science Research
TLDR
There is growing appreciation for the advantages of experimentation in the social sciences, and changes have been particularly pronounced in development economics, where hundreds of randomized trials have been carried out over the last decade.
...
1
2
...