Underreporting in Psychology Experiments
@article{Franco2016UnderreportingIP, title={Underreporting in Psychology Experiments}, author={Annie Franco and Neil Malhotra and G{\'a}bor Simonovits}, journal={Social Psychological and Personality Science}, year={2016}, volume={7}, pages={12 - 8} }
Many scholars have raised concerns about the credibility of empirical findings in psychology, arguing that the proportion of false positives reported in the published literature dramatically exceeds the rate implied by standard significance levels. A major contributor of false positives is the practice of reporting a subset of the potentially relevant statistical analyses pertaining to a research project. This study is the first to provide direct evidence of selective underreporting in…
49 Citations
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods
- PsychologyAdvances in Methods and Practices in Psychological Science
- 2019
Publication bias and questionable research practices in primary research can lead to badly overestimated effects in meta-analysis. Methodologists have proposed a variety of statistical approaches to…
What Meta-Analyses Reveal About the Replicability of Psychological Research
- PsychologyPsychological bulletin
- 2018
The low power and high heterogeneity that the survey finds fully explain recent difficulties to replicate highly regarded psychological studies and reveal challenges for scientific progress in psychology.
The perceived role of null results in school psychology research and publication
- PsychologySchool Psychology International
- 2019
Researchers appear to assume that published research is limited to significant findings. If that is the case, it may be related to perceived or actual publication bias (i.e., journals publishing only…
Scientific Misconduct in Psychology: A Systematic Review of Prevalence Estimates and New Empirical Data
- PsychologyZeitschrift für Psychologie
- 2019
Spectacular cases of scientific misconduct have contributed to concerns about the validity of published results in psychology. In our systematic review, we identified 16 studies reporting prevalence…
Publication Biases in Replication Studies
- PsychologyPolitical Analysis
- 2020
Abstract One of the strongest findings across the sciences is that publication bias occurs. Of particular note is a “file drawer bias” where statistically significant results are privileged over…
Black and White discrimination in the United States: Evidence from an archive of survey experiment studies
- Psychology
- 2018
This study reports results from a new analysis of 17 survey experiment studies that permitted assessment of racial discrimination, drawn from the archives of the Time-sharing Experiments for the…
Questionable research practices among italian research psychologists
- PsychologyPloS one
- 2017
The similarity of results obtained in the United States, this study, and a related study conducted in Germany suggest that adoption of these practices is an international phenomenon and is likely due to systemic features of the international research and publication processes.
Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis
- Psychology, BiologyPloS one
- 2019
Examination of evidence of publication bias in a large-scale data set of primary studies that were included in 83 meta-analyses published in Psychological Bulletin and 499 systematic reviews from the Cochrane Database of Systematic Reviews found evidence of publications bias that appeared to be similar in both psychology and medicine.
Replication and Reproducibility in Cross-Cultural Psychology
- Psychology
- 2018
Replication is the scientific gold standard that enables the confirmation of research findings. Concerns related to publication bias, flexibility in data analysis, and high-profile cases of academic…
Should Psychology Journals Adopt Specialized Statistical Review?
- PsychologyAdvances in Methods and Practices in Psychological Science
- 2019
Readers of peer-reviewed research may assume that the reported statistical analyses supporting scientific claims have been closely scrutinized and surpass a high-quality threshold. However,…
References
SHOWING 1-10 OF 18 REFERENCES
Underreporting in Political Science Survey Experiments: Comparing Questionnaires to Published Results
- EconomicsPolitical Analysis
- 2015
The accuracy of published findings is compromised when researchers fail to report and adjust for multiple testing. Preregistration of studies and the requirement of preanalysis plans for publication…
Finding the Missing Science : The Fate of Studies Submitted for Review by a Human Subjects Committee
- Psychology
- 2004
Publication bias, including prejudice against the null hypothesis, and other biasing filters may operate on researchers as well as journal editors and reviewers. A survey asked 33 psychology…
False-Positive Psychology
- PsychologyPsychological science
- 2011
It is shown that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings, flexibility in data collection, analysis, and reporting dramatically increases actual false- positive rates, and a simple, low-cost, and straightforwardly effective disclosure-based solution is suggested.
An Agenda for Purely Confirmatory Research
- PsychologyPerspectives on psychological science : a journal of the Association for Psychological Science
- 2012
This article proposes that researchers preregister their studies and indicate in advance the analyses they intend to conduct, and proposes that only these analyses deserve the label “confirmatory,” and only for these analyses are the common statistical tests valid.
Let’s Put Our Money Where Our Mouth Is
- PsychologyPerspectives on psychological science : a journal of the Association for Psychological Science
- 2014
Recommendations include changing the way reviewers respond to imperfections in empirical data, focusing less on individual tests of statistical significance and more on meta-analyses, and attending carefully to the theoretical contribution of a manuscript in addition to its methodological rigor.
Publication bias in the social sciences: Unlocking the file drawer
- EconomicsScience
- 2014
Fully half of peer-reviewed and implemented social science experiments are not published, providing direct evidence of publication bias and identifying the stage of research production at which publication bias occurs: Authors do not write up and submit null findings.
Registered Reports A Method to Increase the Credibility of Published Results
- Psychology
- 2014
Ignoring replications and negative results is bad for science. This special issue presents a novel publishing format – Registered Reports – as a partial solution. Peer review occurs prior to data…
The New Statistics
- PsychologyPsychological science
- 2014
An eight-step new-statistics strategy for research with integrity is described, which starts with formulation of research questions in estimation terms, has no place for NHST, and is aimed at building a cumulative quantitative discipline.
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
- Political Science, BusinessPsychological science
- 2012
It is found that the percentage of respondents who have engaged in questionable practices was surprisingly high, which suggests that some questionable practices may constitute the prevailing research norm.
Promoting Transparency in Social Science Research
- Political ScienceScience
- 2014
There is growing appreciation for the advantages of experimentation in the social sciences, and changes have been particularly pronounced in development economics, where hundreds of randomized trials have been carried out over the last decade.