False-Positive Psychology

@article{Simmons2011FalsePositiveP,
  title={False-Positive Psychology},
  author={Joseph P. Simmons and Leif D. Nelson and Uri Simonsohn},
  journal={Psychological Science},
  year={2011},
  volume={22},
  pages={1359 - 1366}
}
In this article, we accomplish two things. First, we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how… 

Figures and Tables from this paper

Research Practices That Can Prevent an Inflation of False-Positive Rates
  • K. MurayamaR. PekrunK. Fiedler
  • Psychology
    Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc
  • 2014
TLDR
It is argued that an inflation of false-positive rates would diminish, sometimes to a substantial degree, when researchers have explicit a priori theoretical hypotheses, include multiple replication studies in a single paper, and collect additional data based on observed results.
Underreporting in Psychology Experiments
Many scholars have raised concerns about the credibility of empirical findings in psychology, arguing that the proportion of false positives reported in the published literature dramatically exceeds
Theoretical false positive psychology.
A fundamental goal of scientific research is to generate true positives (i.e., authentic discoveries). Statistically, a true positive is a significant finding for which the underlying effect size (δ)
Fact or fiction: reducing the proportion and impact of false positives
TLDR
The concept of researcher ‘degrees-of-freedom’ is described to explain how many false-positive findings arise, and how the various strategies of registration, pre-specification, and reporting standards that are being adopted both reduce and make these visible.
The Crisis of Confidence in Research Findings in Psychology: Is Lack of Replication the Real Problem? Or Is It Something Else?
There have been frequent expressions of concern over the supposed failure of researchers to conduct replication studies. But the large number of meta-analyses in our literatures shows that
The Long Way From α-Error Control to Validity Proper
TLDR
It is argued that, given the current state of affairs in behavioral science, false negatives often constitute a more serious problem and a scientific culture rewarding strong inference is more likely to see progress than a culture preoccupied with tightening its standards for the mere publication of original findings.
Addressing the theory crisis in psychology.
TLDR
Recommendations for addressing the "replication crisis" are critically revisited, including the proposal to distinguish exploratory from confirmatory research, and the preregistration of hypotheses and analysis plans.
Why most psychological research findings are not even wrong
Psychology’s replication crisis is typically conceptualised as the insight that the published literature contains a worrying amount of unreplicable, false-positive findings. At the same time,
Detecting and avoiding likely false‐positive findings – a practical guide
TLDR
It is argued that a culture of ‘you can publish if your study is rigorous’ creates a systematic bias against the null hypothesis which renders meta‐analyses questionable and may even lead to a situation where hypotheses become difficult to falsify.
In press for Perspectives on Psychological Science dummy text An Agenda for Purely Confirmatory Research
The veracity of substantive research claims hinges on the way experimental data are collected and analyzed. Here we discuss an uncomfortable fact that threatens the core of psychology’s academic
...
...

References

SHOWING 1-10 OF 13 REFERENCES
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
TLDR
It is found that the percentage of respondents who have engaged in questionable practices was surprisingly high, which suggests that some questionable practices may constitute the prevailing research norm.
The case for motivated reasoning.
  • Z. Kunda
  • Philosophy
    Psychological bulletin
  • 1990
TLDR
It is proposed that motivation may affect reasoning through reliance on a biased set of cognitive processes--that is, strategies for accessing, constructing, and evaluating beliefs--that are considered most likely to yield the desired conclusion.
Biased evaluation and persistence in gambling.
  • T. Gilovich
  • Psychology
    Journal of personality and social psychology
  • 1983
TLDR
Gamblers may tend to accept wins at face value but explain away or discount losses, and a manipulation of the salience or existence of a critical "fluke" play in a sporting event had a greater impact on the subsequent expectations of those who had bet on the losing team than of those on the winning team.
Unpublished results hide the decline effect
TLDR
Being open about findings that don't make the scientific record could reveal why some effects diminish when tests are repeated, says Jonathan Schooler.
Explaining Bargaining Impasse: The Role of Self-Serving Biases
A major unsolved riddle facing the social sciences is the cause of impasse in negotiations. The consequences of impasse are evident in the amount of private and public resources spent on civil
Attribution of success and failure revisited, or: The motivational bias is alive and well in attribution theory
Do causal attributions serve the need to protect and / or enhance self-esteem? In a recent review, Miller and Ross (1975) proposed that there is evidence for self-serving effect in the attribution of
Group sequential methods in the design and analysis of clinical trials
SUMMARY In clinical trials with sequential patient entry, fixed sample size designs are unjustified on ethical grounds and sequential designs are often impracticable. One solution is a group
Motivated Reasoning and Performance on the was on Selection Task
People tend to approach agreeable propositions with a bias toward confirmation and disagreeable propositions with a bias toward disconfirmation. Because the appropriate strategy for solving the
They saw a game: a case study.
Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011).
TLDR
It is concluded that Bem's p values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.
...
...