An Agenda for Purely Confirmatory Research

@article{Wagenmakers2012AnAF,
  title={An Agenda for Purely Confirmatory Research},
  author={Eric-Jan Wagenmakers and Ruud Wetzels and Denny Borsboom and H.L.J. van der Maas and Rogier Andrew Kievit},
  journal={Perspectives on Psychological Science},
  year={2012},
  volume={7},
  pages={632 - 638}
}
The veracity of substantive research claims hinges on the way experimental data are collected and analyzed. In this article, we discuss an uncomfortable fact that threatens the core of psychology’s academic enterprise: almost without exception, psychologists do not commit themselves to a method of data analysis before they see the actual data. It then becomes tempting to fine tune the analysis to the data in order to obtain a desired result—a procedure that invalidates the interpretation of the… 
View on SAGE
ejwagenmakers.com
659 Citations
Highly Influential Citations
22
Background Citations
151
Methods Citations
52
Results Citations
10

Figures and Topics from this paper

659 Citations

Citation Type

Citation Type

There's more than one way to conduct a replication study: Beyond statistical significance.
TLDR
A variety of additional "replication goals" are developed that will allow researchers to develop a more nuanced understanding of replication that can be flexible enough to answer the various questions that researchers might seek to understand.
Open Peer Commentary
With the growing number of fraudulent and nonr replicability of experiments performed in laboratories world practices that investigators may be using to increase the replicability. We laud them for…
Psychology's Renaissance
TLDR
It is shown that the scientific practices of experimental psychologists have improved dramatically and is argued that meta‐analytical thinking increases the prevalence of false positives.
Examining Psychological Science Through Systematic Meta-Method Analysis: A Call for Research
  • M. Elson
  • Psychology
    Advances in Methods and Practices in Psychological Science
  • 2019
Research synthesis is based on the assumption that when the same association between constructs is observed repeatedly in a field, the relationship is probably real, even if its exact magnitude can…
Don’t Forget the Psychology in Analyses of Psychological Data: The Case of Sequential Testing
Sequential testing enables researchers to monitor and analyze data as it arrives, and decide whether or not to continue data collection depending on the results. Bayesian approaches can mitigate many…
Flexible yet fair: blinding analyses in experimental psychology
TLDR
How researchers may conduct blinded analyses is discussed, which breaks the feedback loop between the analysis plan and analysis outcome, thereby preventing cherry-picking and significance seeking and retaining the flexibility to account for unexpected peculiarities in the data.
From Discovery to Justification: Outline of an Ideal Research Program in Empirical Psychology
TLDR
It is argued that the replicability crisis is “home-made” because more sophisticated strategies can deliver results the successful replication of which is sufficiently probable, and can be overcome by integrating empirical results into genuine research programs.
A purely confirmatory replication study of structural brain-behavior correlations
TLDR
This is the first multi-study confirmatory replication of SBB correlation studies comprising a total of 17 effects and hopes to encourage other researchers to undertake similar replication attempts.
Addressing the theory crisis in psychology.
TLDR
Recommendations for addressing the "replication crisis" are critically revisited, including the proposal to distinguish exploratory from confirmatory research, and the preregistration of hypotheses and analysis plans.
Underreporting in Psychology Experiments
Many scholars have raised concerns about the credibility of empirical findings in psychology, arguing that the proportion of false positives reported in the published literature dramatically exceeds…
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 51 REFERENCES
Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011).
TLDR
It is concluded that Bem's p values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.
Must psychologists change the way they analyze their data?
TLDR
It is argued that Wagenmakers, Wetzels, Borsboom, and van der Maas have incorrectly selected an unrealistic prior distribution for their analysis and that a bayesian analysis using a more reasonable distribution yields strong evidence in favor of the psi hypothesis.
False-Positive Psychology
TLDR
It is shown that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings, flexibility in data collection, analysis, and reporting dramatically increases actual false- positive rates, and a simple, low-cost, and straightforwardly effective disclosure-based solution is suggested.
A practical solution to the pervasive problems ofp values
TLDR
The BIC provides an approximation to a Bayesian hypothesis test, does not require the specification of priors, and can be easily calculated from SPSS output.
The Rules of the Game Called Psychological Science
TLDR
This paper considers 13 meta-analyses covering 281 primary studies in various fields of psychology and finds indications of biases and/or an excess of significant results in seven, highlighting the need for sufficiently powerful replications and changes in journal policies.
Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa
Abstract There is some evidence that in fields where statistical tests of significance are commonly used, research which yields nonsignificant results is not published. Such research being unknown to…
Bayesian Versus Orthodox Statistics: Which Side Are You On?
  • Z. Dienes
  • Psychology, Medicine
    Perspectives on psychological science : a journal of the Association for Psychological Science
  • 2011
TLDR
This article presents some common situations in which Bayesian and orthodox approaches to significance testing come to different conclusions; the reader is shown how to apply Bayesian inference in practice, using free online software, to allow more coherent inferences from data.
Circular analysis in systems neuroscience: the dangers of double dipping
TLDR
It is argued that systems neuroscience needs to adjust some widespread practices to avoid the circularity that can arise from selection, and 'double dipping' the use of the same dataset for selection and selective analysis is suggested.
Why Most Published Research Findings Are False
TLDR
Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true.
Publication decisions revisited: the effect of the outcome of statistical tests on the decision to p
TLDR
Evidence that published results of scientific investigations are not a representative sample of results of all scientific studies is presented and practice leading to publication bias have not changed over a period of 30 years is indicated.
...
1
2
3
4
5
...