A Bayes factor meta-analysis of Bem’s ESP claim

@article{Rouder2011ABF,
  title={A Bayes factor meta-analysis of Bem’s ESP claim},
  author={Jeffrey N. Rouder and Richard Donald Morey},
  journal={Psychonomic Bulletin \& Review},
  year={2011},
  volume={18},
  pages={682-689}
}
In recent years, statisticians and psychologists have provided the critique that p-values do not capture the evidence afforded by data and are, consequently, ill suited for analysis in scientific endeavors. The issue is particular salient in the assessment of the recent evidence provided for ESP by Bem (2011) in the mainstream Journal of Personality and Social Psychology. Wagenmakers, Wetzels, Borsboom, and van der Maas (Journal of Personality and Social Psychology, 100, 426–432, 2011) have… 
View on Springer
link.springer.com
175 Citations
Highly Influential Citations
17
Background Citations
44
Methods Citations
66
Results Citations
4

175 Citations

Commentary: How Bayes factors change scientific practice
TLDR
Dienes's main argument is one of best-fit-for-purpose: the Bayes factor pits the probability of the data under one hypothesis against that under another on equal ground, providing a symmetric assessment--the data may favor either hypothesis, or neither--as a continuous measure of evidence in the form of odds.
Why Bayesian Psychologists Should Change the Way They Use the Bayes Factor
TLDR
It is argued that a thorough investigation of problems of the Bayesian factor in the context of more encompassing hypotheses and statistical models is called for if Bayesian psychologists want to add a well-founded Bayes factor to the tool kit of psychological researchers.
Meta Analysis of Bayes Factors
Bayes Factors, the Bayesian tool for hypothesis testing, are receiving increasing attention in the literature. Compared to their frequentist rivals (p-values or test statistics), Bayes Factors have…
A Bayes factor meta-analysis of recent extrasensory perception experiments: comment on Storm, Tressoldi, and Di Risio (2010).
TLDR
This analysis differs from Storm et al.'s in that it relies on Bayes factors, a Bayesian approach for stating the evidence from data for competing theoretical positions, and argues that this value is unpersuasive in the context of psi because there is no plausible mechanism and because there are almost certainly omitted replication failures.
Feeling the future: A meta-analysis of 90 experiments on the anomalous anticipation of random future events
TLDR
A meta-analysis of 90 experiments from 33 laboratories in 14 countries which yielded an overall effect greater than 6 sigma is reported, indicating that the database is not significantly compromised by either selection bias or by intense “ p-hacking”—the selective suppression of findings or analyses that failed to yield statistical significance.
Feeling the future: A meta-analysis of 90 experiments on the anomalous anticipation of random future events.
TLDR
A meta-analysis of 90 experiments from 33 laboratories in 14 countries which yielded an overall effect greater than 6 sigma is reported, concluding that the database is not significantly compromised by either selection bias or by " p-hacking"-the selective suppression of findings or analyses that failed to yield statistical significance.
Must psychologists change the way they analyze their data?
TLDR
It is argued that Wagenmakers, Wetzels, Borsboom, and van der Maas have incorrectly selected an unrealistic prior distribution for their analysis and that a bayesian analysis using a more reasonable distribution yields strong evidence in favor of the psi hypothesis.
Using Bayes to get the most out of non-significant results
TLDR
It is argued Bayes factors allow theory to be linked to data in a way that overcomes the weaknesses of the other approaches, and provides a coherent approach to determining whether non-significant results support a null hypothesis over a theory, or whether the data are just insensitive.
Bayes factor approaches for testing interval null hypotheses.
TLDR
This article develops several Bayes factor 1-sample tests for the assessment of approximate equality and ordinal constraints, which are alternatives to previously developed Bayes factors, which do not allow for interval null hypotheses.
A Primer on Bayesian Model-Averaged Meta-Analysis
TLDR
Bayesian model-averaged meta-analysis therefore avoids the need to select either a fixed-effect or random-effects model and instead takes into account model uncertainty in a principled manner.
...
...

References

SHOWING 1-10 OF 25 REFERENCES
THE STRENGTH OF EVIDENCE VERSUS THE POWER OF BELIEF: ARE WE ALL BAYESIANS?
Although statisticians have the job of making conclusions based on data, for many questions in science and society prior beliefs are strong and may take precedence over data when people make…
A practical solution to the pervasive problems ofp values
TLDR
The BIC provides an approximation to a Bayesian hypothesis test, does not require the specification of priors, and can be easily calculated from SPSS output.
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Abstract The problem of testing a point null hypothesis (or a “small interval” null hypothesis) is considered. Of interest is the relationship between the P value (or observed significance level) and…
Bayesian t tests for accepting and rejecting the null hypothesis
TLDR
To facilitate use of the Bayes factor, an easy-to-use, Web-based program is provided that performs the necessary calculations and has better properties than other methods of inference that have been advocated in the psychological literature.
The Bayesian Two-Sample t Test
This article shows how the pooled-variance two-sample t statistic arises from a Bayesian formulation of the two-sided point null testing problem, with emphasis on teaching. We identify a reasonable…
The importance of proving the null.
TLDR
The simple computations and the intuitive graphic representation of the analysis are illustrated by the analysis of diverse examples from the current literature.
Meta-analysis of free-response studies, 1992-2008: assessing the noise reduction model in parapsychology.
TLDR
It was found that selected participants (believers in the paranormal, meditators, etc.) had a performance advantage over unselected participants, but only if they were in the ganzfeld condition.
The Theory of Probability
TLDR
This book is a searching analysis of the fundamental principles of the theory of probability and of the particular judgments involved in its application to concrete problems and is in agreement with the views expressed by Dr. Wrinch and the present reviewer.
Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect.
  • D. Bem
  • Psychology
    Journal of personality and social psychology
  • 2011
TLDR
9 experiments are reported, involving more than 1,000 participants, that test for retroactive influence by "time-reversing" well-established psychological effects so that the individual's responses are obtained before the putatively causal stimulus events occur.
Bayes Factors in Practice
Computational advances have facilitated application of various Bayesian methods, including the assessment of evidence in favor of a scientific theory. This is accomplished by calculating a Bayes…
...
...