Which Data to Meta-Analyze, and How?: A Specification-Curve and Multiverse-Analysis Approach to Meta-Analysis

@article{Voracek2019WhichDT,
  title={Which Data to Meta-Analyze, and How?: A Specification-Curve and Multiverse-Analysis Approach to Meta-Analysis},
  author={Martin Voracek and Michael Kossmeier and Ulrich S. Tran},
  journal={Zeitschrift f{\"u}r Psychologie},
  year={2019},
  volume={227},
  pages={64–82}
}
Which data to analyze, and how, are fundamental questions of all empirical research. As there are always numerous flexibilities in data-analytic decisions (a “garden of forking paths”), this poses perennial problems to all empirical research. Specification-curve analysis and multiverse analysis have recently been proposed as solutions to these issues. Building on the structural analogies between primary data analysis and meta-analysis, we transform and adapt these approaches to the meta… 

Figures and Tables from this paper

A Survey of Tasks and Visualizations in Multiverse Analysis Reports
Analysing data from experiments is a complex, multi‐step process, often with multiple defensible choices available at each step. While analysts often report a single analysis without documenting how
Conducting a meta-analysis in the age of open science: Tools, tips, and practical recommendations.
TLDR
This tutorial describes why open science is important in the context of meta-analysis in psychology, and suggests how to adopt the 3 main components of open science: preregistration, open materials, and open data.
Charting the landscape of graphical displays for meta-analysis and systematic reviews: a comprehensive review, taxonomy, and feature analysis
TLDR
A comprehensive overview of available graphs allows researchers to make better-informed decisions on which graphs suit their needs and therefore facilitates using the meta-analytic tool kit of graphs to its full potential.
Null and Void? Errors in Meta-analysis on Perceptual Disfluency and Recommendations to Improve Meta-analytical Reproducibility
TLDR
Here a framework applicable to pre- and post-publication review on improving the Methods Reproducibility of meta-analysis is presented, and applying reproducibility checks to TRANSFER performance as the chosen outcome variable is found to be (rather) robust.
Of differing methods, disputed estimates and discordant interpretations: the meta-analytical multiverse of brain volume and IQ associations
Brain size and IQ are positively correlated. However, multiple meta-analyses have led to considerable differences in summary effect estimations, thus failing to provide a plausible effect estimate.
Future Objectivity Requires Perspective and Forward Combinatorial Meta-Analyses
This manuscript contributes to a future definition of objectivity by bringing together recent statements in epistemology and methodology. It outlines how improved objectivity can be achieved by
2D:4D Does Not Predict Economic Preferences: Evidence from a Large, Representative Sample
The digit ratio (2D:4D) is considered a proxy for testosterone exposure in utero, and there has been a recent surge of studies testing whether 2D:4D is associated with economic preferences. Although
Exploring the efficacy of psychological treatments for depression: a multiverse meta-analysis protocol
Introduction In the past four decades, over 700 randomised controlled trials (RCTs) and 80 meta-analyses have examined the efficacy of psychological treatments for depression. Overwhelming evidence
Meta-analyzing intelligence and religiosity associations: Evidence from the multiverse
Over the past century, a remarkable body of research about the relationship of intelligence and religiosity has accumulated. So far, the majority of studies that investigated this relationship showed
2D:4D and Self-Employment Using SOEP Data: A Replication Study
The 2 D:4D digit ratio, the ratio of the length of the 2nd digit to the length of the 4th digit, is often considered a proxy for testosterone exposure in utero. A recent study by Nicolaou et al.
...
...

References

SHOWING 1-10 OF 180 REFERENCES
Specification Curve: Descriptive and Inferential Statistics on All Reasonable Specifications
Empirical results often hinge on data analytic decisions that are simultaneously defensible, arbitrary, and motivated. To mitigate this problem we introduce Specification-Curve Analysis, which
Increasing Transparency Through a Multiverse Analysis
TLDR
It is shown that analyzing a single data set can be misleading and a multiverse analysis is proposed as an alternative practice, which involves performing all analyses across the whole set of alternatively processed data sets corresponding to a large set of reasonable scenarios.
Comment: Why Meta-Analyses Rarely Resolve Ideological Debates
In their meta-analysis Wood, Kressel, Joshi, and Louie (2014) argue little evidence supports shifts in mating preferences across the menstrual cycle. They imply this may represent a critical weakness
A Unified Framework to Quantify the Credibility of Scientific Findings
TLDR
This work outlines a unified framework for estimating the credibility of published research by examining four fundamental falsifiability-related dimensions, and includes a standardized workflow in which the degree to which a finding has survived scrutiny is quantified along these four facets of credibility.
Model Uncertainty and Robustness
TLDR
The architecture and interpretation of multimodel analysis is demonstrated using data on the union wage premium, gender dynamics in mortgage lending, and tax flight migration among U.S. states to illustrate how initial results can be strongly robust to alternative model specifications or remarkably dependent on a knife-edge specification.
Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to
GOSH – a graphical display of study heterogeneity
TLDR
This work uses the summary effect sizes and other statistics produced by the all-subsets meta-analyses to generate graphs that can be used to investigate heterogeneity, identify influential studies, and explore subgroup effects.
Elastic analysis procedures: an incurable (but preventable) problem in the fertility effect literature. Comment on Gildersleeve, Haselton, and Fales (2014).
TLDR
It is argued that if fertility-effect researchers take advantage of the rapidly emerging opportunities for study preregistration, the validity of this literature can be rapidly clarified, and that the possibility that positive results in this literature may have been created by p-hacking receives additional support.
Meta-analyses and p-curves support robust cycle shifts in women's mate preferences: reply to Wood and Carden (2014) and Harris, Pashler, and Mickes (2014).
TLDR
Evidence from both of the meta-analyses and the p-curves strongly supports genuine, robust effects consistent with the ovulatory shift hypothesis and contradicts claims that these effects merely reflect publication bias, p-hacking, or other research artifacts.
Triangulating meta-analyses: the example of the serotonin transporter gene, stressful life events and major depression
TLDR
The variability in results of different meta-analyses on the same topic is demonstrated, and it is argued that triangulating evidence using multiple evidence synthesis methods is preferable where possible, and that every effort should be made for meta-analysis to be as unbiased as possible.
...
...