Small sample sizes reduce the replicability of task-based fMRI studies

@article{Turner2018SmallSS,
  title={Small sample sizes reduce the replicability of task-based fMRI studies},
  author={Benjamin O. Turner and E. Paul and Michael B. Miller and A. Barbey},
  journal={Communications Biology},
  year={2018},
  volume={1}
}
Despite a growing body of research suggesting that task-based functional magnetic resonance imaging (fMRI) studies often suffer from a lack of statistical power due to too-small samples, the proliferation of such underpowered studies continues unabated. Using large independent samples across eleven tasks, we demonstrate the impact of sample size on replicability, assessed at different levels of analysis relevant to fMRI researchers. We find that the degree of replicability for typical sample… Expand

Figures, Tables, and Topics from this paper

The empirical replicability of task-based fMRI as a function of sample size
TLDR
This work uses a large database of 1400 subjects performing four types of tasks from the IMAGEN project to subsample a series of independent samples of increasing size and demonstrates not only a positive effect of sample size, but also a trade-off between spatial resolution and replicability. Expand
Correspondence: fMRI replicability depends upon sufficient individual-level data
TLDR
It is shown that fMRI can have strong replicability even at modest sample sizes when individuals are adequately sampled, but that inadequate individual-level sampling leads to poor replicable. Expand
fMRI replicability depends upon sufficient individual-level data
  • D. E. Nee
  • Psychology, Medicine
  • Communications Biology
  • 2019
TLDR
The authors found that even data sets with 100 or more participants can produce results that do not replicate, suggesting that large sample sizes are necessary for task-based fMRI replicability. Expand
Hyperbolic trade-off: the importance of balancing trial and subject sample sizes in neuroimaging
TLDR
This work systematically explores the different factors that impact effect uncertainty, drawing on evidence from hierarchical modeling, simulations and an FMRI dataset of 42 subjects who completed a large number of trials of a commonly used cognitive task to find that trial sample size has nearly the same impact on statistical efficiency. Expand
Standardized Effect Sizes and Image-Based Meta-Analytical Approaches for fMRI Data
TLDR
This paper discusses how an observed effect size in an fMRI data-analysis can be transformed into a standardized effect size and discusses a general model to aggregate these and different approaches to deal with between-study heterogeneity. Expand
Influence of sample size and analytic approach on stability and interpretation of brain‐behavior correlations in task‐related fMRI data
TLDR
Novel experimental evidence is provided in two independent data sets that the sample size commonly used in fMRI studies of 20–30 participants is very unlikely to be sufficient for obtaining reproducible brain‐behavior correlations, regardless of analytic approach. Expand
Power, positive predictive value, and sample size calculations for random field theory-based fMRI inference
TLDR
Power and positive predictive value (PPV) functions for voxel-and cluster-level inference in both uncorrected single test and corrected multiple testing scenarios are developed and the theoretical results are applied to evaluate the sample size necessary to achieve desired power and PPV levels based on an fMRI pilot study. Expand
Spatial confidence sets for raw effect size images
TLDR
This work extends on a method proposed by Sommerfeld et al. (2018) (SSS) to develop spatial Confidence Sets (CSs) on clusters found in thresholded raw effect size maps, delivering a procedure with superior performance in sample sizes as low as N=60. Expand
Empirical assessment of changing sample‐characteristics in task‐fMRI over two decades: An example from gustatory and food studies
Over the past two decades, functional neuroimaging has not only grown into a large field of research, but also substantially evolved. Here we provide a quantitative assessment of these presumed inExpand
Replication of fMRI group activations in the neuroimaging battery for the Mainz Resilience Project (MARP)
TLDR
Overall, good to excellent replicability of activations at the whole brain level was observed and Tasks with stronger activation in the discovery sample showed better replicable. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 64 REFERENCES
Power and sample size calculations for fMRI studies based on the prevalence of active peaks
TLDR
A simple way to characterize the spatial signal in a fMRI study with just two parameters, and a direct way to estimate these two parameters based on an existing study is presented, allowing one to minimize the cost of an fMRI experiment, while preserving a predefined statistical power. Expand
The relation between statistical power and inference in fMRI
TLDR
This work aimed to clarify the power problem by considering and contrasting two simulated scenarios of such possible brain-behavior correlations: weak diffuse effects and strong localized effects. Expand
What Has Replication Ever Done for Us? Insights from Neuroimaging of Speech Perception
  • S. Evans
  • Psychology, Medicine
  • Front. Hum. Neurosci.
  • 2017
TLDR
Recently in the speech perception domain, there have been some notable replication attempts, here I discuss what has been learnt from them about speech perception and the replication endeavor more generally. Expand
Analysis of a large fMRI cohort: Statistical and methodological issues for group analyses
TLDR
The study shows that inter-subject variability plays a prominent role in the relatively low sensitivity and reliability of group studies and focuses on the notion of reproducibility by bootstrapping. Expand
Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates
TLDR
It is found that the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%. Expand
The secret lives of experiments: Methods reporting in the fMRI literature
  • J. Carp
  • Computer Science, Medicine
  • NeuroImage
  • 2012
TLDR
Significant gaps in methods reporting among fMRI studies are documented, and improved methodological descriptions in research reports would yield significant benefits for the field. Expand
fMRI reliability: Influences of task and experimental design
TLDR
The results suggest that fMRI reliability is a complex construct whose value may be increased or decreased by specific combinations of factors, and which could interact, increasing or decreasing the relative reliability of certain Task × Design combinations. Expand
Power calculation for group fMRI studies accounting for arbitrary design and temporal autocorrelation
TLDR
A flexible power calculation model is introduced that makes fewer simplifying assumptions, leading to a more accurate power analysis that can be used on a wide variety of study designs and can be use to increase understanding of the data as well as calculate power for a future study. Expand
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature
TLDR
In light of the findings, the recently reported low replication success in psychology is realistic, and worse performance may be expected for cognitive neuroscience, and false report probability is likely to exceed 50% for the whole literature. Expand
Revisiting multi-subject random effects in fMRI: Advocating prevalence estimation
TLDR
This work proposes a Gaussian-mixture-random-effect that amortizes between-subject spatial disagreement and quantifies it using the prevalence of activation at each location, and presents a formal definition and an estimation procedure of this prevalence. Expand
...
1
2
3
4
5
...