Small effect size leads to reproducibility failure in resting-state fMRI studies

Xi Ze Jia, Na Zhao, Barek Barton, Roxana Burciu, Nicolas Carrière, Antonio Cerasa, Bo Yu Chen, Jun Chen, Stephen Coombes, Luc Defebvre, Christine Delmaire, Kathy Dujardin, Fabrizio Esposito, Guo Guang Fan, Di Nardo Federica, Yi Xuan Feng, Brett W. Fling, Saurabh Garg, Moran Gilat, Martin GorgesShu Leong Ho, Fay B. Horak, Xiao Hu, Xiao Fei Hu, Biao Huang, Pei Yu Huang, Ze Juan Jia, Christy Jones, Jan Kassubek, Lenka Krajcovicova, Ajay Kurani, Jing Li, Qian Li, Ai Ping Liu, Bo Liu, Hu Liu, Wei Guo Liu, Renaud Lopes, Yu Ting Lou, Wei Luo, Tara Madhyastha, Ni Ni Mao, Grainne McAlonan, Martin J. McKeown, Shirley Y.Y. Pang, Aldo Quattrone, Irena Rektorova, Alessia Sarica, Hui Fang Shang, James Shine, Priyank Shukla, Tomas Slavicek, Xiao Peng Song, Gioacchino Tedeschi, Alessandro Tessitore, David Vaillancourt, Jian Wang, Jue Wang, Z. Jane Wang, Lu Qing Wei, Xia Wu, Xiao Jun Xu, Lei Yan, Jing Yang, Wan Qun Yang, Nai Lin Yao, De Long Zhang, Jiu Quan Zhang, Min Ming Zhang, Yan Ling Zhang, Cai Hong Zhou, Chao Gan Yan, Xi Nian Zuo, Mark Hallett, Tao Wu, Yu Feng Zang*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

Thousands of papers using resting-state functional magnetic resonance imaging (RS-fMRI) have been published on brain disorders. Results in each paper may have survived correction for multiple comparison. However, since there have been no robust results from large scale meta-analysis, we do not know how many of published results are truly positives. The present meta-analytic work included 60 original studies, with 57 studies (4 datasets, 2266 participants) that used a between-group design and 3 studies (1 dataset, 107 participants) that employed a within-group design. To evaluate the effect size of brain disorders, a very large neuroimaging dataset ranging from neurological to psychiatric isorders together with healthy individuals have been analyzed. Parkinson’s disease off levodopa (PD-off) included 687 participants from 15 studies. PD on levodopa (PD-on) included 261 participants from 9 studies. Autism spectrum disorder (ASD) included 958 participants from 27 studies. The meta-analyses of a metric named amplitude of low frequency fluctuation (ALFF) showed that the effect size (Hedges’ g) was 0.19 - 0.39 for the 4 datasets using between-group design and 0.46 for the dataset using within-group design. The effect size of PD-off, PD-on and ASD were 0.23, 0.39, and 0.19, respectively. Using the meta-analysis results as the robust results, the between-group design results of each study showed high false negative rates (median 99%), high false discovery rates (median 86%), and low accuracy (median 1%), regardless of whether stringent or liberal multiple comparison correction was used. The findings were similar for 4 RS-fMRI metrics including ALFF, regional homogeneity, and degree centrality, as well as for another widely used RS-fMRI metric namely seed-based functional connectivity. These observations suggest that multiple comparison correction does not control for false discoveries across multiple studies when the effect sizes are relatively small. Meta-analysis on un-thresholded t-maps is critical for the recovery of ground truth. We recommend that to achieve high reproducibility through meta-analysis, the neuroimaging research field should share raw data or, at minimum, provide un-thresholded statistical images.

Original languageEnglish (US)
JournalUnknown Journal
DOIs
StatePublished - Mar 20 2018

ASJC Scopus subject areas

  • Biochemistry, Genetics and Molecular Biology(all)
  • Agricultural and Biological Sciences(all)
  • Immunology and Microbiology(all)
  • Neuroscience(all)
  • Pharmacology, Toxicology and Pharmaceutics(all)

Fingerprint Dive into the research topics of 'Small effect size leads to reproducibility failure in resting-state fMRI studies'. Together they form a unique fingerprint.

Cite this