Systematic review and evaluation of meta-analysis methods for same data meta-analyses in multiverse

1905 

Abstract Submission 

jeremy Lefort-Besnard¹, Thomas Nichols², Camille Maumet¹

¹Inria, Empenn team, Rennes, France, ²University of Oxford, Oxford, United Kingdom

jeremy Lefort-Besnard  
Inria, Empenn team
Rennes, France

Thomas Nichols  
University of Oxford
Oxford, United Kingdom

Camille Maumet  
Inria, Empenn team
Rennes, France

Researchers using task-fMRI data have access to a wide range of analysis tools to model brain activity. This diversity of analytical approaches has been shown to have substantial effects on neuroimaging results (Botvinik-Nezer et al., 2020; Bowring et al., 2018; Carp, 2012; Glatard et al., 2015). Combined with selective reporting, this analytical flexibility can lead to an inflated rate of false positives and contributes to the irreproducibility of neuroimaging findings (Poldrack et al., 2017). Multiverse analyses are a way to systematically explore and integrate pipeline variation on a given dataset. We focus on the setting where multiple statistic maps are produced as an output of a set of analyses. Meta-analysis is a natural approach to extract consensus inferences from these maps, yet the traditional assumption of independence amongst input datasets does not hold. In this work we consider a suite of methods to conduct meta-analysis in the multiverse setting, accounting for inter-pipeline dependence among the results.

We propose several same data meta-analysis (SDMA) methods based on the traditional 'Stouffer' fixed-effects meta-analysis (Stouffer, 1949):
-SDMA Stouffer in which correlation across pipelines is taken into account,
-Consensus SDMA Stouffer and Consensus Average methods, where the combined inference is calibrated to be as similar to the input pipelines as possible, and
-General Least Squares (GLS) SDMA, where inter-pipeline correlation is used to find the statistically optimal combination of pipeline results.
The validity of these models were assessed in a set of simulations. Here, we focus on false positive control in two scenarios: 1/ independent pipelines with no significant results (null case), and 2/ correlated pipelines with no significant results (null correlated case). These meta-analysis models were also evaluated on a real world dataset from NARPS (Botvinik-Nezer et al., 2020), a multiverse analysis with 70 different statistic maps originating from the same data. Finally, given that these SDMA methods assume that the inter-pipeline correlation is the same across the brain, we measured heterogeneity with the Frobenius norm between the whole brain and a set of several brain regions derived from the AAL atlas.

Simulation results under the null setting of no effect and independent pipelines show that all the tested meta-analysis estimators are valid (Fig 1 top row). However, when data are correlated, as typically observed in a multiverse setting, only the SDMA estimators had valid inferences (Fig 1 lower row), while the conventional meta-analysis approach (Stouffer) dramatically overestimated the number of false-positives. On the real world dataset (Fig 2), the GLS method finds more significant voxels while the 3 other methods all have similar sensitivity. Finally, we found that the Frobenius norm was 0.01% across brain regions, supporting the validity of the consistent correlation assumption.

We compared several methods for combining multiverse results that account for the dependence among inputs. Our findings demonstrated the validity of the SDMA models under inter-pipeline dependence. As expected, the (traditional) Stouffer's method is liberal while the SDMA methods are all valid and present different levels of significance. These different levels illustrated different types of inference that practitioners can choose based on the assumptions of their analyses. As an illustration, in their work, Botvinik-Nezer and colleagues (2020) implemented the Consensus Average model to combine inferences, striving to align them closely with each of the input pipelines, under the assumption that all pipelines were equally valid and thus contributed equally relevant information. This assumption may not hold true in other multiverse settings.

Methods Development ¹

Databasing and Data Sharing ²

Workflows

Meta- Analysis

MRI

Open Data

Open-Source Code

Other - Multiverse, Reproducibility, Variability, Open Science