Enhancing Robustness in Network Neuroscience through Multiverse Analysis - The Comet Toolbox
Poster No:
1442
Submission Type:
Abstract Submission
Authors:
Micha Burkhardt1, Carsten Giessing1
Institutions:
1Carl von Ossietzky Universität Oldenburg, Oldenburg, Germany
First Author:
Co-Author:
Introduction:
Dynamic functional connectivity (dFC) offers a window into the temporal fluctuations of brain networks, promising novel insights into the dynamic architecture of the brain (Preti et al., 2016). However, reproducibility and robustness remain significant challenges, driven by the complexity of neuroimaging data and the multitude of methodological choices involved in dFC analysis. Various methods for estimating dFC from fMRI data and subsequent graph-theoretical analyses have emerged, but the absence of ground truths often forces researchers to make arbitrary (yet defensible) decisions. These choices can result in considerable variability, raising concerns about the generalizability of findings.
To address these challenges, the concept of multiverse analysis has been proposed as a systematic approach to improve robustness and transparency in data analysis (Steegen et al., 2016). Rather than focusing on a single analysis pipeline, multiverse analysis evaluates all defensible combinations of analytical decisions, revealing how these choices influence outcomes. By aggregating results across pipelines, multiverse analysis provides a clear overview of which effects are robust and which are sensitive to methodological choices, addressing key concerns in reproducibility.
We present Comet, an open-source Python toolbox that unifies dFC estimation, graph-theoretical methods, and multiverse analysis workflows. Comet facilitates the systematic exploration of methodological choices, providing researchers with a meta-scientific approach to assess their impact on research outcomes. By introducing a robust and transparent framework, the Comet Toolbox contributes to enhancing robustness and reproducibility in network neuroscience.
To address these challenges, the concept of multiverse analysis has been proposed as a systematic approach to improve robustness and transparency in data analysis (Steegen et al., 2016). Rather than focusing on a single analysis pipeline, multiverse analysis evaluates all defensible combinations of analytical decisions, revealing how these choices influence outcomes. By aggregating results across pipelines, multiverse analysis provides a clear overview of which effects are robust and which are sensitive to methodological choices, addressing key concerns in reproducibility.
We present Comet, an open-source Python toolbox that unifies dFC estimation, graph-theoretical methods, and multiverse analysis workflows. Comet facilitates the systematic exploration of methodological choices, providing researchers with a meta-scientific approach to assess their impact on research outcomes. By introducing a robust and transparent framework, the Comet Toolbox contributes to enhancing robustness and reproducibility in network neuroscience.
Methods:
Currently, the toolbox includes more than 20 static, dynamic, state-based, and edge-centric connectivity measures. These methods were identified through a systematic literature review to ensure broad coverage of existing approaches. Additionally, optimized graph-theoretical algorithms and full integration of the Brain Connectivity Toolbox (Rubinov and Sporns, 2010) are included, enabling comprehensive network analyses.
The connectivity and graph-theoretical measures are integrated into a flexible multiverse analysis workflow. This workflow allows users to specify decision points within their analysis pipelines and systematically evaluate all valid combinations of decisions. Comet's multiverse analysis features also support complex pipeline structures, including the reordering of analytical steps and exclusion of invalid combinations.
Designed with accessibility and maintainability in mind, the Comet Toolbox features both an intuitive graphical user interface (GUI) and comprehensive scripting capabilities. This ensures suitability for users with diverse technical backgrounds. All functionalities can also be used independently outside of the multiverse analysis workflow, providing flexibility for standalone applications. Extensive documentation and demo scripts are available to facilitate integration into existing workflows (https://comet-toolbox.readthedocs.io/).
The connectivity and graph-theoretical measures are integrated into a flexible multiverse analysis workflow. This workflow allows users to specify decision points within their analysis pipelines and systematically evaluate all valid combinations of decisions. Comet's multiverse analysis features also support complex pipeline structures, including the reordering of analytical steps and exclusion of invalid combinations.
Designed with accessibility and maintainability in mind, the Comet Toolbox features both an intuitive graphical user interface (GUI) and comprehensive scripting capabilities. This ensures suitability for users with diverse technical backgrounds. All functionalities can also be used independently outside of the multiverse analysis workflow, providing flexibility for standalone applications. Extensive documentation and demo scripts are available to facilitate integration into existing workflows (https://comet-toolbox.readthedocs.io/).
Results:
Figure 1 highlights the graphical user interface and the performance improvements for a computationally expensive graph measure (nodal efficiency). Figure 2 presents a specification curve as an example output for a multiverse analysis. It illustrates the variability in results across different analysis pipelines for predicting connectivity states from synthetic fMRI data.
Conclusions:
Integrating a wide array of dFC methods, graph-based tools, and multiverse analysis workflows, the Comet Toolbox empowers researchers to systematically explore the impact of analytical decisions. Future developments will further expand its functionality, ensuring Comet remains a dynamic and evolving tool for advancing robust and reproducible analyses in network neuroscience. The toolbox is openly available at https://github.com/mibur1/comet.
Modeling and Analysis Methods:
Connectivity (eg. functional, effective, structural) 2
fMRI Connectivity and Network Modeling 1
Neuroinformatics and Data Sharing:
Workflows
Keywords:
Other - functional MRI; fMRI; functional connectivity; dynamic functional connectivity; graph analysis; network neuroscience; multiverse analysis; robustness; reproducibility
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
Provide references using APA citation style.
Rubinov, M. (2009). Complex network measures of brain connectivity: Uses and interpretations. NeuroImage, 52(3), 1059–1069.
Steegen, S. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702–712.