Bi-cross-validation: a method to compare dynamic functional connectivity models in fMRI
Poster No:
1363
Submission Type:
Abstract Submission
Authors:
Yiming Wei1, Stephen Smith1, Stanislaw Adaszewski2, Stefan Fraessle2, Mark Woolrich1, Rezvan Farahibozorg1
Institutions:
1Wellcome Centre for Integrative Neuroimaging, University of Oxford, Oxford, Oxfordshire, 2Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd, Basel, Basel-Stadt
First Author:
Co-Author(s):
Stephen Smith
Wellcome Centre for Integrative Neuroimaging, University of Oxford
Oxford, Oxfordshire
Wellcome Centre for Integrative Neuroimaging, University of Oxford
Oxford, Oxfordshire
Mark Woolrich
Wellcome Centre for Integrative Neuroimaging, University of Oxford
Oxford, Oxfordshire
Wellcome Centre for Integrative Neuroimaging, University of Oxford
Oxford, Oxfordshire
Rezvan Farahibozorg
Wellcome Centre for Integrative Neuroimaging, University of Oxford
Oxford, Oxfordshire
Wellcome Centre for Integrative Neuroimaging, University of Oxford
Oxford, Oxfordshire
Introduction:
Functional connectivity (FC) quantifies interactions between brain regions. Dynamic functional connectivity (dFC), which captures temporal variations in these interactions during resting-state fMRI, has gained attention (Calhoun et al., 2014). However, evaluating dFC models against each other and selecting optimal configurations remains challenging.
Methods:
We compared three dFC models: the Hidden Markov Model (HMM; Vidaurre et al., 2017), Dynamic Network Modes (DyNeMo; Gohil et al., 2022) and Sliding Window Correlation with K-means clustering (SWC; Allen et al., 2014). These models describe fMRI data using a small number of recurring patterns, collectively referred to as "states" (Figure 1A). These states, also called "modes" in DyNeMo and "centroids" in SWC, have associated time courses and observation model parameters, including state-specific covariances capturing functional connectivity. SWC reduces to static functional connectivity (static FC) when the window length spans the entire session.
To evaluate model performance, we adapted bi-cross-validation (Fu & Perry, 2020), which reshuffles and splits data across "subjects" and "regions" (Figure 1B). This process generates a data quad [X_train,Y_train; X_test, Y_test], which undergoes a four-step validation procedure (Figure 1B). For N_states=1, bi-cross-validation simplifies to inferring a group-level covariance on Y_train and computing the log-likelihood on Y_test. The "reshuffle, split and validation" procedure was repeated 100 times. One cannot use simple standard cross-validation for such evaluations, because aspects of the test data are needed for completing the inference (on the test data), resulting in overfitting.
We validated bi-cross-validation using simulated and real data. Simulation datasets included 500 subjects, each with 50 regions, 6 states, and 1200 time points. Ground-truth state covariances were randomly generated using osl-dynamics (Gohil et al., 2023). Real data came from the Human Connectome Project (HCP) S1200 release (N = 1003, four 15-min scans per subject) (Glasser et al., 2013; Van Essen et al., 2013). Preprocessing involved ICA-FIX denoising, surface alignment using MSMAll (Robinson et al., 2014) and group-level ICA with 50 components (Beckmann & Smith, 2004). This process yielded spatial maps and the associated time series, which were z-scored for each 15-min run. Models were fit using osl-dynamics for HMM and DyNeMo, and a window length of 100 time points for SWC.
To evaluate model performance, we adapted bi-cross-validation (Fu & Perry, 2020), which reshuffles and splits data across "subjects" and "regions" (Figure 1B). This process generates a data quad [X_train,Y_train; X_test, Y_test], which undergoes a four-step validation procedure (Figure 1B). For N_states=1, bi-cross-validation simplifies to inferring a group-level covariance on Y_train and computing the log-likelihood on Y_test. The "reshuffle, split and validation" procedure was repeated 100 times. One cannot use simple standard cross-validation for such evaluations, because aspects of the test data are needed for completing the inference (on the test data), resulting in overfitting.
We validated bi-cross-validation using simulated and real data. Simulation datasets included 500 subjects, each with 50 regions, 6 states, and 1200 time points. Ground-truth state covariances were randomly generated using osl-dynamics (Gohil et al., 2023). Real data came from the Human Connectome Project (HCP) S1200 release (N = 1003, four 15-min scans per subject) (Glasser et al., 2013; Van Essen et al., 2013). Preprocessing involved ICA-FIX denoising, surface alignment using MSMAll (Robinson et al., 2014) and group-level ICA with 50 components (Beckmann & Smith, 2004). This process yielded spatial maps and the associated time series, which were z-scored for each 15-min run. Models were fit using osl-dynamics for HMM and DyNeMo, and a window length of 100 time points for SWC.
Results:
Synthetic data were generated separately using generative models of HMM and DyNeMo, and all three models were applied to both simulations. On HMM-based simulations (Figure 2A, first row), bi-cross-validation identified the true number of states and penalized overfitting for N_states>6. DyNeMo performed slightly worse due to model complexity, while SWC yielded lower log-likelihood values overall despite improvements with additional states. On DyNeMo-based simulations (Figure 2A, second row), bi-cross-validation recommended the ground truth model. DyNeMo identified the correct number of states, while HMM overestimated the number of states to approximate DyNeMo's performance and SWC consistently underperformed.
On HCP data (Figure 2B), bi-cross-validation showed initial performance improvements with increasing states, followed by plateaus or declines. It effectively penalized overfitting in HMM, DyNeMo and static FC, but struggled to determine the optimal state count for SWC, in line with simulation results. Comparing log-likelihood values across models, DyNeMo performed best, followed by SWC, HMM and static FC.
On HCP data (Figure 2B), bi-cross-validation showed initial performance improvements with increasing states, followed by plateaus or declines. It effectively penalized overfitting in HMM, DyNeMo and static FC, but struggled to determine the optimal state count for SWC, in line with simulation results. Comparing log-likelihood values across models, DyNeMo performed best, followed by SWC, HMM and static FC.
Conclusions:
Bi-cross-validation effectively evaluates dFC models by balancing model fit and complexity. On simulated data, it accurately identified ground-truth models and hyperparameters. For HCP resting-state fMRI, it determined the optimal number of states for each model. All dFC models outperformed the static FC baseline, with DyNeMo providing the most accurate data description.
Modeling and Analysis Methods:
Bayesian Modeling
Connectivity (eg. functional, effective, structural)
fMRI Connectivity and Network Modeling 1
Methods Development
Task-Independent and Resting-State Analysis 2
Keywords:
Computational Neuroscience
Data analysis
FUNCTIONAL MRI
Machine Learning
Modeling
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
For human MRI, what field strength scanner do you use?
Which processing packages did you use for your study?
Provide references using APA citation style.
Beckmann, C. F. (2004). Probabilistic independent component analysis for functional magnetic resonance imaging. IEEE Transactions on Medical Imaging, 23(2), 137–152. https://doi.org/10.1109/TMI.2003.822821
Calhoun, V. D. (2014). The Chronnectome: Time-Varying Connectivity Networks as the Next Frontier in fMRI Data Discovery. Neuron, 84(2), 262–274. https://doi.org/10.1016/j.neuron.2014.10.015
Fu, W. (2020). Estimating the Number of Clusters Using Cross-Validation. Journal of Computational and Graphical Statistics, 29(1), 162–173. https://doi.org/10.1080/10618600.2019.1647846
Glasser, M. F. (2013). The minimal preprocessing pipelines for the Human Connectome Project. NeuroImage, 80, 105–124. https://doi.org/10.1016/j.neuroimage.2013.04.127
Gohil, C. (2023). osl-dynamics: A toolbox for modelling fast dynamic brain activity. eLife, 12. https://doi.org/10.7554/eLife.91949.2
Gohil, C. (2022). Mixtures of large-scale dynamic functional brain network modes. NeuroImage, 263, 119595. https://doi.org/10.1016/j.neuroimage.2022.119595
Robinson, E. C. (2014). MSM: A new flexible framework for Multimodal Surface Matching. NeuroImage, 100, 414–426. https://doi.org/10.1016/j.neuroimage.2014.05.069
Van Essen, D. C. (2013). The WU-Minn Human Connectome Project: An overview. NeuroImage, 80, 62–79. https://doi.org/10.1016/j.neuroimage.2013.05.041
Vidaurre, D. (2017). Brain network dynamics are hierarchically organized in time. Proceedings of the National Academy of Sciences of the United States of America, 114(48), 12827–12832. https://doi.org/10.1073/pnas.1705120114