Benchmarking federated learning approaches against siloed and mega-analysis regimes
Poster No:
1146
Submission Type:
Abstract Submission
Authors:
Michelle Wang1, Nikhil Bhagwat1, Alyssa Dai1, Arman Jahanpour1, Brent McPherson1, Sebastian Urchs1, Jean-Baptiste Poline1
Institutions:
1McGill University, Montreal, Quebec
First Author:
Co-Author(s):
Introduction:
Although neuroimaging is seeing a growing number of datasets, the international adoption of strong data privacy frameworks (Marelli & Testa, 2018) has led to many of these datasets remaining in "silos". When data cannot readily be shared, it becomes imperative to develop distributed data processing tools and federated analysis methods to enable large-scale multi-site studies. Here we compare a simple federated analysis setup (i.e. sharing only fitted models) with two traditional experimental setups – siloed (no sharing) and mega- (sharing data) analyses (Fig 1). We evaluate the performance of machine learning (ML) models on three neuroimaging datasets of Parkinson's (PD) and Alzheimer's disease (AD) on two common prediction tasks in neurodegenerative diseases: 1) brain age and 2) cognitive decline. Brain age is a popular approach for subject-level biomarker development for early detection of diseases (Tan et al., 2024). Reliable prognostication of cognitive symptoms is crucial for early intervention and treatment of both AD and PD (Yousaf et al., 2019). We hypothesize that model performance improves as we go from siloed to federated to mega-analysis setups.
Methods:
We use T1w magnetic resonance images, demographic data, as well as Montreal Cognitive Assessment (MoCA) or Mini-Mental State Examination (MMSE) scores from three studies: the Parkinson's Progression Markers Initiative (PPMI) (N=1457) (Marek et al., 2018), the Alzheimer's Disease Neuroimaging Initiative (ADNI) (N=1200) (Jack et al., 2008), and the Quebec Parkinson Network (QPN) (N=102) (Gan-Or et al., 2020). We use Nipoppy to ensure standardized processing in a distributed setup with FreeSurfer (Fischl, 2012) to extract mean cortical thickness (CTh) values from 62 Desikan-Killiany-Tourville regions (Klein & Tourville, 2012) and subcortical volumes (SVo) from 17 regions. We use tools from the Neurobagel ecosystem to search across datasets and create cohorts for our experiment. For the brain age (BA) task, we use the baseline control cohort from each dataset to predict chronological age from CTh values. For the cognitive decline (Cog) task, we use longitudinal patient cohorts to predict cognitive decline (≥ 3-point loss in their MoCA or MMSE scores within 5 years) based on baseline SVo values. We use ridge (BA) and logistic (Cog) regression models and 5-fold cross-validation. In the siloed setup, we train models independently on each dataset. In the mega-analysis setup, we train on all datasets combined together. In the federated setup, we use the weighted average (by sample size) of the parameters of models that were trained independently on each dataset (Fig. 1).
Results:
Fig. 2 shows model performance for each analysis setup and test dataset. We find that, for the BA task, mean absolute errors (averaged across datasets) are lowest (better) for the siloed setup (4.64 ± 0.68), then the mega-analysis setup (4.83 ± 0.73), then the federated setup (5.19 ± 0.96). For the Cog task, balanced accuracy scores are similar between the siloed (.59 ± .10) and federated (.59 ± .09) setups, and highest (best) for the mega-analysis setup (.62 ± .10). We note that, although individually better, the siloed performance does not generalize across other datasets. Moreover, the mega-analysis BA models, contrary to our expectation, perform poorly, possibly due to skewed sample sizes and feature distributions. The federated Cog models performed on par with traditional setups and better than nulls showing promise as a viable analysis regime.
Conclusions:
With a large number of global datasets that cannot be openly shared, we take advantage of the Nipoppy framework and Neurobagel tools to generate standardized cohorts from distributed datasets and explore common ML tasks. We show that, for the datasets and use-cases investigated, the federated setup shows generalizable and comparable performance to traditional analytic approaches, opening the way to a change in analysis paradigm for many studies.
Disorders of the Nervous System:
Neurodegenerative/ Late Life (eg. Parkinson’s, Alzheimer’s) 2
Modeling and Analysis Methods:
Classification and Predictive Modeling 1
Neuroanatomy, Physiology, Metabolism and Neurotransmission:
Cortical Anatomy and Brain Mapping
Subcortical Structures
Keywords:
Aging
Degenerative Disease
Machine Learning
Movement Disorder
Open-Source Software
STRUCTURAL MRI
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
For human MRI, what field strength scanner do you use?
Which processing packages did you use for your study?
Provide references using APA citation style.
Gan-Or, Z., Rao, T., Leveille, E., Degroot, C., Chouinard, S., Cicchetti, F., Dagher, A., Das, S., Desautels, A., Drouin-Ouellet, J., Durcan, T., Gagnon, J.-F., Genge, A., Karamchandani, J., Lafontaine, A.-L., Sun, S. L. W., Langlois, M., Levesque, M., Melmed, C., … Fon, E. A. (2020). The Quebec Parkinson Network: A Researcher-Patient Matching Platform and Multimodal Biorepository. Journal of Parkinson’s Disease, 10(1), 301–313. https://doi.org/10.3233/JPD-191775
Jack, C. R., Bernstein, M. A., Fox, N. C., Thompson, P., Alexander, G., Harvey, D., Borowski, B., Britson, P. J., L Whitwell, J., Ward, C., Dale, A. M., Felmlee, J. P., Gunter, J. L., Hill, D. L. G., Killiany, R., Schuff, N., Fox-Bosetti, S., Lin, C., Studholme, C., … Weiner, M. W. (2008). The Alzheimer’s Disease Neuroimaging Initiative (ADNI): MRI methods. Journal of Magnetic Resonance Imaging: JMRI, 27(4), 685–691. https://doi.org/10.1002/jmri.21049
Klein, A., & Tourville, J. (2012). 101 Labeled Brain Images and a Consistent Human Cortical Labeling Protocol. Frontiers in Neuroscience, 6. https://doi.org/10.3389/fnins.2012.00171
Marek, K., Chowdhury, S., Siderowf, A., Lasch, S., Coffey, C. S., Caspell-Garcia, C., Simuni, T., Jennings, D., Tanner, C. M., Trojanowski, J. Q., Shaw, L. M., Seibyl, J., Schuff, N., Singleton, A., Kieburtz, K., Toga, A. W., Mollenhauer, B., Galasko, D., Chahine, L. M., … Initiative, the P. P. M. (2018). The Parkinson’s progression markers initiative (PPMI) – establishing a PD biomarker cohort. Annals of Clinical and Translational Neurology, 5(12), 1460–1477. https://doi.org/10.1002/acn3.644
Marelli, L., & Testa, G. (2018). Scrutinizing the EU General Data Protection Regulation. Science, 360(6388), 496–498. https://doi.org/10.1126/science.aar5419
Tan, T. W. K., Nguyen, K.-N., Zhang, C., Kong, R., Cheng, S. F., Ji, F., Chong, J. S. X., Yi Chong, E. J., Venketasubramanian, N., Orban, C., Chee, M. W. L., Chen, C., Zhou, J. H., & Yeo, B. T. T. (2024). Evaluation of Brain Age as a Specific Marker of Brain Health. bioRxiv: The Preprint Server for Biology, 2024.11.16.623903. https://doi.org/10.1101/2024.11.16.623903
Yousaf, T., Pagano, G., Niccolini, F., & Politis, M. (2019). Predicting cognitive decline with non-clinical markers in Parkinson’s disease (PRECODE-2). Journal of Neurology, 266(5), 1203–1210. https://doi.org/10.1007/s00415-019-09250-y