Low diagnostic and prognostic utility of Brain-Age prediction across publicly available packages
Poster No:
1091
Submission Type:
Abstract Submission
Authors:
Ruben Dörfel1,2,3, Brice Ozenne2,4, Melanie Ganz2,3, Jonas Svensson1, Pontus Plavén-Sigray1,2
Institutions:
1Center for Psychiatry Research, Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden, 2Neurobiology Research Unit, Copenhagen University Hospital Rigshospitalet, Copenhagen, Denmark, 3Department of Computer Science, University of Copenhagen, Copenhagen, Denmark, 4Section of Biostatistics, Department of Public Health, University of Copenhagen, Copenhagen, Denmark
First Author:
Ruben Dörfel, MSc
Center for Psychiatry Research, Department of Clinical Neuroscience, Karolinska Institutet|Neurobiology Research Unit, Copenhagen University Hospital Rigshospitalet|Department of Computer Science, University of Copenhagen
Stockholm, Sweden|Copenhagen, Denmark|Copenhagen, Denmark
Center for Psychiatry Research, Department of Clinical Neuroscience, Karolinska Institutet|Neurobiology Research Unit, Copenhagen University Hospital Rigshospitalet|Department of Computer Science, University of Copenhagen
Stockholm, Sweden|Copenhagen, Denmark|Copenhagen, Denmark
Co-Author(s):
Brice Ozenne
Neurobiology Research Unit, Copenhagen University Hospital Rigshospitalet|Section of Biostatistics, Department of Public Health, University of Copenhagen
Copenhagen, Denmark|Copenhagen, Denmark
Neurobiology Research Unit, Copenhagen University Hospital Rigshospitalet|Section of Biostatistics, Department of Public Health, University of Copenhagen
Copenhagen, Denmark|Copenhagen, Denmark
Melanie Ganz
Neurobiology Research Unit, Copenhagen University Hospital Rigshospitalet|Department of Computer Science, University of Copenhagen
Copenhagen, Denmark|Copenhagen, Denmark
Neurobiology Research Unit, Copenhagen University Hospital Rigshospitalet|Department of Computer Science, University of Copenhagen
Copenhagen, Denmark|Copenhagen, Denmark
Jonas Svensson
Center for Psychiatry Research, Department of Clinical Neuroscience, Karolinska Institutet
Stockholm, Sweden
Center for Psychiatry Research, Department of Clinical Neuroscience, Karolinska Institutet
Stockholm, Sweden
Pontus Plavén-Sigray
Center for Psychiatry Research, Department of Clinical Neuroscience, Karolinska Institutet|Neurobiology Research Unit, Copenhagen University Hospital Rigshospitalet
Stockholm, Sweden|Copenhagen, Denmark
Center for Psychiatry Research, Department of Clinical Neuroscience, Karolinska Institutet|Neurobiology Research Unit, Copenhagen University Hospital Rigshospitalet
Stockholm, Sweden|Copenhagen, Denmark
Introduction:
Predicting brain age from structural magnetic resonance images is increasingly used as a biomarker of biological aging and brain health [1], [2]. Many groups have published packages and made them publicly available. Often, the difference between predicted age and chronological age, the predicted age deviation (PAD), is used as the outcome of interest. A PAD near zero indicates a brain typical for its age, while a positive PAD suggests an older, potentially unhealthy brain.
We have previously compared six publicly available brain-age software packages, and demonstrated that a subset are highly accurate and reliable in predicting chronological age [3]. However, for brain age to serve as a clinically useful biomarker of brain health and aging, it must demonstrate more than high reliability. The PAD should be diagnostic (e.g., reflect the current state of health) and be prognostic (e.g., be associated with the future onset of disease, changes in cognition, and alterations in brain biology, such as the rate of gray matter atrophy).
We have previously compared six publicly available brain-age software packages, and demonstrated that a subset are highly accurate and reliable in predicting chronological age [3]. However, for brain age to serve as a clinically useful biomarker of brain health and aging, it must demonstrate more than high reliability. The PAD should be diagnostic (e.g., reflect the current state of health) and be prognostic (e.g., be associated with the future onset of disease, changes in cognition, and alterations in brain biology, such as the rate of gray matter atrophy).
Methods:
In this study, we extended our previous assessment of six publicly available brain age prediction packages and evaluated their convergent validity and clinical utility, using data from the Alzheimer's Disease Neuroimaging Initiative (ADNI). Specifically, the PAD outcome from the following packages were tested: brainageR [4], DeepBrainNet [5], brainage [2], ENIGMA [6], pyment [7], and mccqrnn [8]. In addition, we computed the cerebral gray matter volume and used it as a simple reference to assess the relative performance of the packages.
To assess convergent validity, we examined the agreement between the predictions of each package by correlating their PAD outcomes. For diagnostic utility, we evaluated whether the magnitude of the PAD was associated with being cognitively normal or having a diagnosis of mild cognitive impairment (MCI) or Alzheimer's disease (AD), and whether a higher PAD was linked to lower cognitive scores. To assess prognostic utility, we investigated whether a higher PAD at baseline in cognitively normal participants predicted a change in diagnosis within four years, as well as its ability to predict cognitive decline and gray matter atrophy over the same period. Cognitive performance was assessed using the ADNI_MEM composite score [9].
To assess convergent validity, we examined the agreement between the predictions of each package by correlating their PAD outcomes. For diagnostic utility, we evaluated whether the magnitude of the PAD was associated with being cognitively normal or having a diagnosis of mild cognitive impairment (MCI) or Alzheimer's disease (AD), and whether a higher PAD was linked to lower cognitive scores. To assess prognostic utility, we investigated whether a higher PAD at baseline in cognitively normal participants predicted a change in diagnosis within four years, as well as its ability to predict cognitive decline and gray matter atrophy over the same period. Cognitive performance was assessed using the ADNI_MEM composite score [9].
Results:
PAD outcomes across packages showed limited agreement, with the lowest correlation between brainageR and ENIGMA (r=0.41, 95%CI=[0.37,0.44]), and the highest correlation between brainage and ENIGMA (r=0.71, 95%CI=[0.68,0.73]) (Figure 1A). While PAD at baseline significantly differed between clinical groups for all packages, PAD outcomes for cognitively normal participants were not centered around zero and showed considerable variation across packages (Figure 1B). The highest effect sizes (Cohen's d) for each comparison ranged from 0.95 (brainage, gray matter) for cognitive normal/AD, to 0.62 (gray matter) for MCI/AD, and 0.49 (brainage) for cognitive normal/MCI. Notably, simple gray matter volume estimates performed similar to more complex brain age predictions differentiating between clinical groups. Among cognitive normal individuals, PAD at baseline showed no significant correlation with cognitive performance, was not significantly associated with disease onset within four years and was not associated with gray matter atrophy or cognitive decline during the same period for any of the packages (Figure 2).
Conclusions:
The substantial bias in PAD outcomes in cognitive normal subjects and the variability in cross-sectional brain age predictions, along with the lack of association between brain age and future changes in cognition or gray matter atrophy, challenges the utility of existing models for brain age predictions as biomarkers for biological aging in the central nervous system.
Disorders of the Nervous System:
Neurodegenerative/ Late Life (eg. Parkinson’s, Alzheimer’s) 2
Lifespan Development:
Aging
Modeling and Analysis Methods:
Classification and Predictive Modeling 1
Keywords:
Aging
Machine Learning
MRI
Other - brain age estimation
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
For human MRI, what field strength scanner do you use?
Which processing packages did you use for your study?
Provide references using APA citation style.
[2] T. Kaufmann et al., “Common brain disorders are associated with heritable patterns of apparent aging of the brain,” Nat. Neurosci., vol. 22, no. 10, pp. 1617–1623, Sep. 2019, doi: 10.1038/s41593-019-0471-7.
[3] R. P. Dörfel et al., “Prediction of brain age using structural magnetic resonance imaging: A comparison of accuracy and test–retest reliability of publicly available software packages,” Hum. Brain Mapp., vol. 44, no. 17, pp. 6139–6148, 2023, doi: 10.1002/hbm.26502.
[4] J. H. Cole et al., “Predicting brain age with deep learning from raw imaging data results in a reliable and heritable biomarker,” NeuroImage, vol. 163, no. March, pp. 115–124, 2017, doi: 10.1016/j.neuroimage.2017.07.059.
[5] V. M. Bashyam et al., “MRI signatures of brain age and disease over the lifespan based on a deep brain network and 14 468 individuals worldwide,” Brain, vol. 143, no. 7, pp. 2312–2324, Jul. 2020, doi: 10.1093/brain/awaa160.
[6] L. K. M. Han et al., “Brain aging in major depressive disorder: results from the ENIGMA major depressive disorder working group,” Mol. Psychiatry, vol. 26, no. 9, pp. 5124–5139, May 2021, doi: 10.1038/s41380-020-0754-0.
[7] E. H. Leonardsen et al., “Deep neural networks learn general and clinically relevant representations of the ageing brain,” NeuroImage, vol. 256, no. December 2021, p. 119210, 2022, doi: 10.1016/j.neuroimage.2022.119210.
[8] T. Hahn et al., “An uncertainty-aware, shareable, and transparent neural network architecture for brain-age modeling,” Sci. Adv., vol. 8, no. 1, p. eabg9471, Jan. 2022, doi: 10.1126/sciadv.abg9471.
[9] P. K. Crane et al., “Development and assessment of a composite score for memory in the Alzheimer’s Disease Neuroimaging Initiative (ADNI),” Brain Imaging Behav., vol. 6, no. 4, pp. 502–516, Dec. 2012, doi: 10.1007/s11682-012-9186-z.