Amygdala Volumes Estimated by FSL and FreeSurfer Have Poor Consistency
Poster No:
1637
Submission Type:
Abstract Submission
Authors:
Patrick Sadil1, Martin Lindquist1
Institutions:
1Johns Hopkins University, Baltimore, MD
First Author:
Co-Author:
Introduction:
Regional volumes of subcortex have been proposed as biomarkers for several psychopathologies. For example, the volume of the amygdala is predictive of Alzheimer's, depression symptom severity, bipolar disorder, migraine frequency, and several other pathologies (Daftary et al., 2019; Khatri & Kwon, 2022; Liu et al., 2017; Pfeifer et al., 2008; Rogers et al., 2009; Ruocco et al., 2012; Szeszko et al., 2004). As a biomarker, subcortical volumes are advantageous as they are interpretable, explainable, and readily available.
To estimate regional subcortical volumes, two automated techniques are popular: FSL's FIRST and FreeSurfer's ASEG (Fischl, 2012; Patenaude et al., 2011). Given that both tools are often reasonable choices for a study, and the literature contains many reports based on one but not the other, we sought to understand how the tools compare with each other. Our primary concern is whether correlations between volumes and health-related outcomes depend on the method used for automated segmentation.
To estimate regional subcortical volumes, two automated techniques are popular: FSL's FIRST and FreeSurfer's ASEG (Fischl, 2012; Patenaude et al., 2011). Given that both tools are often reasonable choices for a study, and the literature contains many reports based on one but not the other, we sought to understand how the tools compare with each other. Our primary concern is whether correlations between volumes and health-related outcomes depend on the method used for automated segmentation.
Methods:
The single-measurement intraclass correlation coefficient (ICC) was used to measure consistency and agreement (i.e., ICC(C,1) and ICC(A,1)). To assess the impact of ICC, we explored two kinds of issues. First, lower consistency could make it more likely that one but not both methods lead to significant correlations. Second, lower consistency could make it more likely that the two methods produce significant correlations that go in opposite directions.
To investigate the frequency of these two issues, we first simulated experiments with artificial data. In each simulation, datasets with two noisy estimates of volume and a third, outcome, variable were generated such that the two volume estimates had a pre-specified ICC(C,1) with each other (in expectation), and the true volume had a given product-moment correlation with the outcome variable. The estimated volumes were then tested for a correlation with the outcome, and the process was repeated for several intraclass correlations and sample sizes. In all simulations, the true correlation was set to a value that is either typical for neuroimaging research (0.1), small but non-zero (0.01), or large (0.2).
To assess how often these two issues could occur in practice, we repeated the above analyses, subsampling participants from the UKB (Alfaro-Almagro et al., 2018).
To investigate the frequency of these two issues, we first simulated experiments with artificial data. In each simulation, datasets with two noisy estimates of volume and a third, outcome, variable were generated such that the two volume estimates had a pre-specified ICC(C,1) with each other (in expectation), and the true volume had a given product-moment correlation with the outcome variable. The estimated volumes were then tested for a correlation with the outcome, and the process was repeated for several intraclass correlations and sample sizes. In all simulations, the true correlation was set to a value that is either typical for neuroimaging research (0.1), small but non-zero (0.01), or large (0.2).
To assess how often these two issues could occur in practice, we repeated the above analyses, subsampling participants from the UKB (Alfaro-Almagro et al., 2018).
Results:
Across all structures, estimated agreement was lower than estimated consistency (Figure 1). Consistency varies from "good" to "excellent" across most regions (Figure 1), but for the amygdala ICC(C,1) is poor.
With lower consistency, the estimated product-moment correlations were more often on opposite sides of a significance threshold (Figure 2a, left column). Lower consistency also coincided with a higher proportion of experiments in which the two methods correlate in opposite directions (Figure 2a, right column).
Considering differing significance levels, the rates across experiments using the UKB resembled the rates from experiments with artificial data (compare left columns of Figure 2 a and b). Considering significant correlations with differing signs, the rates across experiments with the UKB bracketed the rates with artificial data (compare right columns of Figure 2 a and b).
With lower consistency, the estimated product-moment correlations were more often on opposite sides of a significance threshold (Figure 2a, left column). Lower consistency also coincided with a higher proportion of experiments in which the two methods correlate in opposite directions (Figure 2a, right column).
Considering differing significance levels, the rates across experiments using the UKB resembled the rates from experiments with artificial data (compare left columns of Figure 2 a and b). Considering significant correlations with differing signs, the rates across experiments with the UKB bracketed the rates with artificial data (compare right columns of Figure 2 a and b).
Conclusions:
We examined the consistency of subcortical volumes within the UK Biobank, observing that two common methods of estimating the volume of the amygdala have low enough consistency that results may depend on the choice of method. Based on this inconsistency, we make three suggestions:
1) When testing new biomarkers, report relationships with multiple automated methods (e.g., both FSL and FreeSurfer).
2) When reviewing or conducting meta-analyses of relationships with amygdala volume, consider the method that was used to estimate volume.
3) When replicating or extending research on a relationship that involves the volume of the amygdala, use the method reported in the original publications.
1) When testing new biomarkers, report relationships with multiple automated methods (e.g., both FSL and FreeSurfer).
2) When reviewing or conducting meta-analyses of relationships with amygdala volume, consider the method that was used to estimate volume.
3) When replicating or extending research on a relationship that involves the volume of the amygdala, use the method reported in the original publications.
Modeling and Analysis Methods:
Classification and Predictive Modeling 2
Segmentation and Parcellation 1
Keywords:
Data analysis
Machine Learning
Morphometrics
Segmentation
Statistical Methods
STRUCTURAL MRI
Sub-Cortical
Univariate
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Are you Internal Review Board (IRB) certified? Please note: Failure to have IRB, if applicable will lead to automatic rejection of abstract.
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
For human MRI, what field strength scanner do you use?
Which processing packages did you use for your study?
Provide references using APA citation style.
Daftary, S., Van Enkevort, E., Kulikova, A., Legacy, M., & Brown, E. S. (2019). Relationship between depressive symptom severity and amygdala volume in a large community-based sample. Psychiatry Research: Neuroimaging, 283, 77–82.
Fischl, B. (2012). FreeSurfer. NeuroImage, 62(2), 774–781.
Khatri, U., & Kwon, G.-R. (2022). Alzheimer’s Disease Diagnosis and Biomarker Analysis Using Resting-State Functional MRI Functional Brain Network With Multi-Measures Features and Hippocampal Subfield and Amygdala Volume of Structural MRI. Frontiers in Aging Neuroscience, 14, 818871.
Liu, H.-Y., Chou, K.-H., Lee, P.-L., Fuh, J.-L., Niddam, D. M., Lai, K.-L., Hsiao, F.-J., Lin, Y.-Y., Chen, W.-T., Wang, S.-J., & Lin, C.-P. (2017). Hippocampus and amygdala volume in relation to migraine frequency and prognosis. Cephalalgia, 37(14), 1329–1336.
Patenaude, B., Smith, S. M., Kennedy, D. N., & Jenkinson, M. (2011). A Bayesian model of shape and appearance for subcortical brain segmentation. NeuroImage, 56(3), 907–922.
Pfeifer, J. C., Welge, J., Strakowski, S. M., Adler, Calebm., & Delbello, M. P. (2008). Meta-Analysis of Amygdala Volumes in Children and Adolescents With Bipolar Disorder. Journal of the American Academy of Child & Adolescent Psychiatry, 47(11), 1289–1298.
Rogers, M. A., Yamasue, H., Abe, O., Yamada, H., Ohtani, T., Iwanami, A., Aoki, S., Kato, N., & Kasai, K. (2009). Smaller amygdala volume and reduced anterior cingulate gray matter density associated with history of post-traumatic stress disorder. Psychiatry Research: Neuroimaging, 174(3), 210–216.
Ruocco, A. C., Amirthavasagam, S., & Zakzanis, K. K. (2012). Amygdala and hippocampal volume reductions as candidate endophenotypes for borderline personality disorder: A meta-analysis of magnetic resonance imaging studies. Psychiatry Research: Neuroimaging, 201(3), 245–252.
Szeszko, P. R., MacMillan, S., McMeniman, M., Lorch, E., Madden, R., Ivey, J., Banerjee, S. P., Moore, G. J., & Rosenberg, D. R. (2004). Amygdala Volume Reductions in Pediatric Patients with Obsessive–Compulsive Disorder Treated with Paroxetine: Preliminary Findings. Neuropsychopharmacology, 29(4), 826–832.