Quantitative Assessment of ML-Based Feature Attribution Methods for ASD Biomarker Discovery in fMRI
Poster No:
1741
Submission Type:
Abstract Submission
Authors:
Lucas Mahler1, Klaus Scheffler2,3, Gabriele Lohmann3
Institutions:
1Max-Planck-Institute for Biological Cybernetics, Tübingen, Baden-Württemberg, 2University of Tübingen, Tübingen, Germany, 3Max-Planck-Institute for Biological Cybernetics, Tübingen, Baden-Württemberg, Germany
First Author:
Co-Author(s):
Klaus Scheffler
University of Tübingen|Max-Planck-Institute for Biological Cybernetics
Tübingen, Germany|Tübingen, Baden-Württemberg, Germany
University of Tübingen|Max-Planck-Institute for Biological Cybernetics
Tübingen, Germany|Tübingen, Baden-Württemberg, Germany
Gabriele Lohmann
Max-Planck-Institute for Biological Cybernetics
Tübingen, Baden-Württemberg, Germany
Max-Planck-Institute for Biological Cybernetics
Tübingen, Baden-Württemberg, Germany
Introduction:
Mental disorders are a pressing global health concern, highlighting the critical need for reliable biomarkers, particularly in conditions such as autism spectrum disorder (ASD). Despite its prevalence, affecting 1 in 160 children worldwide, the lack of objective biomarkers hinders timely and accurate diagnosis, relying solely on behavioral observations.
This study explores the convergence of deep learning, explainable AI, and ASD diagnostics. We aim to evaluate the potential of extracting meaningful biomarkers from the learned representations of deep learning models applied to rs-fMRI data.
In line with recent advances in explainable AI, our focus is on quantitatively evaluating different methods for interpreting machine learning predictions. The goal is to identify the method that produces the highest quality explanations, thereby improving our understanding of the decision-making process in the model.
Beyond prediction, this study undertakes a quantitative exploration of brain imaging-based biomarkers. Using the selected explainable AI method, we aim to uncover biomarkers that contribute to a deeper understanding of ASD, with implications for refining diagnostic strategies and advancing our understanding of psychiatric disorders.
This study explores the convergence of deep learning, explainable AI, and ASD diagnostics. We aim to evaluate the potential of extracting meaningful biomarkers from the learned representations of deep learning models applied to rs-fMRI data.
In line with recent advances in explainable AI, our focus is on quantitatively evaluating different methods for interpreting machine learning predictions. The goal is to identify the method that produces the highest quality explanations, thereby improving our understanding of the decision-making process in the model.
Beyond prediction, this study undertakes a quantitative exploration of brain imaging-based biomarkers. Using the selected explainable AI method, we aim to uncover biomarkers that contribute to a deeper understanding of ASD, with implications for refining diagnostic strategies and advancing our understanding of psychiatric disorders.
Methods:
In our investigation, we use METAFormer (Mahler et al. 2023), a multi-atlas transformer model for ASD classification, trained on the ABIDE-I (DiMartino et al., 2013) dataset (406 ASD, 476 TC subjects) and preprocessed using the PCP-DPARSF pipeline. METAFormer achieves 83.7% accuracy with functional connectomes from the AAL, CC200, and DOS160 atlases.
To identify critical input ROIs, we use common feature attribution methods - Integrated Gradients, DeepLIFT, Feature Ablation, Gradient SHAP, and DeepLIFT-SHAP (Lundberg & Lee, 2017). Each method provides insight into feature contributions. For quantitative evaluation, we use two metrics (Yeh et al., 2019): 1) Infidelity, measured by the mean squared error between the influence of input perturbations on explanations and corresponding changes in the prediction function, assesses the accuracy of explanations under significant perturbations. 2) Sensitivity measures how attribution is affected by insignificant perturbations from a test point.
We randomly sample training, validation, and test sets for model training and attribution generation, calculating infidelity and sensitivity for each data point and attribution method. Since many attribution methods require a manual choice of baseline, we examine its effect on explanation quality by evaluating methods over a baseline range of [-1, 1].
To identify critical input ROIs, we use common feature attribution methods - Integrated Gradients, DeepLIFT, Feature Ablation, Gradient SHAP, and DeepLIFT-SHAP (Lundberg & Lee, 2017). Each method provides insight into feature contributions. For quantitative evaluation, we use two metrics (Yeh et al., 2019): 1) Infidelity, measured by the mean squared error between the influence of input perturbations on explanations and corresponding changes in the prediction function, assesses the accuracy of explanations under significant perturbations. 2) Sensitivity measures how attribution is affected by insignificant perturbations from a test point.
We randomly sample training, validation, and test sets for model training and attribution generation, calculating infidelity and sensitivity for each data point and attribution method. Since many attribution methods require a manual choice of baseline, we examine its effect on explanation quality by evaluating methods over a baseline range of [-1, 1].
Results:
From Figure 1 we can see that DeepLIFT provides the optimal trade-off between infidelity and sensitivity.
It can also be seen that a baseline choice of 0, representing the mean correlation for a given subject, does not yield optimal explanations for all methods except for Integrated Gradients. To minimize infidelity and sensitivity a baseline value of -1, representing strongest anticorrelation for a given subject, provides the most robuts and faithful explanations.
When looking at the most important features, as shown in Figure 2 and Table 1, the attributions provided by DeepLIFT are very consistent with existing autism literature. The influence of the default mode network, cerebellum, frontal and occipital regions in discriminating between ASD and controls is very prominent.
It can also be seen that a baseline choice of 0, representing the mean correlation for a given subject, does not yield optimal explanations for all methods except for Integrated Gradients. To minimize infidelity and sensitivity a baseline value of -1, representing strongest anticorrelation for a given subject, provides the most robuts and faithful explanations.
When looking at the most important features, as shown in Figure 2 and Table 1, the attributions provided by DeepLIFT are very consistent with existing autism literature. The influence of the default mode network, cerebellum, frontal and occipital regions in discriminating between ASD and controls is very prominent.
Conclusions:
We have shown that DeepLIFT is able to extract ROIs from the trained METAFormer model best in terms of explanation sensitivity and fidelity. The explanations provided are very consistent with existing literature. The demonstrated influence of the default mode network, cerebellum, frontal and occipital regions in discriminating between ASD and controls could be a significant step towards the development of model-based imaging biomarkers. This opens further avenues for quantitatively guided deep learning-based imaging biomarker discovery.
Disorders of the Nervous System:
Psychiatric (eg. Depression, Anxiety, Schizophrenia) 2
Modeling and Analysis Methods:
Activation (eg. BOLD task-fMRI)
Connectivity (eg. functional, effective, structural)
fMRI Connectivity and Network Modeling 1
Methods Development
Keywords:
Autism
Computational Neuroscience
Computing
Data analysis
Informatics
Machine Learning
MRI
Psychiatric
Psychiatric Disorders
Statistical Methods
1|2Indicates the priority used for review
Provide references using author date format
Lundberg, S., & Lee, S. (2017). A unified approach to interpreting model predictions. Neural Information Processing Systems, 30, 4768–4777. https://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf
Yeh, C. K., Hsieh, C. Y., Suggala, A., Inouye, D. I., & Ravikumar, P. K. (2019). On the (in) fidelity and sensitivity of explanations. Advances in Neural Information Processing Systems, 32.
Di Martino, A., Yan, C. G., Li, Q., Denio, E., Castellanos, F. X., Alaerts, K., ... & Milham, M. P. (2014). The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism. Molecular psychiatry, 19(6), 659-667.