Unified Cross-modal Fusion with Explainable Graph Neural Network for Schizophrenia Classification
Poster No:
1143
Submission Type:
Abstract Submission
Authors:
Badhan Mazumder1, Lei Wu2, Vince Calhoun3, Dong Hye Ye1
Institutions:
1Georgia State University, ATLANTA, GA, 2TReNDS Center, Georgia State University, Atlanta, GA, 3GSU/GATech/Emory, Atlanta, GA
First Author:
Co-Author(s):
Introduction:
Clinical research identified abnormalities in both brain structural connectivity (SC) and functional connectivity (FC) in individuals with neuropsychiatric conditions like schizophrenia (SZ) (Mazumder, 2024). To leverage this, in this work, we developed an explainable multi-view graph neural network (GNN) framework coupled with a unified cross-modal attention (UCA) based fusion for classification of SZ individuals from healthy controls (HC). The modality-specific GNN backbones were designed to capture the intra-modal dependencies within each modality by leveraging connectomics data. Following this, our UCA fusion combined the embeddings from each modality, enabling efficient capture of both inter and intra-modal relationships for classification. Experiments conducted with two clinical datasets demonstrated improved performance, underscoring our proposed method's robustness.
Methods:
Dataset and preprocessing
We utilized subsets of the Function Biomedical Informatics Research Network (FBIRN) (Keator, 2016) [165; SZ:93, HC:72] and The Center for Biomedical Research Excellence (COBRE) (Calhoun, 2012) [152; SZ:64, HC:88] dataset.
The Neuromark pipeline (Du, 2020) was used for subject-wise FNC generation from resting-state fMRI. SC were derived from diffusion tensor imaging by computing diffusion tensors with FSL and performing whole-brain deterministic tractography with CAMINO. The NeuroMark atlas was spatially normalized to align with the local space, and the native fractional anisotropy map was warped into MNI space. Finally, streams passing through each pair of atlas ROIs were isolated and counted.
Proposed method
As shown in Figure 1, we treated each brain network from different modalities as distinct views, extracting unique features from each. After constructing modality-specific k-nearest neighbors (k=5) graphs, we defined graph edges utilizing the adjacency matrix and degree profiles as node features. The explainability-enhanced GNN (EE-GNN) block was applied to each view, coupling a base GNN (Kipf, 2016) with an explanation generator (Ying, 2019) that generate a modality-specific masks to highlight SZ-specific connections, fine-tuning the base GNN and improving feature representation. For multimodal fusion, we introduced UCA fusion, which adds a third branch to merge the representations of both modalities. The embeddings were concatenated, passed through a fully connected neural network (FCNN) layer for dimensionality reduction, and then fused to capture intra-modal connections. This enhances the model's ability to understand complex relationships and reduces sensitivity to noise, improving performance. Finally, the six feature vectors from the UCA fusion block were concatenated and classified using a FCNN layer.
We utilized subsets of the Function Biomedical Informatics Research Network (FBIRN) (Keator, 2016) [165; SZ:93, HC:72] and The Center for Biomedical Research Excellence (COBRE) (Calhoun, 2012) [152; SZ:64, HC:88] dataset.
The Neuromark pipeline (Du, 2020) was used for subject-wise FNC generation from resting-state fMRI. SC were derived from diffusion tensor imaging by computing diffusion tensors with FSL and performing whole-brain deterministic tractography with CAMINO. The NeuroMark atlas was spatially normalized to align with the local space, and the native fractional anisotropy map was warped into MNI space. Finally, streams passing through each pair of atlas ROIs were isolated and counted.
Proposed method
As shown in Figure 1, we treated each brain network from different modalities as distinct views, extracting unique features from each. After constructing modality-specific k-nearest neighbors (k=5) graphs, we defined graph edges utilizing the adjacency matrix and degree profiles as node features. The explainability-enhanced GNN (EE-GNN) block was applied to each view, coupling a base GNN (Kipf, 2016) with an explanation generator (Ying, 2019) that generate a modality-specific masks to highlight SZ-specific connections, fine-tuning the base GNN and improving feature representation. For multimodal fusion, we introduced UCA fusion, which adds a third branch to merge the representations of both modalities. The embeddings were concatenated, passed through a fully connected neural network (FCNN) layer for dimensionality reduction, and then fused to capture intra-modal connections. This enhances the model's ability to understand complex relationships and reduces sensitivity to noise, improving performance. Finally, the six feature vectors from the UCA fusion block were concatenated and classified using a FCNN layer.
Results:
We conducted 5-fold cross-validation with an 80:20 training-testing split, evaluating accuracy, precision, and F1-score. As shown in Figure 2a, the multimodal approach outperformed unimodal setups, with further improvements over two fusion methods when UCA-fusion was applied. Figure 2b highlights that the obtained top 100 notable group-specific connections for each modality showed denser brain network connections in HC compared to SZ individuals. SZ individuals displayed significant interactions in the Default Mode Network (DMN), Cognitive Control Network (CON), and Sensorimotor Network (SMN), supporting the clinical finding of altered brain structural-functional dynamics in SZ, consistent with prior studies (Mazumder, 2024; Fornito, 2012).
Conclusions:
We introduced an explainable GNN framework for neuropsychiatric disorder classification that learns enhanced, modality-specific graph-level representations via a shared explanation mask and integrates a unified cross-modal attention fusion mechanism to generate a joint representation that better captures inter-modal relationships. Our approach was validated through extensive experiments on clinical datasets, yielding notable performance improvements.
Disorders of the Nervous System:
Psychiatric (eg. Depression, Anxiety, Schizophrenia)
Modeling and Analysis Methods:
Classification and Predictive Modeling 1
Connectivity (eg. functional, effective, structural) 2
Keywords:
Data analysis
DISORDERS
FUNCTIONAL MRI
Machine Learning
Psychiatric Disorders
Schizophrenia
STRUCTURAL MRI
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Are you Internal Review Board (IRB) certified? Please note: Failure to have IRB, if applicable will lead to automatic rejection of abstract.
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
Which processing packages did you use for your study?
Provide references using APA citation style.
2. Du, Y., Fu, Z., Sui, J., Gao, S., Xing, Y., Lin, D., ... & Alzheimer's Disease Neuroimaging Initiative. (2020). NeuroMark: An automated and adaptive ICA based pipeline to identify reproducible fMRI markers of brain disorders. NeuroImage: Clinical, 28, 102375.
3. Fornito, A., Zalesky, A., Pantelis, C., & Bullmore, E. T. (2012). Schizophrenia, neuroimaging and connectomics. Neuroimage, 62(4), 2296-2314.
4. Kanyal, A., Mazumder, B., Calhoun, V. D., Preda, A., Turner, J., Ford, J., & Ye, D. H. (2024). Multi-modal deep learning from imaging genomic data for schizophrenia classification. Frontiers in Psychiatry, 15, 1384842.
5. Keator, D. B., van Erp, T. G., Turner, J. A., Glover, G. H., Mueller, B. A., Liu, T. T., ... & Potkin, S. G. (2016). The function biomedical informatics research network data repository. Neuroimage, 124, 1074-1079.
6. Kipf, T. N., & Welling, M. (2016). Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
7. Mazumder, B., Kanyal, A., Wu, L., D. Calhoun, V., & Hye Ye, D. (2024, October). Physics-Guided Multi-view Graph Neural Network for Schizophrenia Classification via Structural-Functional Coupling. In International Workshop on PRedictive Intelligence In MEdicine (pp. 61-73). Cham: Springer Nature Switzerland.
8. Mazumder, B., Tripathy, D. K., Yeates, K. O., Beauchamp, M. H., Craig, W., Doan, Q., ... & Ye, D. H. (2023, October). Multimodal Deep Learning for Pediatric Mild Traumatic Brain Injury Detection. In 2023 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI) (pp. 1-4). IEEE.
9. Ying, Z., Bourgeois, D., You, J., Zitnik, M., & Leskovec, J. (2019). Gnnexplainer: Generating explanations for graph neural networks. Advances in neural information processing systems, 32.