TractGPT: Using transformer models to predict along-tract profiles
Poster No:
1311
Submission Type:
Late-Breaking Abstract Submission
Authors:
Remika Mito1, Daniel Wells1, David Wood2, James Cole3
Institutions:
1University of Melbourne, Melbourne, Victoria, 2Kings College London, London, London, 3University College London, London, London
First Author:
Co-Author(s):
Late Breaking Reviewer(s):
Rosanna Olsen
Rotman Research Institute, Baycrest Academy for Research and Education
Toronto, Ontario
Rotman Research Institute, Baycrest Academy for Research and Education
Toronto, Ontario
Introduction:
Diffusion MRI (dMRI) is highly valuable for investigating changes to the brain's white matter fibre tracts. Variability in dMRI measures can be investigated along the length of named white matter tracts, and these along-tract profiling approaches can be valuable for detecting subtle abnormalities in both individual patients and patient cohorts (Chamberland et al. 2021). Given the large number of brain white matter tracts and tract segments, the dimensionality of the derived data can be challenging, and the non-independence and sequential nature of dMRI measures along these tracts are often disregarded during analysis. Here, we present a novel approach for learning normative tract profiles, using a tailored deep learning transformer model. Transformers are particularly well-suited to modelling sequential data (like these along-tract measures) and have been widely adopted in large language models.
Methods:
dMRI data from the Human Connectome Project (HCP) Development (n=495) and Aging (n=685) studies were included. In brief, multi-shell dMRI data (b=1500/3000s/mm2) were preprocessed using the HCP minimal preprocesing pipeline (Glasser et al., 2013), modelled using the diffusion tensor (computed using the b1500 shell only), and a measure of fractional anisotropy (FA) computed (Veraart et al., 2013). TractSeg (Wasserthal et al., 2018) was performed to extract key white matter fibre tracts, and tractometry performed to extract mean FA along the length of 50 tracts for each individual (Fig. 1).
Following this, the along-tract FA data were used to train a tailored transformer model (which we call 'TractGPT'), based on the bidirectional encoder representation from transformers (BERT) model (Devlin et al, 2019). The model architecture included an encoder-only structure, embeddings for tract identity, segment location, and participant age and sex. Training was performed on a training set (75% of full dataset, 10% validation set) using a masking approach (akin to masked language models), whereby 15% of the input data were masked at random, and the model had to predict masked values. Model performance was assessed on an unseen test set (15%; n=178).
Following this, the along-tract FA data were used to train a tailored transformer model (which we call 'TractGPT'), based on the bidirectional encoder representation from transformers (BERT) model (Devlin et al, 2019). The model architecture included an encoder-only structure, embeddings for tract identity, segment location, and participant age and sex. Training was performed on a training set (75% of full dataset, 10% validation set) using a masking approach (akin to masked language models), whereby 15% of the input data were masked at random, and the model had to predict masked values. Model performance was assessed on an unseen test set (15%; n=178).
·Figure 1
Results:
Figure 2A shows performance metrics for our TractGPT model on the test set, compared to using mean FA values across the training set (either the mean across the whole tract, or the mean value per segment) to predict FA. Figure 2B shows true versus predicted along-tract FA for a sample participant in the test set using the TractGPT model in the left IFOF. Figure 2C and 2D shows the explained variance per tract (for a few select tracts in plots), as well as per type of tract.
·Figure 2
Conclusions:
Along-tract profiles of dMRI measures capture sequential, non-independent information about diffusion properties (or microstructural properties) along the length of white matter tracts. This makes the resultant tractometry data well suited for transformer models. Here, we provide a preliminary result of a tailored transformer architecture, training a bidirectional transformer on masked tractometry FA values, and demonstrate robust performance on predicting along-tract measures in a test set from the same studies. Although this preliminary work focuses on the modelling normative data from two HCP-Lifespan studies to demonstrate potential utility of applying transformer architectures to such data, the value of such work will likely come from its implementation in anomaly detection frameworks. Future work will focus on whether such models can be used to identify tract-specific or global white matter abnormalities in various clinical cohorts, with comparison to existing deep learning frameworks for anomaly detection.
Lifespan Development:
Early life, Adolescence, Aging
Modeling and Analysis Methods:
Diffusion MRI Modeling and Analysis 1
Neuroanatomy, Physiology, Metabolism and Neurotransmission:
White Matter Anatomy, Fiber Pathways and Connectivity 2
Keywords:
Data analysis
Machine Learning
MRI
NORMAL HUMAN
White Matter
WHITE MATTER IMAGING - DTI, HARDI, DSI, ETC
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
For human MRI, what field strength scanner do you use?
Which processing packages did you use for your study?
Provide references using APA citation style.
Devlin, J., Chang, M.-W., Lee, K., Google, K. T., & Language, A. I. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North, 4171–4186. https://doi.org/10.18653/V1/N19-1423
Glasser, M. F., Sotiropoulos, S. N., Wilson, J. A., Coalson, T. S., Fischl, B., Andersson, J. L., Xu, J., Jbabdi, S., Webster, M., Polimeni, J. R., Van Essen, D. C., & Jenkinson, M. (2013). The minimal preprocessing pipelines for the Human Connectome Project. NeuroImage, 80, 105–124. https://doi.org/10.1016/J.NEUROIMAGE.2013.04.127
Veraart, J., Sijbers, J., Sunaert, S., Leemans, A., & Jeurissen, B. (2013). Weighted linear least squares estimation of diffusion MRI parameters: Strengths, limitations, and pitfalls. NeuroImage, 81, 335–346. https://doi.org/10.1016/J.NEUROIMAGE.2013.05.028
Wasserthal, J., Neher, P., & Maier-Hein, K. H. (2018). TractSeg-Fast and accurate white matter tract segmentation. NeuroImage, 183, 239–253.