Structural MRI modality transfer across age using image synthesis techniques
Poster No:
1505
Submission Type:
Abstract Submission
Authors:
Ho-kyun Kim1, Bo-yong Park2
Institutions:
1Inha university, Incheon, Incheon, 2Korea University, Seoul, Seoul
First Author:
Co-Author:
Introduction:
Changes in brain morphology during neurodevelopment and neurodegenerative processes provide valuable insights into the mechanisms of disease progression during these periods (Raji et al., 2009; Shaw et al., 2013). However, conducting in-depth investigations of structural changes across different ages remains challenging due to the resource-intensive and time-consuming nature of obtaining multimodal structural magnetic resonance imaging (MRI), particularly in young children and the elderly. Advances in deep learning enable condition-specific image synthesis, such as generating T2-weighted (T2w) MRI from T1-weighted (T1w) MRI, providing richer information with limited data (Frangou et al., 2022). In this study, we propose a framework for cross-generating T1w and T2w MRI across the entire lifespan using conditional image synthesis techniques.
Methods:
1.Data
We obtained a total of 3,190 T1w and T2w MRI scans, along with sex and age information, from three independent databases: the Human Connectome Project (HCP), comprising young adults (mean ± SD age: 28.84±3.69 years; female: 54%); HCP-Development (HCP-D), consisting of children (mean ± SD age: 14.44±4.06 years; female: 54%); and HCP-Aging (HCP-A), involving middle-aged and older adults (mean ± SD age: 60.36±15.7 years; female: 56%) (Harms et al., 2018; Van Essen et al., 2012).
2.Model architecture
The cross-modal synthesis model consisted of three components: (i) variational autoencoder with generative adversarial network (VAE-GAN), (ii) latent diffusion model (LDM), and (iii) ControlNet (Zhang et al., 2023) (Fig. 1). The VAE-GAN encodes the 3D input data to obtain latent vectors sampled from an estimated the normal distribution, reconstructs the original data through the decoder, and distinguishes real data from generated data using a Pix2PixHD-based discriminator (Kingma, 2013; Wang et al., 2018). The latent vector generated by the VAE-GAN is fed into the LDM that incorporates 3D-ResNet blocks-based U-Net-structured diffusion model (Guo et al., 2024). Age, sex, and modality information was added via a linear embedding module (Rombach et al., 2022). After training the LDM, ControlNet was employed to incorporate conditional inputs into the diffusion model. Target modality data were entered into the U-Net-based diffusion model, where the encoder was copied from the trained LDM and the decoder was a zero convolution decoder to ensure stable training for condition-specific image restoration.
3.Model training and performance
The data were randomly split within each database at a 9:1 ratio to ensure the training and test sets were equally proportioned. The performance of the model was evaluated using peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and mean squared error (MSE) between the actual and synthesized T1w or T2w data.
We obtained a total of 3,190 T1w and T2w MRI scans, along with sex and age information, from three independent databases: the Human Connectome Project (HCP), comprising young adults (mean ± SD age: 28.84±3.69 years; female: 54%); HCP-Development (HCP-D), consisting of children (mean ± SD age: 14.44±4.06 years; female: 54%); and HCP-Aging (HCP-A), involving middle-aged and older adults (mean ± SD age: 60.36±15.7 years; female: 56%) (Harms et al., 2018; Van Essen et al., 2012).
2.Model architecture
The cross-modal synthesis model consisted of three components: (i) variational autoencoder with generative adversarial network (VAE-GAN), (ii) latent diffusion model (LDM), and (iii) ControlNet (Zhang et al., 2023) (Fig. 1). The VAE-GAN encodes the 3D input data to obtain latent vectors sampled from an estimated the normal distribution, reconstructs the original data through the decoder, and distinguishes real data from generated data using a Pix2PixHD-based discriminator (Kingma, 2013; Wang et al., 2018). The latent vector generated by the VAE-GAN is fed into the LDM that incorporates 3D-ResNet blocks-based U-Net-structured diffusion model (Guo et al., 2024). Age, sex, and modality information was added via a linear embedding module (Rombach et al., 2022). After training the LDM, ControlNet was employed to incorporate conditional inputs into the diffusion model. Target modality data were entered into the U-Net-based diffusion model, where the encoder was copied from the trained LDM and the decoder was a zero convolution decoder to ensure stable training for condition-specific image restoration.
3.Model training and performance
The data were randomly split within each database at a 9:1 ratio to ensure the training and test sets were equally proportioned. The performance of the model was evaluated using peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and mean squared error (MSE) between the actual and synthesized T1w or T2w data.
Results:
The synthesis performance showed PSNR values of 19.43±2.30 (T1w-to-T2w) and 21.39±3.42 (T2w-to-T1w), SSIM values of 0.90±0.015 and 0.927±0.015, and MSE values of 0.024±0.012 and 0.021±0.011. Additionally, visual inspection of the generated images across ages revealed slight decreases in cortical thickness and increases in the cerebrospinal fluid area.
Conclusions:
In this study, we proposed an MRI synthesis framework capable of generating age-specific T1w and T2w images by utilizing VAE-GAN, LDM, and ControlNet-based approach. These findings suggest that our framework can serve as a valuable tool for augmenting existing MRI datasets, addressing data availability challenges, and potentially enabling more comprehensive longitudinal studies on neurodevelopment and aging.
Funding: Institute for Information and Communications Technology Planning and Evaluation (IITP) funded by the Korea Government (MSIT) (No. 2022-0-00448/RS-2022-II220448, Deep Total Recall: Continual Learning for Human-Like Recall of Artificial Neural Networks; RS-2021-II212068, Artificial Intelligence Innovation Hub), Institute for Basic Science (IBS-R015-D1).
Funding: Institute for Information and Communications Technology Planning and Evaluation (IITP) funded by the Korea Government (MSIT) (No. 2022-0-00448/RS-2022-II220448, Deep Total Recall: Continual Learning for Human-Like Recall of Artificial Neural Networks; RS-2021-II212068, Artificial Intelligence Innovation Hub), Institute for Basic Science (IBS-R015-D1).
Modeling and Analysis Methods:
Methods Development 1
Other Methods 2
Keywords:
Computational Neuroscience
Data analysis
Machine Learning
STRUCTURAL MRI
Other - Image Synthesis
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
For human MRI, what field strength scanner do you use?
Which processing packages did you use for your study?
Provide references using APA citation style.
Guo, P., Zhao, C., Yang, D., Xu, Z., Nath, V., Tang, Y., Simon, B., Belue, M., Harmon, S., Turkbey, B., & others. (2024). MAISI: Medical AI for Synthetic Imaging. ArXiv Preprint ArXiv:2409.11169.
Harms, M. P., Somerville, L. H., Ances, B. M., Andersson, J., Barch, D. M., Bastiani, M., Bookheimer, S. Y., Brown, T. B., Buckner, R. L., Burgess, G. C., & others. (2018). Extending the Human Connectome Project across ages: Imaging protocols for the Lifespan Development and Aging projects. Neuroimage, 183, 972–984.
Kingma, D. P. (2013). Auto-encoding variational bayes. ArXiv Preprint ArXiv:1312.6114.
Raji, C. A., Lopez, O. L., Kuller, L. H., Carmichael, O. T., & Becker, J. T. (2009). Age, Alzheimer disease, and brain structure. Neurology, 73(22), 1899–1905.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10684–10695.
Shaw, P., Malek, M., Watson, B., Greenstein, D., De Rossi, P., & Sharp, W. (2013). Trajectories of cerebral cortical development in childhood and adolescence and adult attention-deficit/hyperactivity disorder. Biological Psychiatry, 74(8), 599–606.
Van Essen, D. C., Ugurbil, K., Auerbach, E., Barch, D., Behrens, T. E. J., Bucholz, R., Chang, A., Chen, L., Corbetta, M., Curtiss, S. W., & others. (2012). The Human Connectome Project: a data acquisition perspective. Neuroimage, 62(4), 2222–2231.
Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Tao, A., Kautz, J., & Catanzaro, B. (2018). High-resolution image synthesis and semantic manipulation with conditional gans. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8798–8807.
Zhang, L., Rao, A., & Agrawala, M. (2023). Adding conditional control to text-to-image diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision, 3836–3847.