Face Decoding from 7T fMRI Using Diffusion Generative Model Improves with Hemodynamic Deconvolution
Poster No:
1107
Submission Type:
Abstract Submission
Authors:
Nguyen Huynh1, Joshita Majumdar1, Gopikrishna Deshpande1,2,3,4,5,6
Institutions:
1Auburn University Neuroimaging Center, Department of Electrical & Computer Engineering, Auburn, AL, 2Department of Psychological Sciences, Auburn University, Auburn, AL, 3Alabama Advanced Imaging Consortium, Birmingham, AL, 4Center for Neuroscience, Auburn University, Auburn, AL, 5Department of Psychiatry, National Institute of Mental Health and Neurosciences, Bangalore, India, 6Department of Heritage Science and Technology, Indian Institute of Technology, Hyderabad, India
First Author:
Nguyen Huynh
Auburn University Neuroimaging Center, Department of Electrical & Computer Engineering
Auburn, AL
Auburn University Neuroimaging Center, Department of Electrical & Computer Engineering
Auburn, AL
Co-Author(s):
Joshita Majumdar
Auburn University Neuroimaging Center, Department of Electrical & Computer Engineering
Auburn, AL
Auburn University Neuroimaging Center, Department of Electrical & Computer Engineering
Auburn, AL
Gopikrishna Deshpande
Auburn University Neuroimaging Center, Department of Electrical & Computer Engineering|Department of Psychological Sciences, Auburn University|Alabama Advanced Imaging Consortium|Center for Neuroscience, Auburn University|Department of Psychiatry, National Institute of Mental Health and Neurosciences|Department of Heritage Science and Technology, Indian Institute of Technology
Auburn, AL|Auburn, AL|Birmingham, AL|Auburn, AL|Bangalore, India|Hyderabad, India
Auburn University Neuroimaging Center, Department of Electrical & Computer Engineering|Department of Psychological Sciences, Auburn University|Alabama Advanced Imaging Consortium|Center for Neuroscience, Auburn University|Department of Psychiatry, National Institute of Mental Health and Neurosciences|Department of Heritage Science and Technology, Indian Institute of Technology
Auburn, AL|Auburn, AL|Birmingham, AL|Auburn, AL|Bangalore, India|Hyderabad, India
Introduction:
Information integration across cortical layers within the dorsal visual cortex during face decoding remains unexplored. Understanding the neural circuitry of face perception may aid individuals with face recognition difficulties, such as those with stroke. In this work, we employed a diffusion generative model on laminar fMRI data to reconstruct the face frames from a subject watching a music video. Our goal was to determine which layers within the visual regions along the dorsal pathway are most effective at reconstructing face images. Given that fMRI is not a direct measure of neural activity, we employed a blind deconvolution method, which reveals latent neural activity induced by the stimuli. Our findings indicate that deconvolution can significantly enhance face decoding accuracy from laminar fMRI data.
Methods:
The data was acquired on a Siemens Terra X 7T scanner with a 32-channel head coil. The whole brain anatomical image was acquired using an MPRAGE sequence (288 slices, voxel size: 0.6 mm isotropic, TR/TE: 4000/3.45 ms, FOV read: 248 mm, flip angle: 4o). Functional images were collected while a 35-year-old female subject watched a 4-minute music video, using a multiband gradient-echo echo planar imaging (EPI) sequence with the following parameters: 92 slices, voxel size: 0.8 mm isotropic, TR/TE: 1500/23 ms, flip angle: 70°, FOV read: 136 mm, A>P phase-encoding, iPAT GRAPPA acceleration factor of 4, and a total of 167 volumes. Scans were obtained weekly from the same subject, with 30 weeks of functional data analyzed in this work.
The structural image was preprocessed using FreeSurfer for skull stripping, intensity normalization, and segmentation with manual white matter (WM) and gray matter (GM) corrections in ITK-Snap to improve accuracy. The equi-volume method (Waehnert, 2010) in LayNii was then used to define the superficial, middle, and deep layers for V1, V2, V3 and V4. A standard pipeline, including motion and slice-time correction, outlier removal, nuisance regression of 6 motion parameters, WM and CSF mean signals, and linear detrending, was performed on the functional data. A blind deconvolution technique (Wu, 2013) was used to obtain the latent neural signals for each layer, and a bandpass filter (0.01–0.1 Hz) was applied before and after the deconvolution method for comparison.
Our diffusion model, based on the Latent Diffusion Model (Rombach, 2022), reconstructs images from brain activity. Gaussian noise is incrementally added to images, and a UNet model reverses this process. Using contrastive learning, compressed image representations are aligned with latent laminar features, while mismatched pairs are minimized. A Multilayer Perceptron predicted image embeddings from brain embeddings, which were used as inputs for the diffusion model to generate the corresponding scenes.
From 4,800 video scenes (3,420 face images), data was split 80:20 for training and testing. After training, test set MRI data reconstructed scenes from laminar neural activity, with accuracy defined as Pearson correlation > 0.6 between reconstructed and original images.
The structural image was preprocessed using FreeSurfer for skull stripping, intensity normalization, and segmentation with manual white matter (WM) and gray matter (GM) corrections in ITK-Snap to improve accuracy. The equi-volume method (Waehnert, 2010) in LayNii was then used to define the superficial, middle, and deep layers for V1, V2, V3 and V4. A standard pipeline, including motion and slice-time correction, outlier removal, nuisance regression of 6 motion parameters, WM and CSF mean signals, and linear detrending, was performed on the functional data. A blind deconvolution technique (Wu, 2013) was used to obtain the latent neural signals for each layer, and a bandpass filter (0.01–0.1 Hz) was applied before and after the deconvolution method for comparison.
Our diffusion model, based on the Latent Diffusion Model (Rombach, 2022), reconstructs images from brain activity. Gaussian noise is incrementally added to images, and a UNet model reverses this process. Using contrastive learning, compressed image representations are aligned with latent laminar features, while mismatched pairs are minimized. A Multilayer Perceptron predicted image embeddings from brain embeddings, which were used as inputs for the diffusion model to generate the corresponding scenes.
From 4,800 video scenes (3,420 face images), data was split 80:20 for training and testing. After training, test set MRI data reconstructed scenes from laminar neural activity, with accuracy defined as Pearson correlation > 0.6 between reconstructed and original images.
·The architecture of the diffusion generative model
Results:
Figure 2 shows the number of accurately reconstructed face scenes across cortical layers in visual areas, both with and without deconvolution. The results show that using the deconvolution significantly increases the number of reconstructed images. This is likely because the deconvolution method likely improves the effective temporal resolution of the neural activity estimates by deblurring the smoothing effect of the hemodynamic response, as well as removing draining vein effects that are pronounced at laminar resolution, thus aligning the data more closely with the underlying neural events that generated the BOLD response.
·Upper Half: The images generated by the diffusion model (right), using laminar features as inputs. Lower Half: Comparison of the number of accurately reconstructed face scenes across cortical depths
Conclusions:
We used a diffusion generative model to study cortical layer roles in the dorsal stream for face decoding using laminar fMRI. We found that using the deconvolution method to identify latent neural activity greatly improves reconstruction accuracy.
Modeling and Analysis Methods:
Classification and Predictive Modeling 1
Task-Independent and Resting-State Analysis
Neuroanatomy, Physiology, Metabolism and Neurotransmission:
Cortical Cyto- and Myeloarchitecture 2
Novel Imaging Acquisition Methods:
BOLD fMRI
Keywords:
Cortical Layers
Data analysis
FUNCTIONAL MRI
HIGH FIELD MR
Machine Learning
Other - Diffusion Generative Model; Laminar fMRI
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Are you Internal Review Board (IRB) certified? Please note: Failure to have IRB, if applicable will lead to automatic rejection of abstract.
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
For human MRI, what field strength scanner do you use?
Which processing packages did you use for your study?
Provide references using APA citation style.
2. Waehnert, M. D., Dinse, J., Weiss, M., Streicher, M. N., Waehnert, P., Geyer, S., ... & Bazin, P. L. (2014). Anatomically motivated modeling of cortical laminae. Neuroimage, 93, 210-220.
3. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695).