A NeuroGen extension: synthetic generation of images towards precise visual system activation
Poster No:
772
Submission Type:
Abstract Submission
Authors:
Daniel Chong1, Keith Jamison2, Mert Sabuncu3, Amy Kuceyeski4
Institutions:
1Cornell University, Ithaca, NY, 2Weill Cornell Medicine, New York City, NY, 3Cornell Tech, New York City, NY, 4Weill Cornell Medicine, Ithaca, NY
First Author:
Co-Author(s):
Introduction:
Functional Magnetic Resonance Imaging (fMRI) combined with advancements in machine learning and computer vision have allowed new insights into how the human visual system processes information. One tool in this realm called NeuroGen couples an encoding model of human vision with an image generator to allow creation of synthetic images that maximally drive or suppress visual regions' activity levels (Gu, 2022). Building on this foundation, our work extends NeuroGen's capabilities to generate synthetic images that can match a user-provided, continuous-valued target vector of brain activations for every region in the visual cortex.
Methods:
In this study, we utilized the Natural Scenes Dataset (NSD), which consists of fMRI collected from 8 subjects while viewing 10,000 naturalistic images (Allen, 2022). The fMRI data was preprocessed and general linear models provided estimates for voxel-level activations (beta maps) for each subject and image. Here, voxel activations from 21 early and late visual regions were averaged to obtain a single activation response value per region for each subject and each image.
To determine the target activation values for our image generation process, we relied on a previously published novel clustering approach using Shared Decodable Components, which results in clusters of images with similar semantic concepts (Efird, 2024). For our study, we focused on 5 out of the paper's 33 clusters labeled "Animals," "Faces," "Food," "Hard (objects)," and "Movement." For each cluster, we identified the top 10 images closest to the centroid and averaged their beta weights, after which we took the voxel average within the 21 visual regions. These 5 activation vectors, each 21 x 1 in dimension, served as our target activation vectors in the next step.
We adapted NeuroGen to generate images that would result in a brain activity pattern as similar as possible to the 5 target activation vectors described above, specifically by maximizing the correlation between the predicted activity and the target activity. The image generator we use, BigGan-Deep, requires selection of an image category from the possible 1000 ImageNet classes. During image optimization, 100 images were generated for each of the 1000 ImageNet classes, fed through the encoding model to obtain predicted activities for each visual region, which were then correlated with the target pattern of activities. The top 10 ImageNet classes with the highest average correlation (with the target vector in question) across the 100 images was identified. For a given image category, the base noise vector is further fine-tuned through gradient descent to maximize the correlation between the predicted region activation and the target region activation. In the end, we generated 10 images for each subject and each of the 5 target activations.
To determine the target activation values for our image generation process, we relied on a previously published novel clustering approach using Shared Decodable Components, which results in clusters of images with similar semantic concepts (Efird, 2024). For our study, we focused on 5 out of the paper's 33 clusters labeled "Animals," "Faces," "Food," "Hard (objects)," and "Movement." For each cluster, we identified the top 10 images closest to the centroid and averaged their beta weights, after which we took the voxel average within the 21 visual regions. These 5 activation vectors, each 21 x 1 in dimension, served as our target activation vectors in the next step.
We adapted NeuroGen to generate images that would result in a brain activity pattern as similar as possible to the 5 target activation vectors described above, specifically by maximizing the correlation between the predicted activity and the target activity. The image generator we use, BigGan-Deep, requires selection of an image category from the possible 1000 ImageNet classes. During image optimization, 100 images were generated for each of the 1000 ImageNet classes, fed through the encoding model to obtain predicted activities for each visual region, which were then correlated with the target pattern of activities. The top 10 ImageNet classes with the highest average correlation (with the target vector in question) across the 100 images was identified. For a given image category, the base noise vector is further fine-tuned through gradient descent to maximize the correlation between the predicted region activation and the target region activation. In the end, we generated 10 images for each subject and each of the 5 target activations.
Results:
Across all subjects and categories chosen, there was a significant improvement in the correlation between predicted and target activations during the fine-tuning step (See Figure 1). Images generated during the optimization procedure are shown in Figure 2, where we first show the top 10 images closest to that target centroid and then show one example image per subject for that target category. The semantic information contained in the images does appear to be consistent between the target and the optimized images.
Conclusions:
This study extends the capabilities of NeuroGen, enhancing its utility as a tool for neuroscience discovery. Our framework enables the creation of synthetic images optimized to align with continuous-valued target vectors of activation (rather than purely minimizing or maximizing activation as was done before). This improvement holds potential for applications in neuroscience and clinical settings, particularly in crafting synthetic stimuli for rare or unconventional regional activation patterns that otherwise may be difficult to elicit with natural images.
Higher Cognitive Functions:
Imagery 1
Modeling and Analysis Methods:
Activation (eg. BOLD task-fMRI) 2
Connectivity (eg. functional, effective, structural)
fMRI Connectivity and Network Modeling
Methods Development
Keywords:
Data analysis
Design and Analysis
FUNCTIONAL MRI
Machine Learning
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Are you Internal Review Board (IRB) certified? Please note: Failure to have IRB, if applicable will lead to automatic rejection of abstract.
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
Provide references using APA citation style.
Allen, E. J. (2022). A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25(1), 116–126.
Efird, C.(2024). Finding Shared Decodable Concepts and their Negations in the Brain. arXiv [Cs.LG].