Movies Illuminate Minds: Functional Connectivity-Based Behavior Mapping During Naturalistic Viewing
Poster No:
1378
Submission Type:
Abstract Submission
Authors:
Yunrui Zhang1, Mert Sabuncu2, Amy Kuceyeski3
Institutions:
1Cornell University, Ithaca, NY, 2Cornell Tech, Weill Cornell Medicine, New York, NY, 3Cornell University, Weill Cornell Medicine, Ithaca, NY
First Author:
Co-Author(s):
Introduction:
Functional MRI (fMRI) scanning during naturalistic viewing offers a distinct approach compared to traditional resting-state or task-based paradigms; it resembles what we experience in our everyday life - we watch, we listen, and we think about it. Movie-watching fMRI, in particular, has emerged as a powerful tool for deriving individual-specific functional connectivity patterns. Recent research suggests that functional connectivity during movie-watching is more predictive of cognition than resting-state measures, as demonstrated by linear prediction models (Finn, 2021; Gal 2022; Guan 2023; Vanderwal 2017). However, questions still remain about which connections in the brain are most predictive, what features of the movies are most stimulating, and whether neural synchrony across individuals during movie-watching affects prediction accuracy. In this study, we propose a novel deep learning framework for (1) predicting cognition and sex from movie-watching functional connectivity, (2) identifying the brain connections and associated movie features driving these predictions, and (3) examining the relationship between neural synchrony and prediction performance.
Methods:
To explore the complex relationships between functional connectivity (FC) and behavioral attributes, including cognition and sex, we developed a deep neural network (DNN) model that integrates principal component analysis (PCA) and a multi-layer perceptron (MLP) with three hidden layers, each containing 40 nodes and employing the ReLU activation function. The dataset used for this study consists of 176 participants from the Human Connectome Project (HCP) 7T release (Vu, 2015). This DNN model takes in movie FC and predicts cognition or sex. Once the model is trained, we can identify the functional connectivity features that most strongly influenced model predictions. To do so, we computed the propagation of feature contributions through the prediction model, generating an importance score for each FC edge.
Neural synchrony of brain activity across individuals was also quantified for each pair of subjects by computing the Pearson correlation of their regional activity over time. High-level movie features, including the duration of human face and voice presence, were hand-coded for each clip (Betzel 2020) . The DNN models' prediction accuracies were compared to both neural synchrony and movie features for each movie clip to test if the accuracy of brain-behavior models was related to the content or across-subject synchrony of neural activity during the movie.
Neural synchrony of brain activity across individuals was also quantified for each pair of subjects by computing the Pearson correlation of their regional activity over time. High-level movie features, including the duration of human face and voice presence, were hand-coded for each clip (Betzel 2020) . The DNN models' prediction accuracies were compared to both neural synchrony and movie features for each movie clip to test if the accuracy of brain-behavior models was related to the content or across-subject synchrony of neural activity during the movie.
Results:
Certain movie clips demonstrated higher predictability for cognition and sex compared to resting-state FC, while others were either comparable or less predictive. Among the clips, Social Network achieved the highest Spearman correlation for cognition (0.39), followed by Ocean's 11 (0.38) and Inception (0.32). For sex prediction, most movies performed similarly, with Ocean's 11, Flower, and Pockets being the most predictive, with area under the curve (AUC) scores of 0.75, 0.75, and 0.74, respectively.
The most predictive FC edges varied across movie clips but consistently involved the prefrontal cortex (higher order mental functions including executive attention), parietal lobe, temporal lobe, and occipital lobe.
Regions involved in audio and visual processing exhibited particularly high levels of neural synchrony across nearly all clips. Moreover, better prediction performance was positively correlated with higher overall neural synchrony and strongly associated with longer duration of human face and voice presence in the clips.
The most predictive FC edges varied across movie clips but consistently involved the prefrontal cortex (higher order mental functions including executive attention), parietal lobe, temporal lobe, and occipital lobe.
Regions involved in audio and visual processing exhibited particularly high levels of neural synchrony across nearly all clips. Moreover, better prediction performance was positively correlated with higher overall neural synchrony and strongly associated with longer duration of human face and voice presence in the clips.
Conclusions:
Our results revealed the valuable brain connectivity dynamics elicited by movie-watching, their potential generalizability among individuals viewing the same content, and the critical role of social and sensory cues in shaping behaviorally relevant brain responses. These findings highlight the value of naturalistic paradigms in increasing the reliability and accuracy of brain-behavior models.
Emotion, Motivation and Social Neuroscience:
Social Cognition
Higher Cognitive Functions:
Imagery
Modeling and Analysis Methods:
Activation (eg. BOLD task-fMRI)
Connectivity (eg. functional, effective, structural) 2
fMRI Connectivity and Network Modeling 1
Keywords:
Cognition
Computational Neuroscience
FUNCTIONAL MRI
Machine Learning
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Are you Internal Review Board (IRB) certified? Please note: Failure to have IRB, if applicable will lead to automatic rejection of abstract.
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
For human MRI, what field strength scanner do you use?
Provide references using APA citation style.
2. Finn, E.S. (2021). Movie-watching outperforms rest for functional connectivity-based prediction of behavior. NeuroImage.
3. Gal, S. (2022). Act natural: Functional connectivity from naturalistic stimuli fMRI outperforms resting-state in predicting brain activity. NeuroImage.
4. Guan, Y. (2023). The abilities of movie-watching functional connectivity in individual identifications and individualized predictions. Brain Imaging and Behavior.
5. Shen, X. (2013). Groupwise whole-brain parcellation from resting-state fMRI data for network node identification. NeuroImage, 82, 403-415.
6. Vanderwal, T. (2017). Individual differences in functional connectivity during naturalistic viewing conditions. NeuroImage, 157, 521–530.
7. Vu, A.T. (2017). Tradeoffs in pushing the spatial resolution of fMRI for the 7T Human Connectome Project. NeuroImage, 154, 23-32.