Cognitive map-like representations of semantic structure during movie watching

970 

Abstract Submission 

Siyang Li^1,2, Weiyang Shi^3,1, Yongfu Hao¹, Xia Liang², Zhang Yu¹, Tianzi Jiang^3,1

¹Zhejiang Lab, Hangzhou, Zhejiang, China, ²Harbin Institute of Technology, Harbin, Heilongjiang, China, ³Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China

Siyang Li  
Zhejiang Lab|Harbin Institute of Technology
Hangzhou, Zhejiang, China|Harbin, Heilongjiang, China

Weiyang Shi  
Brainnetome Center, Institute of Automation, Chinese Academy of Sciences|Zhejiang Lab
Beijing, China|Hangzhou, Zhejiang, China

Yongfu Hao  
Zhejiang Lab
Hangzhou, Zhejiang, China

Xia Liang  
Harbin Institute of Technology
Harbin, Heilongjiang, China

Zhang Yu  
Zhejiang Lab
Hangzhou, Zhejiang, China

Tianzi Jiang  
Brainnetome Center, Institute of Automation, Chinese Academy of Sciences|Zhejiang Lab
Beijing, China|Hangzhou, Zhejiang, China

Neuroscience aims to unravel the link between cognition and neural activity, with a growing focus on how the brain represents naturalistic stimuli like movie-watching (Kringelbach, Perl et al. 2023). Studies shown that the brain segments continuous stimuli into discrete events and forms narrative graphs, impacting memory performance in behaviors (Lee and Chen 2022). However, the precise characterization and neural representations of such naturalistic information in the human brain remains unclear. This study investigates cognitive map-like brain representations during movie watching, based on the theory that the brain encodes and organizes experiences in a relational map (Tolman 1948, Behrens, Muller et al. 2018). In this study, we explored the cognitive map-like representations in the brain during movie watching and their convergence with the underlying semantic structure of movie lines.

We used the 7T movie-watching fMRI data acquired from the Human Connectome Project (HCP) dataset (Elam, Glasser et al. 2021). Building upon our prior research (Li et al., under revision), we identified brain states within cognitive-map networks (Figure 1). Initially, we computed the similarity matrix (W) for brain-state occurrences, followed by t-SNE for dimensional reduction and k-means for clustering analysis. Subsequently, we embedded all clusters of brain-state occurrences into a two-dimensional (2D) space. Next, we constructed a predictive map (M) of brain-state occurrences, employing the successor representation (SR) technique. Finally, we estimated the predictivity score based on the backward skewness of SR fields (i.e., columns of M). To validate the cognitive-map-like representation of naturalistic stimuli, we extracted the semantic features from each movie segment by applying the wave2vec model to movie lines. We then calculated the semantic similarity matrix (S) between segments. Subsequently, we assessed the association of these semantic similarities with the predictive map.

We identified two alternating brain states using movie-watching fMRI data (Figure 2). Among which, State1 predominantly involved DMN and hippocampus, while State2 exhibited strong activation in sensorimotor regions and hippocampus. Upon estimating the representation similarity between brain-state occurrences and semantic features, we observed significantly lower representational similarity in State1. Notably, during movie watching, this representational similarity exhibited an increase trend in State1, probably updating internal representations with information gained from the movie segment, but still lower than State2. When embedding the brain-state occurrences into a 2D state space, we found higher semantic similarity (shorter distance) within occurrence clusters than between clusters for both states, but with higher within-cluster similarity in State1. Subsequent SR modeling of State1 revealed typical 'place fields' of occurrence clusters, suggesting a cognitive-map-like representation. Notably, the predictivity score of SR fields (backward skewness of SR) was strongly covaried with individual vocabulary skills and verbal comprehension, evaluated by the Picture Vocabulary Test.

Our study has identified two alternating brain states associated with internal and external focus during movie watching. State2 predominantly responds to sensory processing of external stimuli, while State1 integrates abstract relational representations of the movie segment into internal knowledge maps. This supports the cognitive map theory, wherein 'modules' within the semantic structure are represented as brain-state occurrence clusters, exhibiting characteristic 'place field' patterns. Notably, the predictivity score of SR fields was strongly associated with participants' verbal comprehension in the Picture Vocabulary Test, suggesting an underlying semantic representation mechanism of natural stimuli during internal cognitive processing.

Higher Cognitive Functions Other ¹

Learning and Memory Other ²

Cognition

Computational Neuroscience

FUNCTIONAL MRI

Memory