Neural correlates of sentence comprehension using sentence embedding on narrative fMRI data
Poster No:
801
Submission Type:
Abstract Submission
Authors:
Seiji Kawashima1, Sudesna Chakraborty1
Institutions:
1Aoyama Gakuin University, Sagamihara, Kanagawa
First Author:
Co-Author:
Introduction:
The human brain processes narrative stories by integrating linguistic information hierarchically, from individual words to complete stories (Chang et al., 2022). Previous research utilizing word embedding models, such as word2vec, has demonstrated correlations between word representations and cortical activity (Huth et al., 2016). However, natural language comprehension involves more complex processes, such as sentence integration, that go beyond word-level associations. The recent development of Sentence Transformers (SBERT) has enabled the generation of sentence-level embeddings, opening new avenues for exploring narrative comprehension. In this study, we hypothesized that cumulative sentence embeddings correlate with BOLD signal patterns in narrative fMRI data, specifically within the Yeo Network's (Yeo et al., 2011) Dorsal Attention Network, Ventral Attention Network, and Default Mode Network-regions associated with narrative understanding and cognitive processes. Furthermore, we examined the differences in neural activity when participants listen to narratives presented in a coherent, paragraph-level ordered compared to when the paragraph order is randomized. By leveraging sentence embeddings, we aim to uncover how the brain processes and integrates narrative information under different conditions.
Methods:
We used an openly available story fMRI dataset from Princeton Dataspace: "Pie Man" (https://dataspace.princeton.edu/jspui/handle/88435/dsp015d86p269k). The dataset included a total of 105 subjects with the following details: intact condition had 36 subjects (25 females), paragraph scrambled had 18 subjects (6 males). Further details of the dataset and the preprocessing can be found in Simony et al., 2016.
To reduce computational demand, we parcellated the data using Schaefer parcellation (Schaefer et al., 2018) (3-mm resolution), and focused on 3 of the Yeo's 7 intrinsic connectivity networks (Yeo et al., 2011): the dorsal attention, ventral attention, and default mode networks. BOLD signals were averaged across subjects and the regions of interest within each network, resulting in a one-dimensional time series for each network. Given the one-dimensional nature of these time series, feature construction incorporated both the raw signal values and their first-order differences.
For the sentence embedding, we employed Sentence BERT embeddings (all-MiniLM-L6-v2) (Reimers & Gurevych, 2019) to represent sentences as 384-dimensional fixed-length vectors. Sentences were first tokenized into individual words, and these words were cumulatively stacked from sentence onset to create continuous segments. For each stacked segment, the sentence embedding was obtained, and changes between consecutive embeddings were computed using the Euclidean distance. If consecutive functional words (e.g., prepositions, conjunctions) yielded a change value below 0.2, the corresponding segment was excluded.
Principal Component Analysis (PCA) was performed on both the sentence embeddings and the BOLD signals to determine the number of components that collectively accounted for at least 80% of the cumulative variance. Using the retained number of components, a second PCA was conducted, and the first principal components were extracted to represent the time series for each network and paragraph. Spearman's rank-order correlation analyses were then applied to these principal component time series, and corresponding p-values were computed to assess statistical significance (see Fig.1 for the workflow).This procedure was repeated for both the intact and temporally scrambled paragraph condition.
To reduce computational demand, we parcellated the data using Schaefer parcellation (Schaefer et al., 2018) (3-mm resolution), and focused on 3 of the Yeo's 7 intrinsic connectivity networks (Yeo et al., 2011): the dorsal attention, ventral attention, and default mode networks. BOLD signals were averaged across subjects and the regions of interest within each network, resulting in a one-dimensional time series for each network. Given the one-dimensional nature of these time series, feature construction incorporated both the raw signal values and their first-order differences.
For the sentence embedding, we employed Sentence BERT embeddings (all-MiniLM-L6-v2) (Reimers & Gurevych, 2019) to represent sentences as 384-dimensional fixed-length vectors. Sentences were first tokenized into individual words, and these words were cumulatively stacked from sentence onset to create continuous segments. For each stacked segment, the sentence embedding was obtained, and changes between consecutive embeddings were computed using the Euclidean distance. If consecutive functional words (e.g., prepositions, conjunctions) yielded a change value below 0.2, the corresponding segment was excluded.
Principal Component Analysis (PCA) was performed on both the sentence embeddings and the BOLD signals to determine the number of components that collectively accounted for at least 80% of the cumulative variance. Using the retained number of components, a second PCA was conducted, and the first principal components were extracted to represent the time series for each network and paragraph. Spearman's rank-order correlation analyses were then applied to these principal component time series, and corresponding p-values were computed to assess statistical significance (see Fig.1 for the workflow).This procedure was repeated for both the intact and temporally scrambled paragraph condition.
Results:
The results (Fig. 2) highlight the distinct involvement of the Dorsal Attention, Ventral Attention, and Default Mode Networks, which varied depending on the sentence structure and content within paragraphs and between intact and scrambled conditions.
Conclusions:
The present study demonstrated that variations in paragraph structure and content influence narrative comprehension.
Higher Cognitive Functions:
Higher Cognitive Functions Other
Language:
Language Comprehension and Semantics 1
Speech Perception 2
Keywords:
Language
Open Data
Other - Natural Language Processing
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
Provide references using APA citation style.
Huth, A. G., De Heer, W. A., Griffiths, T. L., Theunissen, F. E., & Gallant, J. L. (2016). Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600), 453–458.
Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In arXiv [cs.CL]. arXiv. https://aclanthology.org/D19-1410.pdf
Schaefer, A., Kong, R., Gordon, E. M., Laumann, T. O., Zuo, X.-N., Holmes, A. J., Eickhoff, S. B., & Yeo, B. T. T. (2018). Local-Global Parcellation of the Human Cerebral Cortex from Intrinsic Functional Connectivity MRI. Cerebral Cortex , 28(9), 3095–3114.
Simony, E., Honey, C. J., Chen, J., Lositsky, O., Yeshurun, Y., Wiesel, A., & Hasson, U. (2016). Dynamic reconfiguration of the default mode network during narrative comprehension. Nature Communications, 7(May 2015), 12141.
Yeo, B. T. T., Krienen, F. M., Sepulcre, J., Sabuncu, M. R., Lashkari, D., Hollinshead, M., Roffman, J. L., Smoller, J. W., Zöllei, L., Polimeni, J. R., Fischl, B., Liu, H., & Buckner, R. L. (2011). The organization of the human cerebral cortex estimated by intrinsic functional connectivity. Journal of Neurophysiology, 106(3), 1125–1165.