Mapping individual and shared cortical language representations during real-time natural dialogues

1052 

Abstract Submission 

Zaid Zada¹, Samuel Nastase¹, Sebastian Speer¹, Laetitia Mwilambwe-Tshilobo¹, Lily Tsoi², Shannon Burns³, Uri Hasson¹, Diana Tamir¹

¹Princeton University, Princeton, NJ, ²Caldwell University, Caldwell, NJ, ³Pomona College, Claremont, CA

Zaid Zada  
Princeton University
Princeton, NJ

Samuel Nastase  
Princeton University
Princeton, NJ

Sebastian Speer  
Princeton University
Princeton, NJ

Laetitia Mwilambwe-Tshilobo  
Princeton University
Princeton, NJ

Lily Tsoi  
Caldwell University
Caldwell, NJ

Shannon Burns  
Pomona College
Claremont, CA

Uri Hasson  
Princeton University
Princeton, NJ

Diana Tamir  
Princeton University
Princeton, NJ

How is language encoded in the brain during everyday conversations, and how is that linguistic encoding shared across interlocutors? Typical studies of the neural basis of language present subjects with predetermined, isolated words or sentences (Price, 2010), and do not consider the role of spontaneous language production nor linguistic neural coupling (Garrod & Pickering, 2004). Here, we aim to address both gaps and map brain areas involved in both speech production and comprehension during natural dialogue.

We developed a hyperscanning paradigm to collect simultaneous fMRI data in 30 dyads (60 subjects, 41F) as they freely discussed 10 topics across 5 runs (Fig 1A) (Speer et al., 2023). Topics were presented as a starting point, but each dyad was free to pursue the discussion in different ways. Our goal was to characterize the linguistic content encoded in the brain within subjects. To that end, we estimated voxel-wise encoding models to predict held-out BOLD signals during speech production or comprehension from 6 feature spaces: nuisance task structure regressors, mel-spectral features, phonetic articulatory features, head motion parameters, and word embeddings extracted from the GPT-2 language model (Fig 1B) (la Tour et al., 2022). To account for turn-taking during natural conversations, we split all regressors into separate sets for speaking and listening, and fit both submodels jointly using banded ridge regression; this allows the model to learn different weights for each process, and allows us to quantify the relative contribution of each submodel. Then, we correlated the actual and predicted BOLD activity for left-out runs in each voxel only from the language model word embedding, quantifying the extent of linguistic content in the signal for production or comprehension time points separately (Fig 1C).

We found strong encoding performance bilaterally throughout the language network-superior temporal cortex, middle frontal gyrus, inferior frontal gyrus, and angular gyrus-as well as somatomotor and precuneus areas. When evaluating the feature sets separately, we found that the task structure, spectral, and articulation features all recruited the somatomotor cortex during speech production, while speech comprehension more strongly recruited the superior temporal cortex (Fig 2C). While head motion features predicted a typical halo in the superior area. With that variance accounted for, the language model embeddings predicted posterior temporal cortex and middle frontal regions the most (Fig 1A). Surprisingly, we found better overall accuracy in the right hemisphere than left, and the right angular gyrus is heavily recruited for production. The superior temporal region depicted a ventral accuracy gradient for comprehension only (orange), to joint processing (white), and then only production (blue).

In order to test for linguistic coupling across dyads, we developed a model-based coupling method where we correlate the production/comprehension model predictions of one subject to the comprehension/production responses of their partner. We found that model-based predictions can generalize between subjects in several hubs across the language network from the early auditory cortex, to temporal regions, and to precuneus (Fig 2B).

Our findings lay the foundation for assessing model-based, brain-to-brian coupling between speakers and listeners. We showed that cortical language representations during interactive natural dialogues can be predicted by language model embeddings, revealing both shared and selective encoding for speaking and listening. While this concurs with existing literature for language comprehension (Caucheteux & King, 2022; Schrimpf et al., 2021), we also show this in production during dialogue (Goldstein et al., 2023; Yamashita et al., 2023). We further extend the production–perception relationship by formally modeling the shared speaker–listener alignment on linguistic features (Zada et al., 2023).

Social Interaction

Language Comprehension and Semantics ²

Speech Perception

Speech Production ¹

Classification and Predictive Modeling

Computational Neuroscience

FUNCTIONAL MRI

Language

Perception

Social Interactions