A hierarchical processing pipeline to explain the brain activity while movie viewing vs listening
Poster No:
1470
Submission Type:
Abstract Submission
Authors:
Iqra Ejaz1, Sadia Shakil2
Institutions:
1Institute of space Technology (IST), Islamabad, Pakistan, 2The Chinese University of Hong, Hong Kong
First Author:
Co-Author:
Introduction:
Explainability and interpretability of the results while studying the brain dynamics under naturalistic stimuli is critical for both reproducibility and processing steps. However, employing various methods, the intermediate results are often poorly explained limiting the reproducibility and subsequent use of processing pipeline by other studies. Here we proposed a novel approach of a hierarchical processing pipeline integrating multiple analysis methods from different domains to study brain activity to interpret the differences while listening vs viewing a movie.
Methods:
We used fMRI dataset (Zadbood, 2017) of 36 participants listening (narration) and watching two audio-visual movies, with 1-18 listening to Sherlock and watching Merlin, and vice versa for 19-36. It was reported in (Zadbood, 2017) that scene specific neural patterns between movie viewing and listening were significantly correlated and more evident in the higher overlapping areas of DMN. We are also targeting DMN, however our focus is to use a hierarchical processing pipeline to explain the differences in the incoming stimuli modality (listeners vs viewers) through associated brain activity.
The data was pre-processed using SPM12 and DMN's regions-of-interest (ROIs: posterior cingulate cortex (PCC); anterior cingulate and medial prefrontal cortex (ACMP); inferior parietal (IP); lateral temporal cortex (LTC)) were extracted (Raichle, 2015; Buckner, 2008).
The pipeline consists of three layers as shown in equation below. Layer 1 (L1) extract static and dynamic aspects of DMN functional connectivity (FC) using Pearson correlation (PCorr) and layer 2 (L2), build upon L1, provides structural evolution of DMN FC using persistent homology (PH). Layer 3 (L3), build upon first 2 layers, extracts significant connectivity and topological features as covariates to differentiate between stimuli modalities, using binary logistic regression (BinLR).
y=f(x)=BinLR(PH(PCorr(x1,x2,..xn)
Both the whole movie and scene-wise analysis was performed in L1 and L2. L3 uses features from both layers to explain modality differences in DMN activity. Also in L3 scene-wise data was used to increase the data size, as the number participants were not sufficient to implement BinLR. The block diagram of the study is shown in Fig1(1).
The data was pre-processed using SPM12 and DMN's regions-of-interest (ROIs: posterior cingulate cortex (PCC); anterior cingulate and medial prefrontal cortex (ACMP); inferior parietal (IP); lateral temporal cortex (LTC)) were extracted (Raichle, 2015; Buckner, 2008).
The pipeline consists of three layers as shown in equation below. Layer 1 (L1) extract static and dynamic aspects of DMN functional connectivity (FC) using Pearson correlation (PCorr) and layer 2 (L2), build upon L1, provides structural evolution of DMN FC using persistent homology (PH). Layer 3 (L3), build upon first 2 layers, extracts significant connectivity and topological features as covariates to differentiate between stimuli modalities, using binary logistic regression (BinLR).
y=f(x)=BinLR(PH(PCorr(x1,x2,..xn)
Both the whole movie and scene-wise analysis was performed in L1 and L2. L3 uses features from both layers to explain modality differences in DMN activity. Also in L3 scene-wise data was used to increase the data size, as the number participants were not sufficient to implement BinLR. The block diagram of the study is shown in Fig1(1).
Results:
L1 results in Fig1(2) and Fig2(3B,C) shows high static (whole-brain) and dynamic (scene-wise) inter-modality FC consistent with (Zadbood, 2017), was also highest (0.95) in PCC. A key observation from L1 is higher scene-wise co-activation for the same subjects in both movies (Merlin viewers, Sherlock listeners) Fig2(3), suggesting that idiosyncratic activity may contribute to overall co-activation, consistent with findings in (Nguye, 2017; Song, 2021). L1 provides an overview of static and dynamic inter-modality FC in DMN cortices and ROIs but lacks details on FC structural evolution, which L2 (PH) reveals using FC matrices from L1.
L2 results in Fig1(2) validates L1 results, showing closer mean Betti curves for Merlin modalities than Sherlock. However, L1's observation of FC dynamics tied to individual brain states is not evident. Instead, FC evolution appears modality-dependent, highlighted by the results of L3 in Fig2(3G,H) that mCorr, AUC, and slope are significant covariates in differentiating Merlin modalities, while in Sherlock mCorr is found to be significant.
As from results of L3, mCorr significantly differentiates viewers and listeners in both movies. However, structural FC differences are notable in Merlin but not in Sherlock, highlighting the need to analyse FC structure beyond basic connectivity.
L2 results in Fig1(2) validates L1 results, showing closer mean Betti curves for Merlin modalities than Sherlock. However, L1's observation of FC dynamics tied to individual brain states is not evident. Instead, FC evolution appears modality-dependent, highlighted by the results of L3 in Fig2(3G,H) that mCorr, AUC, and slope are significant covariates in differentiating Merlin modalities, while in Sherlock mCorr is found to be significant.
As from results of L3, mCorr significantly differentiates viewers and listeners in both movies. However, structural FC differences are notable in Merlin but not in Sherlock, highlighting the need to analyse FC structure beyond basic connectivity.
Conclusions:
Three-layer hierarchical model highlights the value of a hierarchical processing pipeline, showing that combining multiple methods provides deeper insights than using them individually, and correlation alone is insufficient to fully understand brain FC structure. Hence, validating the significance of hierarchical approach to develop interconnected model.
Emotion, Motivation and Social Neuroscience:
Social Cognition
Higher Cognitive Functions:
Higher Cognitive Functions Other
Modeling and Analysis Methods:
Connectivity (eg. functional, effective, structural) 2
fMRI Connectivity and Network Modeling 1
Methods Development
Keywords:
Cognition
Computational Neuroscience
Data analysis
FUNCTIONAL MRI
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
For human MRI, what field strength scanner do you use?
Which processing packages did you use for your study?
Provide references using APA citation style.
Nguyen, M., Vanderwal, T., & Hasson, U. (2017). Shared understanding is correlated with shared neural responses in the default mode network. bioRxiv, 231019.
Raichle, Marcus E. 2015. “The Brain ’ s Default Mode Network.”
Song, H., Finn, E. S., & Rosenberg, M. D. (2021). Neural signatures of attentional engagement during narratives and its consequences for event memory. Proceedings of the National Academy of Sciences, 118(33), e2021905118.
Zadbood, A. a. (2017). How we transmit memories to other brains: constructing shared neural representations via communication. Cerebral cortex, 4988--5000.