Poster No:
614
Submission Type:
Abstract Submission
Authors:
Chaery Park1, Jongwan Kim1
Institutions:
1Jeonbuk National University, Jeonju, South Korea
First Author:
Chaery Park
Jeonbuk National University
Jeonju, South Korea
Co-Author:
Jongwan Kim
Jeonbuk National University
Jeonju, South Korea
Introduction:
Recently, there has been growing interest in the ability of large language models (LLM) such as ChatGPT, to accurately predict human emotions, particularly for potential applications in clinical settings. However, limited research has explored whether neural responses in humans align with those of artificial intelligence models like ChatGPT. To advance our understanding, we evaluated ChatGPT's interpretation of emotions associated with textual descriptions of video segments and aimed to determine whether brain activity observed in people while watching movies could be used to predict ChatGPT's ratings of valence and arousal.
Methods:
We reanalyzed the fMRI data collected by Lee and Chen (2022a, 2022b) and Lee, Chen, and Hasson (2022c), who investigated the neural activity associated with watching short films and spoken recall. In this study, 20 participants watched 10 short audiovisual films across two fMRI sessions. Additionally, we asked ChatGPT to rate the emotions while reading the descriptions of each segment for each film. The emotion scales were composed of the following eight descriptors: sad, anxious, calm, irritated, melancholic, joyful, relaxed, happy, and angry (7-point Likert scale). We averaged the ratings of the scales based on two dimensions: valence (positive and negative) and arousal (high and low), and used them as labels for training set and test set.
After preprocessing the fMRI data including motion correction, slice timing correction, normalization, and smoothing, we conducted cross-participant classifications to investigate whether the neural representation of affective states was consistent across participants. Cross-participant classification performance was evaluated using 20-fold cross-validation, where data from one participant was left out for testing in each fold. Bonferroni correction was applied to account for multiple comparisons.
We also performed multidimensional scaling (MDS) to examine whether regions of interest (ROIs) were represented within the core affect space. We Procrustes-rotated the initial MDS solution to align it with the design matrix of valence and arousal and calculated Pearson's correlation values between the design matrix and the rotated MDS solution coordinates for each dimension. A permutation test was used to assess significance.
Results:
For valence, cross-participant classification revealed significant activations (ps < .05, Bonferroni corrected) in the left middle frontal gyrus (MFG), left medial prefrontal cortex (mPFC), bilateral anterior and posterior orbitofrontal cortex (OFC; lateral and medial parts), insula, posterior cingulate cortex (PCC), left superior temporal gyrus (STG), and left precuneus among the 22 a priori regions of interest (ROIs) selected. For arousal, significant activations (ps < .05, Bonferroni corrected) were observed in the right MFG, left mPFC, bilateral anterior and posterior OFC (lateral and medial parts), insula, PCC, right precuneus, left STG, and left middle temporal gyrus (MTG).
Based on permutation-correction, MDS results revealed significant valence in the bilateral precuneus and the right anterior OFC (ps < .05). For the arousal dimension, significant regions included the bilateral middle frontal gyrus, bilateral mPFC, bilateral anterior OFC, right posterior OFC, left lateral OFC, bilateral PCC, and bilateral precuneus (ps < .05).
Conclusions:
This study explored if ChatGPT's valence and arousal ratings can be predicted based on human neural responses and if they are consistent across participants. We were able to predict valence and arousal at several ROIs and revealed that the representation of valence and arousal was consistent across individuals based on ChatGPT ratings and human brain responses. These findings underscore the utility of LLM-generated affective ratings as a robust tool for understanding the neural representation of emotions, providing a novel approach to studying affective neuroscience with naturalistic stimuli.
Emotion, Motivation and Social Neuroscience:
Emotional Perception 1
Modeling and Analysis Methods:
Activation (eg. BOLD task-fMRI)
Classification and Predictive Modeling
Multivariate Approaches 2
Keywords:
Computational Neuroscience
Emotions
Multivariate
Perception
Other - Naturalistic Stimuli
1|2Indicates the priority used for review
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
I accept
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information.
Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
I do not want to participate in the reproducibility challenge.
Please indicate below if your study was a "resting state" or "task-activation” study.
Task-activation
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Healthy subjects
Was this research conducted in the United States?
Yes
Are you Internal Review Board (IRB) certified?
Please note: Failure to have IRB, if applicable will lead to automatic rejection of abstract.
Yes, I have IRB or AUCC approval
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel?
NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Yes
Were any animal research approved by the relevant IACUC or other animal research panel?
NOTE: Any animal studies without IACUC approval will be automatically rejected.
Not applicable
Please indicate which methods were used in your research:
Functional MRI
Behavior
For human MRI, what field strength scanner do you use?
3.0T
Which processing packages did you use for your study?
SPM
Provide references using APA citation style.
H. Lee, J. Chen, A generalized cortical activity pattern at internally generated mental context boundaries during un- guided narrative recall, eLife 11 (2022a) e73693, doi: 10.7554/eLife.73693.
H. Lee, J. Chen, Predicting memory from the network structure of naturalistic events, Nat. Commun. 13 (2022b) 4235, doi: 10.1038/s41467- 022- 31965- 2.
H. Lee, J. Chen, U. Hasson Film festival. OpenNeuro, (2022c), doi: 10.18112/openneuro.ds004042.v1.0.0.
No