Whole Brain Mapping of Disturbed Semantic Representation in Autism Spectrum Disorder

Poster No:


Submission Type:

Abstract Submission 


Jong-eun Lee1, KYOUNGSEOB BYEON2, Boris Bernhardt3, Michael Milham2, Hyunjin Park4, Seok-Jun Hong5


1Sungkyunkwan University, Suwon-si, Gyeonggi-do, 2Child Mind Institute, New York, NY, 3Montreal Neurological Institute and Hospital, Montreal, Quebec, 4Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon, Kyeonggi-do, 5Sungkyunkwan University, Gyeonggi-do, Suwon-si

First Author:

Jong-eun Lee  
Sungkyunkwan University
Suwon-si, Gyeonggi-do


Child Mind Institute
New York, NY
Boris Bernhardt  
Montreal Neurological Institute and Hospital
Montreal, Quebec
Michael Milham  
Child Mind Institute
New York, NY
Hyunjin Park  
Center for Neuroscience Imaging Research, Institute for Basic Science
Suwon, Kyeonggi-do
Seok-Jun Hong  
Sungkyunkwan University
Gyeonggi-do, Suwon-si


It is widely accepted that the children with autism spectrum disorder (ASD) differently perceive the external world1. The two leading theories, namely Weak Central Coherence2 and Enhanced Perceptual Functioning3 theories, may provide a possible account for this phenomenon, given that both commonly indicate the detail-oriented sensory processing in ASD. Notably, this atypical cognitive style has been also associated with the failure of prototype-based categorical learning in ASD4. However, how all these cognitive symptoms are mechanistically related and how the atypical perception is represented on the functional brain in ASD remain largely unknown. To address these questions, we implemented a novel analytical framework to probe whole-brain semantic representation based on movie-watching fMRI. By estimating low-dimensional bases and dimensionality of this semantic representation, we sought to reveal the underlying mechanism of atypical perceptual process and the neural correlates for altered categorical learning in ASD.


We analyzed movie-watching fMRI from 123 individuals with ASD and 79 with the typically developing brain (movie clip: 'Despicable Me' [600 secs; TR: 0.8s])5. We manually annotated the temporal events of entities and actions that appear in movie frames (Fig 1a) and used them as regressors for semantic representation. Notably, gaze patterns, decoded from fMRI signals of the eyeball6., were used to model the effect of visual attention, providing a continuous weight for each regressor (Fig 1b). All these regressors were then linearly modeled with whole-brain fMRI signals to derive a semantic coefficient matrix (Fig 1c). To test the generalizability, we split the movie-fMRI into training and test datasets using 3-fold cross-validation and checked the correlation between actual and predicted fMRI signals (Fig 1d). Finally, the coefficient matrix underwent dimensionality reduction (i.e., PCA), which resulted in multiple principal components (=PCs) summarizing the semantic representation (Fig 1e).


Fig 1i shows the semantic axis of PC1, which strongly differentiates between social-vs.-non-social categories, which is validated by independent semantic ratings7 (Fig 1j). By projecting the coefficient matrix onto this axis, the extent to which each brain region reflects the social-vs.-non-social axis was measured. The TPOJ, MT+ complex, and PCC are the brain regions that prominently align with one end of axis, while the early visual, superior parietal, and somatosensory areas are closely associated with the opposite end of axis (Fig 1k,l). Next, the group comparison analysis showed that ASD has significant alterations in PC1 scores mostly in DMN areas (Fig 1m-o). To identify a possible account for this group differences, we further estimated the representational dimensionality (=RD) of a semantic coefficient matrix at each brain area8 (Fig 2a). The whole-brain mean RD for each group is shown in Fig 2b. The RD of ASD showed an increased pattern (i.e., a lack of generalizability) in widespread brain areas, with notable effects primarily emerging from DMN (Fig 2c,d). While both groups showed higher RD in unimodal regions compared to transmodal regions, the RD in the ASD group was consistently higher in both modality systems (Fig 2e-g). Finally, the posthoc analysis revealed that the RD significantly mediates the autistic social behavior through altered PC1, suggesting the existence of their potential mechanistic links for the symptom manifestation (Fig 2h).


This study sheds light on how the unique perception of ASD affects their semantic processing, particularly along the social-vs.-nonsocial axis. Our experimental evidence supporting atypical neurocognitive pathway for social impairment in ASD may offer a potential hint to improve the strategy of future behavioral treatments in this prevailing developmental condition.

Disorders of the Nervous System:

Neurodevelopmental/ Early Life (eg. ADHD, autism) 1

Modeling and Analysis Methods:

Activation (eg. BOLD task-fMRI) 2


Computational Neuroscience
Machine Learning

1|2Indicates the priority used for review
Supporting Image: figure1.png
Supporting Image: figure2.png

Provide references using author date format

1. Robertson, C. E. (2017). Sensory perception in autism. Nature Reviews Neuroscience, 18(11), 671-684.
2. Happé, F. (2006). The weak coherence account: detail-focused cognitive style in autism spectrum disorders. Journal of autism and developmental disorders, 36(1), 5-25.
3. Mottron, L. (2006). Enhanced perceptual functioning in autism: An update, and eight principles of autistic perception. Journal of autism and developmental disorders, 36, 27-43.
4. Vanpaemel, W. (2021). Prototype-based category learning in autism: A review. Neuroscience & Biobehavioral Reviews, 127, 607-618.
5. Alexander, L. M. (2017). An open resource for transdiagnostic research in pediatric mental health and learning disorders. Scientific data, 4(1), 1-26.
6. Frey, M. (2021). Magnetic resonance-based eye tracking using deep neural networks. Nature neuroscience, 24(12), 1772-1779.
7. Wang, S. (2023). A large dataset of semantic ratings and its computational extension. Scientific Data, 10(1), 106.
8. Ito, T. (2023). Multitask representations in the human cortex transform along a sensory-to-motor hierarchy. Nature Neuroscience, 26(2), 306-315.