Brain and machine representations converge in relational reasoning
Poster No:
784
Submission Type:
Abstract Submission
Authors:
Luke Hearne1, Conor Robinson2, Luca Cocchi3, Takuya Ito4
Institutions:
1QIMR Berghofer Medical Research Institute, Herston, Queensland, 2QIMR Berghofer Medical Research Institute, Brisbane, QLD, 3Queensland Institute for Medical Research, Brisbane, QLD, 4IBM Research, Yorktown Heights, NY
First Author:
Co-Author(s):
Introduction:
Relational reasoning is the ability to understand and analyse the relationships between entities (Halford et al., 1998). The ability to learn, abstract and generalise relations is fundamental to problem-solving ability and general intelligence. At first glance, modern Transformer-based (Vaswani et al., 2017) Artificial Neural Networks (ANNs) can perform incredibly well on a range of complex cognitive tasks, such as language comprehension, mathematical reasoning and logical reasoning. However, despite these models' impressive and seemingly human-like abilities, they often lack the capacity to reason in a systematic and interpretable way (Mitchell et al., 2023). In the current work, we study the representations that are key for reliable and generalisable relational reasoning in humans and ANNs by combining functional MRI, a well-validated relational reasoning task and neural network modelling.
Methods:
The study used a previously published dataset (Hearne et al., 2017) wherein 40 participants completed the Latin Square Task (LST, see Figure 1A) (Birney et al., 2006) while undergoing fMRI. Brain imaging data were preprocessed using fMRIprep (Esteban et al., 2018), and trial-specific brain activations were estimated. We used an Encoder-only Transformer architecture with four layers, embedding dimension of 160, and a single attention head (Vaswani, 2017). We used a positional encoding parameter that was specifically designed to enable reasoning in 2D. The models were tested on the same 108 puzzles that participants completed in the scanner.
Results:
Task performance
Human performance on the LST decreased as the relational complexity increased (χ2(2)=62.40, p < .001, Figure 1B). Accuracy in the most difficult condition (3-vectors) correlated with performance on a fluid intelligence test (The Ravens Advanced Progressive Matrices, rs=.48, p=.002). The Transformer achieved high accuracy in the test data (Mean performance=97.65%). A complexity effect was also observed (χ2(2)=11.47, p=.003).
Representational Similarity Analysis
Representational Similarity Analysis (RSA) aims to model the similarity structure of activation patterns in response to differing tasks, conditions, or stimuli (Figure 1C) (Diedrichsen & Kriegeskorte, 2017). We observed significant alignment of representations between specific regions of the visual cortex and the first two layers of the Transformer (Figure 1D). In the later Transformer layers, significant alignment was observed within fronto-parietal, cingulo-opercular, and default-mode brain regions. Interestingly, this was not true for the attention mechanism in the Transformer, where significant alignment between the Transformer and empirical brain data were confined to the visual cortex across all layers (Figure 1E).
Representational alignment between brain and machine depends on initialisation of positional encoding
We allowed the Transformer to learn the positional encoding (i.e., the shape of the task grid) with rich (small σ) and lazy (large σ) initializations (Woodworth et al., 2020). Initialisation parameters correlated with test performance, such that low σ (i.e., rich) models performed better than high σ models (rs=.96, p < .001). Likewise, brain-Transformer alignment was related to the initialisation parameter, such that low σ demonstrated higher brain correspondence (unimodal regions, layer 1; rs=.74, p < .001, transmodal regions, layer 4; rs=.51, p=.02).
Human performance on the LST decreased as the relational complexity increased (χ2(2)=62.40, p < .001, Figure 1B). Accuracy in the most difficult condition (3-vectors) correlated with performance on a fluid intelligence test (The Ravens Advanced Progressive Matrices, rs=.48, p=.002). The Transformer achieved high accuracy in the test data (Mean performance=97.65%). A complexity effect was also observed (χ2(2)=11.47, p=.003).
Representational Similarity Analysis
Representational Similarity Analysis (RSA) aims to model the similarity structure of activation patterns in response to differing tasks, conditions, or stimuli (Figure 1C) (Diedrichsen & Kriegeskorte, 2017). We observed significant alignment of representations between specific regions of the visual cortex and the first two layers of the Transformer (Figure 1D). In the later Transformer layers, significant alignment was observed within fronto-parietal, cingulo-opercular, and default-mode brain regions. Interestingly, this was not true for the attention mechanism in the Transformer, where significant alignment between the Transformer and empirical brain data were confined to the visual cortex across all layers (Figure 1E).
Representational alignment between brain and machine depends on initialisation of positional encoding
We allowed the Transformer to learn the positional encoding (i.e., the shape of the task grid) with rich (small σ) and lazy (large σ) initializations (Woodworth et al., 2020). Initialisation parameters correlated with test performance, such that low σ (i.e., rich) models performed better than high σ models (rs=.96, p < .001). Likewise, brain-Transformer alignment was related to the initialisation parameter, such that low σ demonstrated higher brain correspondence (unimodal regions, layer 1; rs=.74, p < .001, transmodal regions, layer 4; rs=.51, p=.02).
·Figure 1
Conclusions:
We employed a validated relational reasoning task to show that Transformer-based ANNs, which use the attention mechanisms to process multiple objects in parallel, excel in achieving generalized performance. Additionally, we observed that models with human-aligned representations-whether pre-defined or learned-achieved the strongest relational reasoning performance. These findings highlight the critical role of interpretable and reliable representations in enabling robust, human-like relational reasoning.
Higher Cognitive Functions:
Reasoning and Problem Solving 1
Modeling and Analysis Methods:
fMRI Connectivity and Network Modeling 2
Keywords:
ADULTS
Cognition
Computational Neuroscience
FUNCTIONAL MRI
Other - reasoning
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
For human MRI, what field strength scanner do you use?
Which processing packages did you use for your study?
Provide references using APA citation style.
Diedrichsen, J., & Kriegeskorte, N. (2017). Representational models: A common framework for understanding encoding, pattern-component, and representational-similarity analysis. PLOS Computational Biology, 13(4), e1005508.
Esteban, O., Markiewicz, C. J., Blair, R. W., Moodie, C. A., Isik, A. I., Erramuzpe, A., Kent, J. D., Goncalves, M., DuPre, E., Snyder, M., Oya, H., Ghosh, S. S., Wright, J., Durnez, J., Poldrack, R. A., & Gorgolewski, K. J. (2018). fMRIPrep: A robust preprocessing pipeline for functional MRI. Nature Methods, 1. https://doi.org/10.1038/s41592-018-0235-4
Halford, G. S., Wilson, W. H., & Phillips, S. (1998). Processing capacity defined by relational complexity: Implications for comparative, developmental, and cognitive psychology. Behavioral and Brain Sciences, 21(6), 803–831.
Hearne, L. J., Cocchi, L., Zalesky, A., & Mattingley, J. B. (2017). Reconfiguration of brain network architectures between resting state and complexity-dependent cognitive reasoning. The Journal of Neuroscience, 0485–17. https://doi.org/10.1523/JNEUROSCI.0485-17.2017
Holyoak, K. J., & Lu, H. (2021). Emergence of relational reasoning. Current Opinion in Behavioral Sciences, 37, 118–124. https://doi.org/10.1016/j.cobeha.2020.11.012
Mitchell, M., Palmarini, A. B., & Moskvichev, A. (2023, November). Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks. arXiv. http://arxiv.org/abs/2311.09247
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is All you Need. Advances in Neural Information Processing Systems, 30. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
Woodworth, B., Gunasekar, S., Lee, J. D., Moroshko, E., Savarese, P., Golan, I., Soudry, D., & Srebro, N. (2020). Kernel and Rich Regimes in Overparametrized Models. Conference on Learning Theory, 3635–3673. http://proceedings.mlr.press/v125/woodworth20a.html
Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., & Kim, Y. (2023). Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks. arXiv Preprint arXiv:2307.02477.