Print Close

Spatially specific reward signals in visual areas during closed-loop naturalistic interaction

Presented During:

Advanced investigations of perceptual and attention systems

Wednesday, June 26, 2024: 11:30 AM - 12:45 PM

COEX

Room: Grand Ballroom 104-105

Poster No:

2555

Submission Type:

Abstract Submission

Authors:

Royoung Kim¹^,2, Jiwoong Park¹^,2^,3, Kendrick Kay⁴, WON MOK SHIM¹^,2^,3

Institutions:

¹Center for Neuroscience Imaging Research, Institute of Basic Science (IBS), Suwon, Korea, Republic of, ²Dept. of Intelligent Precision Healthcare Convergence, Sungkyunkwan University (SKKU), Suwon, Korea, Republic of, ³Dept. of Biomedical Engineering, Sungkyunkwan University (SKKU), Suwon, Korea, Republic of, ⁴Center for Magnetic Resonance Research, Department of Radiology, University of Minnesota, Minneapolis, MN

First Author:

Royoung Kim
Center for Neuroscience Imaging Research, Institute of Basic Science (IBS)|Dept. of Intelligent Precision Healthcare Convergence, Sungkyunkwan University (SKKU)
Suwon, Korea, Republic of|Suwon, Korea, Republic of

Co-Author(s):

Jiwoong Park
Center for Neuroscience Imaging Research, Institute of Basic Science (IBS)|Dept. of Intelligent Precision Healthcare Convergence, Sungkyunkwan University (SKKU)|Dept. of Biomedical Engineering, Sungkyunkwan University (SKKU)
Suwon, Korea, Republic of|Suwon, Korea, Republic of|Suwon, Korea, Republic of

Kendrick Kay
Center for Magnetic Resonance Research, Department of Radiology, University of Minnesota
Minneapolis, MN

Won Mok Shim
Center for Neuroscience Imaging Research, Institute of Basic Science (IBS)|Dept. of Intelligent Precision Healthcare Convergence, Sungkyunkwan University (SKKU)|Dept. of Biomedical Engineering, Sungkyunkwan University (SKKU)
Suwon, Korea, Republic of|Suwon, Korea, Republic of|Suwon, Korea, Republic of

Introduction:

Our visual system actively gathers information from the environment to facilitate actions aligned with behavioral goals, and reward information plays a significant role in connecting sensory inputs with optimal actions. While previous animal studies have demonstrated that visually responsive brain regions, ranging from the primary visual cortex (V1) [1,2] to the frontal eye field (FEF) [3], show sensitivity to reward values within their receptive fields [4,5], the mechanism through which potential rewards modulate visual representations during goal-directed actions in dynamic naturalistic settings remains poorly understood, particularly in humans. To address this gap, we introduced an innovative Minecraft-based 3D interactive task, where participants strategically plan to achieve goals while navigating a virtual world. We hypothesized that rewards would elicit spatially specific responses, prioritizing the processing of important stimuli and supporting efficient visually guided actions towards them.

Methods:

We collected data from shepherding task and population receptive field (pRF) experiments (N=7) as part of the 7T Naturalistic Perception, Action, and Cognition (NatPAC) dataset. In the shepherding task, participants continuously formulated plans and made actions within a Minecraft-based 3D interactive environment to herd as many sheep (i.e. potential rewards) as possible to a specified location while avoiding puddles and a fox (Fig 1A). This naturalistic task promotes deep engagement-a characteristic not typically associated with conventional laboratory tasks.
We constructed a voxel-wise pRF model [6,7] from the pRF experiment data (Fig 1B) to estimate the pRFs for each voxel. Combining these pRFs with high-precision gaze estimates obtained through a novel head motion-corrected eye-tracking analysis method, we generated predicted BOLD signals based on gaze-centered visual inputs (Fig 1C, D). Subtracting these predicted signals from the actual BOLD signals allowed us to remove the effects of retinotopic visual stimulation (Fig 1D). We then modeled the residuals with regressors related to rewards. To examine the spatial specificity of rewards, we segmented the visual field into eight radial bins and explained their variance with the reward-related regressors (Fig 2A).

·Figure 1. Task and analysis procedure.

Results:

Figs 2B and 2C show the spatial specificity of the coefficients for reward loss in V1, V2, V3, and the intraparietal sulcus (IPS) and FEF. Significant spatial specificity for reward loss was observed in V1 (p = .029), V3 (p = .037), IPS (p = .007), and FEF (p = .009). These results indicated that reward loss exhibits distinct spatial tuning in the visual field, with the actual BOLD signal being larger than the predicted BOLD signal based solely on retinotopic visual input. Notably, this effect was not evident for other attention-grabbing events, indicating the spatially specific effects were not due to attention in general but were specific to the anticipation of reward. In addition, our correlation analysis revealed that a higher level of reward-based spatial specificity was associated with more successful loss-avoidant behaviors in V1 (r = 0.70, p = .08), V2 (r = 0.71, p = .07), and V3 (r = 0.60, p = .15) (Fig 2D). This suggests that the integration of reward information into visual representations may facilitate optimal behaviors that effectively maximize rewards.

·Figure 2. Spatial specificity of reward loss.

Conclusions:

Using an immersive naturalistic task and high-precision eye-tracking data, we discovered reward-specific spatial responses within both visual and frontoparietal regions. This suggests that visual representations encoded in early visual areas and higher-order frontoparietal regions may effectively integrate cognitive information, such as the reward values associated with stimuli. This study illustrates the interplay of perception, action, and cognition as an integrative process unfolding in naturalistic settings, which is rarely studied in conventional laboratory paradigms.

Perception, Attention and Motor Behavior:

Perception: Visual ¹

Perception and Attention Other ²

Keywords:

FUNCTIONAL MRI

Vision

Other - Reward; Naturalistic; Eye-tracking; pRF; Visual cortex; Visual field; Closed-loop; Goal-directed action

^1|2Indicates the priority used for review

Provide references using author date format

[1] Stănişor, L., Togt, C. van der, Pennartz, C. M. A., & Roelfsema, P. R. (2013). A unified selection signal for attention and reward in primary visual cortex. Proceedings of the National Academy of Sciences, 110(22), 9136–9141.
[2] Henschke, J. U., Dylda, E., Katsanevaki, D., Dupuy, N., Currie, S. P., Amvrosiadis, T., Pakan, J. M. P., & Rochefort, N. L. (2020). Reward Association Enhances Stimulus-Specific Representations in Primary Visual Cortex. Current Biology, 30(10), 1866-1880.e5.
[3] Glaser, J. I., Wood, D. K., Lawlor, P. N., Ramkumar, P., Kording, K. P., & Segraves, M. A. (2016). Role of expected reward in frontal eye field during natural scene search. Journal of Neurophysiology, 116(2), 645–657.
[4] Hayhoe, M. M. (2016). Vision and Action. Annual Review of Vision Science, 3(1), 1–25.
[5] Frankó, E., Seitz, A. R., & Vogels, R. (2010). Dissociable Neural Effects of Long-term Stimulus–Reward Pairing in Macaque Visual Cortex. Journal of Cognitive Neuroscience, 22(7), 1425–1439.
[6] Dumoulin, S. O., & Wandell, B. A. (2008). Population receptive field estimates in human visual cortex. NeuroImage, 39(2), 647–660.
[7] Kay, K. N., Winawer, J., Mezer, A., & Wandell, B. A. (2013). Compressive spatial summation in human visual cortex. Journal of Neurophysiology, 110(2), 481–494.