Neural Foundations of Habitual Behavior: Integrating Reinforcement Learning and Choice Frequency
Poster No:
727
Submission Type:
Abstract Submission
Authors:
Stephan Nebe1, Viktor Timokhov1, Philippe Tobler1, Hugo Fluhr1
Institutions:
1University of Zurich, Zurich, Switzerland
First Author:
Co-Author(s):
Introduction:
Habits permeate our daily lives. While there are models of how habits develop at the behavioral and neural level, there is no consensus on how this process takes place. One crucial part of this mechanism might be the influence of behavioral repetition, which was frequently underexplored in previous studies (Miller et al., 2019). We addressed this issue by building a new computational model of habitual behavior in an instrumental learning paradigm that manipulates behavioral frequency independent from reinforcement of choice (Nebe et al., 2024). This suggests that the frequency of past choice, independent of reinforcement by rewards, influences habitual behavior. The present study aims to replicate and extend these behavioral findings on the role of choice frequency by examining its neural correlates in fMRI.
Methods:
For this pre-registered study (https://osf.io/wbs8n/), we adapted the binary-choice Reward Pairs task (Nebe et al., 2024) for fMRI (e.g., longer intertrial intervals and presentation durations, sequential presentation of stimuli). Participants (N=72; 34 male; m(age)=23.7 years) performed this task that involves learning stimulus-response-outcome associations by trial-and-error under time pressure while undergoing fMRI. Computational models of reinforcement learning (RL), a choice kernel (CK) operationalizing the effect of previous choice frequency, and a model combining RL and CK were fitted to behavioral data during training (192 trials) and subsequently used to explain choice behavior during a test phase (136 trials). Models were compared via BIC values. We used generalized linear mixed effects models (GLMM) to estimate the effect of choice frequency during training on choices in the test phase on a trial-by-trial level. At the neural level, we examined whether BOLD responses in a priori regions of interest (ROI; ventral striatum, ventromedial prefrontal cortex for RL; putamen, parietal cortex for CK) correlate with RL and CK values during performance of the test phase. Preprocessing of fMRI data was done using fMRIprep (Esteban et al., 2019) and included quality control using MRIQC (Esteban et al., 2017). First-level general linear models included onset regressors of the stimuli, motor response, choice feedback, and intertrial interval, in addition to several nuisance regressors. The onset regressor of the presentation of stimuli was parametrically modulated by the subjective values computed by the RL and CK models. A two-sided false-discovery-rate corrected significance threshold of α=.005 was used. After applying the pre-registered exclusion criteria, behavioral analyses were based on a sample size of n=67 and fMRI analyses on n=62.
Results:
Model comparisons indicated a model combining RL and CK to outperform all others when predicting choices in the test phase (BIC_RL+CK=481.87; BIC_RL=497.48; BIC_CK=718.07). In the GLMM, a statistically significant effect of choice frequency indicated that participants preferred stimuli that had been chosen more frequently before, regardless of their reinforcement value (z=2.875, p=0.004). ROI analyses of the fMRI data showed BOLD correlates of RL values in ventral striatum (z=3.79), but no other BOLD correlate in the ROIs survived FDR correction.
·Figure 1. Choice proportions during test phase.
Conclusions:
Our results replicated previous effects of choice frequency on behavior and an association of RL values with BOLD responses in ventral striatum. However, no BOLD correlate of prior choice frequency-based subjective value was found in our ROIs. Thus, the neural basis for an effect of previous choice frequency could not be identified. This leaves open the possibility of representation in a different brain region or by the strength of connectivity between neurons. Future analyses could focus on functional connectivity between candidate regions (e.g., putamen and (pre-)motor cortex) and examine the evolution of CK values during the learning phase.
Higher Cognitive Functions:
Executive Function, Cognitive Control and Decision Making 1
Learning and Memory:
Learning and Memory Other
Modeling and Analysis Methods:
Activation (eg. BOLD task-fMRI) 2
Keywords:
Cognition
Computational Neuroscience
FUNCTIONAL MRI
Modeling
Other - habits
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
For human MRI, what field strength scanner do you use?
Which processing packages did you use for your study?
Provide references using APA citation style.
Esteban, O., Markiewicz, C. J., Blair, R. W., Moodie, C. A., Isik, A. I., Erramuzpe, A., Kent, J. D., Goncalves, M., DuPre, E., Snyder, M., Oya, H., Ghosh, S. S., Wright, J., Durnez, J., Poldrack, R. A., & Gorgolewski, K. J. (2019). fMRIPrep: A robust preprocessing pipeline for functional MRI. Nature Methods, 16(1), 111. https://doi.org/10.1038/s41592-018-0235-4
Miller, K. J., Shenhav, A., & Ludvig, E. A. (2019). Habits without values. Psychological Review, 126(2), 292–311. https://doi.org/10.1037/rev0000120
Nebe, S., Kretzschmar, A., Brandt, M. C., & Tobler, P. N. (2024). Characterizing Human Habits in the Lab. Collabra: Psychology, 10(1), 92949. https://doi.org/10.1525/collabra.92949