Developing a battery of reliable computational assays for studying psychopathology
Poster No:
533
Submission Type:
Abstract Submission
Authors:
Povilas Karvelis1, Sophie Rashid-Cocker1, Pamina Laessing1,2,3, Oreoluwa Ogundipe4, Jacob Koudys5, Anthony Ruocco6,5, Andreea Diaconescu1,2,7,8
Institutions:
1Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health, Toronto, Canada, 2Institute of Medical Sciences, University of Toronto, Toronto, Canada, 3Max Planck Institute for Biological Cybernetics, Tübingen, Germany, 4Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health, Toronto, Canda, 5Department of Psychological Clinical Science, University of Toronto, Toronto, Canada, 6Department of Psychology, University of Toronto Scarborough, Toronto, Canada, 7Department of Psychiatry, University of Toronto, Toronto, Canada, 8Department of Psychology, University of Toronto, Toronto, Canada
First Author:
Povilas Karvelis
Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health
Toronto, Canada
Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health
Toronto, Canada
Co-Author(s):
Sophie Rashid-Cocker
Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health
Toronto, Canada
Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health
Toronto, Canada
Pamina Laessing
Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health|Institute of Medical Sciences, University of Toronto|Max Planck Institute for Biological Cybernetics
Toronto, Canada|Toronto, Canada|Tübingen, Germany
Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health|Institute of Medical Sciences, University of Toronto|Max Planck Institute for Biological Cybernetics
Toronto, Canada|Toronto, Canada|Tübingen, Germany
Oreoluwa Ogundipe
Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health
Toronto, Canda
Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health
Toronto, Canda
Anthony Ruocco
Department of Psychology, University of Toronto Scarborough|Department of Psychological Clinical Science, University of Toronto
Toronto, Canada|Toronto, Canada
Department of Psychology, University of Toronto Scarborough|Department of Psychological Clinical Science, University of Toronto
Toronto, Canada|Toronto, Canada
Andreea Diaconescu
Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health|Institute of Medical Sciences, University of Toronto|Department of Psychiatry, University of Toronto|Department of Psychology, University of Toronto
Toronto, Canada|Toronto, Canada|Toronto, Canada|Toronto, Canada
Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health|Institute of Medical Sciences, University of Toronto|Department of Psychiatry, University of Toronto|Department of Psychology, University of Toronto
Toronto, Canada|Toronto, Canada|Toronto, Canada|Toronto, Canada
Introduction:
Computational assays, which combine behavioral tasks with computational models, are increasingly used to study mental disorders. However, their development rarely undergoes rigorous psychometric validation, with recent emerging evidence showing the assays to have poor reliability and construct validity (Karvelis et al., 2023). To address this, we are developing a battery of gamified cognitive tasks by iteratively assessing and improving their psychometric properties.
Methods:
Our 4-task battery is aimed at targeting all cognitive domains within the Research Domain Criteria (RDoC), and include: (i) Goin' Fishing (a probabilistic learning task); (ii) Balloon Analogue Risk Task (BART) (a risk taking task); (iii) Space Explorer (a cognitive control and negative valence task), and (iv) Jungle Quest (a social cognition task). To model task performance we use Hierarchical Gaussian Filter, Reinforcement Learning, and Active Inference frameworks.
Tasks are deployed on Cognition.run, and participants are recruited through Prolific for remote data collection. To assess associations with psychopathology, participants complete the Hierarchical Taxonomy of Psychopathology (HiTOP) questionnaire (Kotov et al., 2017). Test-retest reliability is evaluated using the Intraclass Correlation Coefficient (ICC) over two sessions spaced two weeks apart.
Tasks are deployed on Cognition.run, and participants are recruited through Prolific for remote data collection. To assess associations with psychopathology, participants complete the Hierarchical Taxonomy of Psychopathology (HiTOP) questionnaire (Kotov et al., 2017). Test-retest reliability is evaluated using the Intraclass Correlation Coefficient (ICC) over two sessions spaced two weeks apart.
Results:
In our first sample of 189 participants, we found behavioral measures across all tasks to have poor to moderate test-retest reliability (ICC = 0.25 – 0.6), with behavioral measures based on continuous responses (eg., reaction time) showing higher reliability than the ones based on binary ones (eg., response accuracy). Further analysis showed that practice effects impacted reliability only minimally (reducing ICC by only 0.05 in the worst cases), whereas the number of trials emerged as a more significant factor limiting reliability (Fig. 1). In contrast, HiTOP demonstrated strong reliability, with most scales achieving ICCs of 0.75 - 0.9 (Fig. 2).
Conclusions:
Our findings underscore the importance of refining and validating cognitive tasks and computational assays to improve their reliability in assessing cognitive processes underlying mental disorders. This work aims to not only deliver a psychometrically robust battery of tasks but also to serve as a model for iterative task development. Additionally, our results provide empirical support for the reliability of the newly developed HiTOP framework.
Disorders of the Nervous System:
Neurodevelopmental/ Early Life (eg. ADHD, autism)
Psychiatric (eg. Depression, Anxiety, Schizophrenia) 1
Emotion, Motivation and Social Neuroscience:
Social Cognition
Higher Cognitive Functions:
Decision Making 2
Executive Function, Cognitive Control and Decision Making
Keywords:
Cognition
Computational Neuroscience
Design and Analysis
DISORDERS
Experimental Design
Learning
Modeling
Psychiatric Disorders
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
Provide references using APA citation style.
Kotov, R., Krueger, R. F., Watson, D., Achenbach, T. M., Althoff, R. R., Bagby, R. M., ... & Zimmerman, M. (2017). The Hierarchical Taxonomy of Psychopathology (HiTOP): A dimensional alternative to traditional nosologies. Journal of abnormal psychology, 126(4), 454.