Prevalent brain-behavior modeling practices lead to feature overinterpretation and irreproducibility
Poster No:
1157
Submission Type:
Abstract Submission
Authors:
Brendan Adkinson1, Matt Rosenblatt2, Link Tejavibulya3, Huili Sun3, Stephanie Noble4, Dustin Scheinost1
Institutions:
1Yale School of Medicine, New Haven, CT, 2Yale University, New Havent, CT, 3Yale University, New Haven, CT, 4Northeastern University, Boston, MA
First Author:
Co-Author(s):
Introduction:
'Brain-behavior' predictive models relate differences in brain structure and function to individual behavioral phenotypes (Dubois & Adolphs, 2016). Neurobiological interpretability-the degree to which results can be mapped to known neural circuits and processes-is a key metric of model utility (Dhamala et al., 2023). To obtain more manageable representations of high dimensionality brain connectivity data, the field commonly employs methods such as feature selection and highly penalized models that favor sparsity (Mwangi et al., 2014). Resultant networks are interpreted as biologically meaningful and theoretically interpretable. For instance, a model predicting substance craving might identify a "craving network" composed of specific brain regions purportedly central to the craving experience (Garrison et al., 2023). The field often accepts identified networks as definitive. Here, we show that alternative sets of features can yield similar predictive power and lead to fundamentally different neurobiological insights.
Methods:
Our analyses span 12,203 participants across PNC, HBN, HCPD, and ABCD, both functional and diffusion MRI, and 20 outcomes including age, sex, cognitive abilities, developmental measures, and psychiatric phenotypes. We use a novel paradigm wherein, within each fold of cross-validation, connectome features are divided into ten deciles based on the strength of their association with a target phenotype (Fig. 1). Each non-overlapping decile was subsequently used for connectome-based predictive modeling. Model performance was evaluated across 100 iterations with Pearson's correlation (r), representing the correspondence between predicted and actual behavioral scores. Significance was determined via permutation testing. All imaging data (motion<0.2mm) were processed using BioImage Suite (Papademetris et al., 2006). Connectivity matrices were created using the Shen 268x268 atlas. Brain networks that predicted each decile were compared with those that predicted every other decile using the means of edgewise regression coefficients averaged across folds, normalized by network size.
Results:
Unexpectedly, lower-ranked feature deciles commonly neglected during neurobiological interpretation continued to demonstrate meaningful predictive power (Fig. 1). For PNC executive function, the first decile achieved prediction at r=0.33 (p<0.01, MSE=1.24, q2=0.09), the second and third at r=0.32, the fourth through sixth at r=0.31, and the seventh through tenth at r's=0.18-0.30. This phenomenon survived external validation and was reproducible across imaging modalities and phenotypic domains. The networks underlying predictions were different between deciles (Fig. 2). For PNC executive function, connectivity between the visual association and frontal parietal networks was prominent in decile 1 but less informative for deciles 2 through 5. While networks demonstrated moderate resemblance (r's=0.15-0.81) between neighboring deciles, non-neighboring deciles exhibited marked dissimilarity (r's=-0.21-0.22).
Conclusions:
We show that, surprisingly, multiple subsets of non-overlapping features overlooked by standard methodological approaches can yield similar prediction accuracies but markedly different biological interpretations. For interventions such as TMS, DBS, and real-time fMRI neurofeedback that target brain regions with the strongest associations with symptoms, our results suggest alternative features, some of which may be more anatomically accessible, might yield additional therapeutic effects. Results were robust to the substantial variations (i.e., "dataset shifts") in imaging acquisition, behavioral assessment, recruitment geography, and clinical symptom burden between PNC, HBN, HCPD, and ABCD. Collectively, our findings suggest that prevalent practices lead to overinterpretation and misrepresentation of feature sets, potentially resulting in inaccurate or irreproducible conclusions about the neurobiological bases of the behaviors being studied.
Disorders of the Nervous System:
Neurodevelopmental/ Early Life (eg. ADHD, autism)
Higher Cognitive Functions:
Executive Function, Cognitive Control and Decision Making
Lifespan Development:
Early life, Adolescence, Aging
Modeling and Analysis Methods:
Classification and Predictive Modeling 1
Connectivity (eg. functional, effective, structural) 2
Keywords:
Cognition
Development
FUNCTIONAL MRI
Modeling
Other - feature selection
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Are you Internal Review Board (IRB) certified? Please note: Failure to have IRB, if applicable will lead to automatic rejection of abstract.
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
For human MRI, what field strength scanner do you use?
Which processing packages did you use for your study?
Provide references using APA citation style.
Dhamala, E., Yeo, B. T. T., & Holmes, A. J. (2023). One size does not fit all: Methodological considerations for brain-based predictive modeling in psychiatry. Biological Psychiatry, 93(8), 717–728.
Mwangi, B., Tian, T. S., & Soares, J. C. (2014). A review of feature reduction techniques in neuroimaging. Neuroinformatics, 12(2), 229–244. https://doi.org/10.1007/s12021-013-9204-3
Garrison, K. A., Sinha, R., Potenza, M. N., Gao, S., Liang, Q., Lacadie, C., & Scheinost, D. (2023). Transdiagnostic connectome-based prediction of craving. The American Journal of Psychiatry, 180(6), 445–453. https://doi.org/10.1176/appi.ajp.21121207
Papademetris, X., Jackowski, M. P., Rajeevan, N., DiStasio, M., Okuda, H., Constable, R. T., & Staib, L. H. (2006). BioImage Suite: An integrated medical image analysis suite: An update. The Insight Journal, 2006, 209.