Poster No:
1837
Submission Type:
Abstract Submission
Authors:
Peer Herholz1, Kevin Sitek1
Institutions:
1Northwestern University, Evanston, IL
First Author:
Co-Author:
Introduction:
Artificial intelligence, deep neural networks (DNNs) in particular, has become vital to neuroscientific research, including auditory processing [1]. Here, DNNs are increasingly used to model auditory perception from environmental sounds over music to speech and language [2] by evaluating the predictive performance of activations from different layers of the DNN models in response to auditory stimuli and comparing them to brain and behavioral responses to the same stimuli [3]. However, the growing number of DNNs and their scattered implementations across platforms, software packages and limited meta-data create prominent barriers for researchers. The application of and activation extraction from these DNNs are particularly error-prone and cumbersome.
To address these problems, we developed CAFENAP, a free and open-source python package.
Methods:
CAFENAP streamlines and standardizes the application of and activation extraction from DNNs focused on auditory processing by providing a common interface to a wide range of models and entailing many utility functions. It facilitates the following key tasks:
Model Implementation: It simplifies accessing and applying existing DNNs to auditory data via organizing and utilizing them in a standardized API, including model setup and quality control. While the first allows users to either use DNNs in a pre-trained state (by downloading respective weights) or in a randomly-initialized state, the second provides users with a brief comparison of the data the DNN was trained on and the data it is applied on.
Activation Extraction: After DNNs are applied, activations in response to the auditory stimuli can be automatically extracted from the DNN layers.
Comparative Analysis: Layer activations can be prepared for the comparison with brain and/or behavioral data by either time-averaging or summing activations with a temporal dimension or computing representational dissimilarity matrices [4] per layer. The respective outcomes can then be submitted to different analysis approaches (e.g., regression or representational similarity analysis (RSA) [4] ) in commonly used neuroimaging software packages such as nilearn [5] or mne [6] or general packages such as scikit-learn [7].
To increase reproducibility and reusability, obtained outputs are accompanied by meta-data files including DNN and layer information and follow a BIDS [8] -like organization.
Results:
Using several open datasets, DNN layer activations were extracted and compared to brain and behavioral representations via ridge regression and RSA to demonstrate the utility of CAFENAP.
This replicated the results of prior studies that suggest that DNNs trained on tasks related to auditory processing exhibit a correspondence to auditory processing in biological agents, i.e. brains; that is, earlier and middle layer activations best predicted primary auditory cortex activations and later layer activations non-primary auditory cortex activations [2]. Furthermore, CAFENAP, allowed a straightforward comparison of pre-trained DNNs against randomly-initialized DNNs, which are commonly considered as baseline models [9].
Conclusions:
CAFENAP enables users to utilize DNNs focused on auditory processing in neuroscientific investigations by streamlining and standardizing DNN applications and respective layer activation extraction in a user-friendly and reproducible manner, as well as preparing layer activations for the comparison with brain and behavioral responses.
Neuroinformatics and Data Sharing:
Informatics Other 1
Perception, Attention and Motor Behavior:
Perception: Auditory/ Vestibular 2
Keywords:
Computational Neuroscience
Informatics
Modeling
Open-Source Code
Open-Source Software
Other - auditory perception, deep neural networks
1|2Indicates the priority used for review
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
I accept
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information.
Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
I do not want to participate in the reproducibility challenge.
Please indicate below if your study was a "resting state" or "task-activation” study.
Other
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Healthy subjects
Was this research conducted in the United States?
Yes
Are you Internal Review Board (IRB) certified?
Please note: Failure to have IRB, if applicable will lead to automatic rejection of abstract.
Not applicable
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel?
NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Not applicable
Were any animal research approved by the relevant IACUC or other animal research panel?
NOTE: Any animal studies without IACUC approval will be automatically rejected.
Not applicable
Please indicate which methods were used in your research:
Functional MRI
EEG/ERP
MEG
Behavior
Computational modeling
Provide references using APA citation style.
[1] Kanwisher, N., Khosla, M., & Dobs, K. (2023). Using artificial neural networks to ask ‘why’questions of minds and brains. Trends in Neurosciences, 46(3), 240-254.
[2] Tuckute, G., Feather, J., Boebinger, D., & McDermott, J. H. (2023). Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions. Plos Biology, 21(12), e3002366.
[3] Kell, A. J., Yamins, D. L., Shook, E. N., Norman-Haignere, S. V., & McDermott, J. H. (2018). A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron, 98(3), 630-644.
[4] Kriegeskorte, N., Mur, M., & Bandettini, P. A. (2008). Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2, 249.
[5] Abraham, A., Pedregosa, F., Eickenberg, M., Gervais, P., Mueller, A., Kossaifi, J., ... & Varoquaux, G. (2014). Machine learning for neuroimaging with scikit-learn. Frontiers in neuroinformatics, 8, 71792.
[6] Gramfort, A., Luessi, M., Larson, E., Engemann, D. A., Strohmeier, D., Brodbeck, C., ... & Hämäläinen, M. (2013). MEG and EEG data analysis with MNE-Python. Frontiers in Neuroinformatics, 7, 267.
[7] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., ... & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. the Journal of machine Learning research, 12, 2825-2830.
[8] Gorgolewski, K. J., Auer, T., Calhoun, V. D., Craddock, R. C., Das, S., Duff, E. P., ... & Poldrack, R. A. (2016). The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Scientific data, 3(1), 1-9.
[9] Storrs, K. R., Kietzmann, T. C., Walther, A., Mehrer, J., & Kriegeskorte, N. (2021). Diverse deep neural networks all predict human inferior temporal cortex well, after training and fitting. Journal of cognitive neuroscience, 33(10), 2044-2064.
No