MultiPy: An open–source python toolbox for multimodal real-time analysis
Poster No:
1979
Submission Type:
Abstract Submission
Authors:
Franziska Klein1, Julien Räker1, Seyed Amir Ali Mohaddes1, Lennart Scheidsteger1, Frerk Müller-von Aschwege1, Andreas Hein2
Institutions:
1OFFIS e.V. - Institute for Computer Science, Oldenburg, Germany, 2University of Oldenburg, Oldenburg, Germany
First Author:
Co-Author(s):
Introduction:
In mobile brain-computer interfaces (BCI) and neurofeedback (NFB), real-time data processing is essential for interpreting neural signals (Klein, 2024). Most current systems are based on unimodal approaches such as electroencephalography (EEG) or functional near-infrared spectroscopy (fNIRS) (Sitaram et al., 2017). EEG offers high temporal resolution and captures fast electrical brain activity, but is sensitive to noise and artifacts and provides limited spatial information. In contrast, fNIRS measures hemodynamic responses such as blood flow and oxygenation and offers higher spatial specificity and lower sensitivity to artifacts, but with lower temporal resolution and delays due to the slow nature of hemodynamic signals (Pinti et al., 2020; Liu et al., 2021; Li et al., 2022).
Individually, these modalities face limitations, however, combining these two techniques could provide complementary insights, improve accuracy, robustness, and might overcome challenges such as BCI illiteracy (Liu et al., 2021; Li et al., 2022, Klein et al., 2023). Despite these advantages, real-time multimodal systems remain technically challenging due to a lack of existing software for simultaneous integration and analysis (Klein et al., 2023). To fill this gap, we introduce MultiPy, an open-source Python toolbox that seamlessly integrates and analyzes EEG and fNIRS data in real-time, improving accessibility and fostering collaboration in developing multimodal solutions for BCI and NFB applications.
Individually, these modalities face limitations, however, combining these two techniques could provide complementary insights, improve accuracy, robustness, and might overcome challenges such as BCI illiteracy (Liu et al., 2021; Li et al., 2022, Klein et al., 2023). Despite these advantages, real-time multimodal systems remain technically challenging due to a lack of existing software for simultaneous integration and analysis (Klein et al., 2023). To fill this gap, we introduce MultiPy, an open-source Python toolbox that seamlessly integrates and analyzes EEG and fNIRS data in real-time, improving accessibility and fostering collaboration in developing multimodal solutions for BCI and NFB applications.
Methods:
MultiPy is a graphical user interface (GUI) toolbox designed to integrate real-time data streams from multiple modalities via the Lab Streaming Layer (LSL) protocol (Kothe et al., 2024). It consists of three interconnected modules, each providing real-time capabilities and the flexibility to work independently or in combination.
The fNIRS module facilitates visualization and preprocessing of fNIRS data (Klein, 2024). Key features include channel quality assessment, channel pruning, and light intensity to hemoglobin concentration conversion using the modified Beer-Lambert law (mBLL). The module also includes motion artifact correction and extracerebral systemic activity correction, with or without the use of short-distance channels. In addition, it supports the combination of EEG and fNIRS data using a generalized linear model (GLM), where EEG features can serve as regressors for fNIRS signals, enabling EEG-based fNIRS GLM analysis.
Besides real-time signal visualization, the EEG module provides robust preprocessing tools for EEG data, including artifact subspace reconstruction (ASR) and temporal filtering.
The feature extraction and machine learning (ML) module focuses on extracting time-domain and frequency-domain features from fNIRS and EEG signals. It integrates these multimodal features in real-time using ML algorithms.
The fNIRS module facilitates visualization and preprocessing of fNIRS data (Klein, 2024). Key features include channel quality assessment, channel pruning, and light intensity to hemoglobin concentration conversion using the modified Beer-Lambert law (mBLL). The module also includes motion artifact correction and extracerebral systemic activity correction, with or without the use of short-distance channels. In addition, it supports the combination of EEG and fNIRS data using a generalized linear model (GLM), where EEG features can serve as regressors for fNIRS signals, enabling EEG-based fNIRS GLM analysis.
Besides real-time signal visualization, the EEG module provides robust preprocessing tools for EEG data, including artifact subspace reconstruction (ASR) and temporal filtering.
The feature extraction and machine learning (ML) module focuses on extracting time-domain and frequency-domain features from fNIRS and EEG signals. It integrates these multimodal features in real-time using ML algorithms.
·Figure 1: GUI of MultiPy in the fNIRS module, showing channel selection, real-time data visualization and available preprocessing options.
Results:
Preliminary tests of MultiPy have already been able to integrate and synchronize EEG and fNIRS data streams in real time. The toolbox successfully deployed preprocessing pipelines to improve signal quality and used advanced algorithms to mitigate motion artifacts and noise. Its intuitive interface and user-friendly design make it an accessible tool and provide a versatile platform for developing and testing multimodal real-time applications.
Conclusions:
The open-source MultiPy toolbox facilitates real-time integration and analysis of EEG and fNIRS data and is designed to support any device that enables real-time data streaming via the LSL protocol (Kothe et al., 2024), thus aiming to increase the reach of NFB and BCI applications. This promotes accessibility and standardization in the field and allows researchers to easily develop and test multimodal approaches. Since MultiPy is open-source software, the community can participate in the development, provide feedback, and make MultiPy a versatile and comprehensive tool for advancing neurotechnology.
Modeling and Analysis Methods:
EEG/MEG Modeling and Analysis
Motion Correction and Preprocessing
Motor Behavior:
Brain Machine Interface
Novel Imaging Acquisition Methods:
EEG 2
NIRS 1
Keywords:
Computational Neuroscience
Electroencephaolography (EEG)
Machine Learning
Near Infra-Red Spectroscopy (NIRS)
Other - Neurofeedback & Brain Computer Interfaces
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
Which processing packages did you use for your study?
Provide references using APA citation style.
Klein, F., Kohl, S. H., Lührs, M., Mehler, D. M. A., & Sorger, B. (2024). From lab to life: challenges and perspectives of fNIRS for haemodynamic-based neurofeedback in real-world environments. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 379(1915), 20230087.
Li, R., Yang, D., Fang, F., Hong, K. S., Reiss, A. L., & Zhang, Y. (2022). Concurrent fNIRS and EEG for Brain Function Investigation: A Systematic, Methodology-Focused Review. Sensors (Basel, Switzerland), 22(15), 5865.
Klein, F., Müller-Von Aschwege, F., Elfert, P., Räker, J., Philipsen, A., Braun, N., Selaskowski, B., Wiebe, A., Guth, M., Spallek, J., Seuss, S., Storey, B., Geppert, L. N., Lück, I., & Hein, A. (2023). Developing Advanced AI Ecosystems to Enhance Diagnosis and Care for Patients with Depression. Studies in health technology and informatics, 309, 18–22.
Klein F. (2024). Optimizing spatial specificity and signal quality in fNIRS: an overview of potential challenges and possible options for improving the reliability of real-time applications. Frontiers in neuroergonomics, 5, 1286586.
Pinti, P., Tachtsidis, I., Hamilton, A., Hirsch, J., Aichelburg, C., Gilbert, S., & Burgess, P. W. (2020). The present and future use of functional near-infrared spectroscopy (fNIRS) for cognitive neuroscience. Annals of the New York Academy of Sciences, 1464(1), 5–29.
Liu, Z., Shore, J., Wang, M., Yuan, F., Buss, A., Zhao. X. (2021). A systematic review on hybrid EEG/fNIRS in brain-computer interface. Biomed. Signal Process. Control. 68:102595.
Kothe, C., Shirazi, S. Y., Stenner, T., Medine, D., Boulay, C., Grivich, M. I., Mullen, T., Delorme, A., & Makeig, S. (2024). The Lab Streaming Layer for Synchronized Multimodal Recording. bioRxiv : the preprint server for biology, 2024.02.13.580071.