Open large-scale collaborations to improve reproducibility

When a change in analysis methods leads to different results, what does it mean for our research findings? In this presentation, I will discuss reproducibility in neuroimaging. Neuroimaging studies are characterized by a very large analysis space, and practitioners usually have to choose between different software, software versions, algorithms, parameters, etc. For many years, these choices have been regarded as implementation details, but it is becoming increasingly clear that the exact choices of analytical strategy can lead to different and sometimes contradictory results. I will review our recent efforts to better understand and manage the different sources of this analytical variability in neuroimaging. 

Several theories of visual perception have proposed that the brain samples sensory information via rhythmic alternation of low and high excitability periods in the alpha band (8-13Hz). However, some null findings have recently questioned this hypothesis. An ongoing #EEGManyLabs initiative aims to replicate an influential study on this topic (Mathewson et al., 2009) and provide new evidence on a perennial question regarding the pre-stimulus alpha fluctuations and the detectability of a visual target. Nine independent Labs are collecting EEG data during a masked visual detection task in which a target is briefly presented at a contrast threshold level while participants report on its presence or absence. Here, we will present the most recent results and discuss some insight into the #EEGManyLabs collaboration.  

Transcranial Magnetic Stimulation coupled with Electroencephalography (TMS-EEG) is a powerful technique to probe cortical excitability and connectivity. While widely applied in clinical and basic research to investigate neurophysiological processes and cognitive states, methodological variability in both data acquisition and data processing remains a major challenge, hindering reproducibility and interpretability of outcome measures, e.g., TMS-evoked potentials (TEPs). To address these issues, the Team for TMS-EEG (T4TE) initiative proposes a large-scale, international collaborative approach aimed at improving methodological rigor, reproducibility, and standardization of TMS-EEG measures. Specifically, this talk will focus on the first study launched by T4TE, investigating how methodological variability impacts TEPs following stimulation of the primary motor cortex (M1), a widely studied cortical area. This study has gathered the contribution of more than 30 labs that will work together to reach three aims: 1) Impact of methodological variability on spatio-temporal features of M1-TEPs: By documenting lab-specific acquisition and analysis pipelines, we will assess variability in M1-TEP components across labs and disentangle the contributions to such variability of data acquisition versus data processing; 2) Modulation of M1-TEPs and Cortico-spinal Excitability: We will evaluate how methodological differences influence findings on the relationship between TEP amplitude and motor-evoked potentials (MEPs); 3) Influence of Prestimulus Oscillations: We will investigate how methodological differences influence findings on the association between TEP amplitude and prestimulus oscillatory power, focusing on alpha-band (mu-rhythm) activity in motor areas. By overcoming statistical power limitations and standardizing methodologies, this initiative aims to accelerate advancements in TMS-EEG research, facilitating its clinical and basic science applications. 

Non-invasive brain stimulation (NIBS) refers to a collection of methods for indirectly stimulating neural tissue from across the scalp, including transcranial magnetic stimulation, transcranial electrical stimulation, and transcranial ultrasound stimulation. NIBS techniques are used in a wide range of basic and clinical applications, ranging from investigating the physiology of the motor cortex to treatment of psychiatric and neurological disorders. Despite the widespread use of NIBS, a major challenge facing the field is the low reproducibility of outcomes associated with stimulation, including both neural and behavioural responses. Compounding the issue of reproducibility, there is no widely adopted standard for organising and describing the data collected in NIBS experiments, limiting the direct comparisons of outcomes across the NIBS field.
In response to similar reproducibility issues, the neuroimaging community introduced the brain imaging data structure (BIDS) nearly a decade ago, a standardised format for organising and describing neuroimaging data. The BIDS standard includes recommended file formats, data organisation guidelines, and methods for capturing metadata associated with neuroimaging experiments. Originally designed for data related to magnetic resonance imaging, the BIDS specification has expanded to include data from electroencephalography, magnetoencephalography, positron emission tomography and many other neuroimaging modalities. BIDS-formatted data is also accepted in widely used open-access data repositories, such as openneuro.org, which includes >1,000 neuroimaging datasets.
In this talk, I will overview recent efforts to extend the BIDS specification to include data associated with NIBS experiments. NIBS is unique from other data modalities included in BIDS in that it represents interventions which are often coupled with other neuroimaging or behavioural outcomes, but does not represent a data recording modality itself. As such, a flexible framework for integrating NIBS-related metadata (e.g., position of the coil/electrode/transducer, stimulation parameters etc.) with other data modalities has been developed. Furthermore, so-called ‘offline’ NIBS protocols are not associated with any neuroimaging data files, requiring the development of unique file formats for describing these interventions. Incorporating NIBS within the BIDS framework will provide an avenue to increase data sharing practices across the field and allow the development of automated analysis pipelines and quality assurance protocols. By improving open science practices, we hope to directly address the reproducibility challenge hampering the NIBS field.