Establishing the joint reliability bottleneck for reproducible neuroscience

COEX 

Biomarkers of behavior and mental health continue to remain out of reach for cognitive and clinical neuroscience. Suboptimal reliability of functional magnetic resonance imaging (fMRI) has been cited as a primary culprit for the poor reproducibility of brain-based biomarker discovery, leading to unfeasibly large sample-size recommendations. In response, steps are being taken towards optimizing MRI reliability and increasing sample size, but this will not be enough. We show that optimizing biological measurement reliability and increasing sample sizes are necessary but insufficient steps for biomarker discovery; this focus overlooks the ‘other side of the equation,’ namely that human neuroscience studies need to optimize the reliability of behavioral assessments as well. Through a combination of simulation and empirical studies using neuroimaging data, we demonstrate that the joint reliability of both brain and behavioral measurements should be optimized to ensure biomarkers are reproducible and accurate. Even with the best-case scenario - that is, high-reliability neuroimaging measurements and large sample sizes - we show that behavioral data (e.g., symptoms, cognitive measurements, surveys, objective markers of behavior) often have test-retest reliability levels that are suboptimal for the discovery of reproducible brain-behavior associations and biomarkers. Developing new assessments continue to be critical for improving the validity, specificity, and reliability of our characterization of the brain, behavior, and mental health, but in the short term, other solutions can also be pursued. Specifically, we emphasize the power of using existing assessments in ways that optimize their reliability, for example aggregating across repeated measurements or following established guidelines for improving behavioral data quality. These improvements are becoming increasingly feasible with recent innovations in data acquisition (e.g., web- and smart-phone-based administration, ecological momentary assessment, burst sampling, wearable devices, multimodal recordings). We demonstrate that these relatively simple changes to study design can improve behavioral measurement reliability and achieve better biomarker discovery for a fraction of the cost engendered by enormous samples. Although the current study has been motivated by ongoing developments in neuroimaging, prioritizing reliable measurements of behavior can transform human neuroscience and broader scientific and clinical endeavors focused on the brain and behavior.