MRI economics: Balancing sample size and scan duration in brain wide association studies
Poster No:
1099
Submission Type:
Abstract Submission
Authors:
Shaoshi Zhang1, Leon Ooi2, Csaba Orban1, Thomas Nichols3, Trevor Tan1, Ru Kong1, Scott Marek4, Nico Dosenbach4, Timothy Laumann4, Evan Gordon4, Kwong Hsia Yap1, Fang Ji1, Joanna Su Xian Chong1, Christopher Chen1, Lijun An5, Nicolai Franzmeier6, Römer Sebastian6, Qingyu Hu7, Jianxun Ren7, Hesheng Liu7, Sidhant Chopra8, Carrisa Cocuzza9, Justin Baker10, Juan Helen Zhou1, Danilo Bzdok11, Simon Eickhoff12, Avram Holmes13, Thomas Yeo1
Institutions:
1National University of Singapore, Singapore, Singapore, 2National Univeristy of Singapore, Singapore, Singapore, 3University of Oxford, Oxford, Oxfordshire, 4Washington University, Saint Louis, MO, 5Lund University, Lund, Sweden, 6LMU, Munich, Germany, 7Changping Laboratory, Beijing, China, 8Orygen, Preston, Victoria, 9Yale University, New Haven, CT, 10Harvard Medical School, Boston, MA, 11McGill University, Montreal, Quebec, 12Research Centre Jülich, Jülich, NRW, 13Rutgers University, New Brunswick, NJ
First Author:
Co-Author(s):
Introduction:
A pervasive dilemma in neuroimaging is whether to prioritize sample size or scan time given fixed resources [1]. This trade-off is particularly critical in brain-wide association studies (BWAS) using functional magnetic resonance imaging (fMRI) [2-5]. Ideally, with unlimited resources, both sample size and scan time per participant could be maximized to improve the model performance. However, in reality, researchers have to decide between scanning more participants for shorter durations or fewer participants for longer durations.
Here, we systematically investigate how sample size and scan time in fMRI-based BWAS impact both phenotypic prediction accuracy and cost-effectiveness. Drawing on evidence from multiple large datasets, we provide an empirical reference for future study design.
Here, we systematically investigate how sample size and scan time in fMRI-based BWAS impact both phenotypic prediction accuracy and cost-effectiveness. Drawing on evidence from multiple large datasets, we provide an empirical reference for future study design.
Methods:
First, we used participants from the Adolescent Brain Cognitive Development (ABCD, N=2565) and the Human Connectome Project (HCP, N=792) datasets [6-7]. For each participant, we computed a functional connectivity (FC) matrix using the first N min of resting-state fMRI data. N increased from 2 min to the maximum scan time in steps of 2 minutes. For each dataset, we trained a kernel ridge regression (KRR) model using participants' FC matrices to predict individual-level phenotypic scores as well as cognitive factor scores [8].
We examined the change in prediction accuracies by gradually varying the number of training participants of each dataset (from 200 to 1800 in ABCD; from 200 to 700 in HCP). The number of test participants was kept the same so that the prediction accuracy was comparable across different number of training participants.
Next, we extended our analysis to a total of 9 datasets (6 resting-fMRI datasets + 3 ABCD task-fMRI datasets). We derived a theoretical model that explained empirical prediction accuracies extremely well (R2 > 0.8). Using the theoretical model, we identified the optimal fMRI scan time under various conditions. That is, for a given target prediction accuracy, we calculated the scan time that resulted in the most cost-effective study design, while taking participant overhead costs and scanning fees into account. The optimal scan time was calculated for 108 different conditions: 9 datasets, 3 accuracy targets (80%, 90% or 95% of maximum achievable accuracy), 2 overhead costs ($500, $1000 per participant) and 2 scan costs per hour ($500, $1000). We then calculated the additional costs incurred when a scan time deviated from the optimal scan time.
We examined the change in prediction accuracies by gradually varying the number of training participants of each dataset (from 200 to 1800 in ABCD; from 200 to 700 in HCP). The number of test participants was kept the same so that the prediction accuracy was comparable across different number of training participants.
Next, we extended our analysis to a total of 9 datasets (6 resting-fMRI datasets + 3 ABCD task-fMRI datasets). We derived a theoretical model that explained empirical prediction accuracies extremely well (R2 > 0.8). Using the theoretical model, we identified the optimal fMRI scan time under various conditions. That is, for a given target prediction accuracy, we calculated the scan time that resulted in the most cost-effective study design, while taking participant overhead costs and scanning fees into account. The optimal scan time was calculated for 108 different conditions: 9 datasets, 3 accuracy targets (80%, 90% or 95% of maximum achievable accuracy), 2 overhead costs ($500, $1000 per participant) and 2 scan costs per hour ($500, $1000). We then calculated the additional costs incurred when a scan time deviated from the optimal scan time.
Results:
First, we demonstrated that increasing sample size and/or scan time could lead to higher prediction performance (Pearson's correlation) in cognitive factor scores derived in the ABCD and HCP. Along a black iso-contour line, the prediction accuracy is nearly constant even though scan time and sample size are changing (Fig. 1). We generalized these results by using a larger collection of 9 fMRI datasets (Fig. 2A).
Next, across the 108 scenarios, we observed that the optimal scan time was at least 20 min for 85% of the scenarios (Fig. 2B, right side of the red vertical line). Furthermore, due to the asymmetry of the cost distribution (Fig. 2C), the additional costs incurred with shorter scan times (e.g. ~10 min) were much higher than with longer scan times (e.g. ~30 min). More specifically, scanning 10 min per participant led to an average cost increase of 35.8%, while scanning 30 min per participant was the most cost effective, leading to an average cost increase of only 4.2% (Fig. 2C and D).
Next, across the 108 scenarios, we observed that the optimal scan time was at least 20 min for 85% of the scenarios (Fig. 2B, right side of the red vertical line). Furthermore, due to the asymmetry of the cost distribution (Fig. 2C), the additional costs incurred with shorter scan times (e.g. ~10 min) were much higher than with longer scan times (e.g. ~30 min). More specifically, scanning 10 min per participant led to an average cost increase of 35.8%, while scanning 30 min per participant was the most cost effective, leading to an average cost increase of only 4.2% (Fig. 2C and D).
Conclusions:
In this work, we show that BWAS with less than 10 min fMRI data is not cost efficient. To maximize cost efficiency, it is recommended that researchers doing BWAS should scan at least 20 min. Moreover, given the asymmetry of the cost curve and the potential for incidental loss of scan time (e.g. due to participant discomfort or head motion), we recommend extending the scan time to be longer than 30 min whenever feasible.
Modeling and Analysis Methods:
Classification and Predictive Modeling 1
Connectivity (eg. functional, effective, structural) 2
Keywords:
Cognition
Design and Analysis
Experimental Design
FUNCTIONAL MRI
Machine Learning
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
For human MRI, what field strength scanner do you use?
Which processing packages did you use for your study?
Provide references using APA citation style.
2. Tian, Y., & Zalesky, A. (2021). Machine learning prediction of cognition from functional connectivity: Are feature weights reliable? NeuroImage, 245, 118648.
3. Chen, J., Ooi, L. Q. R., Tan, T. W. K., Zhang, S., Li, J., Asplund, C. L., ... & Yeo, B. T. (2023). Relationship between prediction accuracy and feature importance reliability: An empirical and theoretical study. NeuroImage, 274, 120115.
4. He, T., Kong, R., Holmes, A. J., Nguyen, M., Sabuncu, M. R., Eickhoff, S. B., ... & Yeo, B. T. (2020). Deep neural networks and kernel regression achieve comparable accuracies for functional connectivity prediction of behavior and demographics. NeuroImage, 206, 116276.
5. Schulz, M. A., Yeo, B. T., Vogelstein, J. T., Mourao-Miranada, J., Kather, J. N., Kording, K., ... & Bzdok, D. (2020). Different scaling of linear models and deep learning in UKBiobank brain images versus machine-learning datasets. Nature communications, 11(1), 4238.
6. Smith, S. M., Beckmann, C. F., Andersson, J., Auerbach, E. J., Bijsterbosch, J., Douaud, G., ... & WU-Minn HCP Consortium. (2013). Resting-state fMRI in the human connectome project. Neuroimage, 80, 144-168.
7. Garavan, H., Bartsch, H., Conway, K., Decastro, A., Goldstein, R. Z., Heeringa, S., ... & Zahs, D. (2018). Recruiting the ABCD sample: Design considerations and procedures. Developmental cognitive neuroscience, 32, 16-22.
8. Ooi, L. Q. R., Chen, J., Zhang, S., Kong, R., Tam, A., Li, J., ... & Yeo, B. T. (2022). Comparison of individualized behavioral predictions across anatomical, diffusion and functional connectivity MRI. NeuroImage, 263, 119636.