Evaluation of Brain Extraction Tools on UK Biobank T1-Weighted MRI Data
Poster No:
1631
Submission Type:
Abstract Submission
Authors:
Fidel Alfaro Almagro1, Stephen Smith1, Frederik Lange1
Institutions:
1WiN FMRIB - University of Oxford, Oxford, United Kingdom
First Author:
Co-Author(s):
Introduction:
Accurate brain extraction is a critical preprocessing step in neuroimaging studies. For example, for future re-analysis of UK Biobank data, we need to optimise robustness and accuracy of the initial brain extraction. Multiple tools are available, but no single method performs optimally across all use cases. We compared 10 brain extraction methods, focusing on their robustness, accuracy, and computational efficiency. As the gold-standard, we used brain masks derived from nonlinear warping to a standard space (brain mask), followed by manual QC (i.e., the UK Biobank preprocessing pipeline (1)). For future use in UK Biobank, ROBEX is likely to be the best option for initial brain extraction, with final brain outline being determined by the MMORF warp to OMM standard space.
Methods:
We focused on following the FAST cost functions (Feasible, Aligned with UK Biobank (UKB) data and goals, Suitable for BIP framework, and Testable) (2). Hence, we compared ten methods, in general using default settings.
1.- BET: FSL's BET (3) run with the recursive option (-R flag).
2.- SS simple: SynthStrip (4) on a T1 without any previous processing.
3.- SS head: SynthStrip (4) on a T1 that had been previously FLIRTed (6 dof, whole head) to the OMM-1 standard space. This was done because SynthStrip performs slightly worse if the head in native space is very tilted.
4.- voting: Median of masks 1, 2, and 3, although only "good" masks are used for that median. A "good" mask is defined as ones that have a Dice index higher than 0.9 with the standard space (OMM-1) brain mask flirted (12 dof) to subject space.
5.- ROBEX: ROBEX v1.2 (5) (brain extraction tool from FreeSurfer) on the T1.
6.- MMORF: Brain mask defined in standard space (OMM-1 (6)) taken to T1 space with an MMORF (7) warp (the number of subjects with this brain mask is smaller due to MMORF having been run on a subset of the total number of subjects).
7.- AFNI: 3dSkullStrip (7) (from AFNI 22.1.02) on the T1 after "de-obliquing" it.
8.- ANTs: antsBrainExtraction.sh (8) (from ANTs suite, commit 4db64a74175a16238969020ae0a74aaf980c730d on the 20th of Nov 20, 2024) run in 3D, using OMM-1 as a template.
9.- HD-Bet: HD-BET v1.1 (9) on the T1 run on a single CPU with the recommended options -device cpu -mode fast -tta 0
10.- BSE: BSE (from BrainSuite 23a (10)) on the T1.
We performed brain extraction on 67,428 T1w images from UKB. We used the brain masks from the UKB Pipeline for these subjects as our reference, because these masks had been checked with semi-manual QC. For each method, we generated binary brain masks and assessed their performance by calculating different stats of the Dice coefficients to measure the overlap with the reference masks. We also counted the number of masks with Dice below 0.9 (these numbers reflect disagreement with our gold standard, although not necessarily an objective bad performance as different methods may have been built with a different gold standard), the number of times the tool did not generate any mask, and the number of times there was no overlap between the resulting brain mask and our gold standard (Dice of 0). Additionally, we averaged the processing time for each method over 1,000 runs to evaluate CPU usage.
The code used to produce this abstract is available in: https://git.fmrib.ox.ac.uk/falmagro/bip_brain_extraction_tests.git
1.- BET: FSL's BET (3) run with the recursive option (-R flag).
2.- SS simple: SynthStrip (4) on a T1 without any previous processing.
3.- SS head: SynthStrip (4) on a T1 that had been previously FLIRTed (6 dof, whole head) to the OMM-1 standard space. This was done because SynthStrip performs slightly worse if the head in native space is very tilted.
4.- voting: Median of masks 1, 2, and 3, although only "good" masks are used for that median. A "good" mask is defined as ones that have a Dice index higher than 0.9 with the standard space (OMM-1) brain mask flirted (12 dof) to subject space.
5.- ROBEX: ROBEX v1.2 (5) (brain extraction tool from FreeSurfer) on the T1.
6.- MMORF: Brain mask defined in standard space (OMM-1 (6)) taken to T1 space with an MMORF (7) warp (the number of subjects with this brain mask is smaller due to MMORF having been run on a subset of the total number of subjects).
7.- AFNI: 3dSkullStrip (7) (from AFNI 22.1.02) on the T1 after "de-obliquing" it.
8.- ANTs: antsBrainExtraction.sh (8) (from ANTs suite, commit 4db64a74175a16238969020ae0a74aaf980c730d on the 20th of Nov 20, 2024) run in 3D, using OMM-1 as a template.
9.- HD-Bet: HD-BET v1.1 (9) on the T1 run on a single CPU with the recommended options -device cpu -mode fast -tta 0
10.- BSE: BSE (from BrainSuite 23a (10)) on the T1.
We performed brain extraction on 67,428 T1w images from UKB. We used the brain masks from the UKB Pipeline for these subjects as our reference, because these masks had been checked with semi-manual QC. For each method, we generated binary brain masks and assessed their performance by calculating different stats of the Dice coefficients to measure the overlap with the reference masks. We also counted the number of masks with Dice below 0.9 (these numbers reflect disagreement with our gold standard, although not necessarily an objective bad performance as different methods may have been built with a different gold standard), the number of times the tool did not generate any mask, and the number of times there was no overlap between the resulting brain mask and our gold standard (Dice of 0). Additionally, we averaged the processing time for each method over 1,000 runs to evaluate CPU usage.
The code used to produce this abstract is available in: https://git.fmrib.ox.ac.uk/falmagro/bip_brain_extraction_tests.git
Results:
We show the resulting comparison in Table 1 and Figure 1
·Table 1: Evaluation of accuracy, robustness, and CPU efficiency of the different tools. *Processing time for MMORF is mostly GPU time. **MMORF was only run on a subset of subjects.
·Figure 1: Raincloud plots showing the distribution of Dice coefficients for 10 brain extraction methods evaluated on 67,428 UK Biobank T1-weighted images.
Conclusions:
Based on our results, we observed large differences in how the brain extraction methods perform. HD-BET achieved very high accuracy and robustness, but at the cost of longer processing time. In contrast, ROBEX offered a good balance between high accuracy and robustness vs. processing time, making it suitable for large-scale application. FSL's BET and AFNI showed lower accuracy and more variability. These findings will help us select the most appropriate brain extraction tool based on our needs for accuracy and efficiency in BIP (the upcoming new UKB core pipeline).
Modeling and Analysis Methods:
Image Registration and Computational Anatomy
Segmentation and Parcellation 1
Neuroinformatics and Data Sharing:
Brain Atlases
Workflows 2
Keywords:
Informatics
Open-Source Software
STRUCTURAL MRI
Workflows
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
For human MRI, what field strength scanner do you use?
Which processing packages did you use for your study?
Provide references using APA citation style.
(2) Alfaro-Almagro, F. Smith, S. M. Have your say in the design of BIP — The UK Biobank Brain Imaging Pipeline! Organization for Human Brain Mapping, Jun 2024, Seoul, South Korea. Abstract Book 5: OHBM 2024 Annual Meeting. doi:10.52294/001c.120595, pp. 3672-3674.
(3) Smith, S.M., 2002. Fast robust automated brain extraction. Human brain mapping, 17(3), pp.143-155.
(4) Hoopes, A., Mora, J.S., Dalca, A.V., Fischl, B. and Hoffmann, M., 2022. SynthStrip: skull-stripping for any brain image. NeuroImage, 260, p.119474.
(5) Iglesias, J.E., Liu, C.Y., Thompson, P.M. and Tu, Z., 2011. Robust brain extraction across datasets and comparison with publicly available methods. IEEE transactions on medical imaging, 30(9), pp.1617-1634.
(6) Arthofer, C., Smith, S.M., Douaud, G., Bartsch, A., Alfaro-Almagro, F., Andersson, J. and Lange, F.J., 2024. Internally-consistent and fully-unbiased multimodal MRI brain template construction from UK Biobank: Oxford-MM. Imaging Neuroscience.
(7) Lange, F.J., Arthofer, C., Bartsch, A., Douaud, G., McCarthy, P., Smith, S.M. and Andersson, J.L., 2024. MMORF—FSL’s MultiMOdal Registration Framework. Imaging Neuroscience, 2, pp.1-30.
(8) Avants, B.B., Epstein, C.L., Grossman, M. and Gee, J.C., 2008. Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Medical image analysis, 12(1), pp.26-41.
(9) Isensee, F., Schell, M., Pflueger, I., Brugnara, G., Bonekamp, D., Neuberger, U., Wick, A., Schlemmer, H.P., Heiland, S., Wick, W. and Bendszus, M., 2019. Automated brain extraction of multisequence MRI using artificial neural networks. Human brain mapping, 40(17), pp.4952-4964.
(10) Shattuck, D.W. and Leahy, R.M., 2002. BrainSuite: an automated cortical surface identification tool. Medical image analysis, 6(2), pp.129-142.