QSM-CI: A continuously integrated QSM challenge
Poster No:
1815
Submission Type:
Abstract Submission
Authors:
Ashley Stewart1, Thuy Dao1, Korbinian Eckstein2, Steffen Bollmann1
Institutions:
1The University of Queensland, Brisbane, Australia, 2Medical University of Vienna, Berlin, Germany
First Author:
Co-Author(s):
Introduction:
Quantitative susceptibility mapping (QSM) involves a complex post-processing pipeline that includes an ill-posed inverse problem. This makes QSM evaluation challenging, with the first QSM challenge (Langkammer et al., 2018) using a COSMOS (Liu et al., 2009) acquisition as the ground truth and the second (QSM Reconstruction Challenge 2.0 Organizing Committee et al., 2021) using a realistic in-silico head phantom (Marques et al., 2021). Public challenges like these are great opportunities to involve the QSM community and report on the current landscape of QSM algorithms under a common evaluation framework. However, running these challenges is a lot of effort and there is an opportunity to build an open-source platform for continuous QSM evaluation that is maintained by the community and is always available to submit algorithms, metrics or updated test data and remain relevant into the future. The availability of benchmarked algorithms would also mean that algorithms are straightforward to reuse in future investigations. This work presents a continuous QSM challenge platform called QSM-CI, implemented using GitHub and Back4app, and automated using GitHub Actions. Users can submit algorithms to automatically evaluate against a range of simulated datasets using quantitative metrics and a qualitative Elo rating system.
Methods:
The QSM-CI GitHub project includes instructions on how to generate and download test datasets, run QSM algorithms and compute metrics. The datasets include simulations for gradient-echo (GRE) magnitude and phase images and other necessary data required for QSM reconstruction, formatted using the Brain Imaging Data Structure (Gorgolewski et al., 2016) (BIDS). These simulations include a susceptibility phantom consisting of cylinders with constant susceptibility values, data derived from a realistic in-silico head phantom (Marques et al., 2021), and COSMOS acquisitions (Shi et al., 2022). The algorithms include user-submitted instructions to execute QSM reconstruction pipelines against the BIDS dataset. The current submissions include algorithms available in QSMxT (Stewart et al., 2022). The metrics include the quantitative metrics from the second QSM challenge, including RMSE, NRMSE, HFEN, XSIM (Milovic et al., 2022), MAD, CC and GXE, as well as a qualitative Elo metric. After a user submits or updates one of their algorithms via a pull request, a GitHub Action will automatically run their pipeline against the datasets and evaluate it using quantitative metrics, publishing the metrics to a Parse backend hosted on Back4app. Qualitative metrics remain blank until the community contributes to the anonymized evaluation of the final images in a frontend web interface using Niivue for visualization (see Figure 1).
·QSM-CI overview. The community maintains the QSM-CI repository. Pushing changes triggers an automated evaluation. Results are pushed to the web to benefit the community.
Results:
The community of developers maintains the QSM-CI repository. Pushing new algorithms and other updates triggers an automated evaluation. Evaluation results are published to a web interface to benefit QSM users.
Multiple algorithms have been submitted to QSM-CI with metrics automatically computed and published (see Figure 2). QSM reconstructions can be browsed interactively using the online Niivue-powered viewer, and anonymized qualitative evaluations can be given by users.
Multiple algorithms have been submitted to QSM-CI with metrics automatically computed and published (see Figure 2). QSM reconstructions can be browsed interactively using the online Niivue-powered viewer, and anonymized qualitative evaluations can be given by users.
·Representative QSM results published on QSM-CI based on the currently submitted algorithms.
Conclusions:
A platform for automatically evaluating QSM algorithms was developed and implemented using GitHub. A frontend interface was developed to display the computed metrics for each algorithm against the test datasets, along with an interactive frontend that uses Niivue to allow users to rate image quality and determine a qualitative Elo ranking. The QSM-CI project provides a proof-of-concept for an always-online QSM challenge platform to streamline the evaluation of QSM algorithms. Published algorithms are straightforward to reuse in investigations using BIDS-structured data. QSM-CI is published at https://github.com/QSMxT/QSM-CI.
Modeling and Analysis Methods:
Other Methods
Neuroinformatics and Data Sharing:
Databasing and Data Sharing 1
Workflows 2
Novel Imaging Acquisition Methods:
Imaging Methods Other
Keywords:
Computing
Data analysis
MRI
Open Data
Open-Source Code
Open-Source Software
Workflows
Other - QSM, Susceptibility, Challenge
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
Which processing packages did you use for your study?
Provide references using APA citation style.
2. Liu, T., Spincemaille, P., de Rochefort, L., Kressler, B., & Wang, Y. (2009). Calculation of susceptibility through multiple orientation sampling (COSMOS): A method for conditioning the inverse problem from measured magnetic field map to susceptibility source image in MRI. Magnetic Resonance in Medicine, 61(1), 196–204. https://doi.org/10.1002/mrm.21828;
3. QSM Reconstruction Challenge 2.0 Organizing Committee, et al. (2021). QSM reconstruction challenge 2.0: Design and report of results. Magnetic Resonance in Medicine, 86(3), 1241–1255. https://doi.org/10.1002/mrm.28754;
4. Marques, J. P., et al. (2021). QSM reconstruction challenge 2.0: A realistic in silico head phantom for MRI data simulation and evaluation of susceptibility mapping procedures. Magnetic Resonance in Medicine, 86(1), 526–542. https://doi.org/10.1002/mrm.28716;
5. Gorgolewski, K. J., et al. (2016). The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Scientific Data, 3(1), 160044. https://doi.org/10.1038/sdata.2016.44;
6. Shi, Y., Feng, R., Li, Z., Zhuang, J., Zhang, Y., & Wei, H. (2022). Towards in vivo ground truth susceptibility for single-orientation deep learning QSM: A multi-orientation gradient-echo MRI dataset. NeuroImage, 261, 119522. https://doi.org/10.1016/j.neuroimage.2022.119522;
7. Stewart, A. W., et al. (2022). QSMxT: Robust masking and artifact reduction for quantitative susceptibility mapping. Magnetic Resonance in Medicine, 87(3), 1289–1300. https://doi.org/10.1002/mrm.29048;
8. Milovic, C., Tejos, C., Irarrazaval, P., & Shmueli, K. (2022, May). XSIM, a susceptibility-optimised similarity index metric: Validation with 2016 and 2019 QSM reconstruction challenge datasets. Proceedings of the Joint Annual Meeting ISMRM-ESMRMB 2022 & ISMRT Annual Meeting. London, UK. Available: https://archive.ismrm.org/2022/2367.html