RICE-Net: Multimodal post-treatment tumor classification
Poster No:
1833
Submission Type:
Abstract Submission
Authors:
Robin Peretzke1,2, Maximilian Fischer1,2, Marlin Hanstein1, Lars Wessel3, Christine Jungk4, Laila Koenig3, Peter Neher1, Klaus Maier-Hein1
Institutions:
1Department of Medical Image Computing, German Cancer Research Center, Heidelberg, Baden-Württemberg, Germany, 2Medical Faculty Heidelberg, Heidelberg University, Heidelberg, Baden-Württemberg, Germany, 3Department of Radiation Oncology, University Hospital of Heidelberg, Heidelberg, Baden-Württemberg, Germany, 4Department of Neurosurgery, Heidelberg University Hospital, Heidelberg, Baden-Württemberg, Germany
First Author:
Robin Peretzke
Department of Medical Image Computing, German Cancer Research Center|Medical Faculty Heidelberg, Heidelberg University
Heidelberg, Baden-Württemberg, Germany|Heidelberg, Baden-Württemberg, Germany
Department of Medical Image Computing, German Cancer Research Center|Medical Faculty Heidelberg, Heidelberg University
Heidelberg, Baden-Württemberg, Germany|Heidelberg, Baden-Württemberg, Germany
Co-Author(s):
Maximilian Fischer
Department of Medical Image Computing, German Cancer Research Center|Medical Faculty Heidelberg, Heidelberg University
Heidelberg, Baden-Württemberg, Germany|Heidelberg, Baden-Württemberg, Germany
Department of Medical Image Computing, German Cancer Research Center|Medical Faculty Heidelberg, Heidelberg University
Heidelberg, Baden-Württemberg, Germany|Heidelberg, Baden-Württemberg, Germany
Marlin Hanstein
Department of Medical Image Computing, German Cancer Research Center
Heidelberg, Baden-Württemberg, Germany
Department of Medical Image Computing, German Cancer Research Center
Heidelberg, Baden-Württemberg, Germany
Lars Wessel
Department of Radiation Oncology, University Hospital of Heidelberg
Heidelberg, Baden-Württemberg, Germany
Department of Radiation Oncology, University Hospital of Heidelberg
Heidelberg, Baden-Württemberg, Germany
Christine Jungk
Department of Neurosurgery, Heidelberg University Hospital
Heidelberg, Baden-Württemberg, Germany
Department of Neurosurgery, Heidelberg University Hospital
Heidelberg, Baden-Württemberg, Germany
Laila Koenig
Department of Radiation Oncology, University Hospital of Heidelberg
Heidelberg, Baden-Württemberg, Germany
Department of Radiation Oncology, University Hospital of Heidelberg
Heidelberg, Baden-Württemberg, Germany
Peter Neher
Department of Medical Image Computing, German Cancer Research Center
Heidelberg, Baden-Württemberg, Germany
Department of Medical Image Computing, German Cancer Research Center
Heidelberg, Baden-Württemberg, Germany
Klaus Maier-Hein
Department of Medical Image Computing, German Cancer Research Center
Heidelberg, Baden-Württemberg, Germany
Department of Medical Image Computing, German Cancer Research Center
Heidelberg, Baden-Württemberg, Germany
Introduction:
In the posttherapeutic situation, intracranial tumors pose a complex diagnostic and therapeutic challenge, particularly in differentiating tumor recurrence from radiation-induced changes, such as Radiation-Induced Contrast Enhancements (RICE) on MRI scans, whose occurrence varies and may depend on the radiation techniques used, tumor types, and treatment plans [1-4]. Accurate differentiation is critical for treatment planning, but it often requires interdisciplinary tumor board meetings, which are time-intensive and pose significant challenges [5]. In this work, we aim to develop a multimodal deep learning approach leveraging conventional MRI and radiotherapy (RT) plans to address this issue. Deep learning has demonstrated promising results across a wide range of medical imaging tasks, offering potential to automate and enhance lesion classification [6-9].
Methods:
Figure 1 illustrates the workflow for integrating RICE-NET, trained on different modalities, into the tumor board for decision support.
Our training dataset for RICE-NET comprises 82 subjects who presented with post-treatment tumor appearances following radiation therapy (RT). Among these, 50 cases were identified as tumor recurrence, while 32 were classified as radiation-induced changes. For each subject, we utilized MRI T1 MPRAGE with contrast sequences from two time points-post-operation and first occurrence of RICE-along with the corresponding radiation therapy plans, as input data for model training.
We developed a ResNet-based neural network using a multichannel input composed of these three modalities, applying data augmentation techniques to enhance the training process. Model evaluation was performed using five-fold cross-validation, with the F1 score serving as the primary performance metric. To assess the model's robustness and examine the influence of individual input modalities, we conducted ablation experiments by systematically excluding each modality and analyzing its impact on overall performance.
Our training dataset for RICE-NET comprises 82 subjects who presented with post-treatment tumor appearances following radiation therapy (RT). Among these, 50 cases were identified as tumor recurrence, while 32 were classified as radiation-induced changes. For each subject, we utilized MRI T1 MPRAGE with contrast sequences from two time points-post-operation and first occurrence of RICE-along with the corresponding radiation therapy plans, as input data for model training.
We developed a ResNet-based neural network using a multichannel input composed of these three modalities, applying data augmentation techniques to enhance the training process. Model evaluation was performed using five-fold cross-validation, with the F1 score serving as the primary performance metric. To assess the model's robustness and examine the influence of individual input modalities, we conducted ablation experiments by systematically excluding each modality and analyzing its impact on overall performance.
·Integration of the multimodal trained network RICE-Net to tumorboards for decision support
Results:
Our proposed multimodal deep learning model demonstrated strong performance in distinguishing true tumor progression from RICE. Using all three modalities (MRI post-operative scans, RICE scans, and RT plans) as input, the model achieved an F1 score of 0.9613, indicating high accuracy in lesion classification.
To assess the impact of individual modalities on model performance, we conducted ablation studies by excluding one modality at a time. With MRI RICE scans and RT plans, the F1 score was 0.9613, indicating similar performance to the full model. Using only MRI post-operative scans and RT plans, the model achieved an F1 score of 0.9487, showing a slight decline compared to the full model. When limited to MRI post-operative scans and RICE scans, the model attained an F1 score of 0.7579, highlighting the critical role of RT plans.
To assess the impact of individual modalities on model performance, we conducted ablation studies by excluding one modality at a time. With MRI RICE scans and RT plans, the F1 score was 0.9613, indicating similar performance to the full model. Using only MRI post-operative scans and RT plans, the model achieved an F1 score of 0.9487, showing a slight decline compared to the full model. When limited to MRI post-operative scans and RICE scans, the model attained an F1 score of 0.7579, highlighting the critical role of RT plans.
Conclusions:
Our results show that the proposed deep learning model can effectively differentiate true tumor progression from RICE. The model's ability to achieve high F1 scores (0.94) directly after surgery indicates its potential to provide doctors with early insights into tumor recurrence risk and type, enabling more tailored treatment strategies.
Interestingly, the model performed comparably using MRI RICE scans and RT plans alone compared to using all modalities including post-op, highlighting the critical role of these modalities in classification and guiding future improvements. However, the relatively small cohort size may limit generalizability. Future experiments will address this by incorporating more subjects, exploring ablations and leveraging time-aware models.
Overall, the classification of RICE versus true tumor progression using our deep learning approach demonstrates high accuracy, offering a reliable solution to this challenging diagnostic problem and supporting tumor boards in making more efficient and informed decisions.
Interestingly, the model performed comparably using MRI RICE scans and RT plans alone compared to using all modalities including post-op, highlighting the critical role of these modalities in classification and guiding future improvements. However, the relatively small cohort size may limit generalizability. Future experiments will address this by incorporating more subjects, exploring ablations and leveraging time-aware models.
Overall, the classification of RICE versus true tumor progression using our deep learning approach demonstrates high accuracy, offering a reliable solution to this challenging diagnostic problem and supporting tumor boards in making more efficient and informed decisions.
Neuroinformatics and Data Sharing:
Workflows 2
Informatics Other 1
Keywords:
Computational Neuroscience
Data analysis
Workflows
Other - Deep Learning
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
For human MRI, what field strength scanner do you use?
Which processing packages did you use for your study?
Provide references using APA citation style.
Eichkorn, T., Lischalk, J. W., Schwarz, R., et al. (2024). Radiation-induced cerebral contrast enhancements strongly share ischemic stroke risk factors. International Journal of Radiation Oncology Biology Physics, 118(9), 1192–1205. https://doi.org/10.1016/j.ijrobp.2023.12.044
Ton, M., Deng, M., Meixner, E., et al. (2024). Efficacy and toxicity of photon, proton, and carbon ion radiotherapy in the treatment of intracranial solitary fibrous tumor/hemangiopericytoma. Radiation Oncology, 19(42). https://doi.org/10.1186/s13014-024-02434-5
Jaspers, et al. (2023). Early and late contrast-enhancing lesions after photon radiotherapy for IDH-mutated grade 2 diffuse glioma. Radiotherapy and Oncology. https://doi.org/10.1016/j.radonc.2023.109674
Krupinski, E. A., et al. (2018). A new software platform to improve multidisciplinary tumor board workflows and user satisfaction: A pilot study. Journal of Pathology Informatics, 9(1), 26.
Huang, S.-C., Jensen, M., Yeung-Levy, S., et al. (2024). Multimodal foundation models for medical imaging: A systematic review and implementation guidelines.
Ulrich, C., et al. (2024). MultiTalent: A multi-dataset approach to medical image segmentation. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. https://doi.org/10.1007/978-3-031-43898-1_62
Du, Y., et al. (2023). SegVol: Universal and interactive volumetric medical image segmentation. ArXiv Preprint.
Ma, J., et al. (2024). Segment anything in medical images. Nature Communications, 15, 654. https://doi.org/10.1038/s41467-024-44824-z