Developing a scalable and easy-to-use neuroimaging container build system
Poster No:
1847
Submission Type:
Abstract Submission
Authors:
Joshua Scarsbrook1, Thuy Dao1, Ashley Stewart1, Aswin Narayanan2, Steffen Bollmann3
Institutions:
1University of Queensland, Brisbane, Queensland, 2Australian National Imaging Facility, Brisbane, Queensland, 3The University of Queensland, Brisbane, Queensland
First Author:
Co-Author(s):
Introduction:
Over the past few years, the Neurodesk project has packaged over 100 neuroimaging software containers. In this abstract, we provide a brief background on how we build containers today and highlight some of the challenges, as well as highlight new solutions we have been developing.
Methods:
Currently, containers in NeuroDesk are built using NeuroDocker to generate a docker file based on a bash script. All the recipes are stored on a GitHub repository and packaged by an automated build process.
However, this approach has significant challenges: accessibility and build time.
A significant issue is that the docker builds can sometimes take hours. With an automated build process, this issue is mitigated but slows the rate at which a user can iterate on new containers. These issues become evident when, for example, a GUI packages needs to be added. This often leads to encountering errors that needed multiple iterations and rebuilding to result in a correctly working container and the current build process does not allow for that.
Another significant issue is accessibility. Right now, the barrier to contributing containers to NeuroDesk limits the amount of community contributed and maintained containers. The goal is to enable more users and tool developers to build and contribute containers. This requires a more straightforward and robust approach to building containers.
However, this approach has significant challenges: accessibility and build time.
A significant issue is that the docker builds can sometimes take hours. With an automated build process, this issue is mitigated but slows the rate at which a user can iterate on new containers. These issues become evident when, for example, a GUI packages needs to be added. This often leads to encountering errors that needed multiple iterations and rebuilding to result in a correctly working container and the current build process does not allow for that.
Another significant issue is accessibility. Right now, the barrier to contributing containers to NeuroDesk limits the amount of community contributed and maintained containers. The goal is to enable more users and tool developers to build and contribute containers. This requires a more straightforward and robust approach to building containers.
Results:
To address these issues the new container build system has the following goals: Split container builds into independent cacheable steps that are shareable across containers and support existing packages without building everything from scratch.
Docker already splits containers into a series of layers, each a snapshot of the filesystem as each step in the build process is completed. This means Docker can cache steps that have yet to be modified. The problem is that modifying a base layer invalidates all layers after it, requiring a complete rebuild. Our solution creates smaller independent layers from each package installed on the system, which can be extracted and reused efficiently.
The next issue is supporting a variety of existing binary packages. Significant progress has already been made in fast cacheable builds in the form of Nix. However, besides accessibility issues, Nix presents an alternative filesystem layer that is incompatible with most Linux applications without modifications. Instead, we use standard package managers like APK from Alpine Linux and APT from Ubuntu/Debian for the caching implementation.
Docker already splits containers into a series of layers, each a snapshot of the filesystem as each step in the build process is completed. This means Docker can cache steps that have yet to be modified. The problem is that modifying a base layer invalidates all layers after it, requiring a complete rebuild. Our solution creates smaller independent layers from each package installed on the system, which can be extracted and reused efficiently.
The next issue is supporting a variety of existing binary packages. Significant progress has already been made in fast cacheable builds in the form of Nix. However, besides accessibility issues, Nix presents an alternative filesystem layer that is incompatible with most Linux applications without modifications. Instead, we use standard package managers like APK from Alpine Linux and APT from Ubuntu/Debian for the caching implementation.
·TinyRange+NeuroContainers Workflow
Conclusions:
Instead of a bash file that describes the build steps we now use simplified YAML files (Figure 1). These YAML files can also be generated and parsed easily for a later interactive web interface. These YAML files are built into Docker layers using TinyRange (https://github.com/tinyrange/tinyrange).
·TinyRange YAML format.
Modeling and Analysis Methods:
Methods Development
Neuroinformatics and Data Sharing:
Workflows 1
Informatics Other 2
Keywords:
Computational Neuroscience
Computing
Data analysis
Data Organization
Design and Analysis
Open Data
Open-Source Code
Open-Source Software
Workflows
1|2Indicates the priority used for review
Abstract Information
By submitting your proposal, you grant permission for the Organization for Human Brain Mapping (OHBM) to distribute your work in any format, including video, audio print and electronic text through OHBM OnDemand, social media channels, the OHBM website, or other electronic publications and media.
The Open Science Special Interest Group (OSSIG) is introducing a reproducibility challenge for OHBM 2025. This new initiative aims to enhance the reproducibility of scientific results and foster collaborations between labs. Teams will consist of a “source” party and a “reproducing” party, and will be evaluated on the success of their replication, the openness of the source work, and additional deliverables. Click here for more information. Propose your OHBM abstract(s) as source work for future OHBM meetings by selecting one of the following options:
Please indicate below if your study was a "resting state" or "task-activation” study.
Healthy subjects only or patients (note that patient studies may also involve healthy subjects):
Was this research conducted in the United States?
Were any human subjects research approved by the relevant Institutional Review Board or ethics panel? NOTE: Any human subjects studies without IRB approval will be automatically rejected.
Were any animal research approved by the relevant IACUC or other animal research panel? NOTE: Any animal studies without IACUC approval will be automatically rejected.
Please indicate which methods were used in your research:
For human MRI, what field strength scanner do you use?
Which processing packages did you use for your study?
Provide references using APA citation style.
Gau, R., Kaczmarzyk, J., Ghosh, S., Bollmann, S., Halchenko, Y., Goncalves, M., Visconti di Oleggio Castello, M., Jarecka, D., Wighton, P., Notter, M., Markiewicz, C., Nielson, D., Vaillant, G., araikes, T., Close, T., Ahn, S., Mitchell, R., Cieslak, M., Scarsbrook, J., … Kent, J. (2024). ReproNim/neurodocker: 1.0.1 (1.0.1) [Software]. Zenodo. https://doi.org/10.5281/zenodo.12675111