Notes on Containers
All jobs in a given Fuzzball workflow run within containers. You can find containers for your jobs in public registries like Docker Hub, Amazon Elastic Container Registry (ECR) Public, or NVIDIA NGC.
You can also create your own containers using Apptainer, Docker, Podman or other container platforms. You can push those containers to public or private registries in either the Singularity Image File (SIF) format, or using the Open Container Initiative (OCI) standard.
If all you need is a public container plus a few extra packages or configuration files, you can skip the external build and registry push. Describe the image in your Fuzzfile and let Fuzzball build it as part of the workflow.
Fuzzball locates and pulls containers for jobs based on URIs you provide in your Fuzzfile. The URIs you use to specify containers follow the format used by Apptainer. A fully specified container URI looks like one of the following:
[PROTOCOL]://<REGISTRY_HOST>/<NAMESPACE>/[REPOSITORY]:<TAG>
[PROTOCOL]://<REGISTRY_HOST>/<NAMESPACE>/[REPOSITORY]@sha256:<HASH>
Fuzzball will accept shorter versions of registry URIs and will try to supply sensible default values where appropriate. You can use the following information to help you specify URIs for your containers.
You must explicitly specify which protocol to use when pulling containers. Fuzzball supports four image URI schemes:
docker://: This protocol pulls containers in OCI format.oras://: This protocol pulls containers in SIF format (using the OCI Registry As Storage or ORAS protocol).fb://andfuzzball://: These pull an image already held in the object cache, including one the workflow builds for itself.
An image URI must use one of those schemes. A Fuzzfile that names an image with any other
scheme, such as file:// or local://, or with no protocol at all (a bare alpine:latest), is
rejected when it is submitted.
The registry host specifies the location from which Fuzzball will pull your container. When using
the docker:// protocol the values under unqualified-search-registries within
/etc/containers/registries.conf (on the Fuzzball Substrate node) are searched in order. On Rocky
9, the default search order is ["registry.access.redhat.com", "registry.redhat.io", "docker.io"].
When you use the oras:// protocol, Fuzzball supplies no default registry – you must specify
the complete registry host.
Here are some example registry hosts that are commonly used:
- docker.io: This is the correct host for Docker Hub.
- nvcr.io: The NVIDIA container registry has a lot of HPC-specific containers many of which are optimized to use GPUs.
- ghcr.io: The GitHub container registry is a popular place for users to host their containers.
- public.ecr.aws: These are containers hosted on the AWS public registry.
- us-west1-docker.pkg.dev: This example shows how you might pull a container from a private registry on Google Cloud. You can use any appropriate URI to pull from a private OCI registry.
OCI registries are organized into namespaces. In the case of common public registries like Docker Hub, these are usually the organization or user who pushed the image. On some registries like Docker Hub there are special official images that can be pulled without supplying a namespace.
This is the name of the actual repository with the container image that you want to pull. Along with the protocol, this value is always required.
You can use this value to specify the version of the container image that you want to access. If
you omit this value, Fuzzball pulls the latest tag. Some images, such as the official Rocky
Linux image, have no latest tag and fail if you pull them without supplying one.
Instead of specifying a tag (which might be a moving target) you can pull your images by hash to
help ensure that you always get a consistent container. The syntax @sha256:<HASH> allows you to do
this. Pulling containers by hash is the most reproducible option.
Digest URIs work in a hand-written Fuzzfile and from the CLI. The Workflow Editor’s image field validates tags only, so paste a digest URI into the YAML tab rather than the Environment tab.
Every compute node reports its CPU architecture as a device: cpu/x86 for x86_64 nodes and
cpu/arm64 for arm64 nodes. On a cluster with both kinds of node, a job whose image was published
for only one architecture must run on a node of that architecture. Placed on the other kind, the
image pull fails, for example with no child with platform linux/arm64 or
SIF primary filesystem architecture mismatch.
Fuzzball does not contact the registry to find out which architectures an image supports. Instead it reads the image URI, and when the URI names an architecture, Fuzzball requests the matching device for every job and service that runs the image. A job that requests no architecture and whose image URI names none can be placed on either kind of node, and the scheduler favors the lower-cost one.
Fuzzball recognizes two patterns in docker:// and oras:// URIs:
| Pattern | Example | Device requested |
|---|---|---|
The tag is, or ends with - or _ followed by, amd64 or x86_64 | oras://ghcr.io/org/blast:2.17.0-1-amd64 | cpu/x86 |
The tag is, or ends with - or _ followed by, arm64, aarch64 or arm64v8 | docker://ghcr.io/org/tool:1.2_aarch64 | cpu/arm64 |
A Docker Hub image in the amd64 namespace | docker://amd64/alpine:3.22 | cpu/x86 |
A Docker Hub image in the arm64v8 namespace | docker://arm64v8/ubuntu:24.04 | cpu/arm64 |
The tag match ignores case and only looks at the end of the tag, so 1.0-AMD64 matches but
amd64-compat and 1.0amd64 do not. A URI with both a tag and a digest, such as
ghcr.io/org/tool:1.0-amd64@sha256:..., is matched on its tag. An architecture in the
repository name, such as
ghcr.io/org/tool-arm64:1.0, is not recognized. An image the workflow builds in its images:
section is judged by the URI in its from field.
Fuzzball adds the device only when the job or service does not already request a cpu/ device,
so an explicit request always wins.
Request the architecture device yourself when the image URI does not follow one of these patterns,
for example a tag such as 1.0, an image pulled by digest alone, or your own image built only
for x86_64. Add the device to the job’s resource section:
version: v4
jobs:
hello:
image:
uri: docker://ghcr.io/org/tool:1.0
script: |
#!/bin/sh
uname -m
resource:
cpu:
cores: 1
memory:
size: 1GB
devices:
cpu/x86: 1
To pin every job and service in a workflow, set the device once under defaults.resource.devices.
A defaults.resource block must also set cpu and memory:
version: v4
defaults:
resource:
cpu:
cores: 1
memory:
size: 1GB
devices:
cpu/x86: 1
A job or service that requests its own architecture device must then request the same one: the
default is added to every job and service that does not already request it, so a job asking for
cpu/arm64 under a cpu/x86 default would ask for both and match no node.
From the CLI, pass the same device with --device:
$ fuzzball run --image docker://ghcr.io/org/tool:1.0 --device cpu/x86=1 -- uname -mAn architecture device is not used up by the jobs that request it, so request a count of 1; any
number of jobs can run on the same node with it. If no provision definition in the cluster offers
the requested architecture, whether you requested it or Fuzzball inferred it from the image URI,
the workflow is rejected when it is submitted with no suitable provision definition found.
Images published for both architectures, such as most official Docker Hub images, need no
architecture request.
Containers can be pushed to and stored on registries in OCI format or in SIF format. The latter uses the OCI Registry as Storage (ORAS) protocol that allows arbitrary data to be stored in container registries.
When you submit a workflow to Fuzzball, Orchestrate creates one or more stages to pull your container(s) using the URI(s) listed in your Fuzzfile. Whatever the format on the registry, the node that runs the stage converts the image to EROFS, the read-only filesystem it runs containers from:
- For an OCI image (
docker://), each layer is downloaded and converted to an EROFS layer as it streams in. Layers are kept in the node’s image store and shared between every image that uses them, so pulling a new tag of an image you already have converts only the layers that changed. - For a SIF image (
oras://), the root filesystem in the SIF is converted to a single EROFS image.
The conversion runs on the node, within the CPU and memory allocated to the stage, and reports its
progress as the stage’s events (fuzzball workflow events). Nothing is unpacked to disk along the
way: neither format requires the image to be extracted and repacked, so the space and time a
conversion needs scale with the size of the image itself rather than with a multiple of it.
Converted images are cached in the object cache as EROFS archives, and a node that later needs the same image downloads only the layers it does not already have. See Automatic image caching for how the cache is keyed and refreshed.
Encrypted SIF images are the exception.
An image with a decryption-secret is downloaded as it is, kept in SIF format and decrypted by the
node when the container starts, so it is neither converted to EROFS nor cached in the object cache.