Notes on Containers

All jobs in a given Fuzzball workflow run within containers. You can find containers for your jobs in public registries like Docker Hub, Amazon Elastic Container Registry (ECR) Public, or NVIDIA NGC.

You can also create your own containers using Apptainer, Docker, Podman or other container platforms. You can push those containers to public or private registries in either the Singularity Image File (SIF) format, or using the Open Container Initiative (OCI) standard.

Fuzzball locates and pulls containers for jobs based on URIs you provide in your Fuzzfile. The URIs you use to specify containers follow the format used by Apptainer. A fully specified container URI looks like one of the following:

[PROTOCOL]://<REGISTRY_HOST>/<NAMESPACE>/[REPOSITORY]:<TAG>

[PROTOCOL]://<REGISTRY_HOST>/<NAMESPACE>/[REPOSITORY]@sha256:<HASH>

Fuzzball will accept shorter versions of registry URIs and will try to supply sensible default values where appropriate. You can use the following information to help you specify URIs for your containers.

You must explicitly specify which protocol to use when pulling containers. Fuzzball currently supports two container-pulling protocols:

• docker://: This protocol pulls containers in OCI format.
• oras://: This protocol pulls containers in SIF format (using the OCI Registry As Storage or ORAS protocol).

An image URI must use one of these two protocols, or refer to an image in the object cache with fuzzball:// or fb://, for example an image built by the workflow. A Fuzzfile that names an image with any other scheme, such as file:// or local://, or with no protocol at all (a bare alpine:latest), is rejected when it is submitted.

The registry host specifies the location from which Fuzzball will pull your container. When using the docker:// protocol the values under unqualified-search-registries within /etc/containers/registries.conf (on the Fuzzball Substrate node) are searched in order. On Rocky 9, the default search order is ["registry.access.redhat.com", "registry.redhat.io", "docker.io"]. Note that when using the oras:// protocol, no default registry is provided—you must explicitly specify the complete registry host.

Here are some example registry hosts that are commonly used:

• docker.io: This is the correct host for Docker Hub.
• nvcr.io: The NVIDIA container registry has a lot of HPC-specific containers many of which are optimized to use GPUs.
• ghcr.io: The GitHub container registry is a popular place for users to host their containers.
• public.ecr.aws: These are containers hosted on the AWS public registry.
• us-west1-docker.pkg.dev: This is just an example showing how you might pull a container from a private registry on Google cloud. You can use any appropriate URI to pull from a private OCI registry.

OCI registries are organized into namespaces. In the case of common public registries like Docker Hub, these are usually the organization or user who pushed the image. On some registries like Docker Hub there are special official images that can be pulled without supplying a namespace.

This is the name of the actual repository with the container image that you want to pull. Along with the protocol, this value is always required.

You can use this value to specify the version of the container image that you want to access. If you omit this value, the “latest” tag will be pulled. Some images (like the official Rocky Linux image) do not have the tag “latest” and will error if you try to pull them without supplying a tag.

Later versions of the Fuzzball workflow editor will complain if you do not supply a tag for your container URI.

Instead of specifying a tag (which might be a moving target) you can pull your images by hash to help ensure that you always get a consistent container. The syntax @sha256:<HASH> allows you to do this. Pulling containers by hash is considered the best strategy for reproducibility.

Containers can be pushed to and stored on registries in OCI format or in SIF format. The latter uses the OCI Registry as Storage (ORAS) protocol that allows arbitrary data to be stored in container registries.

When you submit a workflow to Fuzzball, Orchestrate creates one or more stages to pull your container(s) using the URI(s) listed in your Fuzzfile. Whatever the format on the registry, the node that runs the stage converts the image to EROFS, the read-only filesystem it runs containers from:

• For an OCI image (docker://), each layer is downloaded and converted to an EROFS layer as it streams in. Layers are kept in the node’s image store and shared between every image that uses them, so pulling a new tag of an image you already have converts only the layers that changed.
• For a SIF image (oras://), the root filesystem in the SIF is converted to a single EROFS image.

The conversion runs on the node, within the CPU and memory allocated to the stage, and reports its progress as the stage’s events (fuzzball workflow events). Nothing is unpacked to disk along the way: neither format requires the image to be extracted and repacked, so the space and time a conversion needs scale with the size of the image itself rather than with a multiple of it.

Converted images are cached in the object cache as EROFS archives, and a node that later needs the same image downloads only the layers it does not already have. See Automatic image caching for how the cache is keyed and refreshed.

Encrypted SIF images are the exception. An image with a decryption-secret is downloaded as it is, kept in SIF format and decrypted by the node when the container starts, so it is neither converted to EROFS nor cached in the object cache.