Fuzzball Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

Object Cache

The Fuzzball Object Cache is a built-in store for sharing files, data, and container images across workflows and clusters. Objects are addressed with fb:// URIs, organized into group and user namespaces, and managed through the fuzzball object commands or the web UI. Objects can be given a time-to-live (TTL) so the cache cleans up old data automatically.

The object cache is also used transparently to speed up container image pulls — see Automatic image caching below.

Object URIs and namespaces

Objects are referenced with an fb:// URI (the fuzzball:// prefix is also accepted):

fb://group/path/to/object
fb://user/path/to/object

There are two namespaces:

  • fb://group/... — objects shared within your currently selected group. Visible to members of the group.
  • fb://user/... — objects private to your user. Objects placed in the user namespace are keyed to your user and do not appear in the group listing, and vice versa.

A reference may be pinned to a specific content digest:

fb://group/images/myapp.sif@sha256:abc123...

Managing objects from the CLI

The fuzzball object command group manages cached objects.

Upload

$ fuzzball object put ./report.csv fb://group/reports/report.csv
FlagDescription
--ttlTTL duration for the object (e.g., 7d, 24h, 30d)
--recursive, -RRecursively upload all files in a directory
--forceReplace an existing object even if the content digest differs

Default behavior: If an object already exists at the destination path, object put succeeds only when the new content has an identical digest. Fuzzball rejects an attempt to overwrite with different content and returns an “already exists” error.

Force mode: Use --force to replace an existing object with new content:

$ fuzzball object put ./data.csv fb://group/data.csv --force

Recursive upload:

$ fuzzball object put ./dataset fb://user/datasets/run1 --recursive --ttl 7d

If the destination URI omits the object name (it ends in a trailing slash, such as fb://group/), the name is inferred from the source basename. The following two commands are equivalent:

$ fuzzball object put ./report.csv fb://group/
$ fuzzball object put ./report.csv fb://group/report.csv

The same applies recursively — fuzzball object put -R ./dataset fb://group/ uploads the tree under fb://group/dataset/.

Common errors: A 401 Unauthorized error means the CLI is not authenticated; run fuzzball context login. A 403 Forbidden error means your account lacks permission on the target namespace; contact your group administrator.

Download

$ fuzzball object get fb://group/reports/report.csv ./report.csv
FlagDescription
--recursive, -RRecursively download all objects under a prefix

If the local destination names a directory (it ends in a trailing slash, or is . / ..), the name is inferred from the object basename. The following two commands are equivalent:

$ fuzzball object get fb://group/reports/report.csv ./
$ fuzzball object get fb://group/reports/report.csv ./report.csv

The same applies recursively — fuzzball object get -R fb://group/dataset ./ downloads the tree into ./dataset/.

List

$ fuzzball object list fb://group/reports

The CLI walks every page automatically and prints the full result set.

FlagDescription
--versionsInclude version history for each object
--page-size, -pServer batch size per request (does not cap total results)
--max-pages, -mHard cap on pages fetched (0 = no cap)

Describe

$ fuzzball object describe fb://group/reports/report.csv

Shows detailed information about an object, including its size, digest, and versions.

An object that Fuzzball’s hosted agent wrote with object_put in a turn that had read untrusted content, such as a workflow log, another object’s contents or a catalog entry, is labelled Tainted. The label sits on the version that turn wrote, so the version history marks it too, and putting the same content again does not clear it. Treat a tainted object’s content as data, not as instructions. The label is not carried to copies of the content, and an unlabelled object is not proof that its content is clean.

Delete

$ fuzzball object delete fb://group/reports/report.csv
FlagDescription
--force, -fSkip the confirmation prompt

Automatic image caching

When a workflow pulls an external container image — for example via a docker:// or oras:// URI — the node converts the image to EROFS, the format it runs containers from. When image cache write-back is enabled, the node also uploads the converted result to the object cache, so subsequent pulls of the same image are served from the cache instead of the external registry, which speeds up workflow startup and reduces dependence on external registry availability. Write-back is disabled by default, so pulled images are not cached unless an administrator turns it on (see cacheWriteBack below). Images built by a workflow are always stored this way, regardless of the setting.

How image caching works

  1. Pull and convert: When a job or service specifies a container image URI (e.g., docker://nvcr.io/nvidia/pytorch:24.01-py3), the node pulls the OCI or SIF image from the registry and converts it to EROFS.

  2. Cache storage: If write-back is enabled (see cacheWriteBack below), the converted image is uploaded to the object cache as an EROFS archive — the image’s metadata followed by its layers — under a key derived from the image URI and target architecture; the blob itself is content-addressed by digest. The cached image is addressable with an fb:// URI if needed for manual inspection. Write-back is disabled by default, so pulled images are not uploaded unless an administrator turns it on; images built by a workflow are uploaded regardless.

  3. Subsequent pulls: When another workflow requests the same image URI, Fuzzball checks the cache first. If a cached image exists for that URI (and target architecture), the node reads the archive’s metadata and downloads only the layers its own image store lacks, without contacting the external registry. A node that already holds every layer downloads nothing. Only archives uploaded by a node are imported this way: an archive uploaded with fuzzball object put and referenced as an image is refused, since its contents would be filed under image and layer identities it chose itself.

  4. Cache refresh: Cached images are keyed by URI, so a mutable tag (e.g., latest) continues to be served from the cache even if the tag moves in the remote registry. The cache picks up a new version only after the cached entry expires or is deleted, triggering a fresh pull on the next request. That fresh pull resolves the tag at the registry and gets the image the tag currently points to, even on a node that still holds an image converted from an earlier version of the tag. Because the TTL counts from the entry’s last use, a frequently pulled tag can stay cached (and stale) indefinitely — to force a refresh, delete the cached entry or pin the image by digest.

Encrypted SIF images are not cached: they are pulled from their registry by every node that needs them and decrypted when the container starts.

When write-back is enabled, cached entries are scoped to the group of the workflow that pulled the image; workflows in other groups populate their own cache entries on their first pull.

Watching the upload to the cache

When write-back is enabled (it is off by default), the upload in step 2 runs on the node right after the conversion, and the image stage finishes once it is done, so a stage can take noticeably longer than the pull itself on a large image. The upload publishes its own workflow events, so fuzzball workflow events shows where the time went:

$ fuzzball workflow events WORKFLOW --kind image
...
[IMAGE] docker://docker.io/library/python:3.12 (arm64): Pulling image: docker://docker.io/library/python:3.12 (uri=docker://docker.io/library/python:3.12)
[IMAGE] docker://docker.io/library/python:3.12 (arm64): Image conversion started: docker://docker.io/library/python:3.12 (uri=docker://docker.io/library/python:3.12)
[IMAGE] docker://docker.io/library/python:3.12 (arm64): Image conversion complete: docker://docker.io/library/python:3.12 (372.5 MB) (size=390594560, uri=docker://docker.io/library/python:3.12)
[IMAGE] docker://docker.io/library/python:3.12 (arm64): Uploading image to object cache: 372.5 MB (size=390596608)
[IMAGE] docker://docker.io/library/python:3.12 (arm64): Image uploaded to object cache: 372.5 MB, 41.213s (attempts=1, digest=sha256:..., duration=41.213s, size=390596608)
[IMAGE] docker://docker.io/library/python:3.12 (arm64): Image pull complete: docker://docker.io/library/python:3.12 (372.5 MB, 1m3.217s) (duration=1m3.217s, size=390594560, uri=docker://docker.io/library/python:3.12)
[IMAGE] docker://docker.io/library/python:3.12 (arm64): Stage finished (name=train-image)

While the stage is still running, fuzzball workflow events WORKFLOW --follow keeps printing events as they arrive until the workflow finishes. In the Web UI, the image stage’s card shows a progress bar with the percentage of the archive sent so far while the upload is in flight.

The upload is best effort. If the object cache cannot be reached or answers with a server error, Fuzzball tries the upload up to three times, ten seconds apart, and abandons an attempt that moves no bytes for two minutes or gets no reply within five minutes of sending the last byte. A rejection that a retry would not change, such as an authentication failure or a response the node cannot read, fails on the first attempt, and the object cache’s explanation, such as running out of space, is included in the reason. Either way the stage reports Image upload to object cache failed with the reason, and with the number of attempts when more than one was made, then finishes anyway: the job runs on the image the node converted, the image is simply not cached, and the next pull of the same URI tries the upload again. Only a pull the object cache lacks uploads; a pull served from the cache shows no upload events at all.

The upload of pulled images is off by default. Administrators can turn it on cluster-wide with the cacheWriteBack setting described in the central configuration reference.

Cache TTL and cleanup

Cached images respect the object cache’s TTL policies. Automatically cached images carry no per-object TTL, so they inherit the configured default (group, organization, or cluster policy); if none is configured, the system default of 7 days applies. A TTL counts from the object’s last access or update, so entries in active use are retained. For manually uploaded objects, pass the --ttl flag to fuzzball object put to enforce a specific lifetime.

Expired entries are removed by a periodic purge job. Two effects of that are worth knowing, because both are visible from a workflow: when the cache is under disk pressure entries are treated as expired at half their configured TTL and removed oldest-accessed first, so a long-idle image can miss sooner than its TTL suggests; and at critical disk usage Fuzzball rejects new uploads outright.

On a deployment whose storage provisioners are split into segments, each segment stages its own copy of a cached image on its own shared filesystem. The object cache still keeps a single shared copy of each converted image – only the substrate-side staging is duplicated. Segments are configured by storage administrators in provisioner definitions; nothing in a workflow changes. See Segments.

Architecture-specific caching

The object cache stores converted images with architecture-specific keys. When a workflow pulls an external container image (for example docker://ubuntu:22.04), the node converts it to EROFS and caches the resulting archive under a reference that includes the runtime architecture, such as amd64 or arm64. Three consequences follow in mixed-architecture clusters:

  • Separate entries. A cluster with both amd64 and arm64 nodes holds one cache entry per architecture for each logical image.
  • Per-architecture warming. Pre-populating the cache on amd64 nodes does not warm it for arm64 nodes; those nodes pull and convert the image on first use.
  • Architecture-local hit rates. Moving a workflow to a different architecture produces a cache miss until that architecture has pulled the image once.

Single-architecture clusters are unaffected. Mixed-architecture deployments should size cache warming and storage capacity for one copy of each image per architecture.

Web UI

The Fuzzball web UI includes an Object Cache browser for uploading, searching, and inspecting objects across the group and user namespaces, including object metadata. Version history is available from the CLI with fuzzball object list --versions.

Switching between namespaces

The browser toolbar carries a namespace selector offering Group and User (private), alongside the search, filter, and refresh controls. The selector controls which namespace the browser displays. Switching namespace returns you to that namespace’s root, discarding your current folder path:

  • Group — shows all objects in the group namespace (fb://group/...), visible to all members of your group.
  • User (private) — shows objects in your personal user namespace (fb://user/...), visible only to you.

Click either option to switch between namespaces. The object list, breadcrumb, and all operations update to reflect the selected namespace.

The breadcrumb bar at the top of the browser shows your current location:

  • In the group namespace, the breadcrumb starts with Group followed by any path segments (e.g., Group > datasets > images).
  • In the user namespace, it starts with User (private) followed by path segments (e.g., User (private) > experiments > run-42).

Click any segment to navigate to that level. Clicking the namespace label (Group or User (private)) returns you to the root of that namespace. All operations in the browser — viewing object details, deleting objects, and creating references — act on the currently selected namespace only.

Searching and filtering objects

Use the Search box above the object table to filter by name within the current folder. The Filters menu next to it narrows the list further:

  • Type — object format, inferred from content type or file extension (CSV, JSON, YAML, HDF5, Parquet, Text, Markdown, PyTorch, or Other).
  • Updated by — the user who last updated the object. Only users present in the current listing appear as filter options.
  • TTL status — Expiring soon (less than 24 hours remaining), Active, or No expiry.
  • Scope — My Objects narrows the list to objects you last updated. Group Objects has no effect on its own (selecting it alone, both, or neither all show everything).

Search and filters apply within the current namespace and folder. While any of them is active, the browser loads the full result set in the background (shown as Loading all objects… (N loaded)) instead of stopping at whatever page has already loaded.

Uploading objects

Click Upload object to open the Upload Object dialog. Uploads are always relative to your current location: the object is stored in the namespace and folder you are browsing, so you do not need to retype the full path.

The dialog has the following fields:

  • File — the file or files to upload. Drag them onto the drop zone, or use Choose File / Choose Folder. You can select multiple files, and folder uploads preserve their directory structure.
  • Object Name — the name the object is stored under. For a single file it is filled in automatically from the file’s name; edit it to store the object under a different name. (Shown only when uploading a single file on its own; folder and multi-file uploads keep each file’s own name.)
  • Prefix — an optional subfolder, relative to your current location. The dialog shows that location — for example, Relative to Group / datasets / images. Leave it blank to upload directly into the current folder, or enter a value such as run-42/ to place the object in a subfolder beneath the current folder.
  • TTL — an optional lifetime after which the object is automatically removed: 1 day, 7 days, 30 days, or No expiry (the default).

Click Upload in the dialog to store the object. For example, while browsing Group > datasets > images with an Object Name of cat.png and a Prefix of train/, the file is stored at fb://group/datasets/images/train/cat.png.

When uploading multiple files or a folder, each file is tracked separately: a progress table lists every file with its status (Queued, Hashing, Uploading, Finalizing, Done, or Failed). A failed file doesn’t stop the rest of the batch — once it finishes, the dialog reports how many files succeeded and how many failed.

For the command-line equivalent, see Upload.

Previewing objects

Click an object row to open its preview alongside the table. Fuzzball renders JSON, YAML, Markdown, and plain-text files as text, CSV files as a table, and common image formats inline.

  • Files over 1 MiB show This file is larger than the preview limit with a Preview anyway action to load the full preview.
  • Unsupported file types show Preview not available for this file type.
  • If the preview fails to load, the panel shows the underlying error message.

From the preview panel you can also Download the object, Copy reference (its fb:// URI), open View details for its full metadata and TTL, or Open in new tab.

URL bookmarks and sharing

The browser encodes the namespace and current path in the URL query string:

  • Group namespace root: no prefix parameter
  • User namespace root: ?prefix=user/
  • Group namespace path: ?prefix=group/datasets/images
  • User namespace path: ?prefix=user/experiments/run-42

You can bookmark or share these URLs; opening a URL with a prefix parameter selects the correct namespace and navigates to the specified path.

URLs pointing to the user namespace are personal. If you share a ?prefix=user/... URL with a colleague, they will see their own user namespace at that path, not yours. To share objects with others, store them in the group namespace.