Object Cache
The Fuzzball Object Cache is a built-in store for sharing files, data, and container
images across workflows and
clusters. Objects are addressed with fb:// URIs, organized into group and user
namespaces, and managed through the fuzzball object commands or the web UI. Objects can
be given a time-to-live (TTL) so the cache cleans up old data automatically.
The object cache is also used transparently to speed up container image pulls — see Automatic image caching below.
Objects are referenced with an fb:// URI (the fuzzball:// prefix is also accepted):
fb://group/path/to/object
fb://user/path/to/object
There are two namespaces:
fb://group/...— objects shared within your currently selected group. Visible to members of the group.fb://user/...— objects private to your user. Objects placed in the user namespace are keyed to your user and do not appear in the group listing, and vice versa.
A reference may be pinned to a specific content digest:
fb://group/images/myapp.sif@sha256:abc123...
The fuzzball object command group manages cached objects.
$ fuzzball object put ./report.csv fb://group/reports/report.csv| Flag | Description |
|---|---|
--ttl | TTL duration for the object (e.g., 7d, 24h, 30d) |
--recursive, -R | Recursively upload all files in a directory |
--force | Replace an existing object even if the content digest differs |
Default behavior: If an object already exists at the destination path, object put succeeds
only when the new content has an identical digest. Fuzzball rejects an attempt to overwrite with
different content and returns an “already exists” error.
Force mode: Use --force to replace an existing object with new content:
$ fuzzball object put ./data.csv fb://group/data.csv --forceRecursive upload:
$ fuzzball object put ./dataset fb://user/datasets/run1 --recursive --ttl 7dIf the destination URI omits the object name (it ends in a trailing slash, such
as fb://group/), the name is inferred from the source basename. The following
two commands are equivalent:
$ fuzzball object put ./report.csv fb://group/
$ fuzzball object put ./report.csv fb://group/report.csvThe same applies recursively — fuzzball object put -R ./dataset fb://group/
uploads the tree under fb://group/dataset/.
Common errors: A 401 Unauthorized error means the CLI is not authenticated; run
fuzzball context login. A 403 Forbidden error means your account lacks permission on the
target namespace; contact your group administrator.
$ fuzzball object get fb://group/reports/report.csv ./report.csv| Flag | Description |
|---|---|
--recursive, -R | Recursively download all objects under a prefix |
If the local destination names a directory (it ends in a trailing slash, or is
. / ..), the name is inferred from the object basename. The following two
commands are equivalent:
$ fuzzball object get fb://group/reports/report.csv ./
$ fuzzball object get fb://group/reports/report.csv ./report.csvThe same applies recursively — fuzzball object get -R fb://group/dataset ./
downloads the tree into ./dataset/.
$ fuzzball object list fb://group/reportsThe CLI walks every page automatically and prints the full result set.
| Flag | Description |
|---|---|
--versions | Include version history for each object |
--page-size, -p | Server batch size per request (does not cap total results) |
--max-pages, -m | Hard cap on pages fetched (0 = no cap) |
$ fuzzball object describe fb://group/reports/report.csvShows detailed information about an object, including its size, digest, and versions.
An object that Fuzzball’s hosted agent wrote with object_put in a turn that had read untrusted
content, such as a workflow log, another object’s contents or a catalog entry, is labelled
Tainted. The
label sits on the version that turn wrote, so the version history marks it too, and putting
the same content again does not clear it. Treat a tainted object’s content as data, not as
instructions. The label is not carried to copies of the content, and an unlabelled object is
not proof that its content is clean.
$ fuzzball object delete fb://group/reports/report.csv| Flag | Description |
|---|---|
--force, -f | Skip the confirmation prompt |
When a workflow pulls an external container image — for example via a docker:// or
oras:// URI — the node converts the image to EROFS, the format it runs
containers from. When image cache write-back is enabled, the node also uploads the converted
result to the object cache, so subsequent pulls of the same image are served from the cache
instead of the external registry, which speeds up workflow startup and reduces dependence on
external registry availability. Write-back is disabled by default, so pulled images are not
cached unless an administrator turns it on (see cacheWriteBack below). Images built by a
workflow are always stored this way, regardless of the setting.
Pull and convert: When a job or service specifies a container image URI (e.g.,
docker://nvcr.io/nvidia/pytorch:24.01-py3), the node pulls the OCI or SIF image from the registry and converts it to EROFS.Cache storage: If write-back is enabled (see
cacheWriteBackbelow), the converted image is uploaded to the object cache as an EROFS archive — the image’s metadata followed by its layers — under a key derived from the image URI and target architecture; the blob itself is content-addressed by digest. The cached image is addressable with anfb://URI if needed for manual inspection. Write-back is disabled by default, so pulled images are not uploaded unless an administrator turns it on; images built by a workflow are uploaded regardless.Subsequent pulls: When another workflow requests the same image URI, Fuzzball checks the cache first. If a cached image exists for that URI (and target architecture), the node reads the archive’s metadata and downloads only the layers its own image store lacks, without contacting the external registry. A node that already holds every layer downloads nothing. Only archives uploaded by a node are imported this way: an archive uploaded with
fuzzball object putand referenced as an image is refused, since its contents would be filed under image and layer identities it chose itself.Cache refresh: Cached images are keyed by URI, so a mutable tag (e.g.,
latest) continues to be served from the cache even if the tag moves in the remote registry. The cache picks up a new version only after the cached entry expires or is deleted, triggering a fresh pull on the next request. That fresh pull resolves the tag at the registry and gets the image the tag currently points to, even on a node that still holds an image converted from an earlier version of the tag. Because the TTL counts from the entry’s last use, a frequently pulled tag can stay cached (and stale) indefinitely — to force a refresh, delete the cached entry or pin the image by digest.
Encrypted SIF images are not cached: they are pulled from their registry by every node that needs them and decrypted when the container starts.
When write-back is enabled, cached entries are scoped to the group of the workflow that pulled the image; workflows in other groups populate their own cache entries on their first pull.
When write-back is enabled (it is off by default), the upload in step 2 runs on the node right
after the conversion, and the image stage finishes once it is done, so a stage can take noticeably
longer than the pull itself on a large image. The upload publishes its own workflow events, so
fuzzball workflow events shows where the time went:
$ fuzzball workflow events WORKFLOW --kind image
...
[IMAGE] docker://docker.io/library/python:3.12 (arm64): Pulling image: docker://docker.io/library/python:3.12 (uri=docker://docker.io/library/python:3.12)
[IMAGE] docker://docker.io/library/python:3.12 (arm64): Image conversion started: docker://docker.io/library/python:3.12 (uri=docker://docker.io/library/python:3.12)
[IMAGE] docker://docker.io/library/python:3.12 (arm64): Image conversion complete: docker://docker.io/library/python:3.12 (372.5 MB) (size=390594560, uri=docker://docker.io/library/python:3.12)
[IMAGE] docker://docker.io/library/python:3.12 (arm64): Uploading image to object cache: 372.5 MB (size=390596608)
[IMAGE] docker://docker.io/library/python:3.12 (arm64): Image uploaded to object cache: 372.5 MB, 41.213s (attempts=1, digest=sha256:..., duration=41.213s, size=390596608)
[IMAGE] docker://docker.io/library/python:3.12 (arm64): Image pull complete: docker://docker.io/library/python:3.12 (372.5 MB, 1m3.217s) (duration=1m3.217s, size=390594560, uri=docker://docker.io/library/python:3.12)
[IMAGE] docker://docker.io/library/python:3.12 (arm64): Stage finished (name=train-image)While the stage is still running, fuzzball workflow events WORKFLOW --follow keeps printing
events as they arrive until the workflow finishes. In the Web UI, the image stage’s card shows a
progress bar with the percentage of the archive sent so far while the upload is in flight.
The upload is best effort. If the object cache cannot be reached or answers with a server error,
Fuzzball tries the upload up to three times, ten seconds apart, and abandons an attempt that moves
no bytes for two minutes or gets no reply within five minutes of sending the last byte. A rejection
that a retry would not change, such as an authentication failure or a response the node cannot
read, fails on the first attempt, and the object cache’s explanation, such as running out of space,
is included in the reason. Either way the stage reports Image upload to object cache failed
with the reason, and with the number of attempts when more than one was made, then finishes anyway:
the job runs on the image the node converted, the image is simply not cached, and the next pull of
the same URI tries the upload again. Only a pull the object cache lacks uploads; a pull served from
the cache shows no upload events at all.
The upload of pulled images is off by default. Administrators can turn it on cluster-wide with the
cacheWriteBack setting described in the
central configuration reference.
Cached images respect the object cache’s TTL policies. Automatically cached images carry no
per-object TTL, so they inherit the configured default (group, organization, or cluster
policy); if none is configured, the system default of 7 days applies. A TTL counts from the
object’s last access or update, so entries in active use are retained. For manually uploaded
objects, pass the --ttl flag to fuzzball object put to enforce a specific lifetime.
Expired entries are removed by a periodic purge job. Two effects of that are worth knowing, because both are visible from a workflow: when the cache is under disk pressure entries are treated as expired at half their configured TTL and removed oldest-accessed first, so a long-idle image can miss sooner than its TTL suggests; and at critical disk usage Fuzzball rejects new uploads outright.
On a deployment whose storage provisioners are split into segments, each segment stages its own copy of a cached image on its own shared filesystem. The object cache still keeps a single shared copy of each converted image – only the substrate-side staging is duplicated. Segments are configured by storage administrators in provisioner definitions; nothing in a workflow changes. See Segments.
The object cache stores converted images with architecture-specific keys. When a workflow pulls
an external container image (for example docker://ubuntu:22.04), the node converts it to EROFS
and caches the resulting archive under a reference that includes the runtime architecture, such
as amd64 or arm64. Three consequences follow in mixed-architecture clusters:
- Separate entries. A cluster with both
amd64andarm64nodes holds one cache entry per architecture for each logical image. - Per-architecture warming. Pre-populating the cache on
amd64nodes does not warm it forarm64nodes; those nodes pull and convert the image on first use. - Architecture-local hit rates. Moving a workflow to a different architecture produces a cache miss until that architecture has pulled the image once.
Single-architecture clusters are unaffected. Mixed-architecture deployments should size cache warming and storage capacity for one copy of each image per architecture.
The Fuzzball web UI includes an Object Cache browser for uploading, searching, and
inspecting objects across the group and user namespaces, including object metadata. Version
history is available from the CLI with fuzzball object list --versions.
The browser toolbar carries a namespace selector offering Group and User (private), alongside the search, filter, and refresh controls. The selector controls which namespace the browser displays. Switching namespace returns you to that namespace’s root, discarding your current folder path:
- Group — shows all objects in the group namespace (
fb://group/...), visible to all members of your group. - User (private) — shows objects in your personal user namespace (
fb://user/...), visible only to you.
Click either option to switch between namespaces. The object list, breadcrumb, and all operations update to reflect the selected namespace.
The breadcrumb bar at the top of the browser shows your current location:
- In the group namespace, the breadcrumb starts with Group followed by any path
segments (e.g.,
Group > datasets > images). - In the user namespace, it starts with User (private) followed by path segments
(e.g.,
User (private) > experiments > run-42).
Click any segment to navigate to that level. Clicking the namespace label (Group or User (private)) returns you to the root of that namespace. All operations in the browser — viewing object details, deleting objects, and creating references — act on the currently selected namespace only.
Use the Search box above the object table to filter by name within the current folder. The Filters menu next to it narrows the list further:
- Type — object format, inferred from content type or file extension (CSV, JSON, YAML, HDF5, Parquet, Text, Markdown, PyTorch, or Other).
- Updated by — the user who last updated the object. Only users present in the current listing appear as filter options.
- TTL status — Expiring soon (less than 24 hours remaining), Active, or No expiry.
- Scope — My Objects narrows the list to objects you last updated. Group Objects has no effect on its own (selecting it alone, both, or neither all show everything).
Search and filters apply within the current namespace and folder. While any of them is active, the browser loads the full result set in the background (shown as Loading all objects… (N loaded)) instead of stopping at whatever page has already loaded.
Click Upload object to open the Upload Object dialog. Uploads are always relative to your current location: the object is stored in the namespace and folder you are browsing, so you do not need to retype the full path.
The dialog has the following fields:
- File — the file or files to upload. Drag them onto the drop zone, or use Choose File / Choose Folder. You can select multiple files, and folder uploads preserve their directory structure.
- Object Name — the name the object is stored under. For a single file it is filled in automatically from the file’s name; edit it to store the object under a different name. (Shown only when uploading a single file on its own; folder and multi-file uploads keep each file’s own name.)
- Prefix — an optional subfolder, relative to your current location. The dialog shows
that location — for example,
Relative to Group / datasets / images. Leave it blank to upload directly into the current folder, or enter a value such asrun-42/to place the object in a subfolder beneath the current folder. - TTL — an optional lifetime after which the object is automatically removed: 1 day, 7 days, 30 days, or No expiry (the default).
Click Upload in the dialog to store the object. For example, while browsing Group > datasets > images with an Object Name of cat.png and a Prefix of train/, the
file is stored at fb://group/datasets/images/train/cat.png.
When uploading multiple files or a folder, each file is tracked separately: a progress table lists every file with its status (Queued, Hashing, Uploading, Finalizing, Done, or Failed). A failed file doesn’t stop the rest of the batch — once it finishes, the dialog reports how many files succeeded and how many failed.
For the command-line equivalent, see Upload.
Click an object row to open its preview alongside the table. Fuzzball renders JSON, YAML, Markdown, and plain-text files as text, CSV files as a table, and common image formats inline.
- Files over 1 MiB show This file is larger than the preview limit with a Preview anyway action to load the full preview.
- Unsupported file types show Preview not available for this file type.
- If the preview fails to load, the panel shows the underlying error message.
From the preview panel you can also Download the object, Copy reference (its
fb:// URI), open View details for its full metadata and TTL, or Open in new
tab.
The browser encodes the namespace and current path in the URL query string:
- Group namespace root: no
prefixparameter - User namespace root:
?prefix=user/ - Group namespace path:
?prefix=group/datasets/images - User namespace path:
?prefix=user/experiments/run-42
You can bookmark or share these URLs; opening a URL with a prefix parameter selects the
correct namespace and navigates to the specified path.
URLs pointing to the user namespace are personal. If you share a?prefix=user/...URL with a colleague, they will see their own user namespace at that path, not yours. To share objects with others, store them in the group namespace.