Fuzzfile Syntax Guide
Workflows are the fundamental unit of work in Fuzzball, orchestrating container execution, data movement, and resource allocation across compute clusters. They are described by Fuzzfiles — YAML 1.2 documents with a well-defined structure and syntax. A workflow definition is built from several top-level sections. Between them they describe how data flows into and out of the workflow, what compute operations to perform, what resources those operations require, and how jobs depend on each other. The top-level sections are:
version: v4
preemptible:
files:
defaults:
annotations:
volumes:
jobs:
services:
Each section serves a specific purpose:
- version: Specifies the workflow syntax version (
v1orv4; new workflows must usev4)[required] - preemptible: Marks all of the workflow’s allocations as eligible for
preemption
[optional] - files: Defines inline file content that can be referenced in volumes and
jobs
[optional] - defaults: Sets default configurations (mounts, environment, policy,
resources) for jobs
[optional] - annotations: Provides workflow-level metadata for scheduling and placement
[optional] - volumes: Defines storage volumes with data ingress (importing) and egress
(exporting) capabilities
[optional] - jobs: Specifies the computational work to be performed, organized as a directed
acyclic graph (DAG)
[required*] - services: Defines long-running container services that support jobs or provide
external endpoints
[required*]
* A workflow must have at least one job or service.
Required. Denotes the workflow syntax version. Valid values are v1 (legacy) and
v4. New workflows must use v4 and the V4 Fuzzfile syntax. The v1 format is
supported only for backward compatibility with existing workflows. Note that all snippets
in this reference use V4 syntax.
Fuzzball makes a best-effort attempt to auto-upgradev1workflows tov4at execution time. V1 volume references (reference: volume://...) and old-format mounts are automatically converted. However, writingversion: v4for new workflows avoids the upgrade step and ensures access to all V4 features.
When creating a new workflow in the web UI (via “Create New” in the Workflow Editor) or hydrating a
workflow template from the Workflow Catalog that lacks an explicit version field, the web UI
automatically defaults to version: v4.
The Workflow Editor works in v4 syntax. When you open a workflow, upload a YAML file, or paste a
workflow into the YAML view, the editor upgrades any v1 version or v1 syntax (volume references
and old-format mounts) to v4 through the same server-side upgrade used when a workflow runs, and
tells you when it has done so.
Structure:
version: v4
Example:
version: v4
If you have existing v1 Fuzzfiles that you want to convert to v4 format permanently
(rather than relying on the automatic upgrade at execution time), use the CLI upgrade
command:
$ fuzzball workflow upgrade my-workflow.fz > my-workflow-v4.fzThe upgrade runs server-side with your identity context, allowing persistent volume names
to be resolved against the provisioners available to your group. The output is YAML by
default; use -o json for JSON output.
To verify that a workflow is valid before submitting it, use validate. It accepts both
V1 and V4 Fuzzfiles. A V1 workflow is auto-upgraded and then validated, exactly as
submission does, so a file that fuzzball workflow start accepts also passes validate.
validate also refuses what submission refuses in the definition itself, such as a named
volume that also sets size or annotations, or a job with no command or script:
$ fuzzball workflow validate my-workflow.fzThe
upgradecommand is useful for batch-converting a library of V1 workflows. Redirect the output to a new file so you can review the changes before replacing the original:$ fuzzball workflow upgrade old-workflow.fz > new-workflow.fz $ diff old-workflow.fz new-workflow.fz
Optional. When true, marks every allocation produced by the workflow — jobs,
services, and the internal image, volume, and file stages — as eligible for preemption
by higher-priority allocations on clusters where preemption is enabled. Defaults to
false.
Setting this field is equivalent to starting the workflow with
fuzzball workflow start --preemptible. A job or service that sets its own
preemptible value explicitly overrides the workflow-level setting.
Structure:
preemptible: <boolean>
Example:
version: v4
preemptible: true
jobs:
batch: # preemptible via the workflow-level setting
image:
uri: docker://alpine:latest
command: [run-batch]
critical:
image:
uri: docker://alpine:latest
command: [run-critical]
preemptible: false # overrides the workflow-level setting
Optional. Defines inline file contents that can be referenced in volume ingress and
jobs using a file://<arbitrary_name> URI scheme. This is useful for embedding small
configuration files, scripts, or data directly in the workflow definition rather than
storing them externally. Note that some content may have to be base64 encoded to ensure
that the resulting yaml file is valid.
Structure:
files:
<file-name>: |
<file-contents>
<another-file-name>: |
<more-contents>
Fields:
- <file-name>: An arbitrary name to identify this inline file. This name is used in
file://URIs to reference the content. Can be any valid YAML key. - <file-contents>: The actual content of the file. Use the
|literal block scalar for multi-line content.
Example:
files:
config.ini: |
[settings]
debug=true
timeout=30
data.csv: |
id,name,value
1,alpha,42
2,beta,3.14
Usage:
Inline files can be referenced in two ways:
In volume ingress - to copy the content into a volume:
volumes: scratch: use: ephemeral size: 50GB ingress: - source: uri: file://config.ini # References the inline file destination: uri: file://config/app.ini # Destination in the volumeIn job files - to bind mount the content directly into a container:
jobs: my-job: image: uri: docker://alpine:latest files: /etc/app/config.ini: file://config.ini # Mounted at this path in container script: | #!/bin/sh cat /etc/app/config.ini
Optional. Sets default configurations that are applied to all jobs in the workflow. Individual jobs can override these defaults by specifying their own values. This is useful for reducing repetition when multiple jobs share common settings like volume mounts or environment variables.
Structure:
defaults:
job:
env: # Optional
- <VAR>=<value>
mounts: # Optional
<mount-path>: <volume-name>
policy: # Optional
timeout:
execute: <duration>
retry:
on: <node | exit code> # a scalar, or a list
attempts: <number>
resource: # Optional
cpu:
cores: <number>
affinity: <CORE|SOCKET|NUMA>
sockets: <number>
threads: <boolean>
memory:
size: <size>
by-core: <boolean>
devices:
<device-type>: <count>
exclusive: <boolean>
annotations:
<key>: <value>
Fields:
- job: Container for job-level defaults
env (optional): List of environment variables in
KEY=VALUEformat that will be set in all job containersmounts (optional): Map of absolute volume mount paths to volume names.
policy (optional): Default execution policies for all jobs
- timeout (optional): Time limits for job execution
- execute: Maximum duration (e.g.,
2h,30m,1h30m)
- execute: Maximum duration (e.g.,
- retry (optional): Whether the step may be run again, and what for.
See Retrying a step
- on: What the step may be retried for.
nodemeans the node stopped hosting it; a number from 1 to 255 is an exit code the job may be retried for. Accepts a scalar or a list - attempts (optional): How many times, one budget shared by every cause. Node retries are also bounded by the cluster’s own ceiling
- on: What the step may be retried for.
A job inherits the default
timeoutwhen it sets none, and the defaultretrywhen it declares neitherretrynor the deprecatedrestart. See Inheriting a default retry- timeout (optional): Time limits for job execution
resource (optional): Default hardware resource requirements
- cpu (required if resource is specified): CPU requirements
- cores: Number of CPU cores
- affinity: Binding strategy -
CORE,SOCKET, orNUMA - sockets (optional): Number of CPU sockets
- threads (optional): Whether to expose hardware threads (hyperthreading).
Inherited by jobs that leave
threadsunset; an explicittrueorfalseon the job wins
- memory (required if resource is specified): Memory requirements
- size: Amount of RAM (e.g.,
4GB,512MB) - by-core (optional): If true, size is per CPU core. Inherited by jobs that
leave
by-coreunset; an explicittrueorfalseon the job wins
- size: Amount of RAM (e.g.,
- devices (optional): Map of device types to counts (e.g.,
nvidia.com/gpu: 1). Requestcpu/x86: 1orcpu/arm64: 1to run on that CPU architecture. Fuzzball requests one automatically when the image URI names an architecture; see Image architecture - exclusive (optional): If true, job gets exclusive access to the node.
Inherited by jobs that leave
exclusiveunset; an explicittrueorfalseon the job wins - annotations (optional): Custom key-value pairs for advanced scheduling
- cpu (required if resource is specified): CPU requirements
Example:
defaults:
job:
env:
- SCRATCH=/scratch
- LOG_LEVEL=info
mounts:
/scratch: scratch
policy:
timeout:
execute: 1h
resource:
cpu:
cores: 2
affinity: CORE
threads: false
memory:
size: 4GB
by-core: false
In this example, all jobs will automatically:
- Have
/scratchandLOG_LEVELenvironment variables set - Mount the
scratchvolume at/scratch - Timeout after 1 hour
- Request 2 CPU cores and 4GB of memory
Individual jobs can override any of these defaults by specifying their own values. Default and
job specified environment variables are merged with job variables taking precedence over defaults.
Boolean resource fields (threads, by-core, exclusive) are inherited only when the job leaves
them unset; a job that sets a flag explicitly, including to false, keeps its own value. The same
rule applies to service defaults under defaults.service and to the top-level defaults.resource
block, which applies to both jobs and services.
Optional. Sets workflow-level annotation defaults as key-value pairs. These annotations are
merged into each job’s allocation annotations; per-job resource.annotations values take
precedence over these defaults when the same key appears in both places. The merged set is what
the scheduler matches key-by-key against each candidate node provisioner’s annotations
map — node provisioners that do not have a matching entry for a given key are rejected for that job.
Structure:
annotations:
<key>: <value>
<another-key>: <another-value>
Fields:
- <key>: String key identifying the annotation
- <value>: String value for the annotation
Example:
annotations:
nvidia.com/gpu.model: A100
nvidia.com/gpu.memory: "81920" # MiB per GPU, so at least 80 GiB
Common Use Cases:
- GPU selection: Require a specific GPU model, family, or minimum memory
- Node pool selection: Target specific sets of compute nodes that expose a matching annotation
Known annotation keys:
Fuzzball understands the following NVIDIA GPU annotations, which describe the GPUs on a node. The
values come from the hardware report each node sends when it boots. Until a cloud instance type has
been booted once, its node provisioner may carry only some of these keys, taken from the cloud
provider’s catalog. The scheduler matches against the node provisioner’s annotations, so to see the
values you can target, run fuzzball node provisioner get <provisioner> and look under
resource.annotations.
| Key | Meaning | Matching |
|---|---|---|
nvidia.com/gpu.model | GPU model name, such as NVIDIA A100 | Case-insensitive substring, so A100 matches NVIDIA A100 |
nvidia.com/gpu.product | GPU product name, with spaces replaced by hyphens (NVIDIA-A100) | Case-insensitive substring |
nvidia.com/gpu.arch | GPU architecture, such as Ampere | Exact |
nvidia.com/gpu.family | GPU family | Exact |
nvidia.com/gpu.memory | Memory per GPU, in MiB | Node value must be at least the requested value |
nvidia.com/gpu.count | Number of GPUs on the node | Node value must be at least the requested value |
nvidia.com/gpu.compute.major | Major version of the GPU’s CUDA compute capability | Node value must be at least the requested value |
nvidia.com/gpu.compute.minor | Minor version of the GPU’s CUDA compute capability | Exact |
Any other key is matched by exact string comparison against the node provisioner’s annotations, so it only selects nodes whose provisioners an administrator has given that annotation.
Keys such asnvidia.com/gpu,amd.com/gpu,cpu/x86,cpu/arm64, andcpu/x86/avxare devices, not annotations. Request them withresource.devices(for example,nvidia.com/gpu: 2), not withannotations.
Annotation matching is strict by default. Every key in the merged allocation annotations (from both workflow-level
annotationsand per-jobresource.annotations) must either be present with a matching value in each candidate node provisioner’sannotationsmap, or the cluster admin must add the key toscheduler.ignoredAnnotationsin the central configuration. The platform’s own internal annotation keys — a specific enumerated set underfuzzball.io/*such asfuzzball.io/workflow.id,fuzzball.io/job.name, andfuzzball.io/provisioner_cluster_id, plus the dynamically keyedfuzzball.io/connect/<service>namespace — are skipped automatically. Apart from the connect namespace this is an allowlist, not a prefix match: thefuzzball.io/prefix is reserved for the platform, and any user-defined key placed under that prefix (e.g.fuzzball.io/my-team-label) is not auto-exempt and would still need to be added toscheduler.ignoredAnnotations. Treat thefuzzball.io/namespace as off-limits for your own annotations.Annotation keys used only for organizational metadata, auditing, or policy routing should not appear in either
annotationsorresource.annotationsunless the cluster admin has added them toscheduler.ignoredAnnotations; otherwise the scheduler will reject any node provisioner that does not carry those keys.
See Scheduler annotation matching in the configuration reference for built-in GPU matchers and configuration details.
Optional. Defines storage volumes that jobs can access. Volumes provide persistent or ephemeral storage for workflow data and support data ingress (importing data at workflow start) and egress (exporting data at workflow end).
Structure:
volumes:
<volume-name>:
use: <provisioner-name> # Optional
name: <persistent-name> # Optional (only for persistent volumes)
size: <capacity> # Optional (only for ephemeral volumes)
annotations: # Optional (only for ephemeral volumes)
<key>: <value>
ingress: # Optional
- source:
uri: <source-uri>
secret: <secret-ref> # Optional
destination:
uri: <dest-uri>
policy: # Optional
timeout:
execute: <duration>
egress: # Optional
- source:
uri: <source-uri>
destination:
uri: <dest-uri>
secret: <secret-ref> # Optional
policy: # Optional
timeout:
execute: <duration>
Fields:
- <volume-name>: Arbitrary name for the volume, used to reference it in job
mountssections (e.g.,scratch,data) - use (optional): Storage provisioner name. Can also
be
ephemeralorpersistentfor automatic provisioner selection - reference (deprecated): Legacy V3 volume URI format (
volume://scope/class/name). Still supported for backward compatibility butuse/name/size/annotationsis preferred - ingress (optional): List of files to import into the volume at workflow start
- source (required): Where to fetch the data from
- uri: Source location. Supported schemes:
s3://,http://,https://,file://(references inline files from thefilessection),fuzzball://andfb://(fuzzball://group/<path>orfuzzball://user/<path>, referencing an object stored in the cluster’s object cache), andhf://(or the longerhuggingface://alias) pulling a model, dataset, or space repository from the HuggingFace Hub — see HuggingFace ingress. Workflow submission rejects any other scheme, including the undocumentedlocal:// - secret (optional): Reference to credentials for accessing the source (format:
secret://<scope>/<name>)
- uri: Source location. Supported schemes:
- destination (required): Where to place the data in the volume
- uri: Destination path in the volume (format:
file://<path>). When jobs mount this volume, the path will be relative to the mount point
- uri: Destination path in the volume (format:
- policy (optional): Execution policies for the transfer
- timeout (optional): Time limit for the transfer
- execute: Maximum duration (e.g.,
5m,1h)
- execute: Maximum duration (e.g.,
- Transfers are not retried, so
retry.onis refused here;retrywith onlyattemptsis accepted and has no effect
- timeout (optional): Time limit for the transfer
- source (required): Where to fetch the data from
- egress (optional): List of files to export from the volume at workflow end. Structure
is similar to ingress, but source is the file in the volume and destination is the
external storage location. Egress runs after every job and service that mounts the
volume is done. A job, or a service that is not an autoscaled replica pool, is done
when it reaches FINISHED. A non-persist service reaches FINISHED when Fuzzball stops
it (once nothing that depends on it is still running), so egress data produced by work
the service does after readiness should be ordered with
dependsOn. Apersist: trueservice reaches FINISHED when its own command exits successfully; if it never exits, egress for that volume does not run and cancelling the workflow discards it. An autoscaled replica-pool service is done when the pool is idle: every replica has been torn down on its node or is waiting unstarted in the pool, and any service whose scale-up triggers target the pool has reached FINISHED. A pool woken after egress is released, for example by a request to one of its endpoints, may write more data, but those writes are not guaranteed to be included in the export.
Ephemeral volume fields — these fields define a volume that is created when the workflow runs and destroyed when it completes:
- size (optional): Requested volume capacity (e.g.,
10GB,1TiB) - annotations (optional): Key-value pairs for provisioner auto-selection. Fuzzball matches these against provisioner annotations to select the right storage backend
Thesizeandannotationsfields are only valid for ephemeral volumes. They cannot be combined with thenamefield.
Persistent volume fields — these fields reference an existing volume that was created before the workflow (via the CLI, web UI, or API):
- name (optional): Name of an existing persistent volume to bind. The volume must
already exist on the provisioner specified by
use, or on any accessible provisioner ifuseis omitted. Workflows that reference a volume that does not exist are rejected at submission time.
Thenamefield is the signal that distinguishes a persistent volume from an ephemeral one. Whennameis set, the volume is persistent. Whennameis absent, the volume is ephemeral. Persistent volumes are never created dynamically — they must be created ahead of time.
Example:
volumes:
scratch:
use: my-provisioner
size: 50GB
ingress:
- source:
uri: s3://my-bucket/input-data.tar.gz
secret: secret://group/AWS_CREDENTIALS
destination:
uri: file://inputs/data.tar.gz
policy:
timeout:
execute: 10m
- source:
uri: file://config.ini # References inline file
destination:
uri: file://config/app.ini
egress:
- source:
uri: file://results/output.tar.gz
destination:
uri: s3://my-bucket/results/output.tar.gz
secret: secret://group/AWS_CREDENTIALS
data:
use: my-provisioner
name: research-data
In this example:
scratchis ephemeral (noname, hassize), downloads data from S3 at start, and uploads results at enddatais persistent (hasname), survives workflow completion with no data transfers
Path Resolution Example:
If a job mounts scratch at /scratch, a file ingressed to file://inputs/data.tar.gz
will be available in the job container at /scratch/inputs/data.tar.gz.
Some provisioners use per-node storage rather than shared storage. A volume on one of those exists on a single node, so Fuzzball places everything that uses it on that node. This applies to ephemeral and persistent volumes alike, and it rules out anything that needs more than one node at once.
Fuzzball rejects these at submission rather than letting them fail part-way through:
- Multinode jobs. The job spans more nodes than the volume exists on.
- Task arrays running more than one task at once. Each concurrent task needs its own
node.
concurrencydefaults to the size of the array, so an array of more than one task is rejected unless it setsconcurrency: 1. - Autoscaled services whose
autoscaler.replicas.maxis greater than1. Fuzzball places at most one replica of a pool per node. - Two node-local volumes that live on different nodes. No single placement reaches both. This is assessed across the jobs that tie the volumes together, so it applies even when no one job mounts both.
- A persistent node-local volume with no recorded holding node, used alongside another. Fuzzball cannot confirm the two share a node.
A persistent node-local volume is also refused on its own when it has no recorded holding node on a cluster of more than one node, or when the node it is recorded on is not currently registered.
To check whether a provisioner is node-local, run fuzzball volume provisioner info <name> and look for local: true under driver:; the field is omitted when the storage
is shared. See When the path is not shared for how
node-local provisioners are configured.
Naming Constraint: Job and service names must be unique within a workflow — a job and a service cannot share the same name. Images, volumes, and ingress/egress file IDs share that same namespace, so none of them may reuse a job or service name either. The constraint exists because allocation IDs and dependency references are keyed by name alone. Fuzzball rejects a workflow with a name conflict at submission and returns a validation error:
workflow validation failed: name "<name>" is used by both a job and a service; job and service names must be unique within a workflowFor example, this workflow definition is invalid and will be rejected:
version: v4 jobs: database: image: uri: docker://alpine:latest command: ["echo", "job"] resource: cpu: cores: 1 memory: size: 1GB services: database: # ERROR: name already used by a job image: uri: docker://postgres:16 resource: cpu: cores: 2 memory: size: 4GBEnsure all job and service names are distinct before submission.
Optional (but at least one job or service is required). Defines the computational work to be performed as a directed acyclic graph (DAG) of compute steps. Jobs can run scientific software, data processing scripts, or any containerized application. Jobs are the fundamental execution unit in a workflow.
Structure:
jobs:
<job-name>:
image: # Required
uri: <image-uri>
secret: <secret-ref> # Optional
decryption-secret: <secret-ref> # Optional
command: [<arg1>, <arg2>, ...] # One of command or script required (mutually exclusive with script)
script: | # One of command or script required (mutually exclusive with command)
<script-content>
args: [<arg1>, <arg2>, ...] # Optional
env: # Optional
- <VAR>=<value>
cwd: <working-directory> # Optional
mounts: # Optional
<mount-path>: <volume-name>
files: # Optional
<container-path>: <file-uri>
resource: # Optional
cpu: # Required if resource specified
cores: <number>
affinity: <CORE|SOCKET|NUMA>
sockets: <number> # Optional
threads: <boolean> # Optional
memory: # Required if resource specified
size: <size>
by-core: <boolean> # Optional
devices: # Optional
<device-type>: <count>
exclusive: <boolean> # Optional
annotations: # Optional
<key>: <value>
policy: # Optional
timeout:
execute: <duration>
retry: # Optional
on: <node | exit code> # a scalar, or a list
attempts: <number> # Optional
requires: [<job1>, <job2>, ...] # Optional (deprecated, use depends-on)
depends-on: # Optional
- name: <job-name>
status: <RUNNING|FINISHED|FAILED>
description: <text> # Optional
multinode: # Optional (mutually exclusive with task-array, network)
nodes: <number>
implementation: <ompi|openmpi|mpich|gasnet|generic>
procs-per-node: <number> # Optional
network: # Optional (mutually exclusive with multinode, task-array)
isolated: <boolean> # Required (must be true)
expose-tcp: [<number>, <number>, ...] # Optional
expose-udp: [<number>, <number>, ...] # Optional
task-array: # Optional (mutually exclusive with multinode, network)
start: <number>
end: <number>
concurrency: <number> # Optional
preemptible: <boolean> # Optional
Fields:
- <job-name>: Arbitrary name identifying this job (e.g.,
preprocess-data,train-model). Must be a valid DNS subdomain and must be unique within a workflow. This name appears infuzzball workflow statusand is used in dependency specifications - image (required): Container image specification
- uri: Image location. Supported schemes:
docker://for OCI containers,oras://for SIF images (e.g.,oras://depot.ciq.com/fuzzball/fuzzball-applications/curl.sif:latest) - secret (optional): Credentials for private registries (format:
secret://<scope>/<name>) - decryption-secret (optional): Secret to decrypt encrypted SIF images
- uri: Image location. Supported schemes:
- command (required, mutually exclusive with script): The program to run and its arguments, as
a list (e.g.,
[python3, script.py, --input, /data]). The first element must be an executable. Fuzzball runs the list as given and does not prepend the image’sENTRYPOINT - script (required, mutually exclusive with command): Multi-line shell script to execute.
If no
#!shebang line is present,#!/bin/shis assumed. Include an explicit shebang for other interpreters (e.g.,#!/bin/bash,#!/usr/bin/python3) - args (optional): Additional arguments passed to command or script
- env (optional): List of environment variables in
KEY=VALUEformat. Fuzzball also sets variables of its own, such asFB_WORKFLOW_IDandFB_JOB_NAME, in every container; see the Environment Variables Reference - cwd (optional): Working directory for the job. Must be an absolute path. Defaults to the
image’s working directory or
/ - mounts (optional): Map of absolute volume mount paths to volume names (from the
volumessection) - files (optional): Map of container paths to inline file URIs. Bind mounts inline files
directly into the container (e.g.,
/etc/config.ini: file://my-config) - resource (optional): Hardware resource requirements for scheduling
- cpu (required if resource specified): CPU requirements
- cores: Number of CPU cores (must be > 0)
- affinity: Binding strategy -
CORE(any cores),SOCKET(same socket), orNUMA(same NUMA domain). Defaults toCORE - sockets (optional): Number of physical CPU sockets
- threads (optional): Whether to expose hardware threads (hyperthreading)
- memory (required if resource specified): Memory requirements
- size: Amount of RAM with units (e.g.,
4GB,512MB,2GiB) - by-core (optional): If true, size is per CPU core, i.e. total size is
cores * size
- size: Amount of RAM with units (e.g.,
- devices (optional): Map of device types to counts (e.g.,
nvidia.com/gpu: 2). Requestcpu/x86: 1orcpu/arm64: 1to run on that CPU architecture. Fuzzball requests one automatically when the image URI names an architecture; see Image architecture - exclusive (optional): If true, job gets exclusive node access
- annotations (optional): Custom key-value pairs for advanced node selection (e.g., GPU model). See Known annotation keys
- cpu (required if resource specified): CPU requirements
- policy (optional): Execution policies
- timeout (optional): Time limits
- execute: Maximum job duration (e.g.,
2h,30m)
- execute: Maximum job duration (e.g.,
- retry (optional): Whether the job may be run again, and what for. See
Restart Tolerance
- on:
node, an exit code from 1 to 255, or a list of them - attempts (optional): The most retries, shared by every cause
- on:
- timeout (optional): Time limits
- requires (optional, deprecated): List of job names that must complete before this job
starts. Use
depends-oninstead - depends-on (optional): List of concrete dependencies with status requirements. Note that
depending on a job array to finish will wait for all tasks to finish.
- name: Job or service name to depend on
- status: Required status -
FINISHED(job/service completed successfully),RUNNING(job/service is running), orFAILED(job/service errored). AFAILEDdependency is an “on failure” dependency: the dependent starts only when the named job/service errors, which is useful for recovery or notification steps. It does not trigger on user cancellation or preemption. If the named job/service instead succeeds, the on-failure dependent is not needed and is skipped. AFAILEDdependency is supported only when the named target is a single-node job or service; multi-node targets are rejected at submission. For a service target, the on-failure dependent fires if the service errors while another job is using it; if the service is reaped or stops normally it is recorded as finished and the on-failure dependent is skipped. See Workflow Status Values for how these map to the CLI status labels and theSTAGE_STATUS_*enum. - description (optional): Human-readable explanation of the dependency
- multinode (optional, mutually exclusive with task-array/network): Multi-node parallel
execution
- nodes: Number of nodes to allocate
- implementation: MPI/communication implementation -
ompi,openmpi,mpich,gasnet, orgeneric - procs-per-node (optional): Processes per node. Defaults to number of allocated CPUs
- network (optional, mutually exclusive with multinode and task-array): If present, job is run
inside its own (isolated) network namespace. Each exposed container port is published on the node
at a randomly assigned host port.
- isolated: Must be
truewhen thenetworksection is specified. - expose-tcp (optional): List of container TCP ports to expose
- expose-udp (optional): List of container UDP ports to expose
- isolated: Must be
- task-array (optional, mutually exclusive with multinode and network): Embarrassingly parallel
execution
- start: Starting task ID (inclusive, must be > 0)
- end: Ending task ID (inclusive, must be >= start)
- concurrency (optional): Maximum parallel tasks (max 200). Each task receives
$FB_TASK_ID
- preemptible (optional): If
true, this job may be preempted by higher-priority allocations on clusters where preemption is enabled. When unset, the job inherits the workflow-levelpreemptiblesetting (or the--preemptiblestart flag); an explicittrueorfalseoverrides it
Example:
jobs:
preprocess:
image:
uri: docker://python:3.11
script: |
#!/bin/sh
python3 preprocess.py --input /data/raw/${FB_TASK_ID} --output /data/processed/${FB_TASK_ID}
env:
- PYTHONUNBUFFERED=1
mounts:
/data: data
resource:
cpu:
cores: 4
affinity: NUMA
memory:
size: 8GB
policy:
timeout:
execute: 30m
task-array:
start: 1
end: 1000
concurrency: 100
train-multinode:
image:
uri: docker://nvcr.io/nvidia/pytorch:24.01-py3
secret: secret://user/NGC_API_KEY
script: |
#!/bin/bash
python -m torch.distributed.run train.py --input /data/processed
depends-on:
- name: preprocess
status: FINISHED
resource:
cpu:
cores: 32
affinity: SOCKET
memory:
size: 128GB
devices:
nvidia.com/gpu: 4
annotations:
nvidia.com/gpu.model: NVIDIA L40
multinode:
nodes: 4
implementation: openmpi
procs-per-node: 4
This example demonstrates:
- preprocess: Basic job task array with resource requests
- train-multinode: Multi-node MPI job with GPUs and explicit dependencies
Naming Constraint: Job and service names must be unique within a workflow — a job and a service cannot share the same name. Images, volumes, and ingress/egress file IDs share that same namespace, so none of them may reuse a job or service name either. The constraint exists because allocation IDs and dependency references are keyed by name alone. Fuzzball rejects a workflow with a name conflict at submission and returns a validation error:
workflow validation failed: name "<name>" is used by both a job and a service; job and service names must be unique within a workflowSee the jobs section for a detailed example of this validation constraint.
Optional (but at least one job or service is required). Defines long-running container services that support jobs or provide external endpoints. Unlike jobs which complete and exit, services run continuously for the workflow duration (or until dependent jobs finish). Services are useful for databases, web servers, message queues, interactive computing (e.g. jupyter or Rstudio), AI inference servers, or any persistent service that jobs need to access.
Structure:
services:
<service-name>:
image: # Required
uri: <image-uri>
secret: <secret-ref> # Optional
decryption-secret: <secret-ref> # Optional
command: [<arg1>, <arg2>, ...] # One of command or script required (mutually exclusive with script)
script: | # One of command or script required (mutually exclusive with command)
<script-content>
args: [<arg1>, <arg2>, ...] # Optional
env: # Optional
- <VAR>=<value>
cwd: <working-directory> # Optional
mounts: # Optional
<mount-path>: <volume-name>
files: # Optional
<container-path>: <file-uri>
resource: # Optional
cpu:
cores: <number>
affinity: <CORE|SOCKET|NUMA>
sockets: <number> # Optional
threads: <boolean> # Optional
memory:
size: <size>
by-core: <boolean> # Optional
devices: # Optional
<device-type>: <count>
exclusive: <boolean> # Optional
annotations: # Optional
<key>: <value>
requires: [<job1>, <svc1>, ...] # Optional (deprecated, use depends-on)
depends-on: # Optional
- name: <job-or-service-name>
status: <RUNNING|FINISHED|FAILED>
description: <text> # Optional
multinode: # Optional
nodes: <number>
implementation: <ompi|openmpi|mpich|gasnet|generic>
procs-per-node: <number> # Optional
network: # Optional
host: <boolean> # Optional
ports:
- name: <port-name>
port: <port-number>
protocol: <tcp|udp> # Optional
endpoints: # Optional
- name: <endpoint-name>
port-name: <port-name> # References port name above
protocol: <http|https> # Optional. Defaults to http
type: <subdomain|path> # Optional. Defaults to path
scope: <endpoint-scope> # Optional. One of: user, group, organization, public
annotations: # Optional
<key>: <value>
per-replica: <boolean> # Optional; autoscaled pools only
persist: <boolean> # Optional
readiness-probe: # Optional
exec: # One of: exec, http-get, tcp-socket, grpc
command: [<arg1>, <arg2>]
http-get:
path: <path>
port: <port-number>
scheme: <HTTP|HTTPS> # Optional
http-headers: # Optional
- name: <header-name>
value: <header-value>
tcp-socket:
port: <port-number>
grpc:
port: <port-number>
service: <service-name> # Optional
initial-delay-seconds: <seconds> # Optional
period-seconds: <seconds> # Optional
timeout-seconds: <seconds> # Optional
success-threshold: <number> # Optional
failure-threshold: <number> # Optional
autoscaler: # Optional
replicas: # Required for a self-scaling service
min: <number> # Optional (0 enables scale-to-zero)
max: <number> # Required, >= 1 and >= min
metrics: # Optional
enabled: <boolean>
path: <metrics-path>
port: <port-number>
interval: <seconds>
includes: [<metric1>, ...]
scale-up: # Optional
triggers:
- metrics-query: <promql-expression>
cooldown-period: <seconds> # Optional
services: [<svc1>, ...] # Optional (cross-service scaling)
scale-down: # Optional
metrics-query: <promql-expression>
cooldown-period: <seconds> # Optional
drain-period: <seconds> # Optional (default 30, max 600)
dynamic-config: # Optional
path: <container-path>
services: [<svc1>, <svc2>]
script: |
<bash-script>
preemptible: <boolean> # Optional
Fields:
Services share many fields with jobs (image, command, script, args, env, cwd, mounts, files, resource, depends-on, multinode, preemptible). See the jobs section for details on these common fields. Service-specific fields are:
Important: A multinode service is not designed to run multiple identical instances of a service with load-balanced client connections. It is intended for distributed services (e.g., MPI-based, vLLM cluster …), where the endpoint is served exclusively on rank 0. Clients always connect to rank 0, which coordinates with the other instances internally.
<service-name>: Arbitrary name identifying this service. Must be a valid DNS subdomain and must be unique within a workflow.
network (optional): Network configuration for service exposure. If present and the list of exposed ports is not empty, service is run inside its own (isolated) network namespace (multinode services excepted – they always run on the host network, see host below), and each declared port is published on the node at the declared port. Other workflow members reach the service at
<service>.svc.<workflow-id>.<account-id>.fuzzball:<declared-port>(the service name alone also resolves, via the workflow’s DNS search path). Because the declared port is claimed on the node, two services that declare the same port number cannot share a node: the scheduler places them on different nodes. A service whose declared port is taken on every node with room for it waits in the queue – the stage reports ascheduling_blockedevent with reasonport_conflict, andfuzzball workflow whyreportsPORT_CONFLICT– until a holder ends or a node joins. A port a finished or cancelled service held stays claimed for a few seconds more, while the node releases its container.- Autoscaled replica pools (services with
autoscaler.replicas) are the exception: each replica’s declared port is published at a randomly assigned host port, so same-port replicas can pack onto one node. Clients of a pool must not connect to<node>:<declared-port>directly — discover each replica’s host port through the DNSSRVrecords named_<port-name>._<protocol>.<service>.autoscaler.<workflow-id>.fuzzball, or throughdynamic-configarrays. See Service Autoscaling. - Endpoints and
fuzzball workflow connectroute to the published host port transparently. Endpoints are supported on replica pools, but must usetype: subdomainand reference a named port – thepathform selects a single row and cannot address a pool. - host (optional): If
true, service serves on the host network namespace. Ports are used as declared (no host-port assignment), so two host-network services that bind the same port cannot share a node; the scheduler keeps them apart the same way. Forced totruefor services with amultinodeblock. - ports: List of ports the service listens on
- name: Identifier for this port (used in endpoints and SRV records)
- port: Port number (1-65535)
- protocol (optional):
tcporudp. Defaults totcp
- endpoints (optional): List of external endpoints to create
name: Endpoint identifier (255 bytes max)
port-name: References a port name from the ports list
protocol (optional):
httporhttps. Defaults tohttp. The endpoint proxy forwards over HTTP, so stream protocols cannot be servedtype (optional): Endpoint style -
subdomain(creates<endpoint-id>.endpoints.<cluster-domain>) orpath(creates/endpoints/<endpoint-id>). Defaults topathscope (optional): Determines who can access the endpoint. One of
user(only the workflow creator),group(anyone in the same group),organization(anyone in the same organization),public(anyone without authentication). Defaults togroupif not specified.annotations (optional): Map of key-value pairs stored with the endpoint and returned by
fuzzball endpoint listandget. Fuzzball does not interpret them; clients discovering endpoints use them to tell which ones they should consume. Limits: at most 32 entries per endpoint, keys up to 253 characters, values up to 1024 characters, and 8192 bytes of keys and values combined. A key is an optional DNS subdomain prefix followed by/and an alphanumeric name of up to 63 characters (dashes, underscores and dots allowed within), for exampleexample.com/roleorrole. Thefuzzball.ioprefix, and any subdomain of it, is reserved for the platform and rejected in your own annotations.per-replica (optional): If
true, every live replica of an autoscaled pool also gets its own endpoint, alongside the pool’s. Requires a service that definesautoscaler.replicasandtype: subdomain. See addressing individual replicas.persistent (optional): If
true, the endpoint serves your persistent endpoint of the samename, which has to exist, instead of an endpoint of the workflow’s own. An omittedtype,protocolorscopethen takes the persistent endpoint’s, and a declared one must match it. Not supported on an autoscaled service. Defaults tofalse.
portsandendpointsare YAML lists: each entry begins with a-, and a service may declare more than one. Write them as sequence items (as shown in the structure above), not as a single mapping.- Autoscaled replica pools (services with
persist (optional): If
true, service continues running even after all dependent jobs finish and continues until the workflow is cancelled. Iffalse(default), service stops when no jobs/services depend on it.“No dependents” includes a service that never had any. A service with
persistunset and nothing declaringdependsOnit is stopped as soon as it reports ready – it reaches RUNNING, then stops, and the workflow finishes. Setpersist: truefor a service you intend to run on its own – for example a model server you connect to directly, or a service under test.The rule applies to every replica of an autoscaled service. Endpoint traffic, cross-service
scale-uptriggers, anddynamic-configdo not count as dependents, so an autoscaled service needspersist: true. Without it, each replica is stopped as soon as it becomes ready instead of remaining in the serving pool.policy (deprecated): Accepted only so that workflows written when services supported it still run. It has no effect: Fuzzball ignores it and returns a warning naming the service. Services have no execution timeout; to bound how long a service runs, run its command under
timeoutin the service’s script.readiness-probe (optional): Kubernetes-style health check to determine when service is ready. Service status transitions from STARTED to RUNNING only after probe succeeds
- exec: Run a command in the container. Success if exit code is 0
- command: Command to execute
- http-get: HTTP GET request. Success if status code is 200-399
- path: HTTP path
- port: Port number
- scheme (optional):
HTTPorHTTPS - http-headers (optional): Custom HTTP headers
- tcp-socket: TCP connection attempt. Success if connection establishes
- port: Port number
- grpc: gRPC health check. Success per gRPC health checking protocol
- port: Port number
- service (optional): gRPC service name
- initial-delay-seconds (optional): Delay before first probe
- period-seconds (optional): Frequency of probes
- timeout-seconds (optional): Probe timeout
- success-threshold (optional): Consecutive successes needed
- failure-threshold (optional): Consecutive failures before marking unhealthy
- exec: Run a command in the container. Success if exit code is 0
autoscaler (optional): Automatically scales the number of replicas of this service up and down based on PromQL queries evaluated against the service’s metrics. See Service Autoscaling for a full guide.
- replicas: Bounds the replica count for a self-scaling service. Required unless the
service’s
scale-uptriggers target other services.- min (optional): Minimum number of replicas.
0enables scale-to-zero (the service runs no replicas until a trigger fires). - max: Maximum number of replicas. Must be at least
1and not less thanmin.
- min (optional): Minimum number of replicas.
- metrics (optional): Configures collection of the service’s own metrics, used by the scale
policies.
- enabled: Set to
trueto scrape the service’s metrics endpoint. - path: HTTP path of the metrics endpoint (e.g.
/metrics). - port: Container port exposing the metrics endpoint.
- interval: Scrape interval, in seconds (default
15). Also sets the sampling interval for the default container CPU/memory metrics, even whenenabledis not set. - includes (optional): Restricts collection to the named metrics. All metrics are collected when empty.
- enabled: Set to
- scale-up (optional): One or more triggers that add replicas (including from zero).
- triggers: List of independent scale-up conditions. Each trigger:
- metrics-query: A PromQL expression. The trigger fires when the query returns a true
(non-zero) result. Use the
boolmodifier so the comparison yields1/0(e.g.vllm:num_requests_waiting >= bool 2). - cooldown-period (optional): Minimum number of seconds between consecutive scaling actions for the targeted service.
- services (optional): Names of other services this trigger scales, instead of this service’s own replicas (cross-service scaling — see the autoscaling guide). Leave empty to scale the owning service itself.
- metrics-query: A PromQL expression. The trigger fires when the query returns a true
(non-zero) result. Use the
- triggers: List of independent scale-up conditions. Each trigger:
- scale-down (optional): A single policy that removes replicas of this service.
- metrics-query: PromQL expression; the policy fires when it returns a true result.
- cooldown-period (optional): Minimum number of seconds between consecutive scale-downs.
- drain-period (optional): Number of seconds a retiring replica keeps running after it is
removed from the service’s DNS records and dependent dynamic-configs, so requests in flight
to it can complete. Default
30, maximum600. Only valid onscale-down. Omitting the field (or setting0) selects the default; draining cannot be disabled — usedrain-period: 1for the shortest drain.
- replicas: Bounds the replica count for a self-scaling service. Required unless the
service’s
dynamic-config (optional): Renders a configuration file from the live addresses of other services and bind-mounts it into this service’s container, re-rendering it as those services' replicas scale up and down.
- path: Destination path of the rendered file inside the container.
- services: Services whose addresses are made available to the script. Each service
nameis exposed as a bash array namedSERVICES_<NAME>(uppercased, non-identifier characters replaced with_) holding the ready replicas’ IP addresses; for example serviceworkerbecomes${SERVICES_WORKER[@]}. Each of the watched service’s named TCP ports additionally yields aSERVICES_<NAME>_<PORT-NAME>array ofip:portpairs carrying each replica’s assigned host port (see network above); for example porthttpof serviceworkerbecomes${SERVICES_WORKER_HTTP[@]}. Use theip:portarrays to address replicas — the assigned host port differs per replica. - script: A bash script whose standard output becomes the file contents. It runs in a
restricted interpreter: builtins and heredocs only — no external commands (except a no-argument
catforcat <<EOFtemplating) and no filesystem access.
Example:
version: v4
volumes:
db:
use: ephemeral
size: 10GB
services:
postgres:
image:
uri: docker://postgres:16
command: ["docker-entrypoint.sh", "postgres"]
env:
- POSTGRES_PASSWORD=secret
- POSTGRES_DB=myapp
mounts:
/var/lib/postgresql/data: db
resource:
cpu:
cores: 4
memory:
size: 8GB
network:
ports:
- name: postgres
port: 5432
protocol: tcp
readiness-probe:
tcp-socket:
port: 5432
initial-delay-seconds: 5
period-seconds: 10
persist: true
api-server:
image:
uri: docker://mycompany/api:v1.2.3
secret: secret://group/REGISTRY_CREDS
command: ["/usr/local/bin/api-server", "--port", "8080"]
env:
- DATABASE_URL=postgresql://postgres:5432/myapp
depends-on:
- name: postgres
status: RUNNING
description: "API needs database connection"
resource:
cpu:
cores: 2
memory:
size: 4GB
network:
ports:
- name: http
port: 8080
endpoints:
- name: api
port-name: http
protocol: https
type: subdomain
scope: group
readiness-probe:
http-get:
path: /health
port: 8080
initial-delay-seconds: 10
period-seconds: 5
failure-threshold: 3
jobs:
data-processor:
image:
uri: docker://mycompany/processor:latest
script: |
#!/bin/sh
python process.py --api-url http://api-server:8080
depends-on:
- name: api-server
status: RUNNING
resource:
cpu:
cores: 8
memory:
size: 16GB
This example shows:
- postgres: Persistent database service with readiness probe
- api-server: REST API depending on postgres, exposed via HTTPS subdomain endpoint at account scope
- data-processor: Job that depends on api-server being running before it starts
A workflow, a job, or
another stage reports the same underlying state under
different names depending on where you look: the status column the
CLI prints, the raw WorkflowStatus enum returned by
fuzzball workflow describe and the SDK, the depends-on: status: field in a Fuzzfile, and the
--status filter.
| State | CLI status | WorkflowStatus enum | Fuzzfile depends-on: status: | --status filter |
|---|---|---|---|---|
| Waiting for resources or dependencies | Pending | STAGE_STATUS_UNSPECIFIED | — | pending |
| Running | Started | STAGE_STATUS_STARTED | RUNNING | started |
| Completed without errors | Finished | STAGE_STATUS_FINISHED | FINISHED | finished |
| Ended with errors | Errored | STAGE_STATUS_FAILED | FAILED | errored (alias failed) |
| Cancelled by a user, never started because its workflow was cancelled or failed, or a skipped on-failure dependent | Canceled | STAGE_STATUS_CANCELED | — | canceled (alias cancelled) |
| Autoscaled service replica scaled to zero | Down | STAGE_STATUS_DOWN | — | — |
A few things to watch for:
- The running state is written
RUNNINGin a Fuzzfiledepends-on, but shown asStarted/STAGE_STATUS_STARTEDin CLI and enum output — the same state, a different word.- For job and service dependencies,
depends-on: status:acceptsRUNNING,FINISHED, andFAILED. Any other value (for exampleSTARTED) is rejected and the workflow fails validation. AFAILEDdependency starts the dependent only when its target errors – an “on failure” dependency for recovery or notification steps; it does not trigger on cancellation or preemption, and if the target succeeds the on-failure dependent is skipped. A skipped dependent never runs; it shows asCanceled, with a reason that startsskipped:and names the dependency.FAILEDis currently supported only against single-node jobs and services; depending on a multi-node job’s failure is not yet supported. Stages that depend on something other than a job or service, such as a volume, are not restricted to these values.- The “ended with errors” and “cancelled” states are spelled differently in each column on purpose (for example
Canceledin the CLI vsSTAGE_STATUS_CANCELEDin the enum) — match the spelling to the context you’re working in.
Fuzzball sometimes has to take a running step off the node it is on:
- The node goes offline – power loss, a killed agent, an instance terminated underneath Fuzzball. This needs no configuration and is the most common of the three.
- An operator drains the node with
--active. - The node’s hardware reports a fault the cluster is configured to act on.
What happens next depends on whether the step can be safely run again.
Fuzzball does not checkpoint. Running a step again runs all of it again, including anything it has already written. Only the workflow’s author knows whether that is safe, so it is declared per step:
jobs:
simulation:
image:
uri: docker://example/simulation:latest
policy:
retry:
on: node
attempts: 3
resource:
cpu:
cores: 8
memory:
size: 32GB
on value | Retried when | Runs again where |
|---|---|---|
node | The node stops hosting the work – an evict hold, or the node going offline | Another node; this one is being emptied |
An exit code, 1 to 255 | The job’s process exits on its own with exactly that code | Wherever the scheduler places it, as if new |
on accepts a scalar or a list, and a scalar is exactly a one-element list –
on: node and on: [node] mean the same thing, as do on: 137 and
on: [137]. Causes can be mixed: on: [node, 137, 1]. Listing the same
cause twice is an error, because a repeated code is usually a typo for a
different one.
on is required. Without it nothing is retried, whatever else retry
says. A definition carrying only attempts retries nothing, which is how that
field has always behaved.
node covers deliberate and involuntary loss alike, deliberately. An evict
hold and a crashed node leave the step in the same position, and an author who
wants their work to survive losing a node should not have to enumerate the ways
a node can be lost.
attempts is one budget, shared by every cause in on: a step that has been
moved off one node and retried once for an exit code has used two. Node retries
are additionally bounded by a cluster-wide ceiling,
nodeHealth.maxEvictionRestarts, which counts node retries only, and the lower
of the two applies – so a workflow can lower its own allowance but cannot raise
it past what the operator permits. Exhausting the budget fails the step,
recording which cause exhausted it.
When attempts is unset, node retries are bounded by the cluster ceiling alone,
and exit-code retries by a budget of 3, the default of that ceiling, shared
with any node retries the step has had.
An exit code in on retries the job when its process exits on its own with
exactly that code – an out-of-memory kill (137), say, or a code your own
program uses to mean “transient, try again”:
jobs:
fetch:
image:
uri: docker://example/fetch:latest
command: [/usr/bin/fetch, --all]
policy:
retry:
on: [node, 75, 137]
attempts: 3
Write exit codes as YAML integers (137) or numeric strings ("137"); both
mean the same. The job runs again from the beginning, placed wherever the
scheduler finds room, exactly as a job moved off a node is.
Some codes are rejected when the workflow is submitted:
0: success is not a failure, so there is nothing to retry.- Anything outside 1-255: not an exit code a process can have.
130and143: a job that ends on SIGINT (130) or SIGTERM (143) is treated as a clean stop rather than a failure – that is how a cancelled job ends – so it never fails with either code, and can never be retried for it.
Only a failure that carries the process’s exit code is matched. A job that failed for any other reason – its node went away, its time limit ran out, it never started – is not retried for an exit code, and neither is a failure reported by a compute node running a Fuzzball release that predates exit-code reporting.
A multi-node job is retried as a whole. Its first rank to fail fails the group, and that rank’s exit code decides: if it is listed, every rank is stopped and the job is requeued as one, which is also what a node retry does. A task array is requeued the same way a node retry requeues it: the failed task and any tasks still running start again, and tasks that already finished do not.
Each retry is recorded on the stage as a retried workflow event carrying the
exit code, the rank and node it came from, and which retry this is out of how
many. A step that exits with a listed code after using its whole budget fails
as it would have without retry, and gets a retries_exhausted event with the
cause exit_code and the limit source policy.retry.attempts.
defaults.job.policy is applied field by field. A job takes the default
timeout when it sets none of its own, and the default’s retry declaration –
its retry, or the deprecated restart if that is what the defaults set – when
it declares neither a retry nor a restart. Setting only a timeout on a job
no longer drops the workflow’s default retry for it.
This changes what an existing workflow does. Before, a job with any
policyof its own inherited only the defaulttimeout, so a workflow whosedefaults.job.policysetrestart: on-node-fault(or aretry) ran a job that set only atimeoutwith no restart at all. That job is now moved off a failing node and run again from the beginning. To keep a job from inheriting the default, give it aretryof its own with no cause, which retries nothing:policy: timeout: execute: 10m retry: attempts: 1An empty
retry: {}does not do this: it is dropped when the workflow is read, and the job inherits the default.
The default retry is inherited whole – on and attempts together – or not
at all. A job that declares retry: {attempts: 5} has declared its own retry
and takes none of the default’s causes, so its budget and the default’s causes
are never mixed into a combination neither declaration states. A job that
declares the deprecated restart keeps it, and does not inherit the default
retry, because the two may not be set together.
defaults:
job:
policy:
retry:
on: [node, 137]
attempts: 2
jobs:
inherits: # takes on: [node, 137] and attempts: 2
image:
uri: docker://alpine:latest
command: [/bin/sh, -c, ./step.sh]
policy:
timeout:
execute: 10m
overrides: # retried on exit code 3 only, up to the default 3 times
image:
uri: docker://alpine:latest
command: [/bin/sh, -c, ./step.sh]
policy:
retry:
on: 3
policy.restart asked a subset of the same question and is deprecated. It
still works, and existing workflow definitions keep behaving exactly as they
did:
| Written today | Treated as |
|---|---|
restart: on-node-fault | retry: {on: node} |
restart: never | No retry |
restart unset | No retry, unchanged |
Setting both restart and retry.on is an error rather than one silently
winning, because a silent winner means one of the two declarations is ignored
and the author cannot tell which – on a question about whether their work may
be run twice.
Declaring nothing at all stays the conservative default: a routine drain leaves the step running, and losing the node fails it rather than repeating work that may not be safe to repeat.
Only declareretry.onfor steps that are safe to repeat. A step that appends to a file, submits an order, or writes partial output to a shared volume will do all of it a second time – Fuzzball does not checkpoint, so a retry runs the step from the beginning.
A step that is restarted keeps its identity – same job name, same workflow – so its logs and status continue to be reported in the same place. There is a cluster-wide limit on how many times one step may be moved by node faults; beyond it the step is failed rather than moved again, so work cannot circulate through failing hardware indefinitely.
When a node ends a step, the error names the node rather than reporting only that the job stopped responding. It names no fault class: Fuzzball does not diagnose hardware, and a rack losing power, an agent being killed and an operator emptying a node look the same from the control plane.
A step that was moved rather than failed says so, so the two outcomes are not mistaken for one another:
node gpu-14 stopped hosting this work; moved to another node
A step that had not declared policy.retry.on: node is failed, and told how to
opt in:
node gpu-14 stopped hosting this work; job failed. Set policy.retry.on to node for work that is safe to run again from the beginning
A step that had declared it, and has been moved as often as the cluster allows, is failed with the count instead – repeating the advice would be telling the author to set something they already set:
node gpu-14 stopped hosting this work; job failed after 3 node restart(s), the limit for this step
Where a hold caused the move, the hold’s own reason is carried instead, because somebody wrote that sentence about this node deliberately:
node gpu-14 was held: failing disk, replacing Friday; job failed after 3 node restart(s), the limit for this step
Each move and failure is also recorded on the stage as an evicted workflow
event naming the node and, for a hold, the hold’s id, source and reason; a step
that ran out of restarts gets a retries_exhausted event saying which limit it
reached – policy.retry.attempts or the cluster’s
nodeHealth.maxEvictionRestarts – and the cause of the eviction that found
the restarts spent. A moved step
that cannot find room to run again reports a scheduling_blocked event with the
reason requeued_awaiting_capacity. See
Reading evicted and retries_exhausted Events.
See Node Health Monitoring for how an administrator configures which faults trigger this, and Nodes that go offline for the case that needs no configuration at all.