Fuzzball Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

Fuzzfile Syntax Guide

Overview

Workflows are the fundamental unit of work in Fuzzball, orchestrating container execution, data movement, and resource allocation across compute clusters. They are described by Fuzzfiles — YAML 1.2 documents with a well-defined structure and syntax. A workflow definition is built from several top-level sections. Between them they describe how data flows into and out of the workflow, what compute operations to perform, what resources those operations require, and how jobs depend on each other. The top-level sections are:

version: v4
preemptible:
files:
defaults:
annotations:
volumes:
jobs:
services:

Each section serves a specific purpose:

  • version: Specifies the workflow syntax version (v1 or v4; new workflows must use v4) [required]
  • preemptible: Marks all of the workflow’s allocations as eligible for preemption [optional]
  • files: Defines inline file content that can be referenced in volumes and jobs [optional]
  • defaults: Sets default configurations (mounts, environment, policy, resources) for jobs [optional]
  • annotations: Provides workflow-level metadata for scheduling and placement [optional]
  • volumes: Defines storage volumes with data ingress (importing) and egress (exporting) capabilities [optional]
  • jobs: Specifies the computational work to be performed, organized as a directed acyclic graph (DAG) [required*]
  • services: Defines long-running container services that support jobs or provide external endpoints [required*]

* A workflow must have at least one job or service.

Top-Level Sections

version

Required. Denotes the workflow syntax version. Valid values are v1 (legacy) and v4. New workflows must use v4 and the V4 Fuzzfile syntax. The v1 format is supported only for backward compatibility with existing workflows. Note that all snippets in this reference use V4 syntax.

Fuzzball makes a best-effort attempt to auto-upgrade v1 workflows to v4 at execution time. V1 volume references (reference: volume://...) and old-format mounts are automatically converted. However, writing version: v4 for new workflows avoids the upgrade step and ensures access to all V4 features.

Default version in the Web UI

When creating a new workflow in the web UI (via “Create New” in the Workflow Editor) or hydrating a workflow template from the Workflow Catalog that lacks an explicit version field, the web UI automatically defaults to version: v4.

The Workflow Editor works in v4 syntax. When you open a workflow, upload a YAML file, or paste a workflow into the YAML view, the editor upgrades any v1 version or v1 syntax (volume references and old-format mounts) to v4 through the same server-side upgrade used when a workflow runs, and tells you when it has done so.

Structure:

version: v4

Example:

version: v4

Upgrading existing V1 workflows

If you have existing v1 Fuzzfiles that you want to convert to v4 format permanently (rather than relying on the automatic upgrade at execution time), use the CLI upgrade command:

$ fuzzball workflow upgrade my-workflow.fz > my-workflow-v4.fz

The upgrade runs server-side with your identity context, allowing persistent volume names to be resolved against the provisioners available to your group. The output is YAML by default; use -o json for JSON output.

To verify that a workflow is valid before submitting it, use validate. It accepts both V1 and V4 Fuzzfiles. A V1 workflow is auto-upgraded and then validated, exactly as submission does, so a file that fuzzball workflow start accepts also passes validate. validate also refuses what submission refuses in the definition itself, such as a named volume that also sets size or annotations, or a job with no command or script:

$ fuzzball workflow validate my-workflow.fz

The upgrade command is useful for batch-converting a library of V1 workflows. Redirect the output to a new file so you can review the changes before replacing the original:

$ fuzzball workflow upgrade old-workflow.fz > new-workflow.fz
$ diff old-workflow.fz new-workflow.fz

preemptible

Optional. When true, marks every allocation produced by the workflow — jobs, services, and the internal image, volume, and file stages — as eligible for preemption by higher-priority allocations on clusters where preemption is enabled. Defaults to false.

Setting this field is equivalent to starting the workflow with fuzzball workflow start --preemptible. A job or service that sets its own preemptible value explicitly overrides the workflow-level setting.

Structure:

preemptible: <boolean>

Example:

version: v4
preemptible: true
jobs:
  batch:                # preemptible via the workflow-level setting
    image:
      uri: docker://alpine:latest
    command: [run-batch]
  critical:
    image:
      uri: docker://alpine:latest
    command: [run-critical]
    preemptible: false  # overrides the workflow-level setting

files

Optional. Defines inline file contents that can be referenced in volume ingress and jobs using a file://<arbitrary_name> URI scheme. This is useful for embedding small configuration files, scripts, or data directly in the workflow definition rather than storing them externally. Note that some content may have to be base64 encoded to ensure that the resulting yaml file is valid.

Structure:

files:
  <file-name>: |
    <file-contents>
  <another-file-name>: |
    <more-contents>

Fields:

  • <file-name>: An arbitrary name to identify this inline file. This name is used in file:// URIs to reference the content. Can be any valid YAML key.
  • <file-contents>: The actual content of the file. Use the | literal block scalar for multi-line content.

Example:

files:
  config.ini: |
    [settings]
    debug=true
    timeout=30
  data.csv: |
    id,name,value
    1,alpha,42
    2,beta,3.14

Usage:

Inline files can be referenced in two ways:

  1. In volume ingress - to copy the content into a volume:

    volumes:
      scratch:
        use: ephemeral
        size: 50GB
        ingress:
          - source:
              uri: file://config.ini          # References the inline file
            destination:
              uri: file://config/app.ini      # Destination in the volume
    
  2. In job files - to bind mount the content directly into a container:

    jobs:
      my-job:
        image:
          uri: docker://alpine:latest
        files:
          /etc/app/config.ini: file://config.ini   # Mounted at this path in container
        script: |
          #!/bin/sh
          cat /etc/app/config.ini
    

defaults

Optional. Sets default configurations that are applied to all jobs in the workflow. Individual jobs can override these defaults by specifying their own values. This is useful for reducing repetition when multiple jobs share common settings like volume mounts or environment variables.

Structure:

defaults:
  job:
    env:                    # Optional
      - <VAR>=<value>
    mounts:                 # Optional
      <mount-path>: <volume-name>
    policy:                 # Optional
      timeout:
        execute: <duration>
      retry:
        on: <node | exit code>  # a scalar, or a list
        attempts: <number>
    resource:               # Optional
      cpu:
        cores: <number>
        affinity: <CORE|SOCKET|NUMA>
        sockets: <number>
        threads: <boolean>
      memory:
        size: <size>
        by-core: <boolean>
      devices:
        <device-type>: <count>
      exclusive: <boolean>
      annotations:
        <key>: <value>

Fields:

  • job: Container for job-level defaults
    • env (optional): List of environment variables in KEY=VALUE format that will be set in all job containers

    • mounts (optional): Map of absolute volume mount paths to volume names.

    • policy (optional): Default execution policies for all jobs

      • timeout (optional): Time limits for job execution
        • execute: Maximum duration (e.g., 2h, 30m, 1h30m)
      • retry (optional): Whether the step may be run again, and what for. See Retrying a step
        • on: What the step may be retried for. node means the node stopped hosting it; a number from 1 to 255 is an exit code the job may be retried for. Accepts a scalar or a list
        • attempts (optional): How many times, one budget shared by every cause. Node retries are also bounded by the cluster’s own ceiling

      A job inherits the default timeout when it sets none, and the default retry when it declares neither retry nor the deprecated restart. See Inheriting a default retry

    • resource (optional): Default hardware resource requirements

      • cpu (required if resource is specified): CPU requirements
        • cores: Number of CPU cores
        • affinity: Binding strategy - CORE, SOCKET, or NUMA
        • sockets (optional): Number of CPU sockets
        • threads (optional): Whether to expose hardware threads (hyperthreading). Inherited by jobs that leave threads unset; an explicit true or false on the job wins
      • memory (required if resource is specified): Memory requirements
        • size: Amount of RAM (e.g., 4GB, 512MB)
        • by-core (optional): If true, size is per CPU core. Inherited by jobs that leave by-core unset; an explicit true or false on the job wins
      • devices (optional): Map of device types to counts (e.g., nvidia.com/gpu: 1). Request cpu/x86: 1 or cpu/arm64: 1 to run on that CPU architecture. Fuzzball requests one automatically when the image URI names an architecture; see Image architecture
      • exclusive (optional): If true, job gets exclusive access to the node. Inherited by jobs that leave exclusive unset; an explicit true or false on the job wins
      • annotations (optional): Custom key-value pairs for advanced scheduling

Example:

defaults:
  job:
    env:
      - SCRATCH=/scratch
      - LOG_LEVEL=info
    mounts:
      /scratch: scratch
    policy:
      timeout:
        execute: 1h
    resource:
      cpu:
        cores: 2
        affinity: CORE
        threads: false
      memory:
        size: 4GB
        by-core: false

In this example, all jobs will automatically:

  • Have /scratch and LOG_LEVEL environment variables set
  • Mount the scratch volume at /scratch
  • Timeout after 1 hour
  • Request 2 CPU cores and 4GB of memory

Individual jobs can override any of these defaults by specifying their own values. Default and job specified environment variables are merged with job variables taking precedence over defaults. Boolean resource fields (threads, by-core, exclusive) are inherited only when the job leaves them unset; a job that sets a flag explicitly, including to false, keeps its own value. The same rule applies to service defaults under defaults.service and to the top-level defaults.resource block, which applies to both jobs and services.

annotations

Optional. Sets workflow-level annotation defaults as key-value pairs. These annotations are merged into each job’s allocation annotations; per-job resource.annotations values take precedence over these defaults when the same key appears in both places. The merged set is what the scheduler matches key-by-key against each candidate node provisioner’s annotations map — node provisioners that do not have a matching entry for a given key are rejected for that job.

Structure:

annotations:
  <key>: <value>
  <another-key>: <another-value>

Fields:

  • <key>: String key identifying the annotation
  • <value>: String value for the annotation

Example:

annotations:
  nvidia.com/gpu.model: A100
  nvidia.com/gpu.memory: "81920"  # MiB per GPU, so at least 80 GiB

Common Use Cases:

  • GPU selection: Require a specific GPU model, family, or minimum memory
  • Node pool selection: Target specific sets of compute nodes that expose a matching annotation

Known annotation keys:

Fuzzball understands the following NVIDIA GPU annotations, which describe the GPUs on a node. The values come from the hardware report each node sends when it boots. Until a cloud instance type has been booted once, its node provisioner may carry only some of these keys, taken from the cloud provider’s catalog. The scheduler matches against the node provisioner’s annotations, so to see the values you can target, run fuzzball node provisioner get <provisioner> and look under resource.annotations.

KeyMeaningMatching
nvidia.com/gpu.modelGPU model name, such as NVIDIA A100Case-insensitive substring, so A100 matches NVIDIA A100
nvidia.com/gpu.productGPU product name, with spaces replaced by hyphens (NVIDIA-A100)Case-insensitive substring
nvidia.com/gpu.archGPU architecture, such as AmpereExact
nvidia.com/gpu.familyGPU familyExact
nvidia.com/gpu.memoryMemory per GPU, in MiBNode value must be at least the requested value
nvidia.com/gpu.countNumber of GPUs on the nodeNode value must be at least the requested value
nvidia.com/gpu.compute.majorMajor version of the GPU’s CUDA compute capabilityNode value must be at least the requested value
nvidia.com/gpu.compute.minorMinor version of the GPU’s CUDA compute capabilityExact

Any other key is matched by exact string comparison against the node provisioner’s annotations, so it only selects nodes whose provisioners an administrator has given that annotation.

Keys such as nvidia.com/gpu, amd.com/gpu, cpu/x86, cpu/arm64, and cpu/x86/avx are devices, not annotations. Request them with resource.devices (for example, nvidia.com/gpu: 2), not with annotations.

Annotation matching is strict by default. Every key in the merged allocation annotations (from both workflow-level annotations and per-job resource.annotations) must either be present with a matching value in each candidate node provisioner’s annotations map, or the cluster admin must add the key to scheduler.ignoredAnnotations in the central configuration. The platform’s own internal annotation keys — a specific enumerated set under fuzzball.io/* such as fuzzball.io/workflow.id, fuzzball.io/job.name, and fuzzball.io/provisioner_cluster_id, plus the dynamically keyed fuzzball.io/connect/<service> namespace — are skipped automatically. Apart from the connect namespace this is an allowlist, not a prefix match: the fuzzball.io/ prefix is reserved for the platform, and any user-defined key placed under that prefix (e.g. fuzzball.io/my-team-label) is not auto-exempt and would still need to be added to scheduler.ignoredAnnotations. Treat the fuzzball.io/ namespace as off-limits for your own annotations.

Annotation keys used only for organizational metadata, auditing, or policy routing should not appear in either annotations or resource.annotations unless the cluster admin has added them to scheduler.ignoredAnnotations; otherwise the scheduler will reject any node provisioner that does not carry those keys.

See Scheduler annotation matching in the configuration reference for built-in GPU matchers and configuration details.

volumes

Optional. Defines storage volumes that jobs can access. Volumes provide persistent or ephemeral storage for workflow data and support data ingress (importing data at workflow start) and egress (exporting data at workflow end).

Structure:

volumes:
  <volume-name>:
    use: <provisioner-name>    # Optional
    name: <persistent-name>    # Optional (only for persistent volumes)
    size: <capacity>           # Optional (only for ephemeral volumes)
    annotations:               # Optional (only for ephemeral volumes)
      <key>: <value>
    ingress:                   # Optional
      - source:
          uri: <source-uri>
          secret: <secret-ref>    # Optional
        destination:
          uri: <dest-uri>
        policy:                # Optional
          timeout:
            execute: <duration>
    egress:                    # Optional
      - source:
          uri: <source-uri>
        destination:
          uri: <dest-uri>
          secret: <secret-ref>    # Optional
        policy:                # Optional
          timeout:
            execute: <duration>

Fields:

  • <volume-name>: Arbitrary name for the volume, used to reference it in job mounts sections (e.g., scratch, data)
  • use (optional): Storage provisioner name. Can also be ephemeral or persistent for automatic provisioner selection
  • reference (deprecated): Legacy V3 volume URI format (volume://scope/class/name). Still supported for backward compatibility but use/name/size/annotations is preferred
  • ingress (optional): List of files to import into the volume at workflow start
    • source (required): Where to fetch the data from
      • uri: Source location. Supported schemes: s3://, http://, https://, file:// (references inline files from the files section), fuzzball:// and fb:// (fuzzball://group/<path> or fuzzball://user/<path>, referencing an object stored in the cluster’s object cache), and hf:// (or the longer huggingface:// alias) pulling a model, dataset, or space repository from the HuggingFace Hub — see HuggingFace ingress. Workflow submission rejects any other scheme, including the undocumented local://
      • secret (optional): Reference to credentials for accessing the source (format: secret://<scope>/<name>)
    • destination (required): Where to place the data in the volume
      • uri: Destination path in the volume (format: file://<path>). When jobs mount this volume, the path will be relative to the mount point
    • policy (optional): Execution policies for the transfer
      • timeout (optional): Time limit for the transfer
        • execute: Maximum duration (e.g., 5m, 1h)
      • Transfers are not retried, so retry.on is refused here; retry with only attempts is accepted and has no effect
  • egress (optional): List of files to export from the volume at workflow end. Structure is similar to ingress, but source is the file in the volume and destination is the external storage location. Egress runs after every job and service that mounts the volume is done. A job, or a service that is not an autoscaled replica pool, is done when it reaches FINISHED. A non-persist service reaches FINISHED when Fuzzball stops it (once nothing that depends on it is still running), so egress data produced by work the service does after readiness should be ordered with dependsOn. A persist: true service reaches FINISHED when its own command exits successfully; if it never exits, egress for that volume does not run and cancelling the workflow discards it. An autoscaled replica-pool service is done when the pool is idle: every replica has been torn down on its node or is waiting unstarted in the pool, and any service whose scale-up triggers target the pool has reached FINISHED. A pool woken after egress is released, for example by a request to one of its endpoints, may write more data, but those writes are not guaranteed to be included in the export.

Ephemeral volume fields — these fields define a volume that is created when the workflow runs and destroyed when it completes:

  • size (optional): Requested volume capacity (e.g., 10GB, 1TiB)
  • annotations (optional): Key-value pairs for provisioner auto-selection. Fuzzball matches these against provisioner annotations to select the right storage backend
The size and annotations fields are only valid for ephemeral volumes. They cannot be combined with the name field.

Persistent volume fields — these fields reference an existing volume that was created before the workflow (via the CLI, web UI, or API):

  • name (optional): Name of an existing persistent volume to bind. The volume must already exist on the provisioner specified by use, or on any accessible provisioner if use is omitted. Workflows that reference a volume that does not exist are rejected at submission time.
The name field is the signal that distinguishes a persistent volume from an ephemeral one. When name is set, the volume is persistent. When name is absent, the volume is ephemeral. Persistent volumes are never created dynamically — they must be created ahead of time.

Example:

volumes:
  scratch:
    use: my-provisioner
    size: 50GB
    ingress:
      - source:
          uri: s3://my-bucket/input-data.tar.gz
          secret: secret://group/AWS_CREDENTIALS
        destination:
          uri: file://inputs/data.tar.gz
        policy:
          timeout:
            execute: 10m
      - source:
          uri: file://config.ini    # References inline file
        destination:
          uri: file://config/app.ini
    egress:
      - source:
          uri: file://results/output.tar.gz
        destination:
          uri: s3://my-bucket/results/output.tar.gz
          secret: secret://group/AWS_CREDENTIALS
  data:
    use: my-provisioner
    name: research-data

In this example:

  • scratch is ephemeral (no name, has size), downloads data from S3 at start, and uploads results at end
  • data is persistent (has name), survives workflow completion with no data transfers

Path Resolution Example:

If a job mounts scratch at /scratch, a file ingressed to file://inputs/data.tar.gz will be available in the job container at /scratch/inputs/data.tar.gz.

Volumes on node-local storage

Some provisioners use per-node storage rather than shared storage. A volume on one of those exists on a single node, so Fuzzball places everything that uses it on that node. This applies to ephemeral and persistent volumes alike, and it rules out anything that needs more than one node at once.

Fuzzball rejects these at submission rather than letting them fail part-way through:

  • Multinode jobs. The job spans more nodes than the volume exists on.
  • Task arrays running more than one task at once. Each concurrent task needs its own node. concurrency defaults to the size of the array, so an array of more than one task is rejected unless it sets concurrency: 1.
  • Autoscaled services whose autoscaler.replicas.max is greater than 1. Fuzzball places at most one replica of a pool per node.
  • Two node-local volumes that live on different nodes. No single placement reaches both. This is assessed across the jobs that tie the volumes together, so it applies even when no one job mounts both.
  • A persistent node-local volume with no recorded holding node, used alongside another. Fuzzball cannot confirm the two share a node.

A persistent node-local volume is also refused on its own when it has no recorded holding node on a cluster of more than one node, or when the node it is recorded on is not currently registered.

To check whether a provisioner is node-local, run fuzzball volume provisioner info <name> and look for local: true under driver:; the field is omitted when the storage is shared. See When the path is not shared for how node-local provisioners are configured.

jobs

Naming Constraint: Job and service names must be unique within a workflow — a job and a service cannot share the same name. Images, volumes, and ingress/egress file IDs share that same namespace, so none of them may reuse a job or service name either. The constraint exists because allocation IDs and dependency references are keyed by name alone. Fuzzball rejects a workflow with a name conflict at submission and returns a validation error:

workflow validation failed: name "<name>" is used by both a job and a service; job and service names must be unique within a workflow

For example, this workflow definition is invalid and will be rejected:

version: v4
jobs:
  database:
    image:
      uri: docker://alpine:latest
    command: ["echo", "job"]
    resource:
      cpu:
        cores: 1
      memory:
        size: 1GB
services:
  database:  # ERROR: name already used by a job
    image:
      uri: docker://postgres:16
    resource:
      cpu:
        cores: 2
      memory:
        size: 4GB

Ensure all job and service names are distinct before submission.

Optional (but at least one job or service is required). Defines the computational work to be performed as a directed acyclic graph (DAG) of compute steps. Jobs can run scientific software, data processing scripts, or any containerized application. Jobs are the fundamental execution unit in a workflow.

Structure:

jobs:
  <job-name>:
    image:                            # Required
      uri: <image-uri>
      secret: <secret-ref>            # Optional
      decryption-secret: <secret-ref> # Optional
    command: [<arg1>, <arg2>, ...]    # One of command or script required (mutually exclusive with script)
    script: |                         # One of command or script required (mutually exclusive with command)
      <script-content>
    args: [<arg1>, <arg2>, ...]       # Optional
    env:                              # Optional
      - <VAR>=<value>
    cwd: <working-directory>          # Optional
    mounts:                           # Optional
      <mount-path>: <volume-name>
    files:                            # Optional
      <container-path>: <file-uri>
    resource:                         # Optional
      cpu:                            # Required if resource specified
        cores: <number>
        affinity: <CORE|SOCKET|NUMA>
        sockets: <number>             # Optional
        threads: <boolean>            # Optional
      memory:                         # Required if resource specified
        size: <size>
        by-core: <boolean>            # Optional
      devices:                        # Optional
        <device-type>: <count>
      exclusive: <boolean>            # Optional
      annotations:                    # Optional
        <key>: <value>
    policy:                           # Optional
      timeout:
        execute: <duration>
      retry:                          # Optional
        on: <node | exit code>        # a scalar, or a list
        attempts: <number>            # Optional
    requires: [<job1>, <job2>, ...]   # Optional (deprecated, use depends-on)
    depends-on:                       # Optional
      - name: <job-name>
        status: <RUNNING|FINISHED|FAILED>
        description: <text>           # Optional
    multinode:                        # Optional (mutually exclusive with task-array, network)
      nodes: <number>
      implementation: <ompi|openmpi|mpich|gasnet|generic>
      procs-per-node: <number>        # Optional
    network:                          # Optional (mutually exclusive with multinode, task-array)
      isolated: <boolean>             # Required (must be true)
      expose-tcp: [<number>, <number>, ...] # Optional
      expose-udp: [<number>, <number>, ...] # Optional
    task-array:                       # Optional (mutually exclusive with multinode, network)
      start: <number>
      end: <number>
      concurrency: <number>           # Optional
    preemptible: <boolean>            # Optional

Fields:

  • <job-name>: Arbitrary name identifying this job (e.g., preprocess-data, train-model). Must be a valid DNS subdomain and must be unique within a workflow. This name appears in fuzzball workflow status and is used in dependency specifications
  • image (required): Container image specification
    • uri: Image location. Supported schemes: docker:// for OCI containers, oras:// for SIF images (e.g., oras://depot.ciq.com/fuzzball/fuzzball-applications/curl.sif:latest)
    • secret (optional): Credentials for private registries (format: secret://<scope>/<name>)
    • decryption-secret (optional): Secret to decrypt encrypted SIF images
  • command (required, mutually exclusive with script): The program to run and its arguments, as a list (e.g., [python3, script.py, --input, /data]). The first element must be an executable. Fuzzball runs the list as given and does not prepend the image’s ENTRYPOINT
  • script (required, mutually exclusive with command): Multi-line shell script to execute. If no #! shebang line is present, #!/bin/sh is assumed. Include an explicit shebang for other interpreters (e.g., #!/bin/bash, #!/usr/bin/python3)
  • args (optional): Additional arguments passed to command or script
  • env (optional): List of environment variables in KEY=VALUE format. Fuzzball also sets variables of its own, such as FB_WORKFLOW_ID and FB_JOB_NAME, in every container; see the Environment Variables Reference
  • cwd (optional): Working directory for the job. Must be an absolute path. Defaults to the image’s working directory or /
  • mounts (optional): Map of absolute volume mount paths to volume names (from the volumes section)
  • files (optional): Map of container paths to inline file URIs. Bind mounts inline files directly into the container (e.g., /etc/config.ini: file://my-config)
  • resource (optional): Hardware resource requirements for scheduling
    • cpu (required if resource specified): CPU requirements
      • cores: Number of CPU cores (must be > 0)
      • affinity: Binding strategy - CORE (any cores), SOCKET (same socket), or NUMA (same NUMA domain). Defaults to CORE
      • sockets (optional): Number of physical CPU sockets
      • threads (optional): Whether to expose hardware threads (hyperthreading)
    • memory (required if resource specified): Memory requirements
      • size: Amount of RAM with units (e.g., 4GB, 512MB, 2GiB)
      • by-core (optional): If true, size is per CPU core, i.e. total size is cores * size
    • devices (optional): Map of device types to counts (e.g., nvidia.com/gpu: 2). Request cpu/x86: 1 or cpu/arm64: 1 to run on that CPU architecture. Fuzzball requests one automatically when the image URI names an architecture; see Image architecture
    • exclusive (optional): If true, job gets exclusive node access
    • annotations (optional): Custom key-value pairs for advanced node selection (e.g., GPU model). See Known annotation keys
  • policy (optional): Execution policies
    • timeout (optional): Time limits
      • execute: Maximum job duration (e.g., 2h, 30m)
    • retry (optional): Whether the job may be run again, and what for. See Restart Tolerance
      • on: node, an exit code from 1 to 255, or a list of them
      • attempts (optional): The most retries, shared by every cause
  • requires (optional, deprecated): List of job names that must complete before this job starts. Use depends-on instead
  • depends-on (optional): List of concrete dependencies with status requirements. Note that depending on a job array to finish will wait for all tasks to finish.
    • name: Job or service name to depend on
    • status: Required status - FINISHED (job/service completed successfully), RUNNING (job/service is running), or FAILED (job/service errored). A FAILED dependency is an “on failure” dependency: the dependent starts only when the named job/service errors, which is useful for recovery or notification steps. It does not trigger on user cancellation or preemption. If the named job/service instead succeeds, the on-failure dependent is not needed and is skipped. A FAILED dependency is supported only when the named target is a single-node job or service; multi-node targets are rejected at submission. For a service target, the on-failure dependent fires if the service errors while another job is using it; if the service is reaped or stops normally it is recorded as finished and the on-failure dependent is skipped. See Workflow Status Values for how these map to the CLI status labels and the STAGE_STATUS_* enum.
    • description (optional): Human-readable explanation of the dependency
  • multinode (optional, mutually exclusive with task-array/network): Multi-node parallel execution
    • nodes: Number of nodes to allocate
    • implementation: MPI/communication implementation - ompi, openmpi, mpich, gasnet, or generic
    • procs-per-node (optional): Processes per node. Defaults to number of allocated CPUs
  • network (optional, mutually exclusive with multinode and task-array): If present, job is run inside its own (isolated) network namespace. Each exposed container port is published on the node at a randomly assigned host port.
    • isolated: Must be true when the network section is specified.
    • expose-tcp (optional): List of container TCP ports to expose
    • expose-udp (optional): List of container UDP ports to expose
  • task-array (optional, mutually exclusive with multinode and network): Embarrassingly parallel execution
    • start: Starting task ID (inclusive, must be > 0)
    • end: Ending task ID (inclusive, must be >= start)
    • concurrency (optional): Maximum parallel tasks (max 200). Each task receives $FB_TASK_ID
  • preemptible (optional): If true, this job may be preempted by higher-priority allocations on clusters where preemption is enabled. When unset, the job inherits the workflow-level preemptible setting (or the --preemptible start flag); an explicit true or false overrides it

Example:

jobs:
  preprocess:
    image:
      uri: docker://python:3.11
    script: |
      #!/bin/sh
      python3 preprocess.py --input /data/raw/${FB_TASK_ID} --output /data/processed/${FB_TASK_ID}
    env:
      - PYTHONUNBUFFERED=1
    mounts:
      /data: data
    resource:
      cpu:
        cores: 4
        affinity: NUMA
      memory:
        size: 8GB
    policy:
      timeout:
        execute: 30m
    task-array:
      start: 1
      end: 1000
      concurrency: 100

  train-multinode:
    image:
      uri: docker://nvcr.io/nvidia/pytorch:24.01-py3
      secret: secret://user/NGC_API_KEY
    script: |
      #!/bin/bash
      python -m torch.distributed.run train.py --input /data/processed
    depends-on:
      - name: preprocess
        status: FINISHED
    resource:
      cpu:
        cores: 32
        affinity: SOCKET
      memory:
        size: 128GB
      devices:
        nvidia.com/gpu: 4
      annotations:
        nvidia.com/gpu.model: NVIDIA L40
    multinode:
      nodes: 4
      implementation: openmpi
      procs-per-node: 4

This example demonstrates:

  • preprocess: Basic job task array with resource requests
  • train-multinode: Multi-node MPI job with GPUs and explicit dependencies

services

Naming Constraint: Job and service names must be unique within a workflow — a job and a service cannot share the same name. Images, volumes, and ingress/egress file IDs share that same namespace, so none of them may reuse a job or service name either. The constraint exists because allocation IDs and dependency references are keyed by name alone. Fuzzball rejects a workflow with a name conflict at submission and returns a validation error:

workflow validation failed: name "<name>" is used by both a job and a service; job and service names must be unique within a workflow

See the jobs section for a detailed example of this validation constraint.

Optional (but at least one job or service is required). Defines long-running container services that support jobs or provide external endpoints. Unlike jobs which complete and exit, services run continuously for the workflow duration (or until dependent jobs finish). Services are useful for databases, web servers, message queues, interactive computing (e.g. jupyter or Rstudio), AI inference servers, or any persistent service that jobs need to access.

Structure:

services:
  <service-name>:
    image:                         # Required
      uri: <image-uri>
      secret: <secret-ref>         # Optional
      decryption-secret: <secret-ref> # Optional
    command: [<arg1>, <arg2>, ...] # One of command or script required (mutually exclusive with script)
    script: |                      # One of command or script required (mutually exclusive with command)
      <script-content>
    args: [<arg1>, <arg2>, ...]    # Optional
    env:                           # Optional
      - <VAR>=<value>
    cwd: <working-directory>       # Optional
    mounts:                        # Optional
      <mount-path>: <volume-name>
    files:                         # Optional
      <container-path>: <file-uri>
    resource:                      # Optional
      cpu:
        cores: <number>
        affinity: <CORE|SOCKET|NUMA>
        sockets: <number>          # Optional
        threads: <boolean>         # Optional
      memory:
        size: <size>
        by-core: <boolean>         # Optional
      devices:                     # Optional
        <device-type>: <count>
      exclusive: <boolean>         # Optional
      annotations:                 # Optional
        <key>: <value>
    requires: [<job1>, <svc1>, ...] # Optional (deprecated, use depends-on)
    depends-on:                     # Optional
      - name: <job-or-service-name>
        status: <RUNNING|FINISHED|FAILED>
        description: <text>        # Optional
    multinode:                     # Optional
      nodes: <number>
      implementation: <ompi|openmpi|mpich|gasnet|generic>
      procs-per-node: <number>     # Optional
    network:                       # Optional
      host: <boolean>              # Optional
      ports:
        - name: <port-name>
          port: <port-number>
          protocol: <tcp|udp>      # Optional
      endpoints:                   # Optional
        - name: <endpoint-name>
          port-name: <port-name>   # References port name above
          protocol: <http|https>     # Optional. Defaults to http
          type: <subdomain|path>   # Optional. Defaults to path
          scope: <endpoint-scope>  # Optional. One of: user, group, organization, public
          annotations:             # Optional
            <key>: <value>
          per-replica: <boolean>   # Optional; autoscaled pools only
    persist: <boolean>             # Optional
    readiness-probe:                # Optional
      exec:                        # One of: exec, http-get, tcp-socket, grpc
        command: [<arg1>, <arg2>]
      http-get:
        path: <path>
        port: <port-number>
        scheme: <HTTP|HTTPS>       # Optional
        http-headers:               # Optional
          - name: <header-name>
            value: <header-value>
      tcp-socket:
        port: <port-number>
      grpc:
        port: <port-number>
        service: <service-name>    # Optional
      initial-delay-seconds: <seconds>  # Optional
      period-seconds: <seconds>        # Optional
      timeout-seconds: <seconds>       # Optional
      success-threshold: <number>      # Optional
      failure-threshold: <number>      # Optional
    autoscaler:                     # Optional
      replicas:                    # Required for a self-scaling service
        min: <number>              # Optional (0 enables scale-to-zero)
        max: <number>              # Required, >= 1 and >= min
      metrics:                     # Optional
        enabled: <boolean>
        path: <metrics-path>
        port: <port-number>
        interval: <seconds>
        includes: [<metric1>, ...]
      scale-up:                    # Optional
        triggers:
          - metrics-query: <promql-expression>
            cooldown-period: <seconds>  # Optional
            services: [<svc1>, ...]     # Optional (cross-service scaling)
      scale-down:                  # Optional
        metrics-query: <promql-expression>
        cooldown-period: <seconds> # Optional
        drain-period: <seconds>    # Optional (default 30, max 600)
    dynamic-config:                 # Optional
      path: <container-path>
      services: [<svc1>, <svc2>]
      script: |
        <bash-script>
    preemptible: <boolean>         # Optional

Fields:

Services share many fields with jobs (image, command, script, args, env, cwd, mounts, files, resource, depends-on, multinode, preemptible). See the jobs section for details on these common fields. Service-specific fields are:

Important: A multinode service is not designed to run multiple identical instances of a service with load-balanced client connections. It is intended for distributed services (e.g., MPI-based, vLLM cluster …), where the endpoint is served exclusively on rank 0. Clients always connect to rank 0, which coordinates with the other instances internally.
  • <service-name>: Arbitrary name identifying this service. Must be a valid DNS subdomain and must be unique within a workflow.

  • network (optional): Network configuration for service exposure. If present and the list of exposed ports is not empty, service is run inside its own (isolated) network namespace (multinode services excepted – they always run on the host network, see host below), and each declared port is published on the node at the declared port. Other workflow members reach the service at <service>.svc.<workflow-id>.<account-id>.fuzzball:<declared-port> (the service name alone also resolves, via the workflow’s DNS search path). Because the declared port is claimed on the node, two services that declare the same port number cannot share a node: the scheduler places them on different nodes. A service whose declared port is taken on every node with room for it waits in the queue – the stage reports a scheduling_blocked event with reason port_conflict, and fuzzball workflow why reports PORT_CONFLICT – until a holder ends or a node joins. A port a finished or cancelled service held stays claimed for a few seconds more, while the node releases its container.

    • Autoscaled replica pools (services with autoscaler.replicas) are the exception: each replica’s declared port is published at a randomly assigned host port, so same-port replicas can pack onto one node. Clients of a pool must not connect to <node>:<declared-port> directly — discover each replica’s host port through the DNS SRV records named _<port-name>._<protocol>.<service>.autoscaler.<workflow-id>.fuzzball, or through dynamic-config arrays. See Service Autoscaling.
    • Endpoints and fuzzball workflow connect route to the published host port transparently. Endpoints are supported on replica pools, but must use type: subdomain and reference a named port – the path form selects a single row and cannot address a pool.
    • host (optional): If true, service serves on the host network namespace. Ports are used as declared (no host-port assignment), so two host-network services that bind the same port cannot share a node; the scheduler keeps them apart the same way. Forced to true for services with a multinode block.
    • ports: List of ports the service listens on
      • name: Identifier for this port (used in endpoints and SRV records)
      • port: Port number (1-65535)
      • protocol (optional): tcp or udp. Defaults to tcp
    • endpoints (optional): List of external endpoints to create
      • name: Endpoint identifier (255 bytes max)

      • port-name: References a port name from the ports list

      • protocol (optional): http or https. Defaults to http. The endpoint proxy forwards over HTTP, so stream protocols cannot be served

      • type (optional): Endpoint style - subdomain (creates <endpoint-id>.endpoints.<cluster-domain>) or path (creates /endpoints/<endpoint-id>). Defaults to path

      • scope (optional): Determines who can access the endpoint. One of user (only the workflow creator), group (anyone in the same group), organization (anyone in the same organization), public (anyone without authentication). Defaults to group if not specified.

      • annotations (optional): Map of key-value pairs stored with the endpoint and returned by fuzzball endpoint list and get. Fuzzball does not interpret them; clients discovering endpoints use them to tell which ones they should consume. Limits: at most 32 entries per endpoint, keys up to 253 characters, values up to 1024 characters, and 8192 bytes of keys and values combined. A key is an optional DNS subdomain prefix followed by / and an alphanumeric name of up to 63 characters (dashes, underscores and dots allowed within), for example example.com/role or role. The fuzzball.io prefix, and any subdomain of it, is reserved for the platform and rejected in your own annotations.

      • per-replica (optional): If true, every live replica of an autoscaled pool also gets its own endpoint, alongside the pool’s. Requires a service that defines autoscaler.replicas and type: subdomain. See addressing individual replicas.

      • persistent (optional): If true, the endpoint serves your persistent endpoint of the same name, which has to exist, instead of an endpoint of the workflow’s own. An omitted type, protocol or scope then takes the persistent endpoint’s, and a declared one must match it. Not supported on an autoscaled service. Defaults to false.

    ports and endpoints are YAML lists: each entry begins with a -, and a service may declare more than one. Write them as sequence items (as shown in the structure above), not as a single mapping.
  • persist (optional): If true, service continues running even after all dependent jobs finish and continues until the workflow is cancelled. If false (default), service stops when no jobs/services depend on it.

    “No dependents” includes a service that never had any. A service with persist unset and nothing declaring dependsOn it is stopped as soon as it reports ready – it reaches RUNNING, then stops, and the workflow finishes. Set persist: true for a service you intend to run on its own – for example a model server you connect to directly, or a service under test.

    The rule applies to every replica of an autoscaled service. Endpoint traffic, cross-service scale-up triggers, and dynamic-config do not count as dependents, so an autoscaled service needs persist: true. Without it, each replica is stopped as soon as it becomes ready instead of remaining in the serving pool.

  • policy (deprecated): Accepted only so that workflows written when services supported it still run. It has no effect: Fuzzball ignores it and returns a warning naming the service. Services have no execution timeout; to bound how long a service runs, run its command under timeout in the service’s script.

  • readiness-probe (optional): Kubernetes-style health check to determine when service is ready. Service status transitions from STARTED to RUNNING only after probe succeeds

    • exec: Run a command in the container. Success if exit code is 0
      • command: Command to execute
    • http-get: HTTP GET request. Success if status code is 200-399
      • path: HTTP path
      • port: Port number
      • scheme (optional): HTTP or HTTPS
      • http-headers (optional): Custom HTTP headers
    • tcp-socket: TCP connection attempt. Success if connection establishes
      • port: Port number
    • grpc: gRPC health check. Success per gRPC health checking protocol
      • port: Port number
      • service (optional): gRPC service name
    • initial-delay-seconds (optional): Delay before first probe
    • period-seconds (optional): Frequency of probes
    • timeout-seconds (optional): Probe timeout
    • success-threshold (optional): Consecutive successes needed
    • failure-threshold (optional): Consecutive failures before marking unhealthy
  • autoscaler (optional): Automatically scales the number of replicas of this service up and down based on PromQL queries evaluated against the service’s metrics. See Service Autoscaling for a full guide.

    • replicas: Bounds the replica count for a self-scaling service. Required unless the service’s scale-up triggers target other services.
      • min (optional): Minimum number of replicas. 0 enables scale-to-zero (the service runs no replicas until a trigger fires).
      • max: Maximum number of replicas. Must be at least 1 and not less than min.
    • metrics (optional): Configures collection of the service’s own metrics, used by the scale policies.
      • enabled: Set to true to scrape the service’s metrics endpoint.
      • path: HTTP path of the metrics endpoint (e.g. /metrics).
      • port: Container port exposing the metrics endpoint.
      • interval: Scrape interval, in seconds (default 15). Also sets the sampling interval for the default container CPU/memory metrics, even when enabled is not set.
      • includes (optional): Restricts collection to the named metrics. All metrics are collected when empty.
    • scale-up (optional): One or more triggers that add replicas (including from zero).
      • triggers: List of independent scale-up conditions. Each trigger:
        • metrics-query: A PromQL expression. The trigger fires when the query returns a true (non-zero) result. Use the bool modifier so the comparison yields 1/0 (e.g. vllm:num_requests_waiting >= bool 2).
        • cooldown-period (optional): Minimum number of seconds between consecutive scaling actions for the targeted service.
        • services (optional): Names of other services this trigger scales, instead of this service’s own replicas (cross-service scaling — see the autoscaling guide). Leave empty to scale the owning service itself.
    • scale-down (optional): A single policy that removes replicas of this service.
      • metrics-query: PromQL expression; the policy fires when it returns a true result.
      • cooldown-period (optional): Minimum number of seconds between consecutive scale-downs.
      • drain-period (optional): Number of seconds a retiring replica keeps running after it is removed from the service’s DNS records and dependent dynamic-configs, so requests in flight to it can complete. Default 30, maximum 600. Only valid on scale-down. Omitting the field (or setting 0) selects the default; draining cannot be disabled — use drain-period: 1 for the shortest drain.
  • dynamic-config (optional): Renders a configuration file from the live addresses of other services and bind-mounts it into this service’s container, re-rendering it as those services' replicas scale up and down.

    • path: Destination path of the rendered file inside the container.
    • services: Services whose addresses are made available to the script. Each service name is exposed as a bash array named SERVICES_<NAME> (uppercased, non-identifier characters replaced with _) holding the ready replicas’ IP addresses; for example service worker becomes ${SERVICES_WORKER[@]}. Each of the watched service’s named TCP ports additionally yields a SERVICES_<NAME>_<PORT-NAME> array of ip:port pairs carrying each replica’s assigned host port (see network above); for example port http of service worker becomes ${SERVICES_WORKER_HTTP[@]}. Use the ip:port arrays to address replicas — the assigned host port differs per replica.
    • script: A bash script whose standard output becomes the file contents. It runs in a restricted interpreter: builtins and heredocs only — no external commands (except a no-argument cat for cat <<EOF templating) and no filesystem access.

Example:

version: v4
volumes:
  db:
    use: ephemeral
    size: 10GB
services:
  postgres:
    image:
      uri: docker://postgres:16
    command: ["docker-entrypoint.sh", "postgres"]
    env:
      - POSTGRES_PASSWORD=secret
      - POSTGRES_DB=myapp
    mounts:
      /var/lib/postgresql/data: db
    resource:
      cpu:
        cores: 4
      memory:
        size: 8GB
    network:
      ports:
        - name: postgres
          port: 5432
          protocol: tcp
    readiness-probe:
      tcp-socket:
        port: 5432
      initial-delay-seconds: 5
      period-seconds: 10
    persist: true

  api-server:
    image:
      uri: docker://mycompany/api:v1.2.3
      secret: secret://group/REGISTRY_CREDS
    command: ["/usr/local/bin/api-server", "--port", "8080"]
    env:
      - DATABASE_URL=postgresql://postgres:5432/myapp
    depends-on:
      - name: postgres
        status: RUNNING
        description: "API needs database connection"
    resource:
      cpu:
        cores: 2
      memory:
        size: 4GB
    network:
      ports:
        - name: http
          port: 8080
      endpoints:
        - name: api
          port-name: http
          protocol: https
          type: subdomain
          scope: group
    readiness-probe:
      http-get:
        path: /health
        port: 8080
      initial-delay-seconds: 10
      period-seconds: 5
      failure-threshold: 3

jobs:
  data-processor:
    image:
      uri: docker://mycompany/processor:latest
    script: |
      #!/bin/sh
      python process.py --api-url http://api-server:8080
    depends-on:
      - name: api-server
        status: RUNNING
    resource:
      cpu:
        cores: 8
      memory:
        size: 16GB

This example shows:

  • postgres: Persistent database service with readiness probe
  • api-server: REST API depending on postgres, exposed via HTTPS subdomain endpoint at account scope
  • data-processor: Job that depends on api-server being running before it starts

Workflow Status Values

A workflow, a job, or another stage reports the same underlying state under different names depending on where you look: the status column the CLI prints, the raw WorkflowStatus enum returned by fuzzball workflow describe and the SDK, the depends-on: status: field in a Fuzzfile, and the --status filter.

StateCLI statusWorkflowStatus enumFuzzfile depends-on: status:--status filter
Waiting for resources or dependenciesPendingSTAGE_STATUS_UNSPECIFIED—pending
RunningStartedSTAGE_STATUS_STARTEDRUNNINGstarted
Completed without errorsFinishedSTAGE_STATUS_FINISHEDFINISHEDfinished
Ended with errorsErroredSTAGE_STATUS_FAILEDFAILEDerrored (alias failed)
Cancelled by a user, never started because its workflow was cancelled or failed, or a skipped on-failure dependentCanceledSTAGE_STATUS_CANCELED—canceled (alias cancelled)
Autoscaled service replica scaled to zeroDownSTAGE_STATUS_DOWN——

A few things to watch for:

  • The running state is written RUNNING in a Fuzzfile depends-on, but shown as Started / STAGE_STATUS_STARTED in CLI and enum output — the same state, a different word.
  • For job and service dependencies, depends-on: status: accepts RUNNING, FINISHED, and FAILED. Any other value (for example STARTED) is rejected and the workflow fails validation. A FAILED dependency starts the dependent only when its target errors – an “on failure” dependency for recovery or notification steps; it does not trigger on cancellation or preemption, and if the target succeeds the on-failure dependent is skipped. A skipped dependent never runs; it shows as Canceled, with a reason that starts skipped: and names the dependency. FAILED is currently supported only against single-node jobs and services; depending on a multi-node job’s failure is not yet supported. Stages that depend on something other than a job or service, such as a volume, are not restricted to these values.
  • The “ended with errors” and “cancelled” states are spelled differently in each column on purpose (for example Canceled in the CLI vs STAGE_STATUS_CANCELED in the enum) — match the spelling to the context you’re working in.

Restart Tolerance

Fuzzball sometimes has to take a running step off the node it is on:

  • The node goes offline – power loss, a killed agent, an instance terminated underneath Fuzzball. This needs no configuration and is the most common of the three.
  • An operator drains the node with --active.
  • The node’s hardware reports a fault the cluster is configured to act on.

What happens next depends on whether the step can be safely run again.

Fuzzball does not checkpoint. Running a step again runs all of it again, including anything it has already written. Only the workflow’s author knows whether that is safe, so it is declared per step:

jobs:
  simulation:
    image:
      uri: docker://example/simulation:latest
    policy:
      retry:
        on: node
        attempts: 3
    resource:
      cpu:
        cores: 8
      memory:
        size: 32GB
on valueRetried whenRuns again where
nodeThe node stops hosting the work – an evict hold, or the node going offlineAnother node; this one is being emptied
An exit code, 1 to 255The job’s process exits on its own with exactly that codeWherever the scheduler places it, as if new

on accepts a scalar or a list, and a scalar is exactly a one-element list – on: node and on: [node] mean the same thing, as do on: 137 and on: [137]. Causes can be mixed: on: [node, 137, 1]. Listing the same cause twice is an error, because a repeated code is usually a typo for a different one.

on is required. Without it nothing is retried, whatever else retry says. A definition carrying only attempts retries nothing, which is how that field has always behaved.

node covers deliberate and involuntary loss alike, deliberately. An evict hold and a crashed node leave the step in the same position, and an author who wants their work to survive losing a node should not have to enumerate the ways a node can be lost.

attempts is one budget, shared by every cause in on: a step that has been moved off one node and retried once for an exit code has used two. Node retries are additionally bounded by a cluster-wide ceiling, nodeHealth.maxEvictionRestarts, which counts node retries only, and the lower of the two applies – so a workflow can lower its own allowance but cannot raise it past what the operator permits. Exhausting the budget fails the step, recording which cause exhausted it.

When attempts is unset, node retries are bounded by the cluster ceiling alone, and exit-code retries by a budget of 3, the default of that ceiling, shared with any node retries the step has had.

Retrying on an Exit Code

An exit code in on retries the job when its process exits on its own with exactly that code – an out-of-memory kill (137), say, or a code your own program uses to mean “transient, try again”:

jobs:
  fetch:
    image:
      uri: docker://example/fetch:latest
    command: [/usr/bin/fetch, --all]
    policy:
      retry:
        on: [node, 75, 137]
        attempts: 3

Write exit codes as YAML integers (137) or numeric strings ("137"); both mean the same. The job runs again from the beginning, placed wherever the scheduler finds room, exactly as a job moved off a node is.

Some codes are rejected when the workflow is submitted:

  • 0: success is not a failure, so there is nothing to retry.
  • Anything outside 1-255: not an exit code a process can have.
  • 130 and 143: a job that ends on SIGINT (130) or SIGTERM (143) is treated as a clean stop rather than a failure – that is how a cancelled job ends – so it never fails with either code, and can never be retried for it.

Only a failure that carries the process’s exit code is matched. A job that failed for any other reason – its node went away, its time limit ran out, it never started – is not retried for an exit code, and neither is a failure reported by a compute node running a Fuzzball release that predates exit-code reporting.

A multi-node job is retried as a whole. Its first rank to fail fails the group, and that rank’s exit code decides: if it is listed, every rank is stopped and the job is requeued as one, which is also what a node retry does. A task array is requeued the same way a node retry requeues it: the failed task and any tasks still running start again, and tasks that already finished do not.

Each retry is recorded on the stage as a retried workflow event carrying the exit code, the rank and node it came from, and which retry this is out of how many. A step that exits with a listed code after using its whole budget fails as it would have without retry, and gets a retries_exhausted event with the cause exit_code and the limit source policy.retry.attempts.

Inheriting a Default Retry

defaults.job.policy is applied field by field. A job takes the default timeout when it sets none of its own, and the default’s retry declaration – its retry, or the deprecated restart if that is what the defaults set – when it declares neither a retry nor a restart. Setting only a timeout on a job no longer drops the workflow’s default retry for it.

This changes what an existing workflow does. Before, a job with any policy of its own inherited only the default timeout, so a workflow whose defaults.job.policy set restart: on-node-fault (or a retry) ran a job that set only a timeout with no restart at all. That job is now moved off a failing node and run again from the beginning. To keep a job from inheriting the default, give it a retry of its own with no cause, which retries nothing:

policy:
  timeout:
    execute: 10m
  retry:
    attempts: 1

An empty retry: {} does not do this: it is dropped when the workflow is read, and the job inherits the default.

The default retry is inherited whole – on and attempts together – or not at all. A job that declares retry: {attempts: 5} has declared its own retry and takes none of the default’s causes, so its budget and the default’s causes are never mixed into a combination neither declaration states. A job that declares the deprecated restart keeps it, and does not inherit the default retry, because the two may not be set together.

defaults:
  job:
    policy:
      retry:
        on: [node, 137]
        attempts: 2
jobs:
  inherits:            # takes on: [node, 137] and attempts: 2
    image:
      uri: docker://alpine:latest
    command: [/bin/sh, -c, ./step.sh]
    policy:
      timeout:
        execute: 10m
  overrides:           # retried on exit code 3 only, up to the default 3 times
    image:
      uri: docker://alpine:latest
    command: [/bin/sh, -c, ./step.sh]
    policy:
      retry:
        on: 3

Deprecated: policy.restart

policy.restart asked a subset of the same question and is deprecated. It still works, and existing workflow definitions keep behaving exactly as they did:

Written todayTreated as
restart: on-node-faultretry: {on: node}
restart: neverNo retry
restart unsetNo retry, unchanged

Setting both restart and retry.on is an error rather than one silently winning, because a silent winner means one of the two declarations is ignored and the author cannot tell which – on a question about whether their work may be run twice.

Declaring nothing at all stays the conservative default: a routine drain leaves the step running, and losing the node fails it rather than repeating work that may not be safe to repeat.

Only declare retry.on for steps that are safe to repeat. A step that appends to a file, submits an order, or writes partial output to a shared volume will do all of it a second time – Fuzzball does not checkpoint, so a retry runs the step from the beginning.

A step that is restarted keeps its identity – same job name, same workflow – so its logs and status continue to be reported in the same place. There is a cluster-wide limit on how many times one step may be moved by node faults; beyond it the step is failed rather than moved again, so work cannot circulate through failing hardware indefinitely.

When a node ends a step, the error names the node rather than reporting only that the job stopped responding. It names no fault class: Fuzzball does not diagnose hardware, and a rack losing power, an agent being killed and an operator emptying a node look the same from the control plane.

A step that was moved rather than failed says so, so the two outcomes are not mistaken for one another:

node gpu-14 stopped hosting this work; moved to another node

A step that had not declared policy.retry.on: node is failed, and told how to opt in:

node gpu-14 stopped hosting this work; job failed. Set policy.retry.on to node for work that is safe to run again from the beginning

A step that had declared it, and has been moved as often as the cluster allows, is failed with the count instead – repeating the advice would be telling the author to set something they already set:

node gpu-14 stopped hosting this work; job failed after 3 node restart(s), the limit for this step

Where a hold caused the move, the hold’s own reason is carried instead, because somebody wrote that sentence about this node deliberately:

node gpu-14 was held: failing disk, replacing Friday; job failed after 3 node restart(s), the limit for this step

Each move and failure is also recorded on the stage as an evicted workflow event naming the node and, for a hold, the hold’s id, source and reason; a step that ran out of restarts gets a retries_exhausted event saying which limit it reached – policy.retry.attempts or the cluster’s nodeHealth.maxEvictionRestarts – and the cause of the eviction that found the restarts spent. A moved step that cannot find room to run again reports a scheduling_blocked event with the reason requeued_awaiting_capacity. See Reading evicted and retries_exhausted Events.

See Node Health Monitoring for how an administrator configures which faults trigger this, and Nodes that go offline for the case that needs no configuration at all.