Fuzzball Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

Creating Fuzzfiles

Fuzzfiles Describe Workflows

Fuzzball runs workflows that are composed of one or more jobs. Each job in a workflow runs some specified command(s), executes within an environment consisting of a container, volumes, environment variables, etc., and requires some set of computational resources. Jobs within a workflow may run independently or may be dependent upon one another forming a Directed Acyclic Graph (DAG).

The first step in creating a workflow is designing it. This involves determining what jobs need to be run, and defining the specific behavior and resource requirements for each of those jobs. These and other specifications are detailed in a special YAML file called a Fuzzfile.

A workflow can also declare services. A service is a container that runs alongside the jobs instead of completing like one: it starts, reports itself ready, and stays up while something depends on it. Services host the things jobs talk to – a database, a model server, a distributed compute cluster. A service that nothing depends on stops as soon as it reports ready, unless it sets persist: true. The last page of this section configures a service that spans several nodes.

A Fuzzfile can describe anything from a single-job workflow, such as a job that prints numbers, to a workflow of many dependent jobs that trains a machine learning model. Like any other text file, a Fuzzfile can be written by hand in a text editor. The Fuzzfile syntax guide documents every available option, including the full services reference.

Using the Workflow Editor

The Fuzzball web UI includes an interactive Workflow Editor. In the editor you create jobs by dragging them into your workflow, declare dependencies by drawing connections between them, and set job commands, environments, and resources through menus.

The Workflow Editor generates the Fuzzfile. A common approach among users is to lay out the general arrangement of jobs and their requirements in the editor, export the generated Fuzzfile, and then fill in the remaining details in a text editor. That way you pick up the YAML syntax as you go.

Images and Connections on the Canvas

The canvas shows the container images your workflow uses as well as its jobs, services, and volumes. Each unique image appears once, in a column to the left of the jobs, with a line to every job and service that uses it. Jobs and services that share an image share one image card. An image declared in the top-level images: section of the Fuzzfile also appears as a card, even when nothing uses it yet.

Image cards are read-only. To change which image a job or service uses, select the job or service and edit its image on the Environment tab of the Properties panel. Selecting an image card shows the workflow properties, because an image has no properties of its own.

When you select any card, the Workflow Editor highlights the lines that connect to it and dims the others, so you can see what depends on what. Select an empty part of the canvas to clear the highlight.

Retrying a Job

A job’s Job tab has a Retry if the node stops hosting the job toggle. Turning it on writes policy.retry.on: node: if the job’s node is evicted or goes offline, Fuzzball runs the job again from the beginning on another node. Below it, Also retry on exit codes takes one exit code per row (Add exit code adds a row): if the job’s process exits with one of them, Fuzzball runs it again from the beginning. Codes run from 1 to 255; the editor rejects 0, 130 and 143, and a code listed twice, as the workflow validator does. Only set either for jobs that are safe to repeat.

While any retry cause is set, an optional Attempts field sets policy.retry.attempts, the most times the job may be retried, one budget shared by every cause. The cluster’s own limit still applies to node retries when it is lower. The Jobs tab of the workflow defaults has the same controls, which write defaults.job.policy.retry. Jobs inherit that default retry unless they declare their own retry or the deprecated restart.

A Fuzzfile opened in the editor keeps its retry settings, including the deprecated policy.restart, which the editor does not offer but does not remove. See Restart Tolerance for how retries behave.

How This Section Is Organized

The section has two halves, and only the first uses the editor.

A tutorial, in the Workflow Editor. The first five pages build a single workflow step by step, from one job to a set of parallel jobs with data ingress and egress.

A reference, written directly in YAML. The last three pages cover Fuzzfile features the Workflow Editor has no equivalent for: encrypted SIF images, building images in a workflow, and host networking for multinode services.