Fuzzball Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

Substrate Installation

This guide installs Fuzzball Substrate on Compute nodes and connects them to Fuzzball Orchestrate. Use the tab that matches the method you used to deploy Orchestrate.

Please select your environment.

A Kubernetes deployment runs Fuzzball Orchestrate in the cluster, but no Compute nodes of its own. This tab adds Compute nodes on separate hosts outside the cluster, which run Fuzzball Substrate and connect to the deployment. Every step is a plain command, so you can also script them with your site’s provisioning tools. Connecting External Static Compute Nodes installs the same files on Compute nodes of a cloud deployment.

Each step is labeled with the host it runs on:

  • CLI host – the machine you deployed from, with the fuzzball CLI, a kubeconfig for the cluster, and the deployment’s Fuzzball context, logged in as a cluster administrator such as the organization owner.
  • Compute node – a separate machine you are adding. Repeat the Compute node steps on each one.

The Compute node commands are written for Rocky Linux and run as root.

The deployment must be ready for the whole procedure: fuzzball cluster kubernetes status shows its Ready condition as True.

Environment variables used in this guide

Several steps use the following variables. Set them in the Compute node’s shell.

VariableValue
DEPOT_USER, ACCESS_KEYYour CIQ Depot credentials
VERSIONYour Orchestrate version, without the leading v
INTERNAL_INTERFACENetwork interface the Compute node uses to reach the cluster (for example enp8s0)
NFS_SERVER, NFS_PATHThe NFS export, as you passed it to deploy with --nfs-server and --nfs-path
NFS_DESTINATIONWhere to mount the export: /mnt/fuzzball-sharedfs, or what you passed to deploy with --nfs-destination

Network requirements

The Compute node and the cluster connect to each other on these ports. Make sure no firewall or security group between them blocks them.

FromToPortUsed for
Compute nodesubstrate-bridge.<domain>TCP 443, or with --ingress nodeport the HTTPS node port (30443 by default)Fetching the addresses of JetStream and DNS
Compute nodeapi.<domain>The same port as substrate-bridge.<domain>Workflow data, through the object cache
Compute nodeJetStreamTCP, at the address and port that substrate-bridge advertisesRegistering with Orchestrate and receiving work
Compute nodeFuzzball DNSTCP, at the address and port that substrate-bridge advertisesResolving workflow service names and job hostnames
Compute nodeNFS serverTCP 2049 (NFSv4)The shared filesystem
The clusterCompute nodeTCP 7331, 7332, and 7333Orchestrate connecting back to the node

Orchestrate connects back from its pods. Most network plugins send that traffic from the Kubernetes nodes’ addresses. Plugins that give pods addresses on the network, such as the AWS VPC CNI that EKS uses, send it from the pods’ own addresses, so allow the pod address range too.

With --ingress nodeport, JetStream and DNS listen on node ports of the Kubernetes nodes. With --ingress loadbalancer, they listen on ports 4222 and 5353 of their load balancer addresses. On load balancers that publish host names, such as AWS’s, the Compute nodes must resolve those names too. To see the addresses and ports, run this on the CLI host:

$ kubectl get service -n fuzzball fuzzball-jetstream-external fuzzball-substrate-bridge-dns

If the Compute nodes can’t reach the Kubernetes nodes’ addresses, see Compute nodes that can’t reach the Kubernetes nodes.

CLI host: generate the node configuration

Generate the files the Compute nodes need to connect to the deployment:

$ fuzzball cluster kubernetes generate-substrate-config ./substrate-config

The command writes four files into ./substrate-config: fuzzball-substrate.yaml, orchestrate.yaml, ca.crt, and nginx-ca.crt. The directory must be new or empty; pass --force to overwrite existing files. orchestrate.yaml is readable only by you, because it holds the key the node uses to authenticate to Orchestrate and the password of substrate-bridge, so copy it only over a secure channel. ca.crt and nginx-ca.crt are copies for debugging: both certificate authorities are already in the configuration, so the nodes don’t need them.

One configuration serves every Compute node of the deployment. When it finishes, the command prints the steps that follow, with the deployment’s substrate-bridge address and image directory filled in.

The command also takes these optional flags:

FlagEffect
--image-dir DIRKeeps the node’s image cache in DIR instead of image under the NFS destination, /mnt/fuzzball-sharedfs/image by default.
--compute-dir DIRSets Substrate’s compute directory, on the node’s local disk, instead of Substrate’s default.
-n, --namespace NAMEThe namespace of the Fuzzball services, if not fuzzball.

Copy the directory to each Compute node, for example with scp -r ./substrate-config root@<compute-node>:. Run the Compute node commands that follow from the directory that contains substrate-config; with the scp example, that’s root’s home directory.

Compute node: install the packages

Enroll the Compute node in CIQ Depot, then install fuzzball-substrate, fuzzball-substrate-orchestrate, and the Fuzzball CLI, which checks the node before Substrate starts. The versions of fuzzball-substrate-orchestrate and fuzzball-cli must match your Orchestrate version. Install the NFS client as well.

# DEPOT_USER="" # populate with your username for CIQ Depot

# ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team

# VERSION="4.4.0-110-g8f0b4aff023d" # matches orchestrate version without the leading v
# dnf install -y https://depot.ciq.com/dlv2/depot-el9.$(uname -m)/depot.rpm

# depot register --user "${DEPOT_USER}" --token "${ACCESS_KEY}"

# depot enable fuzzball

# dnf install -y fuzzball-substrate fuzzball-substrate-orchestrate-${VERSION}-1 fuzzball-cli-${VERSION}-1

# dnf install -y nfs-utils
The $(uname -m) in the Depot client URL selects the x86_64 or aarch64 package. On Rocky Linux 8, use depot-el8 in place of depot-el9. On Rocky Linux 10, install the EL9 package as shown; dnf update depot then moves it to the EL10 build.

If your Compute node has an NVIDIA GPU, install the following package as well.

# dnf install -y fuzzball-substrate-nvidia-gpu-plugin # optional

Don’t start Substrate yet.

Compute node: configure the network

Configure firewalld so it doesn’t interfere with traffic from the cluster. Orchestrate connects back to each Compute node on TCP ports 7331 to 7333, and the nodes of a multinode job talk to each other directly. Either disable firewalld or add the internal interface to the trusted zone.

# INTERNAL_INTERFACE="" # populate with the internal interface of the Compute node (e.g. enp8s0)
# firewall-cmd --permanent --zone=trusted --add-interface=${INTERNAL_INTERFACE}

# firewall-cmd --reload

If a network firewall or security group sits between the cluster and the Compute node, open TCP ports 7331 to 7333 there as well, from the addresses that Network requirements describes.

The Compute node reaches the deployment by name, so substrate-bridge.<domain> and api.<domain> must resolve on the Compute node too, to the same addresses as on the CLI host. Substrate can’t check in until they do. Check them, replacing <domain> with the deployment’s domain:

# getent hosts substrate-bridge.<domain> api.<domain>

If a name is missing from the output, add it to your site’s DNS or to /etc/hosts. A nip.io domain resolves without either.

Compute node: mount the shared filesystem

Every Compute node mounts the deployment’s NFS export at its NFS destination, /mnt/fuzzball-sharedfs unless you deployed with --nfs-destination. Substrate keeps its image cache in image under it, and the deployment’s default storage provisioner keeps volumes there, so all the nodes must see the same files at that path. If you generated the configuration with --image-dir, put that directory on a filesystem every Compute node shares too, and create it in place of ${NFS_DESTINATION}/image below.

Set the export and the mount point as you passed them to deploy. MOUNT_UNIT is the name systemd gives the mount unit, for example mnt-fuzzball\x2dsharedfs.mount:

# NFS_SERVER="" # populate with the NFS server, as passed to deploy with --nfs-server (e.g. 10.0.0.3)

# NFS_PATH="/srv/fuzzball/shared" # the export, as passed to deploy with --nfs-path

# NFS_DESTINATION="/mnt/fuzzball-sharedfs" # the mount point, as passed to deploy with --nfs-destination

# MOUNT_UNIT=$(systemd-escape --path --suffix=mount "${NFS_DESTINATION}")

Create the mount unit, so the share comes back after a reboot:

# cat >"/etc/systemd/system/${MOUNT_UNIT}" <<EOF
[Unit]
Description=Mount the Fuzzball shared filesystem
After=network-online.target remote-fs-pre.target
Wants=network-online.target
Before=remote-fs.target

[Mount]
What=${NFS_SERVER}:${NFS_PATH}
Where=${NFS_DESTINATION}
Type=nfs
Options=_netdev,hard,retry=5,timeo=600,retrans=3
TimeoutSec=300

[Install]
WantedBy=remote-fs.target
EOF

systemd attempts a mount unit only once per start. If the NFS server isn’t reachable or its name doesn’t resolve yet, for example because the Compute node boots before the NFS server, the mount fails and stays failed. The following service retries the mount until it succeeds, waiting 10 seconds after each failed attempt. systemctl accepts the mount point in place of the mount unit’s name.

# cat >/etc/systemd/system/fuzzball-shared-wait.service <<EOF
[Unit]
Description=Mount the Fuzzball shared filesystem, retrying until it is available
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'until systemctl start ${NFS_DESTINATION}; do sleep 10; done'
TimeoutStartSec=infinity
EOF

The following drop-in makes Substrate wait for that service, so Substrate starts only once the share is mounted and doesn’t fail when the share is late.

# mkdir -p /etc/systemd/system/fuzzball-substrate.service.d

# cat >/etc/systemd/system/fuzzball-substrate.service.d/10-require-shared.conf <<EOF
[Unit]
Requires=fuzzball-shared-wait.service
After=fuzzball-shared-wait.service ${MOUNT_UNIT}
PartOf=${MOUNT_UNIT}
EOF

Then create the mount point, start the mount, and create the image directory on the share:

# mkdir -p "${NFS_DESTINATION}"

# systemctl daemon-reload

# systemctl enable --now "${MOUNT_UNIT}"

# install -d -m 0755 "${NFS_DESTINATION}/image"

# findmnt "${NFS_DESTINATION}"

findmnt should show the export mounted with the nfs4 or nfs type. If the mount fails, or files that one node writes to the share don’t show up on another, stop and fix your NFS configuration before you continue.

The compute directory stays on the node’s local disk. Substrate’s default needs no setup; if you generated the configuration with --compute-dir, create that directory with mkdir -p.

Compute node: install the configuration

Install the generated configuration files. Install them after the packages, because installing or updating the fuzzball-substrate package can replace them with its defaults. orchestrate.yaml contains the key the node uses to authenticate to Orchestrate, so only root can read it. Substrate reads fuzzball-substrate.yml, so the first command also renames that file. The package also ships device-plugins.yml in /etc/fuzzball-substrate/; leave it in place.

# install -m 0644 substrate-config/fuzzball-substrate.yaml /etc/fuzzball-substrate/fuzzball-substrate.yml

# install -d -m 0755 /etc/fuzzball-substrate/extension.conf.d

# install -m 0600 substrate-config/orchestrate.yaml /etc/fuzzball-substrate/extension.conf.d/orchestrate.yaml

A node set up with an earlier version of this guide has FB_POLL_CONFIG in /etc/default/fuzzball-substrate.env, which makes Substrate read the file it names and ignore the installed ones. Remove it, and keep FB_MOUNTNS=true, which aids crash recovery:

# sed -i '/^[[:space:]]*FB_POLL_CONFIG=/d' /etc/default/fuzzball-substrate.env

# grep -qx 'FB_MOUNTNS=true' /etc/default/fuzzball-substrate.env || echo 'FB_MOUNTNS=true' >>/etc/default/fuzzball-substrate.env

Compute node: check the node

Check the node before you start Substrate. The check reads the installed configuration as Substrate will, and tests the platform, name resolution and the connection to substrate-bridge, the JetStream and Fuzzball DNS addresses that substrate-bridge advertises, the image registries, the image directory, and the firewall. It needs no kubeconfig. Fix any check that fails, then run it again:

# fuzzball cluster kubernetes preflight node

If it warns that the image directory isn’t on a shared filesystem, the share isn’t mounted where the configuration keeps the image cache; see Compute node: mount the shared filesystem.

CLI host: add a provision definition

Orchestrate assigns work only to nodes that match a provision definition in the central config, and a new deployment has none. Until you add one, Substrate on the node keeps waiting, and the node doesn’t appear in fuzzball node list. One definition can cover many nodes, so you need this step only for the first node it matches.

fuzzball cluster config set replaces the whole configuration, so start from the current one:

$ fuzzball cluster config get >central-config.yaml

On a new deployment, the command says the central config is empty and leaves the file empty. Add a definition for your Compute nodes to the definitions list, creating the list if the file is empty. This one matches nodes named compute01 to compute99, as printed by hostname on each node:

definitions:
  - id: static-compute
    provisioner: static
    provisionerSpec:
      condition: |-
        hostname() matches "^compute[0-9]{2}$"

hostname() matches takes a regular expression, so one definition can cover several Compute nodes. Anchor the pattern with ^ and $: without them, compute1 also matches compute10. Give Compute nodes of each CPU architecture their own definition, for example by adding && uname.machine == "x86_64" to the condition. See the Provisioner Configuration Guide for all options.

Apply the configuration:

$ fuzzball cluster config set central-config.yaml

Compute node: start Substrate

Enable and start Substrate:

# systemctl enable --now fuzzball-substrate.service

If the NFS share isn’t mounted, this command waits until it is. At boot, Substrate likewise waits for the share, and boot doesn’t finish until the share is mounted: systemctl is-system-running reports starting, and units ordered after multi-user.target wait too. Run journalctl -u fuzzball-shared-wait.service -u "${MOUNT_UNIT}" to see the mount attempts and why they fail. To stop retrying and cancel starting Substrate, run systemctl stop fuzzball-shared-wait.service. Once the share is mounted, the service exits, so inactive is its normal state.

On the CLI host, confirm that the node has joined:

$ fuzzball node list

The node shows up as Ready, usually within a minute. Its ID is the address Orchestrate connects back to, followed by /7331. Then run small test workloads on it:

$ fuzzball cluster kubernetes validate --tier functional

Refresh the configuration after a change

The generated configuration holds the deployment’s domain and HTTPS port, the image directory under the NFS destination, the image registries, the certificate authorities, and the credentials Substrate authenticates with. When an update changes any of these, or moves to a new Fuzzball version, regenerate the configuration on the CLI host with the same flags as before, plus --force:

$ fuzzball cluster kubernetes generate-substrate-config ./substrate-config --force

Then, on each Compute node:

  1. For a new Fuzzball version, update the fuzzball-substrate-orchestrate and fuzzball-cli packages to that version first. See Update Substrate on the compute nodes.

  2. Copy the new directory to the node and install the files as in Compute node: install the configuration.

  3. Restart Substrate. Jobs running on the node when it restarts fail, so restart it while the node is idle:

    # systemctl restart fuzzball-substrate.service

Troubleshooting

If the Compute node doesn’t appear in fuzzball node list, or jobs don’t run on it, start with these checks on the Compute node:

# getent hosts substrate-bridge.<domain> api.<domain>

# fuzzball cluster kubernetes preflight node

# journalctl -u fuzzball-substrate -n 100

The first command should print an address for both names. The second checks the node against its installed configuration and reports a fix for each check that fails.

SymptomLikely cause
The node never appears in fuzzball node listNo provision definition matches the node’s hostname, or the node can’t reach substrate-bridge or JetStream. See CLI host: add a provision definition and Network requirements.
preflight node warns that Substrate ignores the installed configurationFB_POLL_CONFIG is set. See Compute node: install the configuration.
Jobs are assigned to the node but never startOrchestrate can’t reach the node on TCP ports 7331 to 7333 at the address in its node ID. See Compute node: configure the network.
Jobs fail on one Compute node because their image is missing, but work on anotherThe Compute nodes don’t share the image directory. See Compute node: mount the shared filesystem.

Next steps

The Compute node is now part of the cluster. Confirm that it matched your provision definition:

$ fuzzball node list -o yaml

Find the node by its host_name and check that its definition_id is static-compute. A node with definition_id: N/A is connected but matches no provision definition, so it never gets work.

If other people will use the cluster, create users for them and organize them into organizations and groups, which control the resources each user can see and share. Follow Initial Entity Management to create an organization, a group, and a user, then see Fuzzball Entity Management for the full reference.

A Docker Compose stack already runs one or more Substrate containers on the Orchestrate host. This tab adds Compute nodes on separate hosts, for compute capacity beyond what the Orchestrate host provides.

Each step is labeled with the host it runs on:

  • Orchestrate host – the machine running the Docker Compose stack, where the fuzzball CLI is installed and logged in.
  • Compute node – a separate machine you are adding. Repeat the Compute node steps on each one.

The commands for both hosts are written for Rocky Linux. On an Ubuntu Orchestrate host, such as DGX OS, open the ports with ufw instead of firewall-cmd, and install the nfs-kernel-server package instead of nfs-utils; Ubuntu starts the NFS server when the package is installed.

The Docker Compose stack must be running for the whole procedure. See Starting and stopping a stack.

The stack’s api.<domain> and nats.<domain> names must also resolve on the Orchestrate host, because several steps run fuzzball commands there. If fuzzball cluster docker-compose info works but other fuzzball commands can’t reach the API, see DNS resolution. For a domain that only resolves through /etc/hosts, deploy with --update-etc-hosts or add the entry yourself.

Environment variables used in this guide

Several steps use the following variables. Set the ones for each host in that host’s shell.

VariableHostValue
ORCHESTRATE_IPBothIP address of the Orchestrate host, as reached from the Compute nodes
COMPUTE_SUBNETOrchestrate hostNetwork range the Compute nodes are on, for the NFS export (for example 10.0.0.0/24)
DEPOT_USER, ACCESS_KEYCompute nodeYour CIQ Depot credentials
INTERNAL_INTERFACECompute nodeNetwork interface the Compute node uses to reach the Orchestrate host (for example enp8s0)

Orchestrate host: open the required ports

The Compute node connects to the following TCP ports on the Orchestrate host. Make sure no firewall between the two hosts blocks them.

PortUsed for
443Fuzzball API, through nginx
4222NATS, which carries the connection between Substrate and Orchestrate
5353Fuzzball DNS, which resolves workflow service names and lets the nodes of a multinode job find each other
2049NFS, if the Orchestrate host serves the shared file system as in the example below

On a Rocky Linux Orchestrate host running firewalld, open the ports in the default zone. The nfs service opens port 2049; leave it out if the Orchestrate host isn’t your NFS server.

# firewall-cmd --permanent --add-port=443/tcp --add-port=4222/tcp --add-port=5353/tcp

# firewall-cmd --permanent --add-service=nfs

# firewall-cmd --reload

Confirm that the ports and services lines include them:

# firewall-cmd --list-all

If another firewall sits between the two hosts, such as a network firewall or a cloud security group, open the same ports there as well.

Deployments created with a Fuzzball CLI older than v4.3.1 do not forward port 5353, so workflow service names and job hostnames don’t resolve on external Compute nodes. Multinode jobs, such as MPI jobs, rely on those names. Upgrade the CLI, then refresh the deployment’s files as described in Refreshing the generated files. Don’t add the port by editing docker-compose.yaml or the nginx configuration yourself; the next refresh overwrites those edits.

Orchestrate host: generate the Substrate configuration

Generate the files the Compute node needs to connect to Orchestrate. Pass the Orchestrate host’s IP address with --orchestrate-ip.

Orchestrate also connects back to each Compute node. By default it uses the address the node sends its own traffic from, which is the node’s address on the route to --orchestrate-ip. If Orchestrate must reach the node at a different address, for example because of NAT or a separate network, also pass --substrate-ip with that address. It becomes the node’s ID and the address Orchestrate connects to, so generate a separate configuration for each Compute node that needs it.

$ ORCHESTRATE_IP="" # populate with the IP address of the Orchestrate host (e.g. 10.0.0.5)

$ fuzzball cluster docker-compose generate-substrate-config ./substrate-config --orchestrate-ip ${ORCHESTRATE_IP}

The command writes five files into ./substrate-config: fuzzball-substrate.yml, orchestrate.yaml, ca.crt, nginx-ca.crt, and hosts. The directory must be empty; pass --force to overwrite existing files. orchestrate.yaml is readable only by you, because it holds the key the node uses to authenticate to Orchestrate.

The command also takes these optional flags:

FlagEffect
--image-dir DIRKeeps the node’s image cache in DIR instead of /mnt/fuzzball-sharedfs/image.
--compute-dir DIRSets Substrate’s compute directory, which is otherwise /var/lib/fuzzball/compute.
--secure-registry HOST[:PORT]Lets the node pull images from a registry whose certificate is signed by a private CA. The command connects to the registry from the Orchestrate host and embeds its CA certificate in the configuration. The port defaults to 443; repeat the flag for each registry.
--image-proxy URLPulls images through the proxy at URL.
--image-no-proxy HOSTPulls images from HOST without the proxy. Repeat the flag or separate hosts with commas.

Copy the directory to the Compute node, for example with scp -r ./substrate-config root@<compute-node>:. Run the Compute node commands that follow from the directory that contains substrate-config; with the scp example, that’s root’s home directory.

Compute node: install Substrate

Enroll the Compute node in CIQ Depot and install the Substrate packages and the Fuzzball CLI, which checks the node before Substrate starts. The versions of fuzzball-substrate-orchestrate and fuzzball-cli must match your Orchestrate version.

# DEPOT_USER="" # populate with your username for CIQ Depot

# ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team

# VERSION="4.4.0-110-g8f0b4aff023d" # matches orchestrate version without the leading v
# dnf install -y https://depot.ciq.com/dlv2/depot-el9.$(uname -m)/depot.rpm

# depot register --user "${DEPOT_USER}" --token "${ACCESS_KEY}"

# depot enable fuzzball

# dnf install -y fuzzball-substrate fuzzball-substrate-orchestrate-${VERSION}-1 fuzzball-cli-${VERSION}-1

If your Compute node has an NVIDIA GPU, install the following package as well.

# dnf install -y fuzzball-substrate-nvidia-gpu-plugin # optional

Don’t start Substrate yet.

Compute node: configure the node

Configure firewalld so it doesn’t interfere with traffic between nodes. Orchestrate connects back to each Compute node on TCP port 7333 to assign and start jobs, and the nodes of a multinode job talk to each other directly. Port 7332 serves Substrate’s health checks and metrics; only a metrics scraper needs to reach it. Either disable firewalld or add the internal interface to the trusted zone.

# INTERNAL_INTERFACE="" # populate with the internal interface of the Compute node (e.g. enp8s0)
# firewall-cmd --permanent --zone=trusted --add-interface=${INTERNAL_INTERFACE}

# firewall-cmd --reload

If the Orchestrate host reaches the Compute node through a different interface, or a network firewall sits between them, open TCP port 7333 from the Orchestrate host there as well. If the address Orchestrate uses differs from the node’s own outbound address, see --substrate-ip in Orchestrate host: generate the Substrate configuration.

The Compute node reaches Orchestrate by name, so api.<domain> and nats.<domain> must resolve to the Orchestrate host on the Compute node too. Substrate can’t check in with Orchestrate until they do. Append the generated hosts entry, which maps both names to --orchestrate-ip, to /etc/hosts on every Compute node:

# cat substrate-config/hosts >>/etc/hosts

Add the entry even with the default <host-ip>.nip.io domain. That domain encodes the Orchestrate host’s primary outbound address, which on a host with more than one network may not be the address the Compute node can reach. Skip this step only if your site’s DNS already resolves both names to an address the Compute node can reach.

Install the NFS client. Substrate uses it to mount volumes from the NFS server, which you set up later, when a job needs them.

# dnf install -y nfs-utils

The generated configuration keeps the node’s container image cache in /mnt/fuzzball-sharedfs/image, or in the directory you passed with --image-dir. Create that directory. With more than one Compute node, you mount the shared image cache there later.

# mkdir -p /mnt/fuzzball-sharedfs/image

Compute node: install the configuration and start Substrate

Install the generated configuration files. Install them after the packages, because installing or updating the Substrate package can replace them with its defaults. orchestrate.yaml contains the key the node uses to authenticate to Orchestrate, so only root can read it.

# install -m 0644 substrate-config/fuzzball-substrate.yml /etc/fuzzball-substrate/fuzzball-substrate.yml

# install -d -m 0755 /etc/fuzzball-substrate/extension.conf.d

# install -m 0600 substrate-config/orchestrate.yaml /etc/fuzzball-substrate/extension.conf.d/orchestrate.yaml

On a node with a GPU, remove the host-rootfs-path line from the generated configuration. It points device discovery at /substrate_hostfs, a path that exists only inside the Docker Compose stack’s own Substrate containers, so GPUs on the Compute node aren’t detected:

# sed -i '/host-rootfs-path:/d' /etc/fuzzball-substrate/fuzzball-substrate.yml

The configuration carries both of the deployment’s certificate authorities, so the node needs nothing in its system trust store. ca.crt and nginx-ca.crt are copies for debugging; don’t install them.

Check the node before you start Substrate. The check reads the installed configuration and tests name resolution, the connections to Orchestrate, the image registries, the image directory, and the firewall. Fix any check that fails, then run it again:

# fuzzball cluster docker-compose preflight node

Enable and start Substrate:

# systemctl enable --now fuzzball-substrate.service
orchestrate.yaml also carries the deployment’s root CA, which the node publishes to workflow containers as /run/fuzzball-substrate/trusted-certs/root-ca.crt so that workflows can verify the API’s certificate (see In-Workflow API Access). A node configured from a file generated by an older CLI has no root-ca.crt. To fix it, regenerate the configuration with --force, replace orchestrate.yaml on the node, and restart Substrate with systemctl restart fuzzball-substrate.service.

Orchestrate host: set up a shared file system

External Compute nodes use shared storage for two separate things:

  • The image cache. Before a job runs, one node in its segment pulls and converts the job’s container image into the image cache, and the node that runs the job then uses it from there. The two aren’t always the same node, so with more than one Compute node, every node must see the same cache directory. A single Compute node can use its local directory.
  • Volumes. Workflows that mount volumes need a storage provisioner the node can serve. The stack’s default provisioner uses a Docker volume that only its own Substrate containers can reach. Workflows without volumes don’t need one.

This guide shows one way to provide both: the Orchestrate host exports two NFS directories. Every Compute node mounts /srv/fuzzball/images as its image cache, and Fuzzball’s NFS storage driver serves volumes from /srv/fuzzball/storage, mounting each volume only when a job needs it. A shared external segment limits these volumes to the external Compute nodes; the stack’s own Substrate containers keep the default provisioner.

Other storage works too, including an existing NFS server or a parallel file system such as Lustre, GPFS, or BeeGFS, but needs a different configuration. See Driver Types and Volume Configuration and Management. To use an existing NFS server, skip the rest of this section and use its address and export paths in the next section and when you add the provisioner.

Install and start the NFS server:

# dnf install -y nfs-utils

# systemctl enable --now nfs-server

Create the directories and export them. Substrate runs as root when it writes the image cache, creates volume directories, and sets their ownership, so the exports need no_root_squash.

# COMPUTE_SUBNET="" # populate with the network range of your Compute nodes (e.g. 10.0.0.0/24)

# mkdir -p /srv/fuzzball/images /srv/fuzzball/storage

# echo "/srv/fuzzball/images ${COMPUTE_SUBNET}(rw,sync,no_subtree_check,no_root_squash)" >>/etc/exports

# echo "/srv/fuzzball/storage ${COMPUTE_SUBNET}(rw,sync,no_subtree_check,no_root_squash)" >>/etc/exports

# exportfs -ra

Compute node: mount the shared image cache

With more than one Compute node, mount the image cache export on every Compute node at the path the generated configuration uses. With a single Compute node, you can skip this section.

Add the mount to /etc/fstab so it returns after a reboot, then mount it and restart Substrate so it uses the shared cache:

# ORCHESTRATE_IP="" # populate with the IP address of the Orchestrate host (e.g. 10.0.0.5)
# echo "${ORCHESTRATE_IP}:/srv/fuzzball/images /mnt/fuzzball-sharedfs/image nfs4 _netdev,hard 0 0" >>/etc/fstab

# systemctl daemon-reload

# mount /mnt/fuzzball-sharedfs/image

# systemctl restart fuzzball-substrate.service

Orchestrate host: add the node to the cluster

Orchestrate assigns work only to nodes that match a provision definition. The stack’s built-in docker-nodes definition matches only its own Substrate containers, so add a definition for the Compute node.

fuzzball cluster config set replaces the whole configuration, so start from the current one to keep the docker-nodes definition:

$ fuzzball cluster config get >provisioner.yaml

Add a definition for the Compute node to the definitions list in provisioner.yaml, replacing <compute-hostname> with the node’s hostname, as printed by hostname on that node:

  - id: compute1
    provisioner: static
    segment: external
    provisionerSpec:
      condition: |-
        hostname() matches "^<compute-hostname>$"

The segment must match the segment of the storage provisioner you add in the next section.

hostname() matches takes a regular expression, so one definition can cover several Compute nodes, for example "^compute[0-9]{2}$" for compute01 to compute99. Anchor the pattern with ^ and $: without them, compute1 also matches compute10. In a regular expression, [01-58] is a set of characters, not a numeric range. See the Provisioner Configuration Guide for all options.

Apply the configuration:

$ fuzzball cluster config set provisioner.yaml

Then restart Substrate on the Compute node so it picks up the new definition:

# systemctl restart fuzzball-substrate.service

Orchestrate host: add a provisioner for the shared file system

Create a storage provisioner that serves volumes from the NFS export. Its segment matches the Compute node’s provision definition, so only the external Compute nodes serve it.

$ cat >shared-storage.yaml <<EOF
shared-storage:
  description: "NFS storage for external Compute nodes"
  driver:
    type: nfs
    target: "${ORCHESTRATE_IP}:/srv/fuzzball/storage"
    version: 4
  segment: external
  access: all
  ephemeral: all
EOF

$ fuzzball volume provisioner add shared-storage -f shared-storage.yaml

version: 4 pins NFSv4, which needs only port 2049. A provisioner’s segment can’t be changed after it is created. See NFS driver for the other driver options and Segments for how segments work.

A deployment created before the default provisioner was scoped to the docker segment still has an unsegmented default provisioner, which is offered to the Compute node even though the node can’t reach its storage. See Upgrades keep the original default provisioner.

Troubleshooting

If the Compute node doesn’t appear in fuzzball node list, or jobs don’t run on it, start with these checks on the Compute node:

# getent hosts api.<domain> nats.<domain>

# fuzzball cluster docker-compose preflight node

# journalctl -u fuzzball-substrate -n 100

The first command should print the Orchestrate host’s address for both names. The second checks the node against its installed configuration and reports a fix for each check that fails.

SymptomLikely cause
The node never appears in fuzzball node listNo provision definition matches the node’s hostname, ports 443 or 4222 on the Orchestrate host are blocked, or the names don’t resolve on the node. See Orchestrate host: add the node to the cluster, Orchestrate host: open the required ports, and Compute node: configure the node.
x509: certificate signed by unknown authority in the Substrate logThe installed configuration doesn’t match the deployment, or, for an image pull, the registry wasn’t passed with --secure-registry. Regenerate the configuration, install it, and restart Substrate. See Orchestrate host: generate the Substrate configuration.
The node is listed, but jobs stay pending and never go to itNo provision definition matches the node. fuzzball node list -o yaml shows definition_id: N/A for it. See Orchestrate host: add the node to the cluster.
Jobs are assigned to the node but never startOrchestrate can’t reach the node on TCP port 7333, or at the address in its node ID. See Compute node: configure the node and --substrate-ip in Orchestrate host: generate the Substrate configuration.
Jobs fail on one Compute node because their image is missing, but work on anotherThe Compute nodes don’t share an image cache. See Compute node: mount the shared image cache.
Multinode jobs fail to resolve each other’s hostnamesPort 5353 isn’t reachable on the Orchestrate host, or the deployment predates v4.3.1. See Orchestrate host: open the required ports.
Jobs that mount a volume fail on the nodeThe node has no storage provisioner it can serve. See Orchestrate host: add a provisioner for the shared file system.

Next steps

The Compute node is now part of the cluster and has storage. Confirm that it is listed and matched to the compute1 definition:

$ fuzzball node list -o yaml

Find the node by its host_name and check that its definition_id is compute1. A node with definition_id: N/A is connected but matches no provision definition, so it never gets work.

If other people will use the cluster, create users for them and organize them into organizations and groups, which control the resources each user can see and share. Follow Initial Entity Management to create an organization, a group, and a user, then see Fuzzball Entity Management for the full reference.

Storage provisioners belong to the organization that created them, so users in a new organization can’t use shared-storage. As an owner of each organization whose users will run jobs on the Compute node, repeat Orchestrate host: add a provisioner for the shared file system. Point each organization’s provisioner at its own directory on the NFS server, such as /srv/fuzzball/storage/<org-name>, and create that directory first; the NFS driver can’t keep organizations apart within one directory. See Tenant Isolation.