Debugging
This section provides documentation for debugging and troubleshooting Fuzzball clusters. These tools are designed for administrators to diagnose issues, monitor service behavior, and perform advanced maintenance tasks.
Debug commands are useful for:
- Viewing service logs to diagnose issues
- Creating debug dumps to share with the support team
You will need to install the Fuzzball CLI (see the CLI installation documentation for details), set up a Fuzzball admin context and authenticate like so:
$ fuzzball context create default-admin-context <api_url> local fuzzball-cli
Configuration for "default-admin-context" created.
Configuration for "default-admin-context" now in use.
$ FUZZBALL_USER="admin_username"
$ read -rs FUZZBALL_PASSWORD # read interactively to set variable without echoing to screen
$ export FUZZBALL_USER FUZZBALL_PASSWORD
$ fuzzball context login
Logging into current cluster context...The fuzzball debug logs command allows administrators to view
logs from Fuzzball services running in the cluster. This is useful for debugging
service issues, monitoring service behavior, and tracking down errors.
$ fuzzball debug logs <SERVICE_NAME> [flags]The following Fuzzball services can be queried for logs:
agent- Fuzzball agent serviceaudit- Audit logging serviceauth- Authentication servicebilling- Billing servicecluster-admin- Cluster administration servicejetstream- JetStream message brokeropenapi- OpenAPI serviceorchestrator- Orchestrator serviceprovision- Provisioning serviceschedule- Scheduling servicestorage- Storage servicesubstrate-bridge- Substrate bridge serviceui- Web UI serviceworkflow- Workflow service
| Flag | Short | Description | Default |
|---|---|---|---|
--since | -s | Show logs since a specific time (RFC3339 or YYYY-MM-DD format) | - |
--since-duration | -d | Show logs since a specific duration (e.g., ‘1h’, ‘30m’) | - |
--tail | -t | Number of lines to show from the end of the logs | 100 |
--follow | -f | Follow the log output (stream logs) | false |
--min-log-level | - | Minimum log level to display (1: Debug, 2: Info, 3: Warning, 4: Error, 5: Fatal) | 0 |
--trace-id | - | Filter logs by trace ID | - |
--ignore-errors | - | Ignore errors when fetching logs | false |
--json | -j | Output in JSON format | false |
View the last 100 lines of logs from the workflow service:
$ fuzzball debug logs workflow
[fuzzball-workflow-0]: 2026-01-13T10:15:23Z INFO Starting workflow service
[fuzzball-workflow-0]: 2026-01-13T10:15:24Z INFO Connected to database
[fuzzball-workflow-0]: 2026-01-13T10:15:25Z INFO Workflow service readyView logs from multiple services:
$ fuzzball debug logs workflow storage
[fuzzball-workflow-0]: 2026-01-13T10:15:23Z INFO Starting workflow service
[fuzzball-storage-0]: 2026-01-13T10:15:22Z INFO Storage service initializedFollow logs in real-time:
$ fuzzball debug logs workflow --follow
[fuzzball-workflow-0]: 2026-01-13T10:15:23Z INFO Starting workflow service
[fuzzball-workflow-0]: 2026-01-13T10:15:24Z INFO Processing workflow requestShow logs since a specific time:
$ fuzzball debug logs workflow --since 2026-01-13T10:00:00ZShow logs from the last hour:
$ fuzzball debug logs workflow --since-duration 1hFilter by minimum log level (errors and above):
$ fuzzball debug logs workflow --min-log-level 4Show the last 500 lines:
$ fuzzball debug logs workflow --tail 500Output logs in JSON format:
$ fuzzball debug logs workflow --json
{"pod_name":"fuzzball-workflow-0","message":"2026-01-13T10:15:23Z INFO Starting workflow service","level":2}
{"pod_name":"fuzzball-workflow-0","message":"2026-01-13T10:15:24Z INFO Connected to database","level":2}- The
--sinceand--since-durationflags cannot be used together - When using
--follow, logs will stream continuously until interrupted (Ctrl+C) - The
--jsonflag is useful for programmatic parsing and integration with log analysis tools - Log levels: 0 (Unspecified), 1 (Debug), 2 (Info), 3 (Warning), 4 (Error), 5 (Fatal)
The fuzzball debug dump command creates a comprehensive dump of the
Fuzzball cluster state, including configuration, logs, and diagnostic
information. This is useful for gathering all relevant information when
reporting issues or performing detailed troubleshooting.
$ fuzzball debug dump [flags]| Flag | Short | Description | Default |
|---|---|---|---|
--dest | -d | Destination path to save the dump | Current directory |
--namespace | -n | Additional namespaces to include in the dump; repeat the flag or pass a comma-separated list | - |
The dump always covers Fuzzball’s own namespaces and those of its
dependencies:
fuzzball, fuzzball-federate, fuzzball-system, fuzzball-database,
fuzzball-identity, cert-manager, and metallb-system. A namespace that
does not exist on your deployment contributes nothing beyond an empty
events.txt.
Create a cluster dump in the current directory:
$ fuzzball debug dump
$ ls -lhd fuzzball-cluster-dump-*
drwxr-xr-x@ 10 user group 320B Jan 15 12:11 fuzzball-cluster-dump-20260115-121100
$ tree -L2 fuzzball-cluster-dump-20260115-121100
fuzzball-cluster-dump-20260115-121100
├── cert-manager
│ ├── certificates
│ ├── configmaps
│ ├── deployments
│ ├── events.txt
│ ├── metrics
│ ├── pods
│ ├── rbac
│ ├── secrets
│ └── services
├── cluster
│ ├── clusterissuers
│ ├── collector-diagnostics.txt
│ ├── crds
│ ├── ingressclasses
│ ├── jetstream
│ ├── metrics
│ ├── nodes
│ ├── rbac
│ └── storage
├── cluster_info.txt
├── errors.txt
├── fuzzball
│ ├── certificates
│ ├── configmaps
│ ├── deployments
│ ├── events.txt
│ ├── ingresses
│ ├── metrics
│ ├── pods
│ ├── rbac
│ ├── scheduler
│ ├── secrets
│ ├── services
│ └── storage
├── fuzzball-database
│ ├── certificates
│ ├── configmaps
│ ├── deployments
│ ├── events.txt
│ ├── health
│ ├── metrics
│ ├── pods
│ ├── rbac
│ ├── secrets
│ ├── services
│ └── storage
├── fuzzball-federate
│ ├── certificates
│ ├── configmaps
│ ├── deployments
│ ├── events.txt
│ ├── ingresses
│ ├── metrics
│ ├── pods
│ ├── rbac
│ ├── secrets
│ ├── services
│ └── storage
├── fuzzball-identity
│ ├── certificates
│ ├── configmaps
│ ├── events.txt
│ ├── ingresses
│ ├── metrics
│ ├── pods
│ ├── secrets
│ └── services
├── fuzzball-system
│ ├── configmaps
│ ├── deployments
│ ├── events.txt
│ ├── metrics
│ ├── pods
│ ├── rbac
│ ├── secrets
│ ├── services
│ └── storage
├── manifest.json
└── metallb-system
└── events.txt
$ du -sh fuzzball-cluster-dump-20260115-121100
23M fuzzball-cluster-dump-20260115-121100 # small dump for a test deployment
$ tar -czf fuzzball-cluster-dump-20260115-121100.tgz \
fuzzball-cluster-dump-20260115-121100 # tar or zip for easy sharingThis creates a directory named fuzzball-cluster-dump-YYYYMMDD-HHMMSS containing the cluster state.
Include an additional namespace beyond the defaults:
$ fuzzball debug dump -n my-workload-namespaceThe cluster dump includes:
- Service logs and pod descriptions from all Fuzzball components, plus an
aggregated
errors.txtcollecting every log line that mentionserror,fatal, orpanic(each pod also gets its ownerrors.txt) - Kubernetes resource configurations (deployments, services, configmaps, RBAC, events; secret values are redacted)
- Node and pod CPU/memory usage from metrics-server
(
cluster/metrics/nodes.txt,<namespace>/metrics/pods.txt); when metrics-server is unavailable, raw kubelet stats summaries are written tocluster/metrics/kubelet-summary/instead - JetStream stream and consumer reports, including per-stream message counts
(
cluster/jetstream/) - The
FuzzballOrchestrateandFuzzballFederatecustom resources (cluster/crds/) - Ingresses and ingress classes (
<namespace>/ingresses/,cluster/ingressclasses/) - Certificates and cluster issuers from cert-manager
(
<namespace>/certificates/,cluster/clusterissuers/) - Storage classes, CSI drivers, persistent volumes, and persistent volume
claims (
cluster/storage/,<namespace>/storage/) - Postgres health checks for in-cluster databases – settings, activity,
connection counts, locks, and long-running queries
(
fuzzball-database/health/) - The scheduler’s view of the connected compute nodes, including each node’s
status, the holds recorded against it, and its recent event history
(
fuzzball/scheduler/) cluster/collector-diagnostics.txt, recording which Kubernetes APIs the dump could reach and therefore which optional collectors could not run; a collector that is skipped also leaves aSKIPPED.txtin its own directorymanifest.json, indexing every artifact and naming any source that failed (see Reading the Manifest under Support Bundle – the same format applies to a plain dump)
- The dump process may take several minutes depending on cluster size and log volume
- Ensure sufficient disk space is available for the dump
- The dump directory is timestamped to prevent overwrites
- Review the dump contents before sharing, as it may contain sensitive information
- Use this command when reporting issues to CIQ support for comprehensive diagnostics
A job that ends with a message naming a node died because that node stopped responding, not because of anything in the workflow:
node 10.0.0.42/2375: memory errors; job stopped responding on this node
Where a fault is named, the node was carrying that condition when the job died. Where only the node is named, the node had no active condition – the job may have hung, been killed for memory, or deadlocked, and the node is reported so you know where it was running.
Fuzzball decides a job has stopped responding in either of two ways, each after about five minutes:
- The node stops answering the job’s lease heartbeat.
- The node is reachable but no longer holds the job’s lease. Restarting Fuzzball Substrate on a node ends every job running there, so this is the usual sign of a substrate restart.
A node the control plane cannot reach is not judged by the second check: its jobs are left alone until it answers again or goes offline.
Start with the node rather than the workflow:
$ fuzzball node show 10.0.0.42/2375
$ fuzzball node events 10.0.0.42/2375 --latestSee Node Health Monitoring for how attribution is decided.
fuzzball debug report collects the cluster dump above and diagnostics
from the compute nodes, into one directory to send to support.
Node diagnostics travel over the substrate channel the nodes already hold, so no SSH access and no inbound port on the node is required. Each node contributes its health report and the recent kernel fault records behind it – the machine checks and GPU Xid records a monitoring rule would have alerted on. See Node Health Monitoring.
$ fuzzball debug report [flags]Press Ctrl+C to stop a collection in progress. Whatever has already been
written stays in the destination directory, but the bundle is left
unfinished: its manifest.json is not amended, and node data is only
written once every node has answered, so an interrupted collection usually
has no node directories at all.
| Flag | Short | Description | Default |
|---|---|---|---|
--dest | -d | Directory to write the bundle into | Current directory |
--nodes | Which nodes to collect from: none, unhealthy, all | none | |
--node | Collect from a single node, whatever its health | ||
--max-fault-lines | Maximum recent kernel fault lines per node | Node default |
--nodes and --node are mutually exclusive.
Collect the control plane and the nodes worth looking at – those that are not ready, unknown, offline, or carrying a hold:
$ fuzzball debug report --nodes unhealthy
bundle written to fuzzball-debug-report-20260812-104233Collect from one specific node, whatever its health:
$ fuzzball debug report --node 10.0.0.42/2375fuzzball-debug-report-20260812-104233
├── cluster/ # the cluster dump described above
├── nodes/
│ ├── gpu-14/
│ │ ├── health.json # the node's own reported signals
│ │ └── kernel-faults.txt # the machine check and Xid records behind them
│ └── gpu-17/
│ └── error.txt # this node did not answer, and why
└── manifest.json
Every bundle carries a manifest.json listing each source, how many artifacts
it produced, and for anything that failed, the reason:
{
"generatedAt": "2026-08-12T10:42:33Z",
"scrubbed": true,
"sources": [
{ "name": "pod-logs", "artifacts": 42 },
{ "name": "nodes", "artifacts": 2 },
{ "name": "node-diagnostics", "artifacts": 0,
"error": "1 of 2 nodes did not answer: 10.0.0.17/7334" }
],
"errors": ["node-diagnostics: 1 of 2 nodes did not answer: 10.0.0.17/7334"]
}
scrubbed reports whether secret values were removed. Every Kubernetes secret
in the bundle keeps its name, type and key names, but each value is replaced
with its length – redacted (7 bytes) rather than the credential. That is
enough to answer whether a key is present, whether it is empty, and whether a
certificate is plausibly a certificate, without the bundle carrying anything
worth stealing. Credential-bearing fields in the dumped FuzzballOrchestrate
and FuzzballFederate custom resources (passwords, access tokens, private
keys) are redacted the same way, and the last-applied-configuration
annotation, which re-embeds the originally applied values, is removed. Each
of their fields, including each item of a list such as cloudInitScripts, is
scrubbed like a ConfigMap entry of the same name, described below, so an
inline private key or a password in an embedded configuration is replaced
too. Container environment variables in Deployments and pod descriptions are
scrubbed the same way: a variable named for a credential
(DEFAULT_ADMIN_HASHED_PASSWORD) is replaced, and so is a credential inside
any other value, such as a URL password. Copies of secret values are matched
only in ConfigMaps, not in these fields or variables. The
last-applied-configuration annotation is removed from every collected object
(and from a pod’s description), and the other annotations, including a
Deployment’s pod template’s, are scrubbed like ConfigMap entries.
ConfigMaps are scrubbed of the credentials Fuzzball writes into them. Values
of credential fields (passwords, tokens, secret and private keys, such as
sessionSecret and spicedb-preshared-key) are replaced with their length,
also in commented-out entries (# password: ..., or ;admin_password = ...
in an INI file). A comment is redacted
like the same text outside a comment, except that a commented-out entry that
reads as a note keeps its value: a word such as required, optional or
none, or, under any field but a password, several words of
prose (# token: the API token issued by the IdP) that include no
credential-like word and don’t start with an authorization scheme such as
Bearer.
In a connection string, everything on the line after the first Password=,
Pwd=, sslpassword=, AccountKey= or SharedAccessKey= that holds a
value is replaced, other fields after it included (Server=db;Password=...
keeps Server=db;, and host=db password=... dbname=x keeps host=db):
connection-string formats delimit and quote values in too many ways to cut
the password out reliably. Only the quotes around a quoted value, and a ;,
, or tag end after its closing quote, are kept. Likewise, in a Go MySQL DSN
(user:...@tcp(host)/db, Azure’s user@server:...@tcp(host)/db, a Cloud
SQL @cloudsql-mysql(...)/db, user:...@/db) everything between the user’s
: after the line’s key and the last such @ on the line is replaced, so
other text with a : before it on the same line, such as a log line’s
timestamp, is replaced too; right after a key: , the key’s : counts, so a
DSN with no password there loses its user name, and words with no : before
a DSN (dial tcp app@tcp(...)) are taken for its password. Also replaced are a
credential in a URL query string (?token=..., a shared-access sig=...),
the password in any URL
(https://user:redacted@host), and any PEM private key, also when
base64-encoded (a kubeconfig’s client-key-data). A credential held in a
secret in the collected namespaces is replaced wherever it appears, raw or
encoded (base64, hex, percent-, JSON- or YAML-escaped), if it is at least
twelve bytes long, or at least eight and mixes at least two of lowercase
letters, uppercase letters, digits and symbols: a shorter or single-word value such as postgres would otherwise be
replaced inside host and image names throughout the bundle.
A placeholder that only refers to a value, such as ${DB_PASSWORD},
$DB_PASSWORD, a {{ .Values.password }} template or Argo CD’s
$oidc.keycloak.clientSecret, is kept (in a connection string, only when it
is quoted or nothing else follows it on the line); one with a default or a
function (${DB_PASSWORD:-...}, {{ .pw | default ... }}) can hold the
credential itself and is replaced. A count under a plural name
(refreshTokens: 3) and a setting such as enabled, true or off are
kept, and so are addresses, IDs, usernames, certificates and the names of Secrets, because a
bundle is often read to check them, even when a secret holds them. A section
named secret keeps only its known settings: the agent’s enabled, db,
kms and kms-config, a pod or projected volume’s secretName, name,
items, optional and defaultMode, a Helm chart’s create,
existingSecret, type, annotations and labels (and key beside a
name), and a host and port, with the credentials in them replaced.
Anything else under it is replaced. A section named authorization (kubelet’s,
Prometheus’s, Strimzi’s ACLs) is kept, except its credential fields and its
basic, bearer, digest, value, header, key and jwt fields. Under any other
credential field, everything is replaced, including a username nested in it.
A ConfigMap value that is a YAML or JSON document is parsed, so a credential field is found whatever YAML or JSON syntax it uses. A value with nothing to redact is kept exactly as it is. One with redactions is written back out, so its formatting, such as indentation and quoting, can differ from the live ConfigMap. Other text, and YAML that does not parse, is scrubbed by pattern, which recognizes fewer forms.
If the secrets of a namespace cannot be listed, no ConfigMaps are collected
from that namespace, and the failure is recorded in the manifest’s errors.
A credential copied from that namespace into another namespace’s ConfigMap is
then found only by its field name.
Scrubbing covers secret values and credential fields, not everything a bundle can contain. Pod logs, events, and container commands and arguments are collected verbatim, and an application that logs a token puts that token in the bundle. In ConfigMaps and environment variables, a credential is not detected under an unrecognized field name; on a command line or in prose (PGPASSWORD=... psql,Bearer ...outside anauthorizationfield); as a URL’s username (a token inhttps://TOKEN@host) or in its path (a chat webhook URL); in formats such as HCL or XML attributes; or base64-encoded together with other text (a Docker configauthentry). Review a bundle before sending it outside your organization.
Collection is best effort by design. A node that does not answer is recorded in the bundle rather than failing it, because during an incident an unreachable node is frequently the fault being investigated – and a tool that returned nothing in that case would be useless exactly when it is needed.
Check
errorsbefore sending a bundle. An empty list means everything asked for was collected. Every source is listed insourceswhether or not it succeeded, so anode-diagnosticsentry with noerrormeans the nodes were collected – not that they were skipped. Note the two distinct sources:nodesis the cluster’s Kubernetes Node objects, whilenode-diagnosticsis the per-node health snapshots gathered by--nodes.node-diagnosticsalways reports0artifacts; read itserrorfield, not its count.
Collecting a bundle does not change what the nodes report next: it reads each node’s most recent signals without taking a new health report, so the node’s heartbeat sequence and its exported metrics are untouched.
When the control plane itself is unreachable, fuzzball debug report cannot
run. Fuzzball ships an sos report plugin
for that case, installed with the fuzzball-substrate-orchestrate package:
$ sos report -o fuzzballIt collects the same host-local sources a node would have contributed –
substrate configuration, service logs, EDAC counters, GPU state and kernel
fault records – and falls back to kubectl for cluster state on a control
plane host. Credentials in configuration files are scrubbed automatically.
When a job is placed, the scheduler publishes a resources_allocated workflow event that
shows both what the job requested and the CPU placement the scheduler computed for it:
$ fuzzball workflow events WORKFLOW
...
Resources allocated (node: 10.0.0.3/7331, requested cpu: 28, requested memory: 28 GB, projected cpus: 28 (ids: 0-6,8-14,16-22,24-30), projected numa nodes: 4 (ids: 0-3))requested cpu,requested memory, andrequested devicesecho the resource request from the workflow definition. When the request setssockets,requested cpuis thecoresvalue, which counts cores per socket;requested devicesis the total device count across all device types.projected cpusandprojected numa nodesare the CPUs the scheduler selected for the job and the NUMA nodes they span, each rendered as a count followed by the specific IDs (count (ids: ...)). When the job requests hardware threads or multiple sockets, the projected CPU count is larger thanrequested cpubecause it counts every schedulable CPU the job receives. For multi-node jobs, per-node values are separated by semicolons in the same order as the comma-separated node list, and each per-node count compares againstrequested cpuindividually.
The projected values are the scheduler’s placement decision, not a reading from the node: the node’s substrate service independently computes the enforced CPU set from the same lease. The two normally match, but the projection is not a guarantee — during upgrades, transient differences in scheduler and node resource accounting, or if placement behavior ever diverges between scheduler and node, the CPUs a job actually runs on can differ from the projection.
When available, the kernel-enforced values appear on the container_started event, read
from the container process on the node when it starts. They use the same count-and-IDs
format, so enforced cpus compares directly against projected cpus and
enforced numa nodes against projected numa nodes:
Container started on node: 10.0.0.3/7331 (enforced cpus: 28 (ids: 0-6,8-14,16-22,24-30), enforced numa nodes: 4 (ids: 0-3))
The enforced values can be omitted when the node’s substrate version predates the feature,
when the container process exits before the values can be read, or when the node cannot be
reached. A container_started event without enforced fields is normal during upgrades or
transient node-communication failures.
Comparing the resources_allocated and container_started events for a stage shows
requested, projected, and enforced values side by side. For multi-node jobs the
container_started values cover the first node only. When the enforced values are
missing from the event, the same values can always be read from inside a job:
$ grep -E 'Cpus_allowed_list|Mems_allowed_list' /proc/self/statusA projected CPU count lower than requested cpu (without hardware threads or sockets in
play) indicates the scheduler could not compute a full placement and should be reported.
When the system takes a running stage off its node for any reason other than
priority, the scheduler publishes an evicted workflow event on that stage. Two
things cause one: an evict hold on the node, placed by fuzzball node evict or by
monitoring, and the node going offline. Each stage gets one event per eviction,
whether it was moved or failed:
$ fuzzball workflow events WORKFLOW
...
Evicted from node 10.0.0.42/2375 by monitoring hold 3f2a9c: GPU 2 uncorrectable ECC; requeued (node restart 1 of 3)
Evicted from node 10.0.0.43/2375: node went offline; stage failedThe event’s attributes, which fuzzball workflow events prints after the message
and -o json returns under metadata:
| Attribute | Meaning |
|---|---|
outcome | requeued when the stage was moved to run again elsewhere, failed when it was ended |
cause | hold or node_offline |
node_id | The node the stage was taken off |
hold_id, hold_source, hold_effect, hold_reason | The hold that caused the move, when cause is hold. hold_source is admin, monitoring, or provisioner. Copied onto the event, so it stays readable after the hold is released. hold_reason is omitted when the hold has none |
restarts, restart_limit | Node restarts the stage has used – counting this move when it was requeued – and how many it may use. When exit_code_retries is present, restarts counts the stage’s retries of every cause instead. Omitted when the stage did not declare policy.retry.on: node |
limit_source | The setting restart_limit came from, as on retries_exhausted. Omitted with restarts |
exit_code_retries | The exit-code retries counted in restarts. Present only when policy.retry.attempts is the limit and the stage has also been retried for an exit code, because that budget is shared |
reason | The sentence the stage was told; for a failed stage, also its error |
trigger | node_drain for an admin hold, node_fault otherwise; the reason label on the preemptions_total metric |
evicted_in_cluster_id | The cluster the eviction happened in. Omitted when the cluster has no configured cluster ID |
A stage that also lists exit codes in policy.retry.on shares one
policy.retry.attempts budget between node restarts and exit-code retries. Once
an exit code has used some of it and that budget is the limit, the move is
counted as a retry instead:
Evicted from node 10.0.0.42/2375: node went offline; requeued (retry 3 of 4)
A stage that declared policy.retry.on: node but has used every node restart it
is allowed is failed rather than moved, and a retries_exhausted event follows
its evicted event:
Retries exhausted: 3 of 3 node restart(s) used, the limit set by nodeHealth.maxEvictionRestarts; cause: monitoring hold 3f2a9c on node 10.0.0.42/2375: GPU 2 uncorrectable ECC
When exit-code retries spent part of a shared budget, the event says how it was spent:
Retries exhausted: 3 of 3 retries used (1 for a node, 2 for an exit code), the limit set by policy.retry.attempts; cause: node 10.0.0.42/2375 went offline
It carries the same cause, node_id, hold_* and evicted_in_cluster_id
attributes, naming the eviction that found the restarts spent – not necessarily
the one that used the last of them – plus restarts, restart_limit,
limit_source and, for a shared budget, exit_code_retries.
limit_source names the setting that produced the limit:
policy.retry.attempts when the stage’s own declaration was the lower (or equal)
of the two, or nodeHealth.maxEvictionRestarts when the cluster’s ceiling was.
Earlier moves appear as earlier evicted events on the same stage. See
Restart Tolerance for how
the two limits combine.
A stage whose job exits with a code listed in policy.retry.on is requeued to run
again from the beginning, wherever the scheduler places it, and the scheduler
publishes a retried event:
Retrying: exited with code 137 on node 10.0.0.42/2375; requeued (retry 1 of 3)
| Attribute | Meaning |
|---|---|
cause | Always exit_code |
exit_code | The code the job exited with |
rank | The rank whose exit decided it – 0 for a single job. A multi-rank job is retried whole either way |
node_id | The node that rank ran on |
restarts, restart_limit | Retries spent, counting this one, and how many the stage may use. The budget is shared: node restarts count against it too |
limit_source | policy.retry.attempts; the limit is 3 when the stage lists exit codes without setting attempts |
reason | The sentence the stage was told |
retried_in_cluster_id | The cluster the retry happened in. Omitted when the cluster has no configured cluster ID |
When the budget is spent, the stage fails as it would have without the retry,
and a retries_exhausted event names the exit code:
Retries exhausted: 3 of 3 retries used, the limit set by policy.retry.attempts; cause: exited with code 137 on node 10.0.0.42/2375
A job that ends on SIGINT or SIGTERM (exit 130 or 143) is treated as a clean stop, so it never produces either event. See Retrying on an Exit Code.
preempted events now mean priority preemption only: the stage was displaced by
higher-priority work, or by an administrator’s preemption. Through v4.3, active
node drains and node evictions also published preempted, with a trigger of
node_drain or node_fault; filters or dashboards that matched those values
should match evicted instead.
When a job’s container exits on its own with a failing code, the stage’s
container_exited and stage_failed events carry that code as the exit_code
attribute, so a filter or script can match on the code instead of parsing the
error message:
$ fuzzball workflow events WORKFLOW
...
Container exited with code: 137
Stage failed: job has exited with code 137 (signal: killed)| Event | exit_code |
|---|---|
container_exited | The code the container exited with. stopped for a service that was shut down |
stage_failed | The code of the container exit that failed the stage |
A code above 128 means a signal ended the container: 128 plus the signal number,
so 137 is SIGKILL (9), which the kernel’s out-of-memory killer sends, and 143 is
SIGTERM (15). A job that exits with 0, 130, or 143 is treated as having stopped
normally, so those codes do not fail it.
exit_code is omitted when the failure was not reported as a container exit –
the node went offline, the job timed out, was evicted, or never started – and
the error message names that cause instead. It is also omitted when the node
could not determine the exit status, and for jobs on nodes running a Fuzzball
release older than the attribute, whose error message still states the code.
The scheduler publishes a scheduling_blocked workflow event when it cannot place a job’s
allocation for a reason worth naming. The event’s reason attribute identifies the cause
and its detail attribute explains it in a sentence, which fuzzball workflow events
shows as the event’s message. The first case below is the pool cap: the allocation is
waiting for node reuse because its provisioner definition’s node pool has reached the
maxNodes cap. That event means the job is queued, not rejected: it proceeds once a node
in the pool becomes free, which is expected when the cluster is at capacity and workflows
are competing for resources. The later subsections cover the other reasons.
- The definition is dynamic and its pool has reached the effective cap
(
min(maxNodes, 128)). - The allocation fits within the cap, but no node is currently free for reuse.
- Two companion events are published, once per blocked allocation:
provisioning_blocked, which carries thedefinition_id,total_nodes, andmax_nodesattributes, and the genericscheduling_blockedevent. - With backfill enabled (the default), an at-cap allocation may instead hold a
reservation and publish a
reservation_heldevent.
Because the events are published once per blocked allocation, judge persistence by whether the allocation stays unscheduled, not by counting events.
Follow the workflow’s events and look for
provisioning_blocked(itsdefinition_idattribute names the affected definition):$ fuzzball workflow events WORKFLOW --followAsk the scheduler why the allocation is not scheduled. A
PROVISIONING_BLOCKEDreason indicates the pool cap; see Queue Observability for the full reasons table:$ fuzzball workflow why WORKFLOWReview the definition’s current
maxNodesvalue:$ fuzzball node provisioner get compute-pool
Increase maxNodes: Raise the
maxNodesvalue on the provisioner definition and apply the updated configuration:definitions: - id: compute-pool provisioner: pbs ttl: 3600 maxNodes: 64 provisionerSpec: cpu: 8 memory: "32GiB" queue: "workq"$ fuzzball cluster config set updated-config.yamlWait for node reuse: Transient blocking resolves on its own as running jobs complete and free their nodes.
Reduce workflow node requirements: If feasible, request fewer nodes so allocations fit within the available pool capacity.
Review provisioner policies: Ensure policy expressions are not concentrating load on a single definition.
A scheduling_blocked event whose reason attribute is node_local_volume_pin means
something different from the pool cap above: the job mounts a volume from a provisioner
marked local,
so it can only run on the one node that holds that volume. The event carries the node in
its required_node attribute, and its detail attribute distinguishes the two cases:
- The node has no capacity yet. Ordinary contention. The job runs once the node frees up, exactly like any other queued allocation.
- The node is not available to the scheduler. The node is down, drained, or has been removed. This does not resolve on its own – the volume exists only on that node, so no other node and no newly provisioned node can run the job.
A related event, reason: node_local_volume_lost, is terminal rather than a wait: it means
the holding node was deprovisioned, so the volume’s data is gone with its disk. The
workflow fails naming the node instead of queueing.
The scheduler does not provision new nodes for a pinned allocation, because a node created now cannot hold a volume created earlier somewhere else. If the node is gone for good, recreate the volume on an available node or restore the original node.
Only volumes on a provisioner withlocal: truepin jobs this way, and both persistent and ephemeral volumes do. A provisioner whose path is genuinely a shared mount across all nodes should leave the field unset, so that jobs are placed without constraint.
A scheduling_blocked event whose reason attribute is lease_grant_failed means the
scheduler picked a node for the job, but the node refused to grant the lease. The
placement is rolled back and the allocation returns to the queue, so the job stays
queued rather than failing immediately. The event carries the node in its node
attribute, the affected rank in rank, how many consecutive attempts have failed in
attempts, and the reason the node gave in detail.
Most refusals clear on their own. A lease from a previous attempt is still being
cleaned up, a node is briefly unreachable, or the node’s actual free capacity no longer
matches what the scheduler projected when it chose the node. The scheduler retries on the
next pass. How many attempts this takes is not fixed — a job with many ranks retried on
every tick can rack up a great many refusals in a short time and still recover — so a
rising attempts count on its own is not a sign of trouble.
A refusal that persists fails the workflow. Once the same allocation has been refused continuously for more than ten minutes and over at least ten attempts, the scheduler stops retrying and fails the workflow with the error the node returned. Both conditions must hold, so on a cluster running the default one-minute scheduler interval it is the elapsed window that decides, and a wedged allocation fails on the first tick past ten minutes — a little past eleven. A longer interval pushes that out proportionally, and past roughly a one-minute loop it is the attempt floor that decides instead. The bound is deliberately generous, never eager. The error appears in the workflow’s status, for example:
failed to grant lease for rank 0 on compute node 10.0.0.97:7331: could not set exclusive mode, other resources in use
“Continuously” is what matters. An allocation that is refused, then is not retried for a while because it is queued behind other work, starts a fresh streak when it is next tried; the run of attempts only counts as continuous while they arrive within three scheduler intervals of each other.
An autoscaled service replica abovereplicas.minis exempt. It is optional capacity, so a persistent refusal returns it to the queue to be retried indefinitely rather than failing the service — the replicas withinreplicas.minkeep running. Thescheduling_blockedevents andworkflow whystill report it as blocked.
Persistent refusal is not the only route to failure. A multi-rank job fails immediately, on the first refusal, if the scheduler also cannot release the leases it had already granted to the job’s other ranks while rolling the placement back — it fails the workflow then rather than retry and leave those leases stranded on their nodes.
To investigate while the job is still queued:
Read the event and the reason the node gave:
$ fuzzball workflow events WORKFLOWAsk the scheduler why the allocation is not running. A
LEASE_GRANT_FAILINGreason names the node and the refusal; see Queue Observability for the full reasons table:$ fuzzball workflow why WORKFLOWCheck the state of the node named in the event:
$ fuzzball node show NODE
A node that looks idle but keeps refusing may still be holding a lease stranded by an earlier attempt. The scheduler handles the common form of this on its own: when a node reports that the lease is already granted, it revokes the stale lease before the next attempt so that retry can grant. That revoke is best-effort, so it does not guarantee the next attempt succeeds. If refusals continue, the node’s view of its own resources has diverged from the scheduler’s and needs an administrator — restarting the Fuzzball agent on that node clears its lease state.
Do not drain the node to clear a refusal. Draining cordons it, which takes it out of service for new placements — including the job you are trying to start.
A scheduling_blocked event whose reason attribute is static_definition_no_nodes
means the allocation is pinned to a static provisioner definition that currently has no
nodes registered at all. The event carries the definition in its definition_id
attribute.
This one does not clear on its own. A static definition never provisions, so no node appears unless an operator brings one back, and the allocation keeps its definition for its whole life – the scheduler chooses a definition once, at submission, and never re-picks it. The job waits indefinitely until a node registers with that definition or the workflow is canceled.
The scheduler will not select an empty static definition for a new submission, so seeing this event means either the allocation was submitted before the nodes went away, or it predates the version that added that check. To resolve it:
Confirm which nodes the definition should have, and whether any are registered:
$ fuzzball node listBring the missing nodes back – the usual cause is that the Fuzzball Substrate daemon is not running on them, or the definition’s matching condition no longer matches the nodes it was written for. See Hardware Grouping Requirements.
If the nodes are gone for good, cancel the workflow and resubmit it once a suitable definition has nodes. Resubmitting is what re-runs definition selection.
A scheduling_blocked event whose reason attribute is
definition_too_few_nodes means the allocation is assigned to a provisioner
definition that has fewer nodes than the job needs. The event carries the
definition in definition_id and both counts in required_nodes and
pool_nodes.
This only affects multi-node jobs, which are gang-scheduled onto a single definition: all of their nodes have to come from one pool, so a pool smaller than the job can never satisfy it however long the job waits. It does not clear on its own, and the allocation keeps the definition it was assigned at submission.
A pool that is merely busy is not reported this way. Occupied, cordoned, and
rebooting nodes all count toward pool_nodes, so a job waiting for a large
enough pool to drain stays quiet and simply runs when capacity frees up.
The scheduler rejects such a job at submission, so seeing this event means the
pool shrank after the job was submitted – nodes deregistered, or the definition
was assigned before that check existed. To resolve it, return nodes to the named
definition, or cancel and resubmit the job with a multinode.nodes the cluster
can host.
A scheduling_blocked event whose reason attribute is insufficient_resources
means the definition has fewer nodes able to hold the request than the job needs,
counting each node’s whole capacity as if nothing ran on it: one node for a job
or a task array, and every rank’s node for a multi-node job. The event carries
the definition in definition_id and the scheduler’s summary of the last
attempt in detail, the same summary fuzzball workflow why reports. When a
node fell short on CPU or memory rather than devices, nearest_node names the
one with the most free memory and detail says what it offers.
Waiting does not clear this. A node that cannot hold the request when empty cannot hold it when it drains either. The usual cause is a request sized against the definition’s advertised resources when the live node has less, such as a cloud instance whose usable memory sits below its nominal size.
A pool that is merely busy is not reported this way. Occupancy does not enter the count, so a job waiting for a large enough node to drain stays quiet and runs when that node frees up, however much backfilled work keeps the pool busy meanwhile.
To resolve it, lower the job’s request to what the definition’s nodes offer, or run it on a definition whose nodes are large enough. The event is published once per cause, not once per tick, so judge persistence by whether the stage stays queued rather than by counting events.
A scheduling_blocked event whose reason attribute is port_conflict means a
host port the service declares is held by a running service on every node of
the definition with room for it. A service’s declared ports are published on
its node at those numbers (bound outright on the host network), so two
services declaring the same port cannot share a node, and the scheduler keeps
them apart. The event carries the definition in definition_id and the
scheduler’s summary in detail – for example
1 with a declared host port already taken; tcp port 8080 on 10.13.0.3/7331 is held by allocation <id> – the same summary fuzzball workflow why reports as
PORT_CONFLICT.
The usual cause is a persistent endpoint takeover
under the ready strategy on a pool too small to run the old and new service
side by side: the running workflow keeps serving and the new one waits. Cancel
the new submission, switch the endpoint to the start strategy, or declare a
different port in each submission. A port a cancelled or finished service held
is reported as still releasing a declared host port for a few seconds, while
the node deletes its container; that clears by itself and raises no event.
A scheduling_blocked event whose reason attribute is device_plugin_failed
means the definition has too few nodes able to hold the request, and the nodes
that report a failed device plugin and lack nothing for the request but a
device kind they have none of would make up the difference. The event
carries the definition in definition_id and the scheduler’s summary in
detail, which names the failed plugins – for example
1 without the requested device type and with a failed device plugin (nvidia)
– the same summary fuzzball workflow why reports. Substrate reports which
plugin failed but not yet which device kind it serves, so the summary states
both facts. A node without the CPU or memory for the request, or with too few
of another requested device kind, is counted as
without the requested device type whatever its plugins.
The same note is appended when a job is refused at submission because no node
in the pool could hold it even when empty, naming the failed plugins on the
same nodes, for example
only 1 of 2 node(s) can hold one rank even when empty but the job needs 2; 1 node(s) report a failed device plugin (nvidia).
Waiting does not clear this. The node keeps reporting and taking work that
needs no device, so nothing about it reads wrong apart from this.
fuzzball node show lists each plugin with its state and error, and the bundled
FuzzballNodeDevicePluginFailed alert fires on the same signal. If the failed
plugin serves the requested kind, repair it and restart fuzzball-substrate:
the device kind reappears on the node’s next report and the job places. If it
serves another kind, the node does not have the requested hardware. See
Device Plugins.
A scheduling_blocked event whose reason attribute is
node_annotations_mismatch means nodes in the definition had capacity for the
request, but not enough of them carry the node annotations the job requires. The
event carries the definition in definition_id, and its detail attribute names
the required annotations and how many nodes had room without them.
A job requires node annotations indirectly: mounting a volume that lives in a storage segment restricts the job to nodes that carry that segment’s annotation. The scheduler does not provision a replacement while a node with room exists, so the job waits until a node carrying the annotations has room. If no node in the definition carries them, that wait never ends.
To resolve it, check which segment the volume’s provisioner is assigned to and whether the definition’s nodes carry it, then either move the volume to a provisioner the definition’s nodes can reach or run the job on a definition whose nodes carry the segment. Like the previous reason, the event is published once per cause, not once per tick.
A scheduling_blocked event whose reason attribute is
requeued_awaiting_capacity means the stage was running, was taken off its node
– by an evicted or preempted event just before it – and the scheduler has
not yet found room to run it again. The event carries the definition in
definition_id; its detail attribute is the scheduler’s summary of the last
placement attempt, or says that the capacity is held for higher-priority work
queued ahead of the stage. When a node fell short on CPU or memory,
nearest_node names the one with the most free memory.
This is the one busy-pool wait that is reported. A stage that has never run waits
for a busy pool in silence; a stage that was running a moment ago and then goes
quiet reads as lost work. The event is published once per eviction, not once per
tick, and the stage runs when capacity frees up – the next event for it is its
resources_allocated. A stage whose provisioner definition can grow provisions a
node instead and reports that with its provisioning events.
The scheduler publishes a provisioning_registration_timeout workflow event when nodes it
provisioned did not register within the definition’s
registration deadline.
Unlike scheduling_blocked, this event is terminal: the scheduler deletes the instances the
provisioning request created and fails the workflow with an error of the form
only 1/3 requested nodes registered within the registration deadline.
The usual cause is a bootstrap failure that the backend never reports. The instance is created and running, but the Fuzzball Substrate on it never comes up – a cloud-init error, a failed package install, an NFS mount that does not appear, or a network path that blocks the Substrate from reaching the cluster.
The event carries the following attributes. The transaction_ attributes refer to the provisioning
request: Fuzzball records each request as a provisioning transaction internally, and that is the
name that appears in event attributes and log messages.
| Attribute | Description |
|---|---|
definition_id | The provisioner definition the nodes were requested from |
transaction_id | The provisioning request whose nodes failed to register |
registered_nodes | How many nodes registered before the deadline expired |
required_nodes | How many nodes the provisioning request asked for |
transaction_age | How long the request had been outstanding when it expired |
A registered_nodes value greater than zero is a partial bootstrap failure: some instances came up
and others did not, which usually points at a per-instance problem (a bad zone, exhausted capacity,
a per-node mount) rather than a broken definition.
Read the event and its attributes from the failed workflow:
$ fuzzball workflow events WORKFLOWCheck whether the definition’s deadline is too short for this workload. Large images, GPU instances, and slow regions can all need longer than the
15mdefault. The deadline lives in the central configuration and is not part offuzzball node provisioner getoutput. This command prints the entire configuration as it was last set and requires cluster-admin credentials; find your definition underdefinitions:. A definition with noregistrationDeadlinekey is on the15mdefault:$ fuzzball cluster config getReproduce the failure with a single-node workflow if you need a live instance to inspect. The instances from the expired request are deleted by the time the workflow fails, so inspect the backend while the reproduction is still running.
Check the Orchestrate logs for the instance cleanup that followed the timeout. A
failed to delete provisioner instances for expired registration transactionmessage means the cleanup did not complete:$ kubectl logs -n fuzzball-system deployment/fuzzball-orchestrator --tail=200
Raise the deadline: If the nodes do eventually register, just later than the deadline allows, raise
registrationDeadlineon the definition and apply the updated configuration:definitions: - id: gpu-pool provisioner: aws ttl: 7200 registrationDeadline: 30m provisionerSpec: instanceType: p5.48xlarge$ fuzzball cluster config set updated-config.yamlFix the bootstrap failure: If the nodes never register no matter how long the deadline, raising it only delays the failure. Investigate the image, the cloud-init or package configuration, and the network path from the node to the cluster.
Account for backend queue time: On Slurm and PBS the deadline starts at
sbatchorqsubsubmission, so time the batch job spends queued in the backend counts against the deadline. See Workflow Stuck in PENDING in the Slurm and PBS troubleshooting guide, which covers both backends in identical sections.
If instance cleanup fails repeatedly, the scheduler stops retrying after three consecutive failures and fails the workflow anyway rather than leaving the workflow hung. Each instance it could not delete is recorded in the Orchestrate logs asfailed to delete instance during transaction cleanup, and those instances may need to be cleaned up manually in the backend.
When image cache write-back is enabled (it is off by default), the node uploads the converted image to the object cache after a cache-miss pull, before the image stage finishes – so a stage can run well past the pull itself on a large image or a slow link. The upload publishes its own events, so read the stage’s events before suspecting the pull:
$ fuzzball workflow events WORKFLOW --followUploading image to object cache marks the hand-off from pull to upload, and the Web UI shows a
progress bar on the stage’s card while it runs. Image upload to object cache failed names the
reason, and the number of attempts when more than one was made; the stage still finishes and the
job runs on the pulled image. The object cache simply does not get a copy. See
Watching the upload to the cache
for the retry and timeout rules, and cacheWriteBack in the
central configuration reference
to turn the upload on where the link to the object cache is fast enough.
For collecting some Fuzzball debugging information, administrators may need to
interact with the Kubernetes cluster directly using
kubectl. In this section we
assume that you have the kubectl tool installed following the general
installation
instructions or
when you configured your local Kubernetes
cluster and configured to access
your Kubernetes cluster.
A full tutorial about Kubernetes and kubectl is beyond the scope of this documentation. Below are some commands useful for debugging.
Direct access to the Kubernetes cluster can be potentially dangerous. Proceed with caution.
Inspect the resource consumption of your pods if you suspect that pods are approaching their limits, are overloaded, or use significantly fewer resources than they have been allocated:
$ kubectl top pods --all-namespaces --sort-by=memory --sum
NAMESPACE NAME CPU(cores) MEMORY(bytes)
fuzzball-identity identity-keycloakx-2 3m 5081Mi
fuzzball-identity identity-keycloakx-1 4m 5057Mi
fuzzball-identity identity-keycloakx-0 3m 4990Mi
ingress kong-kong-7bbf7cc6d7-6wq6b 7m 470Mi
...
fuzzball fuzzball-admin-ui-74f84b6757-m2fwt 1m 5Mi
fuzzball fuzzball-admin-0 1m 0Mi
fuzzball postgres-debug 0m 0Mi
________ ________
2243m 24839MiTo ensure that your Kubernetes nodes are not running out of resources you can check with the following:
$ kubectl top nodes
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
node1 922m 11% 15603Mi 50%
node2 1077m 13% 18847Mi 61%
node3 667m 8% 15156Mi 49%In this example the three node Kubernetes cluster is operating below capacity.
You can check that your fuzzball pods are all in a Running state and not frequently
restarting like so:
$ kubectl get -n fuzzball pods
NAME READY STATUS RESTARTS AGE
fuzzball-admin-0 1/1 Running 0 64m
fuzzball-admin-ui-74f84b6757-m2fwt 1/1 Running 0 64m
fuzzball-agent-7467f6484b-kjwv6 1/1 Running 0 61m
fuzzball-agent-7467f6484b-lkdmf 1/1 Running 0 62m
fuzzball-agent-7467f6484b-qxwfq 1/1 Running 0 62m
fuzzball-audit-68f75847bb-5blmp 1/1 Running 0 64m
...Any states other than Running or Completed may signal problems, as are
pods with fewer than expected Ready replicas.
More detailed information about a particular pod can be obtained with:
$ kubectl describe -n fuzzball pod/fuzzball-agent-7467f6484b-kjwv6
Name: fuzzball-agent-7467f6484b-kjwv6
Namespace: fuzzball
Priority: 0
Service Account: fuzzball-agent
Node: ip-10-0-123-57.us-west-2.compute.internal/10.0.123.57
Start Time: Fri, 16 Jan 2026 19:56:49 -0500
Labels: app.kubernetes.io/instance=fuzzball-agent
app.kubernetes.io/managed-by=pulumi
...A stateful set of 3 or more pods form a
Jetstream cluster used for
message passing between Fuzzball services. A buildup of messages for certain topics
like the workflow-engine may indicate that Orchestrate is overloaded and processing
events like state-changes of workflow stages too slowly. This can be checked with the
following command:
$ kubectl exec -n fuzzball fuzzball-jetstream-0 -- /app/nats stream ls
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Streams │
├────────────────────────────────────────┬─────────────┬─────────────────────┬──────────┬──────┬──────────────┤
│ Name │ Description │ Created │ Messages │ Size │ Last Message │
├────────────────────────────────────────┼─────────────┼─────────────────────┼──────────┼──────┼──────────────┤
│ job-metrics-stream │ │ 2026-01-06 17:00:57 │ 0 │ 0 B │ 56m2s │
│ provisioner-aws-delete-placement-group │ │ 2026-01-06 16:49:54 │ 0 │ 0 B │ never │
│ provisioner-core │ │ 2026-01-06 16:49:54 │ 0 │ 0 B │ 50m14s │
│ substrate-activity-stream │ │ 2026-01-06 17:00:57 │ 0 │ 0 B │ 56m0s │
│ substrate-conn-stream │ │ 2026-01-06 16:48:44 │ 0 │ 0 B │ 50m12s │
│ substrate-lease-stream │ │ 2026-01-06 17:00:56 │ 0 │ 0 B │ 56m0s │
│ substrate-resource-discovery-stream │ │ 2026-01-06 16:48:44 │ 0 │ 0 B │ 56m36s │
│ substrate-resource-stream │ │ 2026-01-06 16:48:44 │ 0 │ 0 B │ 55m27s │
│ substrate-service-stream │ │ 2026-01-06 17:00:57 │ 0 │ 0 B │ 1d5h52m43s │
│ workflow-execution │ │ 2026-01-06 16:49:54 │ 0 │ 0 B │ 56m0s │
│ leader │ │ 2026-01-17 00:26:15 │ 1 │ 39 B │ 4h14m37s │
│ provisioner-aws-pricing-ticker │ │ 2026-01-06 16:49:54 │ 1 │ 60 B │ 10d9h16m27s │
╰────────────────────────────────────────┴─────────────┴─────────────────────┴──────────┴──────┴──────────────╯