Fuzzball Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

Debugging

This section provides documentation for debugging and troubleshooting Fuzzball clusters. These tools are designed for administrators to diagnose issues, monitor service behavior, and perform advanced maintenance tasks.

When to Use Debug Commands

Debug commands are useful for:

  • Viewing service logs to diagnose issues
  • Creating debug dumps to share with the support team

Prerequisites

You will need to install the Fuzzball CLI (see the CLI installation documentation for details), set up a Fuzzball admin context and authenticate like so:

$ fuzzball context create default-admin-context <api_url> local fuzzball-cli
Configuration for "default-admin-context" created.
Configuration for "default-admin-context" now in use.

$ FUZZBALL_USER="admin_username"

$ read -rs FUZZBALL_PASSWORD  # read interactively to set variable without echoing to screen

$ export FUZZBALL_USER FUZZBALL_PASSWORD

$ fuzzball context login
Logging into current cluster context...

Service Logs

The fuzzball debug logs command allows administrators to view logs from Fuzzball services running in the cluster. This is useful for debugging service issues, monitoring service behavior, and tracking down errors.

Basic Usage

$ fuzzball debug logs <SERVICE_NAME> [flags]

Available Services

The following Fuzzball services can be queried for logs:

  • agent - Fuzzball agent service
  • audit - Audit logging service
  • auth - Authentication service
  • billing - Billing service
  • cluster-admin - Cluster administration service
  • jetstream - JetStream message broker
  • openapi - OpenAPI service
  • orchestrator - Orchestrator service
  • provision - Provisioning service
  • schedule - Scheduling service
  • storage - Storage service
  • substrate-bridge - Substrate bridge service
  • ui - Web UI service
  • workflow - Workflow service

Command Flags

FlagShortDescriptionDefault
--since-sShow logs since a specific time (RFC3339 or YYYY-MM-DD format)-
--since-duration-dShow logs since a specific duration (e.g., ‘1h’, ‘30m’)-
--tail-tNumber of lines to show from the end of the logs100
--follow-fFollow the log output (stream logs)false
--min-log-level-Minimum log level to display (1: Debug, 2: Info, 3: Warning, 4: Error, 5: Fatal)0
--trace-id-Filter logs by trace ID-
--ignore-errors-Ignore errors when fetching logsfalse
--json-jOutput in JSON formatfalse

Example Usage

View the last 100 lines of logs from the workflow service:

$ fuzzball debug logs workflow
[fuzzball-workflow-0]: 2026-01-13T10:15:23Z INFO Starting workflow service
[fuzzball-workflow-0]: 2026-01-13T10:15:24Z INFO Connected to database
[fuzzball-workflow-0]: 2026-01-13T10:15:25Z INFO Workflow service ready

View logs from multiple services:

$ fuzzball debug logs workflow storage
[fuzzball-workflow-0]: 2026-01-13T10:15:23Z INFO Starting workflow service
[fuzzball-storage-0]: 2026-01-13T10:15:22Z INFO Storage service initialized

Follow logs in real-time:

$ fuzzball debug logs workflow --follow
[fuzzball-workflow-0]: 2026-01-13T10:15:23Z INFO Starting workflow service
[fuzzball-workflow-0]: 2026-01-13T10:15:24Z INFO Processing workflow request

Show logs since a specific time:

$ fuzzball debug logs workflow --since 2026-01-13T10:00:00Z

Show logs from the last hour:

$ fuzzball debug logs workflow --since-duration 1h

Filter by minimum log level (errors and above):

$ fuzzball debug logs workflow --min-log-level 4

Show the last 500 lines:

$ fuzzball debug logs workflow --tail 500

Output logs in JSON format:

$ fuzzball debug logs workflow --json
{"pod_name":"fuzzball-workflow-0","message":"2026-01-13T10:15:23Z INFO Starting workflow service","level":2}
{"pod_name":"fuzzball-workflow-0","message":"2026-01-13T10:15:24Z INFO Connected to database","level":2}

Notes

  • The --since and --since-duration flags cannot be used together
  • When using --follow, logs will stream continuously until interrupted (Ctrl+C)
  • The --json flag is useful for programmatic parsing and integration with log analysis tools
  • Log levels: 0 (Unspecified), 1 (Debug), 2 (Info), 3 (Warning), 4 (Error), 5 (Fatal)

Cluster Dump

The fuzzball debug dump command creates a comprehensive dump of the Fuzzball cluster state, including configuration, logs, and diagnostic information. This is useful for gathering all relevant information when reporting issues or performing detailed troubleshooting.

Basic Usage

$ fuzzball debug dump [flags]

Command Flags

FlagShortDescriptionDefault
--dest-dDestination path to save the dumpCurrent directory
--namespace-nAdditional namespaces to include in the dump; repeat the flag or pass a comma-separated list-

The dump always covers Fuzzball’s own namespaces and those of its dependencies: fuzzball, fuzzball-federate, fuzzball-system, fuzzball-database, fuzzball-identity, cert-manager, and metallb-system. A namespace that does not exist on your deployment contributes nothing beyond an empty events.txt.

Example Usage

Create a cluster dump in the current directory:

$ fuzzball debug dump

$ ls -lhd fuzzball-cluster-dump-*
drwxr-xr-x@ 10 user  group   320B Jan 15 12:11 fuzzball-cluster-dump-20260115-121100

$ tree -L2 fuzzball-cluster-dump-20260115-121100
fuzzball-cluster-dump-20260115-121100
├── cert-manager
│   ├── certificates
│   ├── configmaps
│   ├── deployments
│   ├── events.txt
│   ├── metrics
│   ├── pods
│   ├── rbac
│   ├── secrets
│   └── services
├── cluster
│   ├── clusterissuers
│   ├── collector-diagnostics.txt
│   ├── crds
│   ├── ingressclasses
│   ├── jetstream
│   ├── metrics
│   ├── nodes
│   ├── rbac
│   └── storage
├── cluster_info.txt
├── errors.txt
├── fuzzball
│   ├── certificates
│   ├── configmaps
│   ├── deployments
│   ├── events.txt
│   ├── ingresses
│   ├── metrics
│   ├── pods
│   ├── rbac
│   ├── scheduler
│   ├── secrets
│   ├── services
│   └── storage
├── fuzzball-database
│   ├── certificates
│   ├── configmaps
│   ├── deployments
│   ├── events.txt
│   ├── health
│   ├── metrics
│   ├── pods
│   ├── rbac
│   ├── secrets
│   ├── services
│   └── storage
├── fuzzball-federate
│   ├── certificates
│   ├── configmaps
│   ├── deployments
│   ├── events.txt
│   ├── ingresses
│   ├── metrics
│   ├── pods
│   ├── rbac
│   ├── secrets
│   ├── services
│   └── storage
├── fuzzball-identity
│   ├── certificates
│   ├── configmaps
│   ├── events.txt
│   ├── ingresses
│   ├── metrics
│   ├── pods
│   ├── secrets
│   └── services
├── fuzzball-system
│   ├── configmaps
│   ├── deployments
│   ├── events.txt
│   ├── metrics
│   ├── pods
│   ├── rbac
│   ├── secrets
│   ├── services
│   └── storage
├── manifest.json
└── metallb-system
    └── events.txt

$ du -sh fuzzball-cluster-dump-20260115-121100
 23M    fuzzball-cluster-dump-20260115-121100   # small dump for a test deployment

$ tar -czf fuzzball-cluster-dump-20260115-121100.tgz \
    fuzzball-cluster-dump-20260115-121100  # tar or zip for easy sharing

This creates a directory named fuzzball-cluster-dump-YYYYMMDD-HHMMSS containing the cluster state.

Include an additional namespace beyond the defaults:

$ fuzzball debug dump -n my-workload-namespace

Dump Contents

The cluster dump includes:

  • Service logs and pod descriptions from all Fuzzball components, plus an aggregated errors.txt collecting every log line that mentions error, fatal, or panic (each pod also gets its own errors.txt)
  • Kubernetes resource configurations (deployments, services, configmaps, RBAC, events; secret values are redacted)
  • Node and pod CPU/memory usage from metrics-server (cluster/metrics/nodes.txt, <namespace>/metrics/pods.txt); when metrics-server is unavailable, raw kubelet stats summaries are written to cluster/metrics/kubelet-summary/ instead
  • JetStream stream and consumer reports, including per-stream message counts (cluster/jetstream/)
  • The FuzzballOrchestrate and FuzzballFederate custom resources (cluster/crds/)
  • Ingresses and ingress classes (<namespace>/ingresses/, cluster/ingressclasses/)
  • Certificates and cluster issuers from cert-manager (<namespace>/certificates/, cluster/clusterissuers/)
  • Storage classes, CSI drivers, persistent volumes, and persistent volume claims (cluster/storage/, <namespace>/storage/)
  • Postgres health checks for in-cluster databases – settings, activity, connection counts, locks, and long-running queries (fuzzball-database/health/)
  • The scheduler’s view of the connected compute nodes, including each node’s status, the holds recorded against it, and its recent event history (fuzzball/scheduler/)
  • cluster/collector-diagnostics.txt, recording which Kubernetes APIs the dump could reach and therefore which optional collectors could not run; a collector that is skipped also leaves a SKIPPED.txt in its own directory
  • manifest.json, indexing every artifact and naming any source that failed (see Reading the Manifest under Support Bundle – the same format applies to a plain dump)

Notes

  • The dump process may take several minutes depending on cluster size and log volume
  • Ensure sufficient disk space is available for the dump
  • The dump directory is timestamped to prevent overwrites
  • Review the dump contents before sharing, as it may contain sensitive information
  • Use this command when reporting issues to CIQ support for comprehensive diagnostics

Jobs That Stopped Responding

A job that ends with a message naming a node died because that node stopped responding, not because of anything in the workflow:

node 10.0.0.42/2375: memory errors; job stopped responding on this node

Where a fault is named, the node was carrying that condition when the job died. Where only the node is named, the node had no active condition – the job may have hung, been killed for memory, or deadlocked, and the node is reported so you know where it was running.

Fuzzball decides a job has stopped responding in either of two ways, each after about five minutes:

  • The node stops answering the job’s lease heartbeat.
  • The node is reachable but no longer holds the job’s lease. Restarting Fuzzball Substrate on a node ends every job running there, so this is the usual sign of a substrate restart.

A node the control plane cannot reach is not judged by the second check: its jobs are left alone until it answers again or goes offline.

Start with the node rather than the workflow:

$ fuzzball node show 10.0.0.42/2375
$ fuzzball node events 10.0.0.42/2375 --latest

See Node Health Monitoring for how attribution is decided.

Support Bundle

fuzzball debug report collects the cluster dump above and diagnostics from the compute nodes, into one directory to send to support.

Node diagnostics travel over the substrate channel the nodes already hold, so no SSH access and no inbound port on the node is required. Each node contributes its health report and the recent kernel fault records behind it – the machine checks and GPU Xid records a monitoring rule would have alerted on. See Node Health Monitoring.

Basic Usage

$ fuzzball debug report [flags]

Press Ctrl+C to stop a collection in progress. Whatever has already been written stays in the destination directory, but the bundle is left unfinished: its manifest.json is not amended, and node data is only written once every node has answered, so an interrupted collection usually has no node directories at all.

Command Flags

FlagShortDescriptionDefault
--dest-dDirectory to write the bundle intoCurrent directory
--nodesWhich nodes to collect from: none, unhealthy, allnone
--nodeCollect from a single node, whatever its health
--max-fault-linesMaximum recent kernel fault lines per nodeNode default

--nodes and --node are mutually exclusive.

Example Usage

Collect the control plane and the nodes worth looking at – those that are not ready, unknown, offline, or carrying a hold:

$ fuzzball debug report --nodes unhealthy
bundle written to fuzzball-debug-report-20260812-104233

Collect from one specific node, whatever its health:

$ fuzzball debug report --node 10.0.0.42/2375

What a Bundle Contains

fuzzball-debug-report-20260812-104233
├── cluster/            # the cluster dump described above
├── nodes/
│   ├── gpu-14/
│   │   ├── health.json         # the node's own reported signals
│   │   └── kernel-faults.txt   # the machine check and Xid records behind them
│   └── gpu-17/
│       └── error.txt           # this node did not answer, and why
└── manifest.json

Reading the Manifest

Every bundle carries a manifest.json listing each source, how many artifacts it produced, and for anything that failed, the reason:

{
  "generatedAt": "2026-08-12T10:42:33Z",
  "scrubbed": true,
  "sources": [
    { "name": "pod-logs", "artifacts": 42 },
    { "name": "nodes", "artifacts": 2 },
    { "name": "node-diagnostics", "artifacts": 0,
      "error": "1 of 2 nodes did not answer: 10.0.0.17/7334" }
  ],
  "errors": ["node-diagnostics: 1 of 2 nodes did not answer: 10.0.0.17/7334"]
}

scrubbed reports whether secret values were removed. Every Kubernetes secret in the bundle keeps its name, type and key names, but each value is replaced with its length – redacted (7 bytes) rather than the credential. That is enough to answer whether a key is present, whether it is empty, and whether a certificate is plausibly a certificate, without the bundle carrying anything worth stealing. Credential-bearing fields in the dumped FuzzballOrchestrate and FuzzballFederate custom resources (passwords, access tokens, private keys) are redacted the same way, and the last-applied-configuration annotation, which re-embeds the originally applied values, is removed. Each of their fields, including each item of a list such as cloudInitScripts, is scrubbed like a ConfigMap entry of the same name, described below, so an inline private key or a password in an embedded configuration is replaced too. Container environment variables in Deployments and pod descriptions are scrubbed the same way: a variable named for a credential (DEFAULT_ADMIN_HASHED_PASSWORD) is replaced, and so is a credential inside any other value, such as a URL password. Copies of secret values are matched only in ConfigMaps, not in these fields or variables. The last-applied-configuration annotation is removed from every collected object (and from a pod’s description), and the other annotations, including a Deployment’s pod template’s, are scrubbed like ConfigMap entries.

ConfigMaps are scrubbed of the credentials Fuzzball writes into them. Values of credential fields (passwords, tokens, secret and private keys, such as sessionSecret and spicedb-preshared-key) are replaced with their length, also in commented-out entries (# password: ..., or ;admin_password = ... in an INI file). A comment is redacted like the same text outside a comment, except that a commented-out entry that reads as a note keeps its value: a word such as required, optional or none, or, under any field but a password, several words of prose (# token: the API token issued by the IdP) that include no credential-like word and don’t start with an authorization scheme such as Bearer.

In a connection string, everything on the line after the first Password=, Pwd=, sslpassword=, AccountKey= or SharedAccessKey= that holds a value is replaced, other fields after it included (Server=db;Password=... keeps Server=db;, and host=db password=... dbname=x keeps host=db): connection-string formats delimit and quote values in too many ways to cut the password out reliably. Only the quotes around a quoted value, and a ;, , or tag end after its closing quote, are kept. Likewise, in a Go MySQL DSN (user:...@tcp(host)/db, Azure’s user@server:...@tcp(host)/db, a Cloud SQL @cloudsql-mysql(...)/db, user:...@/db) everything between the user’s : after the line’s key and the last such @ on the line is replaced, so other text with a : before it on the same line, such as a log line’s timestamp, is replaced too; right after a key: , the key’s : counts, so a DSN with no password there loses its user name, and words with no : before a DSN (dial tcp app@tcp(...)) are taken for its password. Also replaced are a credential in a URL query string (?token=..., a shared-access sig=...), the password in any URL (https://user:redacted@host), and any PEM private key, also when base64-encoded (a kubeconfig’s client-key-data). A credential held in a secret in the collected namespaces is replaced wherever it appears, raw or encoded (base64, hex, percent-, JSON- or YAML-escaped), if it is at least twelve bytes long, or at least eight and mixes at least two of lowercase letters, uppercase letters, digits and symbols: a shorter or single-word value such as postgres would otherwise be replaced inside host and image names throughout the bundle.

A placeholder that only refers to a value, such as ${DB_PASSWORD}, $DB_PASSWORD, a {{ .Values.password }} template or Argo CD’s $oidc.keycloak.clientSecret, is kept (in a connection string, only when it is quoted or nothing else follows it on the line); one with a default or a function (${DB_PASSWORD:-...}, {{ .pw | default ... }}) can hold the credential itself and is replaced. A count under a plural name (refreshTokens: 3) and a setting such as enabled, true or off are kept, and so are addresses, IDs, usernames, certificates and the names of Secrets, because a bundle is often read to check them, even when a secret holds them. A section named secret keeps only its known settings: the agent’s enabled, db, kms and kms-config, a pod or projected volume’s secretName, name, items, optional and defaultMode, a Helm chart’s create, existingSecret, type, annotations and labels (and key beside a name), and a host and port, with the credentials in them replaced. Anything else under it is replaced. A section named authorization (kubelet’s, Prometheus’s, Strimzi’s ACLs) is kept, except its credential fields and its basic, bearer, digest, value, header, key and jwt fields. Under any other credential field, everything is replaced, including a username nested in it.

A ConfigMap value that is a YAML or JSON document is parsed, so a credential field is found whatever YAML or JSON syntax it uses. A value with nothing to redact is kept exactly as it is. One with redactions is written back out, so its formatting, such as indentation and quoting, can differ from the live ConfigMap. Other text, and YAML that does not parse, is scrubbed by pattern, which recognizes fewer forms.

If the secrets of a namespace cannot be listed, no ConfigMaps are collected from that namespace, and the failure is recorded in the manifest’s errors. A credential copied from that namespace into another namespace’s ConfigMap is then found only by its field name.

Scrubbing covers secret values and credential fields, not everything a bundle can contain. Pod logs, events, and container commands and arguments are collected verbatim, and an application that logs a token puts that token in the bundle. In ConfigMaps and environment variables, a credential is not detected under an unrecognized field name; on a command line or in prose (PGPASSWORD=... psql, Bearer ... outside an authorization field); as a URL’s username (a token in https://TOKEN@host) or in its path (a chat webhook URL); in formats such as HCL or XML attributes; or base64-encoded together with other text (a Docker config auth entry). Review a bundle before sending it outside your organization.

Collection is best effort by design. A node that does not answer is recorded in the bundle rather than failing it, because during an incident an unreachable node is frequently the fault being investigated – and a tool that returned nothing in that case would be useless exactly when it is needed.

Check errors before sending a bundle. An empty list means everything asked for was collected. Every source is listed in sources whether or not it succeeded, so a node-diagnostics entry with no error means the nodes were collected – not that they were skipped. Note the two distinct sources: nodes is the cluster’s Kubernetes Node objects, while node-diagnostics is the per-node health snapshots gathered by --nodes. node-diagnostics always reports 0 artifacts; read its error field, not its count.

Collecting a bundle does not change what the nodes report next: it reads each node’s most recent signals without taking a new health report, so the node’s heartbeat sequence and its exported metrics are untouched.

Collecting Without the Control Plane

When the control plane itself is unreachable, fuzzball debug report cannot run. Fuzzball ships an sos report plugin for that case, installed with the fuzzball-substrate-orchestrate package:

$ sos report -o fuzzball

It collects the same host-local sources a node would have contributed – substrate configuration, service logs, EDAC counters, GPU state and kernel fault records – and falls back to kubectl for cluster state on a control plane host. Credentials in configuration files are scrubbed automatically.

Reading resources_allocated Events

When a job is placed, the scheduler publishes a resources_allocated workflow event that shows both what the job requested and the CPU placement the scheduler computed for it:

$ fuzzball workflow events WORKFLOW
...
Resources allocated (node: 10.0.0.3/7331, requested cpu: 28, requested memory: 28 GB, projected cpus: 28 (ids: 0-6,8-14,16-22,24-30), projected numa nodes: 4 (ids: 0-3))
  • requested cpu, requested memory, and requested devices echo the resource request from the workflow definition. When the request sets sockets, requested cpu is the cores value, which counts cores per socket; requested devices is the total device count across all device types.
  • projected cpus and projected numa nodes are the CPUs the scheduler selected for the job and the NUMA nodes they span, each rendered as a count followed by the specific IDs (count (ids: ...)). When the job requests hardware threads or multiple sockets, the projected CPU count is larger than requested cpu because it counts every schedulable CPU the job receives. For multi-node jobs, per-node values are separated by semicolons in the same order as the comma-separated node list, and each per-node count compares against requested cpu individually.

The projected values are the scheduler’s placement decision, not a reading from the node: the node’s substrate service independently computes the enforced CPU set from the same lease. The two normally match, but the projection is not a guarantee — during upgrades, transient differences in scheduler and node resource accounting, or if placement behavior ever diverges between scheduler and node, the CPUs a job actually runs on can differ from the projection.

When available, the kernel-enforced values appear on the container_started event, read from the container process on the node when it starts. They use the same count-and-IDs format, so enforced cpus compares directly against projected cpus and enforced numa nodes against projected numa nodes:

Container started on node: 10.0.0.3/7331 (enforced cpus: 28 (ids: 0-6,8-14,16-22,24-30), enforced numa nodes: 4 (ids: 0-3))

The enforced values can be omitted when the node’s substrate version predates the feature, when the container process exits before the values can be read, or when the node cannot be reached. A container_started event without enforced fields is normal during upgrades or transient node-communication failures.

Comparing the resources_allocated and container_started events for a stage shows requested, projected, and enforced values side by side. For multi-node jobs the container_started values cover the first node only. When the enforced values are missing from the event, the same values can always be read from inside a job:

$ grep -E 'Cpus_allowed_list|Mems_allowed_list' /proc/self/status

A projected CPU count lower than requested cpu (without hardware threads or sockets in play) indicates the scheduler could not compute a full placement and should be reported.

Reading evicted and retries_exhausted Events

When the system takes a running stage off its node for any reason other than priority, the scheduler publishes an evicted workflow event on that stage. Two things cause one: an evict hold on the node, placed by fuzzball node evict or by monitoring, and the node going offline. Each stage gets one event per eviction, whether it was moved or failed:

$ fuzzball workflow events WORKFLOW
...
Evicted from node 10.0.0.42/2375 by monitoring hold 3f2a9c: GPU 2 uncorrectable ECC; requeued (node restart 1 of 3)
Evicted from node 10.0.0.43/2375: node went offline; stage failed

The event’s attributes, which fuzzball workflow events prints after the message and -o json returns under metadata:

AttributeMeaning
outcomerequeued when the stage was moved to run again elsewhere, failed when it was ended
causehold or node_offline
node_idThe node the stage was taken off
hold_id, hold_source, hold_effect, hold_reasonThe hold that caused the move, when cause is hold. hold_source is admin, monitoring, or provisioner. Copied onto the event, so it stays readable after the hold is released. hold_reason is omitted when the hold has none
restarts, restart_limitNode restarts the stage has used – counting this move when it was requeued – and how many it may use. When exit_code_retries is present, restarts counts the stage’s retries of every cause instead. Omitted when the stage did not declare policy.retry.on: node
limit_sourceThe setting restart_limit came from, as on retries_exhausted. Omitted with restarts
exit_code_retriesThe exit-code retries counted in restarts. Present only when policy.retry.attempts is the limit and the stage has also been retried for an exit code, because that budget is shared
reasonThe sentence the stage was told; for a failed stage, also its error
triggernode_drain for an admin hold, node_fault otherwise; the reason label on the preemptions_total metric
evicted_in_cluster_idThe cluster the eviction happened in. Omitted when the cluster has no configured cluster ID

A stage that also lists exit codes in policy.retry.on shares one policy.retry.attempts budget between node restarts and exit-code retries. Once an exit code has used some of it and that budget is the limit, the move is counted as a retry instead:

Evicted from node 10.0.0.42/2375: node went offline; requeued (retry 3 of 4)

A stage that declared policy.retry.on: node but has used every node restart it is allowed is failed rather than moved, and a retries_exhausted event follows its evicted event:

Retries exhausted: 3 of 3 node restart(s) used, the limit set by nodeHealth.maxEvictionRestarts; cause: monitoring hold 3f2a9c on node 10.0.0.42/2375: GPU 2 uncorrectable ECC

When exit-code retries spent part of a shared budget, the event says how it was spent:

Retries exhausted: 3 of 3 retries used (1 for a node, 2 for an exit code), the limit set by policy.retry.attempts; cause: node 10.0.0.42/2375 went offline

It carries the same cause, node_id, hold_* and evicted_in_cluster_id attributes, naming the eviction that found the restarts spent – not necessarily the one that used the last of them – plus restarts, restart_limit, limit_source and, for a shared budget, exit_code_retries. limit_source names the setting that produced the limit: policy.retry.attempts when the stage’s own declaration was the lower (or equal) of the two, or nodeHealth.maxEvictionRestarts when the cluster’s ceiling was. Earlier moves appear as earlier evicted events on the same stage. See Restart Tolerance for how the two limits combine.

Retried for an Exit Code

A stage whose job exits with a code listed in policy.retry.on is requeued to run again from the beginning, wherever the scheduler places it, and the scheduler publishes a retried event:

Retrying: exited with code 137 on node 10.0.0.42/2375; requeued (retry 1 of 3)
AttributeMeaning
causeAlways exit_code
exit_codeThe code the job exited with
rankThe rank whose exit decided it – 0 for a single job. A multi-rank job is retried whole either way
node_idThe node that rank ran on
restarts, restart_limitRetries spent, counting this one, and how many the stage may use. The budget is shared: node restarts count against it too
limit_sourcepolicy.retry.attempts; the limit is 3 when the stage lists exit codes without setting attempts
reasonThe sentence the stage was told
retried_in_cluster_idThe cluster the retry happened in. Omitted when the cluster has no configured cluster ID

When the budget is spent, the stage fails as it would have without the retry, and a retries_exhausted event names the exit code:

Retries exhausted: 3 of 3 retries used, the limit set by policy.retry.attempts; cause: exited with code 137 on node 10.0.0.42/2375

A job that ends on SIGINT or SIGTERM (exit 130 or 143) is treated as a clean stop, so it never produces either event. See Retrying on an Exit Code.

preempted events now mean priority preemption only: the stage was displaced by higher-priority work, or by an administrator’s preemption. Through v4.3, active node drains and node evictions also published preempted, with a trigger of node_drain or node_fault; filters or dashboards that matched those values should match evicted instead.

Reading a Failed Job’s Exit Code

When a job’s container exits on its own with a failing code, the stage’s container_exited and stage_failed events carry that code as the exit_code attribute, so a filter or script can match on the code instead of parsing the error message:

$ fuzzball workflow events WORKFLOW
...
Container exited with code: 137
Stage failed: job has exited with code 137 (signal: killed)
Eventexit_code
container_exitedThe code the container exited with. stopped for a service that was shut down
stage_failedThe code of the container exit that failed the stage

A code above 128 means a signal ended the container: 128 plus the signal number, so 137 is SIGKILL (9), which the kernel’s out-of-memory killer sends, and 143 is SIGTERM (15). A job that exits with 0, 130, or 143 is treated as having stopped normally, so those codes do not fail it.

exit_code is omitted when the failure was not reported as a container exit – the node went offline, the job timed out, was evicted, or never started – and the error message names that cause instead. It is also omitted when the node could not determine the exit status, and for jobs on nodes running a Fuzzball release older than the attribute, whose error message still states the code.

Troubleshooting scheduling_blocked Events

The scheduler publishes a scheduling_blocked workflow event when it cannot place a job’s allocation for a reason worth naming. The event’s reason attribute identifies the cause and its detail attribute explains it in a sentence, which fuzzball workflow events shows as the event’s message. The first case below is the pool cap: the allocation is waiting for node reuse because its provisioner definition’s node pool has reached the maxNodes cap. That event means the job is queued, not rejected: it proceeds once a node in the pool becomes free, which is expected when the cluster is at capacity and workflows are competing for resources. The later subsections cover the other reasons.

When the Events Are Published

  • The definition is dynamic and its pool has reached the effective cap (min(maxNodes, 128)).
  • The allocation fits within the cap, but no node is currently free for reuse.
  • Two companion events are published, once per blocked allocation: provisioning_blocked, which carries the definition_id, total_nodes, and max_nodes attributes, and the generic scheduling_blocked event.
  • With backfill enabled (the default), an at-cap allocation may instead hold a reservation and publish a reservation_held event.

Because the events are published once per blocked allocation, judge persistence by whether the allocation stays unscheduled, not by counting events.

Observing the Events

  1. Follow the workflow’s events and look for provisioning_blocked (its definition_id attribute names the affected definition):

    $ fuzzball workflow events WORKFLOW --follow
  2. Ask the scheduler why the allocation is not scheduled. A PROVISIONING_BLOCKED reason indicates the pool cap; see Queue Observability for the full reasons table:

    $ fuzzball workflow why WORKFLOW
  3. Review the definition’s current maxNodes value:

    $ fuzzball node provisioner get compute-pool

Resolving Persistent Blocking

  • Increase maxNodes: Raise the maxNodes value on the provisioner definition and apply the updated configuration:

    definitions:
      - id: compute-pool
        provisioner: pbs
        ttl: 3600
        maxNodes: 64
        provisionerSpec:
          cpu: 8
          memory: "32GiB"
          queue: "workq"
    
    $ fuzzball cluster config set updated-config.yaml
  • Wait for node reuse: Transient blocking resolves on its own as running jobs complete and free their nodes.

  • Reduce workflow node requirements: If feasible, request fewer nodes so allocations fit within the available pool capacity.

  • Review provisioner policies: Ensure policy expressions are not concentrating load on a single definition.

Blocking on a Node-Local Volume

A scheduling_blocked event whose reason attribute is node_local_volume_pin means something different from the pool cap above: the job mounts a volume from a provisioner marked local, so it can only run on the one node that holds that volume. The event carries the node in its required_node attribute, and its detail attribute distinguishes the two cases:

  • The node has no capacity yet. Ordinary contention. The job runs once the node frees up, exactly like any other queued allocation.
  • The node is not available to the scheduler. The node is down, drained, or has been removed. This does not resolve on its own – the volume exists only on that node, so no other node and no newly provisioned node can run the job.

A related event, reason: node_local_volume_lost, is terminal rather than a wait: it means the holding node was deprovisioned, so the volume’s data is gone with its disk. The workflow fails naming the node instead of queueing.

The scheduler does not provision new nodes for a pinned allocation, because a node created now cannot hold a volume created earlier somewhere else. If the node is gone for good, recreate the volume on an available node or restore the original node.

Only volumes on a provisioner with local: true pin jobs this way, and both persistent and ephemeral volumes do. A provisioner whose path is genuinely a shared mount across all nodes should leave the field unset, so that jobs are placed without constraint.

Blocking on a Refused Lease Grant

A scheduling_blocked event whose reason attribute is lease_grant_failed means the scheduler picked a node for the job, but the node refused to grant the lease. The placement is rolled back and the allocation returns to the queue, so the job stays queued rather than failing immediately. The event carries the node in its node attribute, the affected rank in rank, how many consecutive attempts have failed in attempts, and the reason the node gave in detail.

Most refusals clear on their own. A lease from a previous attempt is still being cleaned up, a node is briefly unreachable, or the node’s actual free capacity no longer matches what the scheduler projected when it chose the node. The scheduler retries on the next pass. How many attempts this takes is not fixed — a job with many ranks retried on every tick can rack up a great many refusals in a short time and still recover — so a rising attempts count on its own is not a sign of trouble.

A refusal that persists fails the workflow. Once the same allocation has been refused continuously for more than ten minutes and over at least ten attempts, the scheduler stops retrying and fails the workflow with the error the node returned. Both conditions must hold, so on a cluster running the default one-minute scheduler interval it is the elapsed window that decides, and a wedged allocation fails on the first tick past ten minutes — a little past eleven. A longer interval pushes that out proportionally, and past roughly a one-minute loop it is the attempt floor that decides instead. The bound is deliberately generous, never eager. The error appears in the workflow’s status, for example:

failed to grant lease for rank 0 on compute node 10.0.0.97:7331: could not set exclusive mode, other resources in use

“Continuously” is what matters. An allocation that is refused, then is not retried for a while because it is queued behind other work, starts a fresh streak when it is next tried; the run of attempts only counts as continuous while they arrive within three scheduler intervals of each other.

An autoscaled service replica above replicas.min is exempt. It is optional capacity, so a persistent refusal returns it to the queue to be retried indefinitely rather than failing the service — the replicas within replicas.min keep running. The scheduling_blocked events and workflow why still report it as blocked.

Persistent refusal is not the only route to failure. A multi-rank job fails immediately, on the first refusal, if the scheduler also cannot release the leases it had already granted to the job’s other ranks while rolling the placement back — it fails the workflow then rather than retry and leave those leases stranded on their nodes.

To investigate while the job is still queued:

  1. Read the event and the reason the node gave:

    $ fuzzball workflow events WORKFLOW
  2. Ask the scheduler why the allocation is not running. A LEASE_GRANT_FAILING reason names the node and the refusal; see Queue Observability for the full reasons table:

    $ fuzzball workflow why WORKFLOW
  3. Check the state of the node named in the event:

    $ fuzzball node show NODE

A node that looks idle but keeps refusing may still be holding a lease stranded by an earlier attempt. The scheduler handles the common form of this on its own: when a node reports that the lease is already granted, it revokes the stale lease before the next attempt so that retry can grant. That revoke is best-effort, so it does not guarantee the next attempt succeeds. If refusals continue, the node’s view of its own resources has diverged from the scheduler’s and needs an administrator — restarting the Fuzzball agent on that node clears its lease state.

Do not drain the node to clear a refusal. Draining cordons it, which takes it out of service for new placements — including the job you are trying to start.

A Static Definition With No Registered Nodes

A scheduling_blocked event whose reason attribute is static_definition_no_nodes means the allocation is pinned to a static provisioner definition that currently has no nodes registered at all. The event carries the definition in its definition_id attribute.

This one does not clear on its own. A static definition never provisions, so no node appears unless an operator brings one back, and the allocation keeps its definition for its whole life – the scheduler chooses a definition once, at submission, and never re-picks it. The job waits indefinitely until a node registers with that definition or the workflow is canceled.

The scheduler will not select an empty static definition for a new submission, so seeing this event means either the allocation was submitted before the nodes went away, or it predates the version that added that check. To resolve it:

  1. Confirm which nodes the definition should have, and whether any are registered:

    $ fuzzball node list
  2. Bring the missing nodes back – the usual cause is that the Fuzzball Substrate daemon is not running on them, or the definition’s matching condition no longer matches the nodes it was written for. See Hardware Grouping Requirements.

  3. If the nodes are gone for good, cancel the workflow and resubmit it once a suitable definition has nodes. Resubmitting is what re-runs definition selection.

A Definition With Too Few Nodes For The Job

A scheduling_blocked event whose reason attribute is definition_too_few_nodes means the allocation is assigned to a provisioner definition that has fewer nodes than the job needs. The event carries the definition in definition_id and both counts in required_nodes and pool_nodes.

This only affects multi-node jobs, which are gang-scheduled onto a single definition: all of their nodes have to come from one pool, so a pool smaller than the job can never satisfy it however long the job waits. It does not clear on its own, and the allocation keeps the definition it was assigned at submission.

A pool that is merely busy is not reported this way. Occupied, cordoned, and rebooting nodes all count toward pool_nodes, so a job waiting for a large enough pool to drain stays quiet and simply runs when capacity frees up.

The scheduler rejects such a job at submission, so seeing this event means the pool shrank after the job was submitted – nodes deregistered, or the definition was assigned before that check existed. To resolve it, return nodes to the named definition, or cancel and resubmit the job with a multinode.nodes the cluster can host.

A Request No Node Can Hold

A scheduling_blocked event whose reason attribute is insufficient_resources means the definition has fewer nodes able to hold the request than the job needs, counting each node’s whole capacity as if nothing ran on it: one node for a job or a task array, and every rank’s node for a multi-node job. The event carries the definition in definition_id and the scheduler’s summary of the last attempt in detail, the same summary fuzzball workflow why reports. When a node fell short on CPU or memory rather than devices, nearest_node names the one with the most free memory and detail says what it offers.

Waiting does not clear this. A node that cannot hold the request when empty cannot hold it when it drains either. The usual cause is a request sized against the definition’s advertised resources when the live node has less, such as a cloud instance whose usable memory sits below its nominal size.

A pool that is merely busy is not reported this way. Occupancy does not enter the count, so a job waiting for a large enough node to drain stays quiet and runs when that node frees up, however much backfilled work keeps the pool busy meanwhile.

To resolve it, lower the job’s request to what the definition’s nodes offer, or run it on a definition whose nodes are large enough. The event is published once per cause, not once per tick, so judge persistence by whether the stage stays queued rather than by counting events.

A Declared Port Taken on Every Node

A scheduling_blocked event whose reason attribute is port_conflict means a host port the service declares is held by a running service on every node of the definition with room for it. A service’s declared ports are published on its node at those numbers (bound outright on the host network), so two services declaring the same port cannot share a node, and the scheduler keeps them apart. The event carries the definition in definition_id and the scheduler’s summary in detail – for example 1 with a declared host port already taken; tcp port 8080 on 10.13.0.3/7331 is held by allocation <id> – the same summary fuzzball workflow why reports as PORT_CONFLICT.

The usual cause is a persistent endpoint takeover under the ready strategy on a pool too small to run the old and new service side by side: the running workflow keeps serving and the new one waits. Cancel the new submission, switch the endpoint to the start strategy, or declare a different port in each submission. A port a cancelled or finished service held is reported as still releasing a declared host port for a few seconds, while the node deletes its container; that clears by itself and raises no event.

A Node Whose Device Plugin Failed

A scheduling_blocked event whose reason attribute is device_plugin_failed means the definition has too few nodes able to hold the request, and the nodes that report a failed device plugin and lack nothing for the request but a device kind they have none of would make up the difference. The event carries the definition in definition_id and the scheduler’s summary in detail, which names the failed plugins – for example 1 without the requested device type and with a failed device plugin (nvidia) – the same summary fuzzball workflow why reports. Substrate reports which plugin failed but not yet which device kind it serves, so the summary states both facts. A node without the CPU or memory for the request, or with too few of another requested device kind, is counted as without the requested device type whatever its plugins.

The same note is appended when a job is refused at submission because no node in the pool could hold it even when empty, naming the failed plugins on the same nodes, for example only 1 of 2 node(s) can hold one rank even when empty but the job needs 2; 1 node(s) report a failed device plugin (nvidia).

Waiting does not clear this. The node keeps reporting and taking work that needs no device, so nothing about it reads wrong apart from this. fuzzball node show lists each plugin with its state and error, and the bundled FuzzballNodeDevicePluginFailed alert fires on the same signal. If the failed plugin serves the requested kind, repair it and restart fuzzball-substrate: the device kind reappears on the node’s next report and the job places. If it serves another kind, the node does not have the requested hardware. See Device Plugins.

Nodes With Room But Not The Requested Annotations

A scheduling_blocked event whose reason attribute is node_annotations_mismatch means nodes in the definition had capacity for the request, but not enough of them carry the node annotations the job requires. The event carries the definition in definition_id, and its detail attribute names the required annotations and how many nodes had room without them.

A job requires node annotations indirectly: mounting a volume that lives in a storage segment restricts the job to nodes that carry that segment’s annotation. The scheduler does not provision a replacement while a node with room exists, so the job waits until a node carrying the annotations has room. If no node in the definition carries them, that wait never ends.

To resolve it, check which segment the volume’s provisioner is assigned to and whether the definition’s nodes carry it, then either move the volume to a provisioner the definition’s nodes can reach or run the job on a definition whose nodes carry the segment. Like the previous reason, the event is published once per cause, not once per tick.

A Requeued Stage Waiting for Capacity

A scheduling_blocked event whose reason attribute is requeued_awaiting_capacity means the stage was running, was taken off its node – by an evicted or preempted event just before it – and the scheduler has not yet found room to run it again. The event carries the definition in definition_id; its detail attribute is the scheduler’s summary of the last placement attempt, or says that the capacity is held for higher-priority work queued ahead of the stage. When a node fell short on CPU or memory, nearest_node names the one with the most free memory.

This is the one busy-pool wait that is reported. A stage that has never run waits for a busy pool in silence; a stage that was running a moment ago and then goes quiet reads as lost work. The event is published once per eviction, not once per tick, and the stage runs when capacity frees up – the next event for it is its resources_allocated. A stage whose provisioner definition can grow provisions a node instead and reports that with its provisioning events.

Troubleshooting provisioning_registration_timeout Events

The scheduler publishes a provisioning_registration_timeout workflow event when nodes it provisioned did not register within the definition’s registration deadline. Unlike scheduling_blocked, this event is terminal: the scheduler deletes the instances the provisioning request created and fails the workflow with an error of the form only 1/3 requested nodes registered within the registration deadline.

The usual cause is a bootstrap failure that the backend never reports. The instance is created and running, but the Fuzzball Substrate on it never comes up – a cloud-init error, a failed package install, an NFS mount that does not appear, or a network path that blocks the Substrate from reaching the cluster.

Event Attributes

The event carries the following attributes. The transaction_ attributes refer to the provisioning request: Fuzzball records each request as a provisioning transaction internally, and that is the name that appears in event attributes and log messages.

AttributeDescription
definition_idThe provisioner definition the nodes were requested from
transaction_idThe provisioning request whose nodes failed to register
registered_nodesHow many nodes registered before the deadline expired
required_nodesHow many nodes the provisioning request asked for
transaction_ageHow long the request had been outstanding when it expired

A registered_nodes value greater than zero is a partial bootstrap failure: some instances came up and others did not, which usually points at a per-instance problem (a bad zone, exhausted capacity, a per-node mount) rather than a broken definition.

Diagnosing the Failure

  1. Read the event and its attributes from the failed workflow:

    $ fuzzball workflow events WORKFLOW
  2. Check whether the definition’s deadline is too short for this workload. Large images, GPU instances, and slow regions can all need longer than the 15m default. The deadline lives in the central configuration and is not part of fuzzball node provisioner get output. This command prints the entire configuration as it was last set and requires cluster-admin credentials; find your definition under definitions:. A definition with no registrationDeadline key is on the 15m default:

    $ fuzzball cluster config get
  3. Reproduce the failure with a single-node workflow if you need a live instance to inspect. The instances from the expired request are deleted by the time the workflow fails, so inspect the backend while the reproduction is still running.

  4. Check the Orchestrate logs for the instance cleanup that followed the timeout. A failed to delete provisioner instances for expired registration transaction message means the cleanup did not complete:

    $ kubectl logs -n fuzzball-system deployment/fuzzball-orchestrator --tail=200

Resolving Persistent Timeouts

  • Raise the deadline: If the nodes do eventually register, just later than the deadline allows, raise registrationDeadline on the definition and apply the updated configuration:

    definitions:
      - id: gpu-pool
        provisioner: aws
        ttl: 7200
        registrationDeadline: 30m
        provisionerSpec:
          instanceType: p5.48xlarge
    
    $ fuzzball cluster config set updated-config.yaml
  • Fix the bootstrap failure: If the nodes never register no matter how long the deadline, raising it only delays the failure. Investigate the image, the cloud-init or package configuration, and the network path from the node to the cluster.

  • Account for backend queue time: On Slurm and PBS the deadline starts at sbatch or qsub submission, so time the batch job spends queued in the backend counts against the deadline. See Workflow Stuck in PENDING in the Slurm and PBS troubleshooting guide, which covers both backends in identical sections.

If instance cleanup fails repeatedly, the scheduler stops retrying after three consecutive failures and fails the workflow anyway rather than leaving the workflow hung. Each instance it could not delete is recorded in the Orchestrate logs as failed to delete instance during transaction cleanup, and those instances may need to be cleaned up manually in the backend.

Image Stages That Outlast Their Pull

When image cache write-back is enabled (it is off by default), the node uploads the converted image to the object cache after a cache-miss pull, before the image stage finishes – so a stage can run well past the pull itself on a large image or a slow link. The upload publishes its own events, so read the stage’s events before suspecting the pull:

$ fuzzball workflow events WORKFLOW --follow

Uploading image to object cache marks the hand-off from pull to upload, and the Web UI shows a progress bar on the stage’s card while it runs. Image upload to object cache failed names the reason, and the number of attempts when more than one was made; the stage still finishes and the job runs on the pulled image. The object cache simply does not get a copy. See Watching the upload to the cache for the retry and timeout rules, and cacheWriteBack in the central configuration reference to turn the upload on where the link to the object cache is fast enough.

Advanced Debugging

Kubernetes-based Orchestrate deployments

For collecting some Fuzzball debugging information, administrators may need to interact with the Kubernetes cluster directly using kubectl. In this section we assume that you have the kubectl tool installed following the general installation instructions or when you configured your local Kubernetes cluster and configured to access your Kubernetes cluster.

A full tutorial about Kubernetes and kubectl is beyond the scope of this documentation. Below are some commands useful for debugging.

Direct access to the Kubernetes cluster can be potentially dangerous. Proceed with caution.

Resource Utilization

Inspect the resource consumption of your pods if you suspect that pods are approaching their limits, are overloaded, or use significantly fewer resources than they have been allocated:

$ kubectl top pods --all-namespaces --sort-by=memory --sum
NAMESPACE           NAME                                                    CPU(cores)   MEMORY(bytes)
fuzzball-identity   identity-keycloakx-2                                    3m           5081Mi
fuzzball-identity   identity-keycloakx-1                                    4m           5057Mi
fuzzball-identity   identity-keycloakx-0                                    3m           4990Mi
ingress             kong-kong-7bbf7cc6d7-6wq6b                              7m           470Mi
...
fuzzball            fuzzball-admin-ui-74f84b6757-m2fwt                      1m           5Mi
fuzzball            fuzzball-admin-0                                        1m           0Mi
fuzzball            postgres-debug                                          0m           0Mi
                                                                            ________     ________
                                                                            2243m        24839Mi

To ensure that your Kubernetes nodes are not running out of resources you can check with the following:

$ kubectl top nodes
NAME     CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
node1    922m         11%      15603Mi         50%
node2    1077m        13%      18847Mi         61%
node3    667m         8%       15156Mi         49%

In this example the three node Kubernetes cluster is operating below capacity.

Kubernetes Pod Health

You can check that your fuzzball pods are all in a Running state and not frequently restarting like so:

$ kubectl get -n fuzzball pods
NAME                                         READY   STATUS      RESTARTS   AGE
fuzzball-admin-0                             1/1     Running     0          64m
fuzzball-admin-ui-74f84b6757-m2fwt           1/1     Running     0          64m
fuzzball-agent-7467f6484b-kjwv6              1/1     Running     0          61m
fuzzball-agent-7467f6484b-lkdmf              1/1     Running     0          62m
fuzzball-agent-7467f6484b-qxwfq              1/1     Running     0          62m
fuzzball-audit-68f75847bb-5blmp              1/1     Running     0          64m
...

Any states other than Running or Completed may signal problems, as are pods with fewer than expected Ready replicas.

More detailed information about a particular pod can be obtained with:

$ kubectl describe -n fuzzball pod/fuzzball-agent-7467f6484b-kjwv6
Name:             fuzzball-agent-7467f6484b-kjwv6
Namespace:        fuzzball
Priority:         0
Service Account:  fuzzball-agent
Node:             ip-10-0-123-57.us-west-2.compute.internal/10.0.123.57
Start Time:       Fri, 16 Jan 2026 19:56:49 -0500
Labels:           app.kubernetes.io/instance=fuzzball-agent
                  app.kubernetes.io/managed-by=pulumi
...

Jetstream Messages

A stateful set of 3 or more pods form a Jetstream cluster used for message passing between Fuzzball services. A buildup of messages for certain topics like the workflow-engine may indicate that Orchestrate is overloaded and processing events like state-changes of workflow stages too slowly. This can be checked with the following command:

$ kubectl exec -n fuzzball fuzzball-jetstream-0 -- /app/nats stream ls
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                   Streams                                                   │
├────────────────────────────────────────┬─────────────┬─────────────────────┬──────────┬──────┬──────────────┤
│ Name                                   │ Description │ Created             │ Messages │ Size │ Last Message │
├────────────────────────────────────────┼─────────────┼─────────────────────┼──────────┼──────┼──────────────┤
│ job-metrics-stream                     │             │ 2026-01-06 17:00:57 │ 0        │ 0 B  │ 56m2s        │
│ provisioner-aws-delete-placement-group │             │ 2026-01-06 16:49:54 │ 0        │ 0 B  │ never        │
│ provisioner-core                       │             │ 2026-01-06 16:49:54 │ 0        │ 0 B  │ 50m14s       │
│ substrate-activity-stream              │             │ 2026-01-06 17:00:57 │ 0        │ 0 B  │ 56m0s        │
│ substrate-conn-stream                  │             │ 2026-01-06 16:48:44 │ 0        │ 0 B  │ 50m12s       │
│ substrate-lease-stream                 │             │ 2026-01-06 17:00:56 │ 0        │ 0 B  │ 56m0s        │
│ substrate-resource-discovery-stream    │             │ 2026-01-06 16:48:44 │ 0        │ 0 B  │ 56m36s       │
│ substrate-resource-stream              │             │ 2026-01-06 16:48:44 │ 0        │ 0 B  │ 55m27s       │
│ substrate-service-stream               │             │ 2026-01-06 17:00:57 │ 0        │ 0 B  │ 1d5h52m43s   │
│ workflow-execution                     │             │ 2026-01-06 16:49:54 │ 0        │ 0 B  │ 56m0s        │
│ leader                                 │             │ 2026-01-17 00:26:15 │ 1        │ 39 B │ 4h14m37s     │
│ provisioner-aws-pricing-ticker         │             │ 2026-01-06 16:49:54 │ 1        │ 60 B │ 10d9h16m27s  │
╰────────────────────────────────────────┴─────────────┴─────────────────────┴──────────┴──────┴──────────────╯