Fuzzball Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

Listing Nodes

List all compute nodes in your Fuzzball cluster.

Please select either the web UI or CLI tab to see the appropriate instructions for your environment.

The Nodes page displays all compute nodes with their cluster affiliation, resource totals, and running job counts. When connected to a Federate cluster, a Cluster column identifies which underlying Orchestrate cluster each node belongs to.

Two columns show the holds on each node. A hold is what actually stops work being placed on a node, so a held node is the one to look at first:

  • Holds: the distinct effects held on the node – Cordon, Evict, or Deprovision. Empty when the node carries no holds. Sorting orders nodes by their strongest effect, so a descending sort brings the nodes being torn down to the top.
  • Held by: who placed the holds – Admin, Monitoring, or Provisioner. Sorting groups nodes held by the same sources together.

Use the filters to narrow the list:

  • Status: most often Unknown, to see the nodes the control plane has stopped hearing from.
  • Hold: Not held, Cordoned, Evicting, or Deprovisioning. A node is classified by its strongest effect, so a node with a cordon hold and an evict hold is Evicting. These are the same states the dashboard counts, so a dashboard number can be reproduced here.
  • Held by: nodes carrying at least one hold from the selected sources.

Filters are carried in the page URL, so a filtered view can be shared.

Click a node to open its details, which include each hold with its source and reason, and the node’s event log – see Node Details.

List all compute nodes in the cluster:

$ fuzzball node list
ID               | CLUSTER | HOSTNAME        | STATUS  | HOLDS        | RUNNING
172.21.0.4/7333  | fb-aws  | worker-01.local | Ready   | -            | 1
172.21.0.5/7333  | fb-aws  | worker-02.local | Ready   | cordon       | 0
172.21.0.6/7333  | fb-aws  | worker-03.local | Unknown | -            | 2
172.21.0.7/7333  | fb-aws  | worker-04.local | Ready   | cordon,evict | 1

The output shows:

  • ID: Unique identifier for the node
  • CLUSTER: Resolved cluster name
  • HOSTNAME: Network hostname
  • STATUS: What Fuzzball has observed about the node – Ready, Not Ready, Offline, or Unknown before its first report
  • HOLDS: What has been asked of the node, as the distinct effects held on it – cordon, evict, deprovision, or - when nothing is
  • STAGES: Number of workflow stages currently running on this node. A stage is any unit of work in a workflow, so this counts compute jobs, services, and the internal data-movement, image, and volume stages that support them

STATUS no longer reports a cordon. A cordoned node reads Ready with a cordon hold, because nothing is wrong with the hardware – somebody has simply asked that no new work go there. Reading STATUS alone therefore does not tell you whether work will be placed on a node; HOLDS does.

Effects rather than a count, because a count cannot answer the question: a node with one cordon hold and a node with one deprovision hold are not in the same situation, and both would read as 1.

See Node Health Monitoring for what each status means and how a silent node is detected.

For the full record – CPU type, total, allocated and available cores and memory, device summaries, and the running fuzzball-substrate version – use -o yaml or -o json. A CLUSTER column is always present; on a Federate cluster it is what tells you which underlying Orchestrate cluster a node belongs to.

Each node record carries the Substrate version the node is running, as substrate itself reports it. The default table does not show it; use fuzzball node show, or -o yaml (field substrate_version) or -o json (field substrateVersion). Nodes running a substrate version that does not report it omit the field in YAML and render it as an empty string in JSON.

Available Resources

Use --available-resources (-a) to replace the table’s job column with each node’s currently free cores, memory, and per-device counts. This costs one extra server round-trip per node. Use --resources (-r) to show the same columns filled with totals instead, which needs no extra round-trip:

$ fuzzball node list --available-resources

JSON Output

For programmatic access or scripting, use the -o json flag:

$ fuzzball node list -o json

The command returns node information in JSON, suitable for parsing with tools like jq:

$ fuzzball node list -o json | jq '.[] | select(.runningJobs > 0)'

JSON output uses camelCase field names and YAML output uses snake_case, so a filter written for one will not match the other. --available-resources is the exception: it serializes its own view rather than the API message, and that view uses snake_case in JSON too (running_jobs, substrate_version).