Listing Nodes
List all compute nodes in your Fuzzball cluster.
The Nodes page displays all compute nodes with their cluster affiliation, resource totals, and running job counts. When connected to a Federate cluster, a Cluster column identifies which underlying Orchestrate cluster each node belongs to.
Two columns show the holds on each node. A hold is what actually stops work being placed on a node, so a held node is the one to look at first:
- Holds: the distinct effects held on the node –
Cordon,Evict, orDeprovision. Empty when the node carries no holds. Sorting orders nodes by their strongest effect, so a descending sort brings the nodes being torn down to the top. - Held by: who placed the holds –
Admin,Monitoring, orProvisioner. Sorting groups nodes held by the same sources together.
Use the filters to narrow the list:
- Status: most often
Unknown, to see the nodes the control plane has stopped hearing from. - Hold:
Not held,Cordoned,Evicting, orDeprovisioning. A node is classified by its strongest effect, so a node with a cordon hold and an evict hold isEvicting. These are the same states the dashboard counts, so a dashboard number can be reproduced here. - Held by: nodes carrying at least one hold from the selected sources.
Filters are carried in the page URL, so a filtered view can be shared.
Click a node to open its details, which include each hold with its source and reason, and the node’s event log – see Node Details.
List all compute nodes in the cluster:
$ fuzzball node list
ID | CLUSTER | HOSTNAME | STATUS | HOLDS | RUNNING
172.21.0.4/7333 | fb-aws | worker-01.local | Ready | - | 1
172.21.0.5/7333 | fb-aws | worker-02.local | Ready | cordon | 0
172.21.0.6/7333 | fb-aws | worker-03.local | Unknown | - | 2
172.21.0.7/7333 | fb-aws | worker-04.local | Ready | cordon,evict | 1The output shows:
- ID: Unique identifier for the node
- CLUSTER: Resolved cluster name
- HOSTNAME: Network hostname
- STATUS: What Fuzzball has observed about the node –
Ready,Not Ready,Offline, orUnknownbefore its first report - HOLDS: What has been asked of the node, as the distinct effects held on
it –
cordon,evict,deprovision, or-when nothing is - STAGES: Number of workflow stages currently running on this node. A stage is any unit of work in a workflow, so this counts compute jobs, services, and the internal data-movement, image, and volume stages that support them
STATUS no longer reports a cordon. A cordoned node reads
Readywith acordonhold, because nothing is wrong with the hardware – somebody has simply asked that no new work go there. Reading STATUS alone therefore does not tell you whether work will be placed on a node; HOLDS does.Effects rather than a count, because a count cannot answer the question: a node with one
cordonhold and a node with onedeprovisionhold are not in the same situation, and both would read as1.
See Node Health Monitoring for what each status means and how a silent node is detected.
For the full record – CPU type, total, allocated and available cores and memory, device
summaries, and the running fuzzball-substrate version – use -o yaml or -o json. A
CLUSTER column is always present; on a
Federate cluster it is what tells
you which underlying Orchestrate cluster a node belongs to.
Each node record carries the
Substrate version the node is
running, as substrate itself reports it. The default table does not show it; use
fuzzball node show, or -o yaml (field substrate_version) or -o json (field
substrateVersion). Nodes running a substrate version that does not report it omit the field
in YAML and render it as an empty string in JSON.
Use --available-resources (-a) to replace the table’s job column with each node’s currently
free cores, memory, and per-device counts. This costs one extra server round-trip per node. Use
--resources (-r) to show the same columns filled with totals instead, which needs no extra
round-trip:
$ fuzzball node list --available-resourcesFor programmatic access or scripting, use the -o json flag:
$ fuzzball node list -o jsonThe command returns node information in JSON, suitable for parsing with tools like jq:
$ fuzzball node list -o json | jq '.[] | select(.runningJobs > 0)'JSON output uses camelCase field names and YAML output uses snake_case, so a filter written
for one will not match the other. --available-resources is the exception: it serializes its
own view rather than the API message, and that view uses snake_case in JSON too
(running_jobs, substrate_version).