Fuzzball Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

Getting Node Details

Get detailed information on a specific compute node in your Fuzzball cluster.

Please select either the web UI or CLI tab to see the appropriate instructions for your environment.

From the Nodes page (see Listing Nodes), click any row to open a Node details dialog showing:

  • Node Information: ID, hostname, CPU, and memory, plus cluster when the node reports a cluster ID, and the substrate version when the node reports it (older substrate versions omit it).
  • Status: what Fuzzball has observed about the node, and when it last reported.
  • Holds: what has been asked of the node – each hold’s effect, the reason recorded on it, who placed it, when it was placed, and when it was last refreshed if that was later. The strongest effect is listed first. Shown only when the node carries any.
  • Attributed job failures: how many jobs stopped responding on this node, and when the last one did. Shown only when there are any.
  • Resource Summary: total, allocated, and available cores and memory.
  • Device Summary: per-device counts (total, requested, and available) when consumable devices are present on the node.
  • Annotations: key/value annotations set on the node (for example, nvidia.com/gpu.model), useful for identifying what to target with a workflow’s resource requirements. Shown only when the node has any.
  • Running Jobs: one row per job, labelled with the job name (or its ID when the job is unnamed), with a View workflow link that opens the job’s workflow at that stage.
  • Events: the node’s recorded history – status changes, holds placed and released, allocations added and removed, evictions, and attributed job failures – with the time and a description of each. Recent shows the latest 50 events, newest first. Full history pages through everything retained, oldest first, the same order as fuzzball node events. Use the type selector to show one kind of event. Events are retained for 90 days.

Above the node’s status, the details offer the hold actions that apply to the node:

  • Cordon: places a cordon hold, so no new work is placed on the node. Work already running is left alone. Shown until you have cordoned the node yourself.
  • Uncordon: releases the cordon holds an administrator placed. Holds placed by monitoring or the provisioner, and evict or deprovision holds, stay in place, and the success message names any holds that remain. Shown only when an administrator cordon exists.
  • Evict: places an evict hold, which takes running work off the node. A step whose policy.retry.on includes node is requeued on another node; anything else is failed. The node stays out of service until the evict hold is released. Shown until the node carries an evict hold.

Each action asks for confirmation first. Cordon and evict also take an optional reason, which is recorded on the hold. If the server leaves the node unchanged, the error message gives its reason. These actions do the same thing as fuzzball node cordon, uncordon, and evict. Only a cluster administrator can use them; other users see them disabled, with a note saying why. Releasing any other hold is done from the CLI. See Node Cordoning and Uncordoning and Evicting and Deprovisioning Nodes.

Get detailed information about a specific node by providing its node ID:

$ fuzzball node show <node-id>

Example:

$ fuzzball node show 172.21.0.6/7333
ID:        172.21.0.6/7333
Hostname:  substrate-localnode-1
CPU:       cpu/arm64
Substrate: v2.7.0
Cluster:   local-dev
Status:    Ready
Reported:  2026-09-17T14:05:00Z
Holds: 1
  7f3c1a90-2b11-4e6d-9a7f-2f0c5d8e1b42  cordon  admin  "failing disk, replacing Friday"
    since 2026-09-14T14:02:11Z
Device plugins: 2
  nvidia  ready  nvidia.com/gpu
  amd  failed  "no AMD GPU found"
Resources:
  Cores:     20 total (1 allocated, 19 available)
  Memory:    127.9 GB total (1.0 GB allocated, 126.9 GB available)
  nvidia.com/gpu:  1 total (0 requested, 1 available)

JOB ID                               | JOB NAME        | REQUESTED CORES | REQUESTED MEMORY (GB) | REQUESTED DEVICES
fab5985c-3157-5e6c-b60b-f2fd8e17fefd   data-processing   2                 4.0                     nvidia.com/gpu:1
e3211b77-117a-54ce-a2d9-420b4c673416   Hidden Job        1                 1.0                     None

The output includes:

Basic Information

  • ID: Node identifier
  • Hostname: Network hostname
  • CPU: CPU model name or architecture
  • Substrate: The fuzzball-substrate version running on the node, as reported by substrate itself. Only shown when the node reports it (older substrate versions omit it).
  • Cluster: Resolved name of the Fuzzball cluster this node belongs to, or the raw cluster_id if the name cannot be resolved. Only shown when cluster_id is present — most visible when connected to a Federate cluster where nodes span multiple underlying Orchestrate clusters.

Holds

Each hold records something that has been asked of the node: its id, the effect (cordon, evict or deprovision), who placed it, an optional reason, and when it was placed. The section is omitted when the node carries no holds.

Holds are separate from status because they answer a different question. Status reports what Fuzzball has observed about the node; a hold reports what somebody decided about it. A node reading Ready with a cordon hold is not a contradiction – the hardware is fine and no new work will be placed there.

EffectMeans
cordonPlace no new work here
evictPlace no new work here, and take off what is already running
deprovisionPlace no new work here, and tear the node down once it has no allocations

Only an administrator can place or release a hold. See Node Cordoning and Uncordoning.

Device Plugins

Each device plugin fuzzball-substrate loaded or tried to load on the node: its name, its state (ready or failed), the device kinds it serves, and the error for one that failed. A failed plugin leaves its device kind out of the node’s resources, so the scheduler passes the node over for every job requesting that kind while the node stays Ready for work that needs no device. The fix is on the node: repair the plugin and restart fuzzball-substrate.

The section is omitted when the node reports no plugins, which is also what a substrate older than v2.7.1 sends. See Device Plugins.

Health

  • Status: What Fuzzball has observed about the node – Ready, Not Ready, Offline, or Unknown before its first report. A cordon is a hold, not a status; see Holds
  • Reported: When the node last sent a health report. A node that has never reported says so rather than showing a zero timestamp.
  • Attributed job failures: How many jobs stopped responding on this node, and when the last one did. Omitted when there are none.

For what each status means and how a silent node is detected, see Node Health Monitoring.

Resource Summary

Cores and Memory

  • Total: Capacity of the node
  • Allocated: Currently in use by running jobs
  • Available: Free capacity for new jobs

Devices

Fuzzball tracks consumable devices per device type:

  • Total Count: Number of devices installed on the node
  • Requested Count: Devices currently requested by jobs
  • Available Count: Devices remaining that are not requested by jobs

Running Jobs

For each job that is currently running on the node:

  • JOB ID: Unique identifier for the job
  • JOB NAME: Job name from the workflow definition
  • REQUESTED CORES: CPU cores requested by the job
  • REQUESTED MEMORY (GB): Memory requested by the job
  • REQUESTED DEVICES: Consumable devices requested by the job

The entries shown are the workflow stages currently executing on this node. A stage is any unit of work in a workflow, so this list can include compute jobs, services, and the internal data-movement, image, and volume stages that support them. When you do not have permission to view a stage’s workflow, it appears as Hidden Job with only its ID visible.

Output Formats

By default the command prints the human-readable summary shown above. For programmatic access or scripting, request structured output with -o json or -o yaml:

$ fuzzball node show <node-id> -o json

The structured output includes additional low-level fields not shown in the default view, suitable for parsing with tools like jq or yq.