Fuzzball Documentation
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

Node Health Monitoring

Fuzzball publishes facts about its compute nodes. It does not judge them.

Each node sends a periodic heartbeat, which is how the control plane tells a live node from a silent one, and exports hardware counters for a monitoring stack to scrape. Which counters matter, how long a fault must persist, and what severity it warrants are decided by Prometheus alerting rules – reviewable in git, tunable per site – not by thresholds inside the product.

When the monitoring stack concludes a node should stop taking work, it says so by placing a hold. That is the hook, and it is the only thing that takes a node out of service.

This split is new. Earlier releases scored each node 0-100, derived a health state from that score, and shipped an opt-in policy engine that could cordon, drain or replace a node on its own. The score, the health state, the condition vocabulary, the policy engine and the Fuzzball-native notification webhook are all retired – see What Was Retired.

Prerequisites

Reading orchestrate service logs requires access to the Fuzzball control plane namespace. Changing the heartbeat miss threshold requires edit access to the FuzzballOrchestrate resource in the fuzzball-system namespace.

What Fuzzball Observes

Four things, and only these:

SignalWhere it appears
Node status – unknown, not ready, ready, offlinefuzzball node list, fuzzball node show, the API
When the node last reportedfuzzball node show, fuzzball_node_last_health_report_timestamp_seconds
Job failures attributed to the nodefuzzball_node_attributed_job_failures
The state of each device plugin on the nodefuzzball node show, fuzzball_node_device_plugin

Every change to a node’s status is recorded as a STATUS_CHANGED event carrying both the previous and the new status, so a node’s history can be rebuilt without parsing the event’s message. Read them with fuzzball node events NODE --type status_changed.

Status reports what Fuzzball has observed about a node, and nothing else:

StatusMeans
unknownFuzzball cannot see the node: no report yet, or reports have gone stale
not readyThe node is reporting, but no provisioner definition has claimed it
readyReporting, claimed, and able to accept work
offlineThe node is gone. Work placed there is released, and the record is removed after 30 minutes – unless it still carries an administrator or monitoring hold, in which case it stays, held, until it is removed or comes back. A monitoring hold usually goes when its alert resolves, which happens once the node stops being scraped, unless the rule is marked fb_release: manual. See Retiring a Node

A cordoned node is not a status. It reads ready and carries a cordon hold, because “nothing is wrong with this machine” and “nobody wants new work on it” are different statements. See Node Cordoning and Uncordoning.

The attributed-failure counter is the one node signal Fuzzball can see that a monitoring stack cannot: it knows which node a job was running on when it stopped responding. It is exported raw, for rules to alert on. Only a job still meant to be running counts: one that was stopped with fuzzball workflow stop, had already finished or reported its exit, or was being moved off the node is going away for its own reasons, and is not counted against the node.

Substrate Version

The orchestrate extension needs a minimum version of Fuzzball Substrate on the node, currently v2.7.0, to pull and build container images. Each node’s extension reports both the version substrate states and the minimum that extension requires. When a node runs an older substrate than its extension requires, or runs an extension too old to report a minimum, the extension keeps running and reporting, and the control plane places a cordon hold on it:

Hold idSourceEffectPlaced when
substrate-versionprovisionercordonThe substrate on the node states a version older than the extension requires, or none at all, or the extension reports no minimum

For an unsupported substrate, the hold’s reason names both versions. While it is held the node refuses image pulls, so workflow stages that need an image fail on it with the same message, and the scheduler places no new work there. Work already running is left alone – the hold cordons, it does not evict: draining would move work that is running fine, and replacing a provisioned node would bring back the same substrate.

An extension that reports no minimum predates v4.4, which converts images to EROFS. It reads every image as a SIF, while current nodes convert images into the image store the nodes share, and an image is pulled separately from the job that uses it. A job placed on such a node could fail on an image another node converted, so once Orchestrate is upgraded the node takes no new work until fuzzball-substrate-orchestrate is updated to match. Dynamically provisioned nodes finish the work they are running and are torn down when idle. Static nodes stay held until their packages are updated, so update them together with Orchestrate, as Updating Fuzzball describes.

This is the one hold Fuzzball places on its own judgement rather than an operator’s or a monitoring rule’s, and it is the exception for a specific reason: whether the extension can serve a node’s image pulls is Fuzzball’s own knowledge, not something a rule reading metrics could determine.

The fix is to upgrade fuzzball-substrate on the node, or fuzzball-substrate-orchestrate when the reason names the extension. The hold is released on the node’s next health report – within 30 seconds by default – and it takes work again. Releasing it by hand with fuzzball node hold release succeeds but achieves nothing: the next report places it again, because the condition it describes is still true.

Device Plugins

Fuzzball Substrate serves each device kind on a node, such as a GPU, through a device plugin. When a plugin fails to initialize, the node reports every other resource normally and the device kind is simply missing, so the scheduler passes the node over for every job that requests it while the node stays ready and keeps taking work that needs no device. Nothing else about the node reads wrong, which is why this is one of the four things Fuzzball reports.

Substrate records every plugin it loaded or tried to load, with its state and the error for one that failed, and the node’s report carries the list. From it, Fuzzball publishes:

  • fuzzball node show prints a Device plugins: block naming each plugin, its state and its error.
  • fuzzball_node_device_plugin{node, node_id, plugin, state} is 1 for the state the plugin is in and 0 for the other, so a repaired plugin reads as a transition. The bundled rule FuzzballNodeDevicePluginFailed fires on it.
  • A job that too few nodes can hold, where the nodes that report a failed plugin and lack nothing for the job but a device kind they have none of would make up the difference, gets a scheduling_blocked event whose reason is device_plugin_failed, and fuzzball workflow why says, for example, 1 without the requested device type and with a failed device plugin (nvidia). A job the pool could not hold even when empty is refused at submission instead, and for such nodes that message carries the same note: 1 node(s) report a failed device plugin (nvidia). Substrate reports which plugin failed but not yet which device kind it serves, so both messages state the two facts side by side.

The node is not held: a cordon would take healthy CPU capacity out of service because its GPU plugin is broken, and draining or replacing the node would not repair the plugin. The fix is on the node – repair the plugin, for example by installing the missing driver, and restart fuzzball-substrate – after which the next report lists the plugin ready, the device kind reappears, and placement resumes.

Reporting plugin state requires Fuzzball Substrate v2.7.1 or later on the node. An older substrate sends no plugin list, so fuzzball node show omits the block and no series is published for that node.

Heartbeat and Liveness

Each compute node reports every 30 seconds by default. When a node misses the configured number of consecutive reports, three by default, the control plane moves it to the unknown status, records a STATUS_CHANGED event, and logs a warning naming the node and the time of its last report.

Heartbeat tracking does not depend on detecting a network disconnect. A node whose hardware has hung, or whose substrate extension has stopped while the host is still reachable, stops sending heartbeats and goes unknown on the same timer as any other silent node.

A node that starts reporting again returns to ready when a provisioner definition has already claimed it, and to not ready when none has – the status it would have had if it had never gone silent. The health report itself does not know whether the node is claimed; that is read from the node’s last substrate resource report.

Only a health report brings a node out of unknown. A substrate resource report for a node whose health reports have stopped does not return it to service, because the two come from different places: substrate publishes the resource, and the orchestrate extension reports the health. A node whose extension has stopped while substrate keeps running would otherwise be handed work again on its next capacity change, and taken back out on the next liveness sweep.

A node that goes offline and re-registers is a different case: its last-report time is cleared with the status, so it comes back as a node that has never reported and is exempt from the sweep until it reports again. Without that, a rebooted node would be condemned on a timestamp from before it left.

A node whose health reporting is switched off after it has reported at least once stays unknown, and out of the placement set, until reporting resumes. Switch health reporting off before the node first reports, or leave it on: a node that has never reported is exempt from the liveness sweep, but one that stops is treated as unreachable.

On the API, that status is NODE_STATUS_UNKNOWN. It is a value of its own rather than NODE_STATUS_UNSPECIFIED, so a client can tell “Fuzzball cannot see this node” from “the server did not set this field”.

Heartbeat Miss Threshold

Set the number of consecutive reports a node can miss in the cluster’s FuzzballOrchestrate custom resource:

apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  name: fuzzball-orchestrate
spec:
  nodeHealth:
    heartbeatMissThreshold: 3

See Node Health Configuration in the CRD reference for the full field description.

The threshold is a count, not a duration, and each node is measured against its own reporting interval. Nodes that report at different intervals therefore need no per-node tuning. With the defaults, a node goes unknown roughly 90 seconds after its last report.

A lower threshold detects a failed node sooner, at the cost of more false positives on a congested or high-latency network. A higher threshold reduces false positives but delays detection.

Node Reporting Interval

The reporting cadence is set per node, in the substrate orchestrate extension configuration rather than on the FuzzballOrchestrate resource. This applies to nodes whose extension configuration you manage directly; nodes provisioned by the Fuzzball operator use the defaults below and do not expose these settings.

binary: fuzzball-substrate-orchestrate
enabled: true
config:
  health:
    disabled: false
    report-interval-seconds: 30
ParameterTypeDefaultDescription
disabledbooleanfalseTurn off health reporting on this node entirely
report-interval-secondsinteger30Seconds between reports. Values below 5 fall back to the default
Turning health reporting off also removes the node from missed-heartbeat detection. A node that has never sent a health report never goes unknown for missing one, because it has missed no heartbeat it promised – the same applies to a node running an extension too old to report health. Nodes that reported and then had reporting disabled are still detected.

The control plane puts a floor under the window it derives from these values, and a ceiling on the interval it derives it from. It waits at least 15 seconds after a node’s last report before recording the node as silent, however short the interval. It also treats any reported interval longer than 24 hours as 24 hours, so a node configured to report very rarely is still recorded as silent after the threshold multiplied by 24 hours – three days at the default threshold.

Image Provider Settings

The same extension configuration carries the settings of the node’s image provider, which converts container images to EROFS and serves them to jobs:

binary: fuzzball-substrate-orchestrate
enabled: true
config:
  image-provider:
    mount-mode: auto
    prune-interval: 6h
ParameterTypeDefaultDescription
mount-modestringautoHow converted images are mounted for containers: fuse serves them through a FUSE daemon, loop through loop devices, and auto picks what the node’s kernel serves best. auto picks loop only where the kernel can also map file ownership on it (Linux 5.19 or newer), and serves any container whose mount the kernel still refuses to map through FUSE, so containers see the same file ownership on every node. With loop set explicitly, such a container fails to start instead
prune-intervalduration6hHow often orphaned layers are removed from the node’s image store, as a Go duration such as 6h or 90m. Nodes sharing a store take a cluster-wide lock, so one prune runs at a time; the store records when it last ran and a restarting node prunes at once if that is older than the interval

Viewing Node State

Status and holds appear in fuzzball node list:

fuzzball node list
ID      | CLUSTER | HOSTNAME  | STATUS  | HOLDS  | RUNNING
node-01 | default | compute-1 | Ready   | -      | 3
node-02 | default | compute-2 | Ready   | cordon | 1
node-03 | default | compute-3 | Unknown | -      | 0

node-03 reading Unknown means Fuzzball cannot see it – it has not reported yet, or its reports have stopped. That is not evidence its hardware is faulty.

node-02 reading Ready with a cordon hold is not a contradiction: the hardware is fine and somebody has asked that no new work be placed there.

fuzzball node show adds when the node last reported, the failures attributed to it, and each hold with its source and reason:

fuzzball node show node-02
ID:        10.0.0.42/9100
Hostname:  compute-2
CPU:       x86_64
Status:    Ready
Reported:  2026-09-17T14:05:00Z
Attributed job failures: 2 (last 2026-09-17T11:20:14Z)
Holds: 1
  7f3c1a90-2b11-4e6d-9a7f-2f0c5d8e1b42  cordon  monitoring  "4 correctable ECC errors in 24h"
    since 2026-09-17T11:22:03Z

Metrics

Node metrics come from two places, joined on a shared node label: the cluster endpoint, for facts only the control plane knows, and the node exporter, for signals only the node can see. Each node also runs the Prometheus node_exporter, bundled with the compute-node package, for standard host metrics such as CPU, memory, disk and network.

node is the node’s host name, falling back to its id when it has not reported one. Both endpoints label their series with it, so a rule can join across them with on(node).

Compute nodes are not Kubernetes objects, so no Kubernetes service discovery can find them, and on a cluster that provisions them per workflow a static target list is stale before you finish writing it. Fuzzball therefore serves its own service discovery, and can relay scrapes to nodes a monitoring stack cannot reach – see Finding Nodes to Scrape.

From the cluster endpoint

MetricLabelsMeaning
fuzzball_node_infonode, node_id, definition, cluster, dynamic, region, zone, segmentConstant 1 carrying the node’s identity
fuzzball_node_statusnode, node_id, status1 for the node’s current status, 0 for the rest. status is one of unknown, not_ready, ready, offline – underscored, unlike the not ready the CLI prints
fuzzball_node_holdnode, node_id, effect, sourceHolds on the node, counted by effect and source
fuzzball_node_running_allocationsnode, node_idAllocations currently running there
fuzzball_node_attributed_job_failuresnode, node_idCumulative jobs that stopped responding on the node
fuzzball_node_last_health_report_timestamp_secondsnode, node_idUnix timestamp of the node’s last health report
fuzzball_node_device_pluginnode, node_id, plugin, state1 for the state a device plugin is in, 0 for the other. state is ready or failed. Absent for nodes running a substrate older than v2.7.1
fuzzball_node_monitoring_holds_capped_totaldefinitionCounter of monitoring holds refused because the provisioner definition was at its ceiling. Each one is a node an alert called faulty that is still in service. definition is empty for nodes no definition owns. What FuzzballNodeHoldCeilingReached fires on

Every series but the last carries both node and node_id; the capped-holds counter is about a pool, not a node. node is the host name and is what joins to an exporter; node_id is the node’s address and is what makes the series unique. A host name is not unique – a substrate that moves to a new address leaves its old record behind until a ghost sweep reaps it, and one host can run more than one substrate – so aggregate with max by (node) when you want one answer per machine, and keep node_id when you need to tell two records apart.

fuzzball_node_info carries only slow-changing identity. Anything that moves while a node is alive gets its own series with a value, so a transition changes a number rather than minting a new series and leaving the old one stranded at its last reading. That is why status and holds are separate series rather than more labels on the info one.

Join exporter series to Fuzzball’s view through node. Which side to collapse depends on what the alert should name.

To name each substrate – what a rule that places holds wants, so the receiver holds exactly the right records – keep node_id on the exporter series (the scrape job promotes it) and join on both labels:

(
  max by (node, node_id) (increase(fuzzball_node_memory_correctable_errors_total[24h]))
  or
  sum by (node, node_id) (increase(node_edac_correctable_errors_total[24h]))
)
  * on(node, node_id) group_left(definition)
    max by (node, node_id, definition) (fuzzball_node_info)
  > 1000

That is the shipped FuzzballNodeMemoryErrors rule. It raises one alert per substrate on the host, each carrying node_id, because a host-wide fault affects every substrate on it. Aggregate by (node, node_id), never by node alone: a host running two substrates is scraped once per substrate, so its counters arrive twice, and a sum by host would double them.

The count comes from Fuzzball’s own exporter, which reads the same EDAC counters as node_exporter and publishes the host’s total. or falls back to node_exporter’s per-controller counters, summed, only for a substrate whose Fuzzball exporter is turned off; never add the two, or every error counts twice. max by (node, node_id, definition) collapses the copies both leaders publish during a failover, and keeps their differing pod labels out of the alert, where a leader change would otherwise change the alert’s identity.

To attach a label to a per-host series instead, use group_left with topk by (node) (1, fuzzball_node_info) on the right. group_left needs exactly one series per node there, and there can be more – both leaders during a failover, or two records sharing a host name – so without collapsing the rule fails outright with many-to-many matching not allowed. topk picks one series per node whatever its labels disagree on; max by (node, definition) only collapses duplicates that agree on definition. Do not carry node_id this way: fuzzball_node_info is always 1, so topk chooses between ties with no stable order, and the node_id it carries can change between evaluations.

fuzzball_node_status and fuzzball_node_hold publish zeroes for the values that do not currently apply, so a rule can say == 0 rather than reasoning about an absent series, and a release or a transition is visible to a range query.

fuzzball_node_last_health_report_timestamp_seconds is what a silent-node alert reads. A timestamp rather than an age, so the value stays true without being republished every scrape; compute the age in the rule:

time() - max by (node, node_id) (fuzzball_node_last_health_report_timestamp_seconds) > 600

It is absent until a node reports once, so pair a staleness rule with an absent() check rather than treating a missing series as a node that never reported. Publishing zero instead would read as “last reported in 1970” and alert on every newly registered node.

Only the leader replica publishes these, and it drops them when it stops being leader, so aggregate with max rather than sum: during a failover both the old and new leader can be scraped, and summing would double count.

Group on (node, node_id) for a staleness rule, not on node alone. max takes the freshest timestamp in the group, so grouping on the host name only would let a live substrate cover for a dead one on a host running two – the alert would never fire for the dead one. Grouping on node_id still dedupes two leaders publishing the same node, because both publish the same id.

That costs nothing in false alerts from ghost records. A substrate that moves address leaves its old record behind, but that record gets a delete event, which marks it offline and clears its last-report timestamp – so its series is dropped rather than left to go stale. Only the liveness sweep leaves a stale timestamp, and that is the case a staleness rule is for.

fuzzball_scheduler_node_attributed_job_failures is still published, with the same value and its original node_id label, so existing dashboards and rules keep working. It is deprecated in favour of fuzzball_node_attributed_job_failures.

From the node exporter

Each compute node serves its own metrics on port 9101. Deliberately narrow: it publishes only what node_exporter – which Fuzzball runs beside it on port 9100 – and DCGM Exporter cannot see for themselves.

MetricLabelsMeaning
fuzzball_node_machine_check_errors_totalnodeMachine check records seen since the extension started
fuzzball_node_machine_check_fatal_errors_totalnodeFatal machine check records
fuzzball_node_gpu_xid_errors_totalnode, gpu, uuid, pci_bus_idXid faults attributed to a GPU, per device
fuzzball_node_gpu_xid_errors_node_totalnodeXid faults seen anywhere on the node, including any that matched no device
fuzzball_node_gpu_hardware_xidnode, gpu, uuid, pci_bus_id1 when an Xid indicating a faulty card has been seen on this device
fuzzball_node_gpu_hardware_xid_nodenode1 when such an Xid has been seen anywhere on the node
fuzzball_node_disk_smart_failingnode, device1 when SMART reports the device failing. Absent when SMART could not be read
fuzzball_node_disk_smart_statusnode, deviceWhether SMART could be read for this device: 1 ok, 0 not available, -1 the probe failed
fuzzball_node_disk_io_errors_totalnode, deviceCounter of block-layer I/O errors the kernel logged against the disk since the extension started, mounted or not
fuzzball_node_disk_scsi_ioerr_cntnode, deviceThe raw SCSI ioerr_cnt for the device. Not a fault signal on its own
fuzzball_node_disk_read_onlynode, device1 when the device has been marked read-only while it backs a mounted filesystem or swap
fuzzball_node_memory_correctable_errors_totalnodeCorrectable ECC errors from the EDAC controllers
fuzzball_node_memory_uncorrectable_errors_totalnodeUncorrectable ECC errors
fuzzball_node_thermal_celsiusnode, zone, zone_idCurrent temperature of a thermal zone
fuzzball_node_thermal_critical_celsiusnode, zone, zone_idThe zone’s critical trip point, absent when the hardware declares none
fuzzball_node_thermal_throttlednode1 when a zone is at or above its trip point
fuzzball_node_substrate_readynode1 when the local substrate runtime is ready
fuzzball_node_run_queue_15snodeRunnable tasks, exponentially averaged over 15 seconds from one-second samples of /proc/loadavg (LSF r15s). Excludes the sampler itself and tasks in uninterruptible sleep, so an idle node reads 0
fuzzball_node_login_sessionsnodeUser sessions in utmp (LSF ls): console and terminal logins, including ssh with a terminal. Batch jobs and ssh commands without a terminal do not count
fuzzball_node_idle_secondsnodeSeconds since the last input on any logged-in terminal (LSF it). With no session that has a terminal (nobody logged in, or only graphical logins), since the last terminal input the exporter observed, or since boot if it has observed none
fuzzball_node_signal_statusnode, signalWhether a collector produced a usable reading. signal is memory, machine_check, disk, thermal, gpu, agent, load or sessions
fuzzball_node_collector_failuresnode, signalHow many of a signal’s collectors failed while others succeeded
fuzzball_node_signals_age_secondsnodeSeconds since these readings were collected
fuzzball_node_exporter_upnode1 when the exporter answered the scrape

The node-wide GPU series are separate names rather than the per-device ones with empty labels, so sum by (node) over the per-device series cannot double count by adding a total to its own components. Use fuzzball_node_gpu_xid_errors_node_total for a node total: a fault whose PCI address matched no device the collector could see still happened, so it can exceed the sum of the per-device series.

zone is the kernel’s name for a thermal zone and is not unique – a two-socket host has two zones both named x86_pkg_temp – so zone_id carries the sysfs directory (thermal_zone0) to tell them apart. Aggregate on zone_id where you need one series per zone.

Machine checks and GPU Xid faults are the reason this exporter exists: both are written only to the kernel ring buffer, which no other exporter reads. SMART needs smartctl, and whether the local substrate runtime is up is Fuzzball’s own knowledge.

fuzzball_node_signal_status distinguishes three things a rule should not conflate:

ValueMeans
1The collector ran and produced a reading
0The hardware or tooling is not present on this machine – expected, not a fault
-1The collector ran and failed – the node is not being fully checked

A signal that could not be read is never reported as passing. Alert on fuzzball_node_signal_status == -1 to catch a node whose telemetry has silently stopped being collected.

Two cases the signal-wide status cannot express, because a signal reports 1 as soon as any of its collectors succeeded:

  • A disk whose SMART probe failed. fuzzball_node_disk_smart_failing is absent for that device rather than 0, and fuzzball_node_disk_smart_status says why.
  • A mixed-vendor GPU host where one vendor’s probe failed. The cards that did report keep their series, and fuzzball_node_collector_failures is non-zero.

Alert on fuzzball_node_collector_failures > 0 as well as on the signal status, or a partly unchecked node reads as fully checked.

The run queue, login session and idle time series are absent unless their signal (load or sessions) reads 1 – before the first sample, or with /proc/loadavg or utmp unreadable – rather than publishing a 0 that would read as an idle, empty node.

Most series are the readings the node’s reporting loop last collected, so they are at most one report interval old – 30 seconds by default. Collecting per scrape would fork smartctl for every disk and nvidia-smi on the request path of an unauthenticated endpoint, worst on exactly the failing hardware that makes those tools hang. fuzzball_node_signals_age_seconds carries the age: a climbing age with unchanging values means collection has stalled, which would otherwise look like healthy, unchanging hardware. The run queue, login sessions, idle time and disk I/O error counts are the exception: they are kept in memory or cheap to read, so each scrape reads them live. A scrape does not count as a heartbeat.

Turn the endpoints off, or move them, in the substrate orchestrate extension configuration. These settings govern both Fuzzball’s exporter and the bundled node_exporter:

binary: fuzzball-substrate-orchestrate
enabled: true
config:
  health:
    metrics-disabled: false
    metrics-port: 9101
    metrics-address: ""
    node-exporter-disabled: false
ParameterTypeDefaultDescription
metrics-disabledbooleanfalseTurn off both Prometheus endpoints – Fuzzball’s exporter and the bundled node_exporter – leaving the heartbeat running
metrics-portinteger9101Port Fuzzball’s exporter listens on. node_exporter always uses 9100
metrics-addressstringall interfacesInterface both endpoints bind. A value that cannot be read as a host, such as [] or a lone bracket, keeps both endpoints closed and is logged as an error
node-exporter-disabledbooleanfalseDo not run the bundled node_exporter, leaving Fuzzball’s exporter running. Set it on a node that runs its own node_exporter

On upgrade, both endpoints are on by default, unauthenticated, and bound to every interface – port 9101 for Fuzzball’s exporter and port 9100 for node_exporter. Both are new to a node upgrading from a release before they shipped. Fuzzball’s serves the node’s host name, GPU UUIDs, PCI bus IDs and disk device names; node_exporter serves the host’s full inventory of CPUs, memory, filesystems, mount points and network interfaces. That default exists because a monitoring stack scrapes them from off the node, so binding loopback only would make them useless for the purpose they have – but it is worth a decision rather than a default.

Three knobs:

  • metrics-address narrows both binds. Set it to one interface (metrics-address: "10.0.0.42") on a node reachable from beyond the cluster network, or to 127.0.0.1 to serve only local scrapes.
  • node-exporter-disabled: true keeps the bundled node_exporter off port 9100.
  • metrics-disabled: true closes both of Fuzzball’s endpoints.

These settings govern only what Fuzzball runs. A node_exporter the site runs itself on 9100 is unaffected by either one, so closing that port is the site’s own configuration.

With proxy-mode service discovery, metrics-address: "127.0.0.1" is the tightest useful setting: the scrape relay runs on the node and reaches both exporters over loopback, so they stay reachable through the authenticated API and nowhere else. Narrowing it to a single non-loopback interface instead puts both out of the relay’s reach – proxied scrapes of fuzzball-node and node then fail with a connection error naming the port, while dcgm keeps working because DCGM Exporter binds every interface by convention. Use direct mode if you need that arrangement, and set metrics-address to the address the node registered with (the address in its node ID): that is the address direct mode connects to.

Also check for a port clash. 9101 is the HAProxy exporter’s conventional port, and a clash is not fatal here: whichever service starts second logs a warning and loses the port. Grep the extension’s log for node metrics endpoint unavailable – it names the address it could not bind.

metrics-disabled is separate from disabled because they answer different questions. Turning off collection makes the node invisible to Fuzzball; turning off the endpoints only stops a monitoring stack scraping them, which is what a site with no Prometheus, or one that objects to the open ports, actually wants. disabled leaves node_exporter running: it serves the host’s metrics, not Fuzzball’s telemetry.

Disk I/O Errors

fuzzball_node_disk_io_errors_total counts the failed requests the kernel logs for a disk, read from the same ring buffer as machine checks. A record is counted when it reports one of the following:

  • A failed request, such as I/O error, dev sda, sector ... or critical medium error, dev sda, sector .... Every kernel from 4.18 (RHEL 8) to 6.x prints this record, with or without a print_req_error: or blk_update_request: prefix.
  • A failed filesystem block read or write, such as Buffer I/O error on dev sda1, .... A partition’s records count against its disk.

The count is a floor rather than a complete history. It starts at zero when the extension starts, the kernel rate-limits these lines, and one failed request can log more than one record. Treat any increase as the signal:

increase(fuzzball_node_disk_io_errors_total[15m]) > 0

Errors are counted whether or not the disk is mounted, because the kernel logs them only for requests that actually failed. A disk that fails badly enough to drop off the bus keeps its series. Loop, optical, floppy, device-mapper and md devices are not counted: for device-mapper and md, the physical disk underneath logs its own record for the same failure, and a virtual machine’s empty floppy logs errors whenever it is probed.

fuzzball_node_signal_status{signal="machine_check"} reports whether the ring buffer is being read. While it is not 1, a disk reading 0 has not been checked.

fuzzball_node_disk_read_only reports a read-only device only while it backs a mounted filesystem or swap. Devices that are read-only by design are excluded, such as config drives, images and virtual CD-ROMs. A device mounted through an alias is recognised by its device number, such as /dev/disk/by-uuid/..., /dev/disk/by-id/... or the /dev/root some images report. A disk under LVM, LUKS or software RAID is mounted through the mapping, not by itself, so it reads as unmounted. A btrfs filesystem mounted through an alias also reads as unmounted, because the mount table reports an anonymous device number for it. A btrfs filesystem mounted by its kernel device name, such as /dev/sdb1, still matches.

From node_exporter

The fuzzball-substrate-orchestrate package, and the orchestrate substrate container images, ship the Prometheus node_exporter – version v1.12.1, pinned and checksum-verified at build time, so nodes never download it. It is installed at /usr/libexec/fuzzball-substrate/node_exporter, with its license and notice under /usr/share/doc/fuzzball-substrate-orchestrate/node_exporter/.

The extension runs it as root on port 9100, on the interface metrics-address names, restarts it with backoff if it exits, and stops it when the extension stops. This is what the node exporter in the discovery inventory and the bundled stack’s node scrape job read, so on an upgraded node that target now answers rather than reading down.

It does not run where it cannot, or should not:

  • Port 9100 already taken. The bundled one stands by rather than fighting for the port: the extension logs port in use; standing by -- the bundled node_exporter starts if it frees, checks again every two minutes, and starts it once the port is free. Whatever holds the port is what gets scraped, so check what it is. An address that cannot be bound at all is handled the same way, logged as a warning.
  • Binary absent. A node on an older package, or an image built without it, runs none, and its node target reads down as before.
  • Switched off. node-exporter-disabled or metrics-disabled – see the configuration above.
A node that runs its own node_exporter should set node-exporter-disabled: true. Nothing orders the substrate service against a site’s node_exporter service at boot, so whichever starts first holds port 9100. If the bundled one wins, the site’s fails to bind – and the bundled one runs with the collector choices below, without the site’s textfile metrics. Standing by only helps when the site’s exporter is already running.

It runs with node_exporter’s default collectors, except where a Fuzzball compute node would make them churn:

  • The textfile collector is off.
  • Mounts beneath /var/lib/fuzzball/ and /run/fuzzball-substrate/ – the per-job container, image and volume mounts – and beneath any erofsmount-<id> directory, where Fuzzball’s image server mounts, are excluded. A filesystem mounted at /var/lib/fuzzball is still reported.
  • The lo and veth* interfaces are excluded. The fb0 bridge, which carries all container traffic in aggregate, is kept.

node_exporter itself logs only warnings and errors. The extension relays each line to its own log at the same level – error for an error, warning otherwise – so a site filtering the extension’s log at warning still sees them.

Node Resource Metrics

How busy a node is, as opposed to whether it is healthy. Each item an LSF lsload user expects maps to a series and a panel on the Fuzzball Node Resources dashboard:

ItemSeriesSourcePanel
Fuzzball statusfuzzball_node_status, fuzzball_node_holdCluster endpointState, Fuzzball status, Holds
CPU utilization (ut)node_cpu_seconds_totalnode_exporterCPU utilization
15-second run queue (r15s)fuzzball_node_run_queue_15sFuzzball exporterRun queue (15 s)
1- and 15-minute load average (r1m, r15m)node_load1, node_load5, node_load15node_exporterLoad average
Page rate (pg)node_vmstat_pgpgin, node_vmstat_pgpgout, node_vmstat_pswpin, node_vmstat_pswpout, node_vmstat_pgmajfaultnode_exporterPaging
Free memory (mem)node_memory_MemAvailable_bytes, node_memory_MemFree_bytesnode_exporterMemory
Filesystem utilization (tmp, for every mount)node_filesystem_avail_bytes, node_filesystem_size_bytesnode_exporterFilesystem utilization
Swap utilization (swp)node_memory_SwapFree_bytes, node_memory_SwapTotal_bytesnode_exporterSwap utilization
Idle time (it)fuzzball_node_idle_secondsFuzzball exporterIdle time
Login sessions (ls)fuzzball_node_login_sessionsFuzzball exporterLogin sessions
Network I/O per interfacenode_network_receive_bytes_total, node_network_transmit_bytes_totalnode_exporterNetwork I/O
Number of GPUsfuzzball_node_gpu_hardware_xid, one series per GPUFuzzball exporterGPUs

The State panel folds status and holds into one LSF-style host state. The most severe applies:

StateWhenLSF
DownStatus offlineunavail
UnknownStatus unknownunreach
Not readyStatus not readyclosed
LockedAny cordon, evict or deprovision holdlockU
OKNone of the aboveok

A held node that is also offline reads Down, because that is what needs acting on first; the Holds panel still lists the hold.

Two of these are not what their names might suggest:

  • The 15-second figure is a run-queue average, not a Linux load average. It follows LSF’s r15s: runnable tasks only, so a node blocked on a dead NFS server reads idle here and busy in node_load1. Compare it against the core count, which the panel draws beside it.
  • Idle time is terminal input. With no session that has a terminal – a headless node nobody logs into, or only graphical logins – it reads as time since the last terminal input the exporter observed, or since boot if it has observed none, as after an extension restart. It answers “is a person at this machine”, not “is the machine doing work”.

Finding Nodes to Scrape

A compute node is not a Kubernetes object, so no Kubernetes service discovery can see it. On a cluster that provisions nodes for individual workflows, a static target list is also useless – a node may exist for two minutes.

Fuzzball serves the target list itself, in Prometheus http_sd_config form:

GET /v4/nodes/-/sd_config?exporter=node

exporter is required, and names one exporter from the cluster’s inventory. Run one Prometheus job per exporter: a single job covering all three would stamp one job label on series from three different exporters, and anything selecting by job would then be selecting by nothing.

The response is a Prometheus target list:

[
  {
    "targets": ["orchestrate.example.com"],
    "labels": {
      "instance": "cn-042:9100",
      "node": "cn-042",
      "__param_node": "cn-042",
      "__param_exporter": "node",
      "__meta_fuzzball_node_id": "10.0.0.42/2375",
      "__meta_fuzzball_definition": "gpu-a100",
      "__meta_fuzzball_cluster": "lab1",
      "__meta_fuzzball_dynamic": "true",
      "__meta_fuzzball_exporter": "node",
      "__meta_fuzzball_sd_mode": "proxy"
    }
  }
]

The __meta_fuzzball_* labels mirror fuzzball_node_info field for field, so a relabel rule can promote any of them and the result joins Fuzzball’s own series without a translation table. They are discovery labels: Prometheus drops them from the final series unless a rule keeps one, so none of them inflate cardinality by default.

Nodes the control plane has written off as offline are omitted. A failed scrape is a problem worth alerting on, and a node already known to be gone is not news – fuzzball_node_status reports that with no scrape involved.

Proxy and Direct Mode

Only your site knows whether Prometheus can open a connection to the compute network at all, so the addressing is configuration:

nodeMetrics:
  serviceDiscovery:
    mode: proxy          # proxy (default) | direct
  exporters:
    - name: node
      port: 9100
    - name: dcgm
      port: 9400
      requires: nvidia-gpu
    - name: fuzzball-node
      port: 9101

In proxy mode every target is the Fuzzball API, and the node and exporter travel as scrape parameters. Prometheus turns the __param_* labels above into a query string, and the API relays the scrape to the node over the connection the node already holds – so nothing needs a route into the compute network, and no node port needs to be reachable from off the node. Pair it with metrics-address: "127.0.0.1" to keep the ports Fuzzball opens, 9100 and 9101, off the network entirely.

In direct mode the target is the node’s own address – the IP address it registered with, as shown in its node ID – and Prometheus connects to it. Compute host names do not need to resolve from where Prometheus runs. Lower latency and no fan-in through the control plane, but it requires a route to that address.

instance is the node’s host name and exporter port in both modes. That is deliberate: instance is the identity of a target in every dashboard and recording rule you have, and pointing it at the proxy would make every series in the fleet share one instance – then changing the mode back would relabel all of them again.

Configuring exporters replaces the defaults rather than adding to them, so a site that does not run DCGM can say so by listing only what it does run. requires: nvidia-gpu omits nodes with no NVIDIA GPU from that exporter’s target list entirely, which is what lets a rule read an absent dcgm target as absence rather than as failure.

For the fuzzball-node exporter the node’s reported port wins over the configured one. It is the only port Fuzzball actually knows: the extension may have been configured elsewhere, failed to bind the port it was given – 9101 is also the HAProxy exporter’s conventional port – or been switched off. A node that has reported no port is omitted, because publishing the configured number would produce a target that fails every scrape and reads as broken hardware rather than as a disabled endpoint.

ParameterTypeDefaultDescription
serviceDiscovery.modestringproxyproxy to scrape through the API, direct to scrape nodes
exporters[].namestring–Exporter name, used as the ?exporter= value
exporters[].portinteger–Port the exporter listens on
exporters[].requiresstring–nvidia-gpu to restrict the exporter to GPU nodes

The Scrape Proxy

In proxy mode Prometheus scrapes:

GET /v4/metrics/node?node=cn-042&exporter=node

The API resolves exporter to a port from the inventory above – never from the request, so the endpoint cannot be used to reach an arbitrary port – and asks the node for that exporter’s output over its existing connection. The node scrapes its own exporter over loopback and compresses the response before sending it back, which is what keeps a large node_exporter payload inside the message limit on that hop.

node is the node’s host name, which is what the discovery response puts in __param_node; the endpoint accepts a node id as well when you call it by hand.

One caveat. A host name does not uniquely identify a node – one host can run more than one substrate, and a moved substrate leaves its old record behind until a ghost sweep reaps it. Where two records share a host name, the relay picks one deterministically: a record that is not offline, then the lowest node id. So a job that labels its series with node_id must also route by it, or both substrates’ targets reach the same record and one of them is labelled with the other’s data. The bundled jobs, and the example job, set __param_node from __meta_fuzzball_node_id for exactly this.

Compute nodes share a single JetStream credential, so the relay trusts that nodes are not adversarial to one another: a node holding that credential could answer a scrape, or a debug request, addressed to another node. The exchange goes through the cluster’s own NATS server, so separating nodes on the network does not prevent it. Add a machine as a compute node only if you would trust it to report for the others.

Proxy mode always relays an exporter’s plain-text exposition format. The scrape’s Accept header is not forwarded to the node, so OpenMetrics and protobuf exposition – and with them exemplars and native histograms – are available only in direct mode.

Upgrade compute nodes before relying on proxy mode. The scrape relay runs in the node’s fuzzball-substrate-orchestrate. A node still on an older version is listed by service discovery but cannot answer a proxied scrape, and its node and dcgm targets fail with an error saying nothing on the node is serving scrape requests. That is expected while a fleet upgrades: those targets recover as each node is upgraded.

Each node’s scrape succeeds or fails on its own: a node that cannot be reached, or whose exporter answered an error, produces a non-200 for that target. This matters more than it sounds. Every proxy-mode target is the same address, so if a missing node answered 200 with an empty body, up would read 1 for every node in the fleet forever and every absence rule would be reading a constant.

Configuring a Prometheus Job

Both the discovery poll and the scrape are authenticated API calls, so both need a bearer token.

The token’s identity needs the cluster scrape_node_metrics permission. Service discovery is open to any authenticated user, but the scrape proxy is not: it relays live exporter output for nodes other tenants may be running on.

Cluster administrators hold the permission, so a cluster administrator user’s credential works. There is currently no API to grant scrape_node_metrics to another identity: the bundled stack’s own scoped identity is provisioned automatically, but a Prometheus you run yourself has to authenticate as a cluster administrator. Refresh that credential the way the warning below describes.

Set refresh_interval explicitly – Prometheus defaults http_sd_config to 60 seconds, and a node that lives two minutes would be scraped once, or not at all. The example below is a proxy-mode job; see the warning after it for what changes in direct mode:

scrape_configs:
  - job_name: fuzzball-node-node
    scrape_timeout: 8s
    authorization:
      credentials_file: /var/run/fuzzball-api/token
    tls_config:
      ca_file: /etc/fuzzball/ca/ca-certificates.crt
    http_sd_configs:
      - url: https://orchestrate.example.com/v4/nodes/-/sd_config?exporter=node
        refresh_interval: 20s
        authorization:
          credentials_file: /var/run/fuzzball-api/token
        tls_config:
          ca_file: /etc/fuzzball/ca/ca-certificates.crt
    relabel_configs:
      # Scrape only targets discovery built for this job's mode. Without it,
      # a job carrying the credential will happily scrape direct-mode targets
      # -- see the warning below.
      - source_labels: [__meta_fuzzball_sd_mode]
        regex: proxy
        action: keep
      # Name the substrate, not just the host -- see below.
      - source_labels: [__meta_fuzzball_node_id]
        target_label: node_id
      # Proxy mode only: route the scrape to that same substrate.
      - source_labels: [__meta_fuzzball_node_id]
        target_label: __param_node

Note what the job does not set. Service discovery emits __scheme__ and __metrics_path__ on every target, and Prometheus honours those over the job’s own settings, so the URL is always right for the mode in force. Stating them yourself is how you get a job that is correct for one mode and wrong for the other: proxy mode addresses this API over HTTPS at a versioned path, direct mode addresses a node’s own exporter over plain HTTP at /metrics.

The credential is the part that does not follow the mode, which is why the keep rule above matters. Prometheus has no per-target credential, so authorization is a property of the job; if discovery is serving direct-mode targets and the job still carries the token, the URL will be correct and the token will go to a node exporter anyway.

ca_file is needed wherever the API’s certificate comes from a private issuer, which is the usual Fuzzball arrangement; drop it if your ingress certificate is publicly trusted.

In direct mode, do not put authorization on the scrape – only on the http_sd_configs entry – and set the keep rule’s regex to direct. Prometheus has no per-target credential, so a job-level one is sent to every target, and in direct mode those targets are node exporters. That would hand a Fuzzball control-plane token to every compute node in the fleet, over plain HTTP, for an endpoint that ignores it – and because it ignores it, every scrape would succeed and nothing would report the leak.

Service discovery emits instance and node as target labels; the second relabel rule adds node_id. node is the host name, and a host can run more than one substrate, so without node_id an alert over an exporter series names a host and leaves the hold receiver to pick one of its substrates. With it, each series names its substrate. The shipped rules rely on it. A host running two substrates is then scraped once per substrate, so aggregate host-level counters by (node, node_id) or take the max across node_id – never a sum by node alone. Promote no other __meta_fuzzball_* label unless a rule selects on it: the shipped rules reach definition by joining to fuzzball_node_info.

For the fuzzball-node exporter’s job only, set honor_labels: true. Fuzzball’s own exporter already labels every series with node, and service discovery attaches node as a target label too. With Prometheus’s default honor_labels: false, a scraped label that collides with a target label is renamed exported_node – even when the two values are identical – so every series would carry a redundant exported_node. The node and dcgm exporters do not label by node, so their jobs leave it unset.

scrape_timeout is below Prometheus’s 10-second default because a proxied scrape is a round trip to the node on top of the API request, and a sample that arrives after the deadline is discarded anyway.

Tokens expire, so something has to refresh the credentials file. Prometheus re-reads credentials_file on every request, so a process that writes a fresh token beside it is enough – it needs no reload. Write through a temporary file and rename it into place; a half-written file read mid-write sends a truncated token and fails an otherwise healthy scrape.

fuzzball context token prints the active context’s token for exactly this. The bundled stack does this for you – see Bundled Monitoring Stack.

Bundled Monitoring Stack

Fuzzball deploys Prometheus, Alertmanager and Grafana alongside the cluster by default, preconfigured against the cluster’s own metrics and every compute node’s exporters. It ships with the rule pack, the Alertmanager routes that turn alerts into node holds, and three dashboards – the same configuration Fuzzball publishes for sites that run their own monitoring.

The stack is what closes the node-health loop: Fuzzball publishes facts about its nodes, Prometheus judges them against the rules, and Alertmanager calls back to place and release holds. A cluster without it publishes facts nobody acts on, unless a site’s own monitoring does the judging.

Upgrading. The stack used to be opt-in. A cluster upgraded to this release that never set monitoring.enabled now gets the stack, in the fuzzball-monitoring namespace, with a 20Gi Prometheus volume and a small Alertmanager volume. To keep the old behaviour, set monitoring.enabled: false in the FuzzballOrchestrate resource before you upgrade the operator.

  • Storage. The volumes use monitoring.storageClassName, or the cluster’s default storage class when that is unset. On a cluster with neither, a resource that never set monitoring.enabled or monitoring.storageClassName deploys without the stack, and the operator records a MonitoringNotDeployed warning event; set monitoring.storageClassName, or monitoring.storageSize: "" to run the stack on emptyDir, to bring it in. A resource that does ask for the stack – monitoring.enabled: true, or a named class that does not exist – is refused with a configuration error naming spec.monitoring.storageClassName.
  • Images. The stack’s images follow the cluster’s image settings like every other component, so with image.exclusive they are pulled from your own repository, under their last path element. Mirror them before upgrading: quay.io/prometheus/prometheus:v2.45.0, quay.io/prometheus-operator/prometheus-config-reloader:v0.66.0, quay.io/prometheus/alertmanager:v0.25.0 and docker.io/grafana/grafana:10.0.1.
  • Existing prometheus-operator CRDs. Earlier releases installed the monitoring.coreos.com CRDs with the stack. This release does not, and leaves any it installed in place rather than deleting them – and with them every ServiceMonitor or PrometheusRule in the cluster.
  • Alertmanager’s volume. On a cluster that already ran the stack, setting monitoring.storageClassName or monitoring.storageSize: "" changes Alertmanager’s volume claim template, which Kubernetes does not allow on an existing StatefulSet. The operator deletes the fuzzball-prometheus-alertmanager StatefulSet first, leaving its pod and volume in place, and the upgrade recreates it. An existing volume keeps the storage class it was created with.

Opting Out

A site that runs its own monitoring turns the stack off and adopts the published configuration instead:

spec:
  monitoring:
    enabled: false

The stack can also run beside an existing kube-prometheus-stack. It installs no CRDs, no DaemonSet, and no host ports, and everything it creates is named after its own Helm releases in its own namespace.

Reaching the Stack

Nothing in the stack is exposed outside the cluster. Reach each UI with a port forward:

$ kubectl -n fuzzball-monitoring port-forward svc/fuzzball-grafana 3000:80
$ kubectl -n fuzzball-monitoring port-forward svc/fuzzball-prometheus-server 9090:80
$ kubectl -n fuzzball-monitoring port-forward svc/fuzzball-prometheus-alertmanager 9093:9093

Unless you set monitoring.grafana.adminSecretRef, Grafana’s admin user is admin and its generated password is stored in the admin-password key of the fuzzball-grafana Secret:

$ kubectl -n fuzzball-monitoring get secret fuzzball-grafana \
    -o jsonpath='{.data.admin-password}' | base64 -d

To choose the credentials yourself, create a Secret in the monitoring namespace holding the user name under admin-user and the password under a key you name, and point monitoring.grafana.adminSecretRef at it. The Secret must be in the monitoring namespace, where Grafana reads it; one named elsewhere is refused.

To publish Grafana under a hostname, enable its ingress through monitoring.grafana.valuesOverride – see the example in the CRD reference.

Customizing the Stack

Retention, storage, the alert webhook and the Grafana admin Secret have their own fields. Anything else either chart supports is set through a valuesOverride per component – monitoring.prometheus, monitoring.alerting and monitoring.grafana – which is merged over the generated values the way Helm layers a values file. The CRD reference lists every field.

Alert Rules

The rule pack keeps the group name fuzzball-node-health, so a site that has already silenced or routed that group keeps working. Three of its rules carry an fb_action label, so on a default install they place holds; see Placing Holds From Alerts for what each label does.

AlertFires whenAction
FuzzballNodeUnknownA node has not reported for 10 minutesAdvisory
FuzzballNodeDevicePluginFailedA device plugin on a node has been in the failed state for 5 minutesAdvisory
FuzzballNodeAttributedJobFailuresFive or more jobs have stopped responding on one node in a dayCordon
FuzzballNodeHealthMetricsAbsentNo node series at all – no leader, or scraping is brokenAdvisory
FuzzballNodeMemoryErrorsA node logs more than 1,000 corrected memory errors in a dayCordon
FuzzballNodeGPUUncorrectableECCA GPU reports an uncorrectable ECC errorEvict, released by hand
FuzzballNodeHoldCeilingReachedFuzzball refused a monitoring hold because the node’s pool is at its ceilingAdvisory

FuzzballNodeMemoryErrors reads the EDAC counters Fuzzball’s node exporter publishes, so it works on a default install; on a node without ECC memory, or with the EDAC modules unloaded, there is nothing to read and it stays quiet. FuzzballNodeGPUUncorrectableECC reads DCGM, which Fuzzball does not deploy. On a cluster without it the rule never fires, and that is absence rather than failure.

FuzzballNodeUnknown and FuzzballNodeDevicePluginFailed ship with the product rather than being left to a site’s own rules, because what they report is only visible to Fuzzball.

A failed scrape is not the same signal. The stack does scrape each node’s exporters, so a silent host does produce an up series that drops – but a failed scrape says only that a target could not be read, which a network fault, the proxy, or an expired credential all produce too. FuzzballNodeUnknown says the specific thing: Fuzzball stopped hearing from the node. In direct mode the two come apart entirely, since the node’s exporter answers Prometheus directly and keeps reporting up while the extension behind it is gone.

Nothing notices a failed device plugin at all. A node with one is otherwise reporting normally, so no exporter has anything to say about it. The plugin alert names the plugin in its labels, so two broken plugins on one node are two alerts.

FuzzballNodeHealthMetricsAbsent exists because every other rule goes quiet when the control plane publishes nothing, and an absent series must never read as a healthy fleet.

Dashboards

Grafana is provisioned with three dashboards in a Fuzzball folder, linked to each other:

  • Fuzzball Fleet Health – nodes by status and by hold effect, who placed the holds in force, hold churn over time, the node-health alerts firing, the nodes that have gone silent, and the nodes with the most attributed job failures or a failed device plugin. It keeps the address of the earlier Fuzzball Node Reliability dashboard, so existing links land here.
  • Fuzzball Node Health – one node, chosen from a list: its status, its holds and their history, the alerts firing on it, its device plugins, and the exporter series behind the hardware alerts – memory and GPU errors, temperature, machine checks, disk I/O errors, disk faults (read-only or failing SMART), whether each exporter can be scraped, and signal coverage: whether each of Fuzzball’s collectors, and SMART on each disk, produced a reading, so a zero on an error panel can be told from an unchecked one.
  • Fuzzball Node Resources – how busy nodes are, as opposed to whether they are healthy. A fleet row (nodes by state, nodes not OK, busiest by CPU, longest run queues) and a per-node row: state, Fuzzball status, holds, running allocations, GPUs, idle time, login sessions, CPU utilization, run queue, load average, paging, memory, filesystem and swap utilization, and network I/O. See Node Resource Metrics for what each panel reads and how State is derived. The panels that read node_exporter need it scraped – the bundled one is, by default.

A hold’s id and reason are not metrics. Read them with fuzzball node hold list and fuzzball node events.

Scraping

The bundled stack scrapes both endpoints: Fuzzball’s own pods, and one job per compute-node exporter using the service discovery above. You do not have to configure either.

If you set nodeMetrics.serviceDiscovery.mode to direct, set monitoring.nodeMetricsMode: direct in the operator configuration to match. The operator cannot read central config, and the two modes need different scrape jobs – direct mode’s targets are unauthenticated node exporters, so the Fuzzball credential is left off the scrape.

A mismatch is caught rather than tolerated: each job keeps only the targets advertising the mode it was built for, so a disagreement leaves the job with no targets – visible immediately on Prometheus’s target page. That interlock matters most in one direction. Left at proxy while central config serves direct, the job would otherwise carry the Fuzzball bearer to every node’s own plaintext exporter, which ignores the header – so every scrape would appear to succeed while a control-plane credential went out in the clear across the compute network.

It handles its own credential. A sidecar beside Prometheus exchanges its projected Kubernetes ServiceAccount token for a Fuzzball token every half hour and writes it where Prometheus reads it, so there is no password anywhere in the stack’s configuration and nothing to rotate by hand.

That token belongs to a scoped fuzzball-monitoring identity, not to a cluster administrator. It holds one permission, scrape_node_metrics, to scrape compute nodes through the control plane, and carries no password, so the ServiceAccount exchange is the only way to obtain it. It cannot place holds: Alertmanager reaches the hold receiver with a client certificate instead – see What the Bundled Stack Configures. On a fresh install the identity is created shortly after the local organization is, so the stack’s first scrapes may be refused for a minute or two before they start succeeding.

The stack validates the Fuzzball API’s certificate against the cluster’s CA bundle only where the operator creates that bundle: when it deploys trust-manager and issues internal certificates itself. Without trust-manager, or with an external internal issuer (tls.internalIssuer.external), there is no bundle for the stack to read, and it trusts public certificate authorities only – so the API’s ingress certificate must be publicly trusted. With a privately issued certificate in that configuration, scrapes fail with a certificate error.

The bundled jobs cover the three built-in exporters. If you replace the inventory with nodeMetrics.exporters, the operator cannot see that – it does not read central config – so the shipped jobs will still name the built-ins, discovery will refuse the ones you removed, and the ones you added will not be scraped. Add jobs for your own exporters through extraScrapeConfigs in monitoring.prometheus.valuesOverride – a string holding the job list, see the example in the CRD reference – or keep the built-in names.

The ServiceAccount is named fuzzball-monitoring in the monitoring namespace. The control plane accepts its projected token alongside the one Fuzzball’s own in-cluster jobs use, and no others – and only on a cluster that runs the bundled stack, so a cluster that opted out grants it nothing.

Published Configuration

The rule pack, the Alertmanager routing and the dashboards are published as files, for a site that runs its own Prometheus, Alertmanager and Grafana. Each file is exactly what the bundled stack deploys in this release:

FileWhat it isAdjust before use
fuzzball-node-health-rules.yamlPrometheus rule fileNothing. Load it with rule_files
fuzzball-alertmanager.yamlAlertmanager configuration: the hold routes and receivers, and a catch-all route to your own receiverReplace alertmanager-receiver.example.com:9443 with the external receiver’s hostname and port and https://alerts.example.com/hook with your receiver. Point cert_file, key_file and ca_file at the client certificate, key and receiver CA the operator issues – see Using Your Own Alertmanager
fuzzball-fleet-health.jsonGrafana fleet dashboardNothing. Import it and pick your Prometheus datasource
fuzzball-node-health.jsonGrafana node dashboardNothing. Import it and pick your Prometheus datasource
fuzzball-node-resources.jsonGrafana node resources dashboardNothing. Import it and pick your Prometheus datasource

The rules need the scrape jobs above in place, and the hardware rules need the exporters they read. If you merge the Alertmanager configuration into an existing one, keep its hold routes ahead of your own and keep continue: true on them, or an alert that places a hold stops notifying you.

Placing Holds From Alerts

Fuzzball accepts Alertmanager notifications and turns them into node holds. Prometheus decides what a node’s metrics mean; Fuzzball provides the facts and the hooks, and the judgement stays in rules you can read and change.

POST https://<receiver>:9443/v4/receivers/alertmanager

The endpoint takes Alertmanager’s webhook payload, version 4. It authenticates by mutual TLS: a caller must present a client certificate issued by one of the cluster’s monitoring client CAs, which the operator creates for this purpose and nothing else. There is one in each namespace an Alertmanager’s credential is issued into – the bundled stack’s namespace, and the Fuzzball namespace when you enable an external receiver – each behind a cert-manager Issuer that signs only for certificates in its own namespace, so creating a certificate anywhere else grants nothing. A connection without a client certificate – or with one from any other CA, including the one that signs the cluster’s server certificates – is refused during the TLS handshake, before any request is read. There is no bearer-token route: the public API answers this path with a 403 naming where the receiver is served, so a cluster-admin token cannot place a monitoring hold.

The receiver is served by the orchestrator on its own port, 9443, not through the cluster ingress: the ingress terminates TLS, so a client certificate would never reach the receiver. Inside the cluster it is fuzzball-orchestrator.fuzzball.svc:9443; outside, it is published only when you ask for it – see Using Your Own Alertmanager.

Holds it writes carry the monitoring source, which no other entry point can write and which this one cannot avoid writing – the source comes from the endpoint, not from the request, so a certificate issued to your monitoring stack cannot place a hold that reads as an operator’s.

The Label Contract

A rule controls what happens to a node through two labels:

LabelEffect
fb_action: cordonPlace a cordon hold: no new work is placed on the node
fb_action: evictPlace an evict hold: no new work, and what is running is taken off
fb_action absentAdvisory. The alert notifies and holds nothing
fb_release: manualThe hold is not released when the alert resolves

The alert names the node with either node_id or node, and its summary annotation becomes the hold’s reason – which is then recorded against any work the hold displaces, so a user whose job moved is told what was wrong with the node.

LabelNames
node_idThe substrate, uniquely. Preferred when both are present
nodeThe host name. Used when node_id is absent

Prefer node_id where your exporter can supply it. A host can run more than one substrate, and they share a host name – so an alert carrying only node on such a host names two records, and the hold lands on one of them. Fuzzball resolves that consistently, so a repeat and its resolve reach the same record, but it is still a node chosen for you rather than the one that is faulty. The rules Fuzzball ships carry node_id for this reason – including the two over third-party exporters, because the bundled scrape jobs promote it onto every exporter series. If you run your own Prometheus, add the same relabel rule; see the scrape job example. A host-wide fault, such as a failing DIMM or GPU, then holds each substrate on that host.

Alertmanager’s own alert fingerprint becomes the hold id, so the notification that resolves an alert releases exactly the hold that the firing notification placed. A firing alert repeats for as long as it fires; each repeat refreshes the hold rather than stacking a second one – on the same record it was placed on, even if a node named by host name has since gone offline, and a resolve releases it there.

A hold id belongs to the source that placed it. Placing a hold through the admin API with an explicit hold_id that a firing alert already uses is refused for that node, so choose another id – fuzzball node hold add generates its own, so it never collides. And a resolved alert only ever releases the monitoring hold it placed. A summary longer than 1,024 bytes is truncated, ending in ..., rather than refused, so a long description never stops a hold being placed.

fb_action: deprovision is refused, and reported as refused rather than as an unrecognised value. A deprovision hold ends with the instance destroyed, and a rule firing on the wrong series would take the hardware with it. Deprovisioning a node stays an operator’s decision – you can still do it by hand after reading the alert.

Nothing is held until a rule says so, and three of the seven shipped rules do say so. FuzzballNodeAttributedJobFailures and FuzzballNodeMemoryErrors carry fb_action: cordon and read series Fuzzball publishes itself, so on a default install they cordon a node that accumulates five job failures in a day, or more than 1,000 corrected memory errors. FuzzballNodeGPUUncorrectableECC also carries an fb_action, but reads DCGM series that Fuzzball does not deploy, so it stays inert until you install DCGM Exporter.

You opt a rule out by dropping its fb_action label, which leaves it notifying and holding nothing.

Releasing a Hold Only When You Choose

A rule labelled fb_release: manual routes to a second receiver that sends no resolved notification. Fuzzball never hears the alert resolve, so the hold it placed stays until an administrator releases it with fuzzball node hold release <fingerprint> --source monitoring. The --source flag is required: a release defaults to administrator holds only, so without it the command reports the hold as not released and the node stays cordoned.

Use it where recovery of the metric does not mean recovery of the node – a GPU that threw an uncorrectable ECC error, say. The error stops being reported; the GPU is still suspect.

What the Bundled Stack Configures

If you deploy the bundled stack, this is already wired. The operator configures two Alertmanager receivers pointed at the receiver’s in-cluster address, https://fuzzball-orchestrator.fuzzball.svc:9443/v4/receivers/alertmanager, and routes alerts carrying fb_action to them. Alertmanager authenticates with a client certificate cert-manager issues into the monitoring namespace (fuzzball-alertmanager-client-tls) from that namespace’s own monitoring client CA, and verifies the receiver against the CA in the fuzzball-alertmanager-receiver-ca ConfigMap beside it. cert-manager renews the certificate before it expires and Alertmanager picks the renewal up without a restart, so there is nothing to rotate by hand. Alertmanager holds no Fuzzball token.

Alerts still reach whatever destination you already route to, because a final catch-all route follows the hold routes – taking a node out of service is not a reason to stop telling anyone about it.

Setting monitoring.alerting.configOverride replaces the whole routing configuration, hold receivers included. If you override it, wire the receivers yourself, keeping the tls_config shown in the published configuration: the certificate and CA are still mounted at /etc/fuzzball/alertmanager-client/ and /etc/fuzzball/alertmanager-receiver-ca/.

Using Your Own Alertmanager

An Alertmanager the operator does not deploy reaches the receiver through an external endpoint the operator publishes on request, with a client certificate the operator issues for it. Enable it in the FuzzballOrchestrate resource – it works whether or not the bundled stack is enabled:

spec:
  monitoring:
    enabled: false          # optional; the bundled stack may stay on as well
    alerting:
      externalReceiver:
        enabled: true
        hostname: alertmanager-receiver.example.com
        # serviceType: NodePort   # default LoadBalancer
        # annotations: {}         # e.g. request an internal load balancer

The operator then creates, in the Fuzzball namespace:

ResourceWhat it is
Service fuzzball-alertmanager-receiverPublishes the receiver on port 9443. A LoadBalancer by default. Point hostname’s DNS at its address
Secret fuzzball-external-alertmanager-client-tlsThe client certificate (tls.crt) and key (tls.key) your Alertmanager presents
ConfigMap fuzzball-alertmanager-receiver-caThe CA (ca.crt) that signs the receiver’s certificate, for your Alertmanager’s ca_file

The client certificate is issued by the Fuzzball namespace’s own monitoring client CA, whose key is in the Secret fuzzball-monitoring-client-ca beside it. Anyone who can read Secrets or create cert-manager Certificate resources in the Fuzzball namespace can therefore obtain a receiver credential; restrict that access as you would the client certificate itself.

hostname defaults to alertmanager-receiver.<domain>; the receiver’s certificate is issued for it, so your Alertmanager must use that name. Copy the credentials to your Alertmanager:

kubectl -n fuzzball get secret fuzzball-external-alertmanager-client-tls \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > fuzzball-client.crt
kubectl -n fuzzball get secret fuzzball-external-alertmanager-client-tls \
  -o jsonpath='{.data.tls\.key}' | base64 -d > fuzzball-client.key
kubectl -n fuzzball get configmap fuzzball-alertmanager-receiver-ca \
  -o jsonpath='{.data.ca\.crt}' > fuzzball-receiver-ca.crt

The client certificate is valid for a year and cert-manager renews it 30 days before it expires. Copy the renewed Secret to your Alertmanager when it does – or sync it with your own tooling. Alertmanager re-reads the files when they change, so no restart is needed.

Then merge the routes and receivers entries below into your existing routing tree, keeping your own root receiver:

route:
  # Your existing default receiver. Alertmanager refuses to load a
  # configuration whose root route has none.
  receiver: your-existing-receiver
  routes:
    - matchers: [ 'fb_action =~ ".+"', 'fb_release = "manual"' ]
      receiver: fuzzball-holds-manual
      group_by: ['alertname', 'node', 'node_id']
      repeat_interval: 15m
      continue: true

    - matchers: [ 'fb_action =~ ".+"', 'fb_release != "manual"' ]
      receiver: fuzzball-holds
      group_by: ['alertname', 'node', 'node_id']
      repeat_interval: 15m
      continue: true

    # Last, and without continue. This is what keeps your own notifications
    # working -- see below.
    - receiver: your-existing-receiver

receivers:
  - name: fuzzball-holds
    webhook_configs:
      - url: https://alertmanager-receiver.example.com:9443/v4/receivers/alertmanager
        send_resolved: true
        http_config: &fuzzball_tls
          tls_config:
            cert_file: /etc/alertmanager/fuzzball/fuzzball-client.crt
            key_file: /etc/alertmanager/fuzzball/fuzzball-client.key
            ca_file: /etc/alertmanager/fuzzball/fuzzball-receiver-ca.crt

  - name: fuzzball-holds-manual
    webhook_configs:
      - url: https://alertmanager-receiver.example.com:9443/v4/receivers/alertmanager
        send_resolved: false
        http_config: *fuzzball_tls

If you reach the receiver by an address other than hostname – a node IP with serviceType: NodePort, say – add server_name: <hostname> to the tls_config so the receiver’s certificate still verifies.

Five details in that configuration are load-bearing:

  • The two hold routes are mutually exclusive rather than ordered, so both can carry continue: true. fb_release != "manual" matches an alert that omits the label entirely, which is the common case. Ordering alone would not work: with continue: true on the first route, the second is still evaluated.
  • The trailing catch-all route is what keeps your existing notifications working, and it is not the same thing as the root route’s receiver. Alertmanager falls back to a route’s own receiver only when none of its children matched, and continue: true means “keep evaluating siblings” – not “also deliver to the parent”. Without that last route, an alert carrying fb_action matches a hold route and is delivered there only, so you would silently stop being told about exactly the alerts that took a node out of service.
  • send_resolved is set explicitly on both hold receivers, because it is the only difference between them.
  • Grouping is by node and node_id, not instance. instance is per exporter, so a node running three exporters would otherwise be held three times for one fault. Both node labels are needed because the shipped rules carry different ones: FuzzballNodeAttributedJobFailures carries only node_id, and grouping on node alone would put every node’s alert from it into one fleet-wide group.
  • repeat_interval on the hold routes is deliberately short. A hold’s refresh timestamp only advances when a notification arrives, and that timestamp is what tells you a still-firing fault from an integration that has stopped talking. Inheriting a typical 12-hour notification cadence would make every live hold look abandoned.

If notifications fail their handshake, the orchestrator logs each refused connection (TLS handshake error) – usually a client certificate that was not copied, has expired, or was issued for another cluster.

The receiver is per-cluster. A hold is written to the node store of the cluster the node belongs to, and two clusters can name a node the same thing, so point each member cluster’s Alertmanager at that cluster’s own receiver. A federate serves no receiver.

A Ceiling on Monitoring Holds

Holds stay until the alert that placed each one resolves, so a rule that misfires across a whole pool – a bad threshold, an exporter reporting nonsense after an upgrade – does not heal itself when the rule is fixed. To keep that from taking a pool out of service, monitoring holds may cover at most 20% of a provisioner definition’s nodes at once. Past that, Fuzzball refuses the next monitoring hold in the pool and raises FuzzballNodeHoldCeilingReached, naming the definition.

  • Nodes are counted, not holds. A firing alert’s repeat, a second alert on a node monitoring already holds, and a hold on an offline node are never refused, since none of them takes another node out of service.
  • Offline nodes are left out of the count entirely.
  • One node is always allowed, so a pool of four or fewer can still hold its one bad node.
  • Administrator holds are never refused and never counted: cordoning half a pool by hand is a decision you have made.
  • A refused alert keeps re-notifying while it fires, so its hold lands on its own once another hold in the pool is released.

When the alert fires, either that many nodes really are failing or a rule is misfiring. fuzzball node hold list shows which alerts placed the holds. Hold the remaining faulty nodes by hand, or raise the ceiling with nodeHealth.maxMonitoringHeldPercent – cluster-wide, or per definition; see the configuration reference. 100 removes it.

Reading What an Alert Did

The response reports one outcome per alert, which answers “the alert fired, so why is the node not held” without reading control-plane logs. Each outcome’s result is ALERTMANAGER_ALERT_OUTCOME_RESULT_ followed by one of:

OutcomeMeaning
PLACEDA new hold was written
REFRESHEDThe hold already existed; its reason and timestamp were updated
RELEASEDA resolved alert removed the hold it had placed
ADVISORYThe alert carried no fb_action, so it held nothing
IGNOREDUnderstood but changed nothing – an unknown node, an unknown or refused action, a hold id another source already holds, a node already carrying 64 holds, a pool at its ceiling, or a release for a hold already gone

A write that genuinely failed is answered with an error instead of this list, so Alertmanager retries. Retrying is safe: placing a hold twice refreshes it, and releasing one twice finds nothing the second time.

Because Alertmanager discards the body of a successful response, these outcomes are also written to the control plane’s log – an IGNORED alert is logged as a warning naming the alert and the reason. That log is where to look when a rule fires and nothing is held; a request made by hand (curl) is where to read the list above directly.

A resolved notification that never reaches Fuzzball – because Alertmanager was down, or the route was misconfigured – leaves a hold in place. Holds do not expire, so check the hold’s refresh timestamp with fuzzball node hold list to tell a fault that is still firing from an integration that stopped talking – see Spotting an Abandoned Hold.

Nodes That Go Offline

When a node disappears – power loss, a killed agent, an instance terminated underneath Fuzzball – the work placed on it is already gone. Fuzzball releases it immediately rather than waiting for each lease to time out one at a time:

  • Steps with policy.retry.on: node are requeued and run again elsewhere.
  • Everything else fails, with the node named.

There is no judgement to make here, so this is not configurable: the node is gone and the work died with it. The only question is whether you are told now or after a timeout. A multinode job in particular no longer holds its remaining nodes for the length of that wait.

The failure names no cause. A node disappearing says nothing about its hardware – a rack losing power and an agent being killed look the same from the control plane – so Fuzzball reports what it knows and no more:

node 10.0.0.42/9100 stopped hosting this work; job failed. Set policy.retry.on to node for work that is safe to run again from the beginning

A step that declared policy.retry.on: node and has already been moved maxEvictionRestarts times is failed with the count rather than that advice:

node 10.0.0.42/9100 stopped hosting this work; job failed after 3 node restart(s), the limit for this step

When a Job Fails on a Node

When a job ends because its node stopped responding, the error names the node:

job stopped responding on node 10.0.0.42/9100

If that node is held at the time, the error names the hold’s reason as well, so a user can tell immediately that the failure was not theirs:

node 10.0.0.42/9100 is held: GPU 2 uncorrectable ECC; job stopped responding on this node

The reason comes from the hold rather than from a diagnosis Fuzzball made: a person or an alerting rule wrote that sentence about this node deliberately. A node with nothing held against it is still named, because knowing which machine a job died on is useful on its own, and claims nothing.

The failure is also counted against the node’s attributed-failure total.

Node Report Storage

The control plane keeps the latest resource report for every attached node in a JetStream key-value bucket named substrate-nodes. Bucket usage grows with the number of nodes and with the size of each report, and a report grows with the node’s core count, its devices and its annotations.

Size the bucket from the node count multiplied by the size of one report. A 192-core node with 8 GPUs and 10 annotations produces a report of roughly 4 KiB, so 3,000 such nodes need about 12 MiB. The bucket defaults to 64 MiB, which holds roughly 16,000 reports of that size.

A full bucket fails quietly – no error surfaces in the CLI, the API or a node’s status. JetStream stops accepting reports from nodes the bucket has not seen before, and stops accepting any report larger than the one it replaces. The cluster stops registering new nodes, and the scheduler keeps planning against the last resources it recorded for the nodes it already has. The only signals are the substrate bridge log, which reports a rejection at error level at most once every 30 seconds, and the metrics in Bucket Metrics.

Set the bucket’s size cap with nodesBucketMaxBytes in the cluster’s FuzzballOrchestrate custom resource:

apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  name: fuzzball-orchestrate
spec:
  fuzzball:
    substrate:
      nodesBucketMaxBytes: 1073741824

See Substrate Configuration in the CRD reference for the full field description.

When the substrate bridge restarts, it applies the new cap, resizing the existing bucket in place. Leaving the field unset changes nothing: the 64 MiB default is the size clusters ran with before the field existed.

Check fuzzball_substrate_nodes_bucket_bytes_used before lowering a cap. Any cap below what the bucket already stores produces the same failure as a full bucket. The field’s 64 MiB minimum only rejects caps below 64 MiB; it does not compare the new cap against what the bucket currently stores, so a bucket already holding more than 64 MiB can still be capped below its own contents.

The cap is a reservation, not a limit on actual use. JetStream subtracts it from the NATS server’s on-disk allowance of 10 GiB whether or not the bucket holds that much, and every other Fuzzball stream on that server draws on the same allowance – including the node event history (1 GiB) and the node allocation history (512 MiB).

That allowance is per server. A high-availability deployment replicates the bucket across three NATS servers, and each server reserves the full cap against its own 10 GiB. A 1 GiB cap therefore costs 1 GiB on every server, not 3 GiB on any one of them.

An oversized reservation stops the substrate bridge from starting rather than degrading quietly, so the cap is bounded at 2 GiB. The Kubernetes API server rejects a nodesBucketMaxBytes value below 64 MiB or above 2 GiB when the custom resource is applied, so a mistyped size fails at apply time. A value that reaches the orchestrate service configuration by another path is clamped into that range rather than preventing the bridge from starting. The bridge logs the cap in force at startup, and logs the configured value alongside it only when that value had to be clamped. Raise nodesBucketMaxBytes as the node count grows; do not set a large value up front.

Bucket Metrics

Three series published by the substrate bridge cover the bucket:

MetricTypeLabelsDescription
fuzzball_substrate_nodes_bucket_bytes_usedgauge-Bytes stored in the substrate-nodes bucket
fuzzball_substrate_nodes_bucket_bytes_maxgauge-Configured cap on the bucket
fuzzball_substrate_nodes_bucket_put_failures_totalcounter-Resource reports the bucket rejected

The two gauges are sampled once a minute, regardless of how many nodes are attached. Alert when fuzzball_substrate_nodes_bucket_bytes_used exceeds 80% of fuzzball_substrate_nodes_bucket_bytes_max. That leaves room to raise the cap and restart the substrate bridge before the bucket fills.

Both gauges are absent until the first successful sample, so absence means usage is unknown rather than zero. Pair a headroom alert with an absent() check rather than treating a missing series as an empty bucket.

Node Event History

Events are retained for 90 days and can be read back per node through the ListNodeEvents API, filtered by time and by event type and returned oldest first in pages. Allocation events (ALLOCATION_ADDED, ALLOCATION_REMOVED) are kept in a separate 512 MiB history, so on a busy cluster the oldest of them can age out before 90 days; hold and status history is unaffected by allocation volume.

Read a node’s history with fuzzball node events, oldest first:

fuzzball node events 10.0.0.42/9100 --since 24h
TIME                 | TYPE                    | DETAIL
2026-09-17T09:14:02Z | HOLD_PLACED             | cordon by monitoring: 4 correctable ECC errors in 24h
2026-09-17T11:37:41Z | JOB_FAILURE_ATTRIBUTED  | job alloc-93f2 failed on this node
2026-09-17T11:52:10Z | STATUS_CHANGED          | status ready -> unknown: no health report for 3 consecutive intervals

Filter by type with --type, repeatable, and use -o json to pull the series out for plotting.

TypeRecorded whenCarries
NODE_PROVISIONEDThe cluster first sees the node, so its history has a beginning–
STATUS_CHANGEDThe node’s status changesThe previous and new status
HOLD_PLACED, HOLD_RELEASEDA hold is placed on or released from the nodeThe hold: id, source, effect, reason
NODE_EVICTEDThe node’s work was taken off it because of an evict holdThe hold that caused it, copied onto the event
NODE_DRAINEDThe node is drained, by an operator or automaticallyWhether running work is left to finish (passive) or taken off (active), and the reason
ALLOCATION_ADDED, ALLOCATION_REMOVEDAn allocation’s rank starts or stops running on the nodeallocation_id and rank
JOB_FAILURE_ATTRIBUTEDA job stopped responding on this nodeallocation_id

A hold is copied onto its events rather than referenced, because by the time the history is read the hold has usually been released. allocation_id is the join into the workflow’s own events, which say what then happened to that work – see fuzzball workflow events.

STATUS_CHANGED is the event to watch for a node going silent. It replaces the two events that used to describe it – a raised HEARTBEAT_MISSED condition and a health-state transition into UNKNOWN – which said the same thing twice in a vocabulary that no longer exists.

What Was Retired

If you are upgrading, these are gone. Nothing needs to be done to remove them; they simply stop being produced.

RetiredReplaced by
The 0-100 reliability score, its penalties, thresholds and explanationPrometheus rules over the raw counters
DEGRADED as a health state, and the closed condition-type listWhatever your rules match on
The policy engine and its observe / cordon / drain / replace modesWhich rules a site enables, and what action each carries
evictOnFaultClassesA rule with an evict action
The score-based placement gateHolds
Automated node replacement and its churn ceilingA deprovision hold; the provisioner supplies capacity again when work needs it
The manual-uncordon suppression windowHold source – an administrator hold is one automation may never clear
The Fuzzball-native signed notification webhookAlertmanager receivers
Score, condition, health-state and policy node eventsHOLD_PLACED, HOLD_RELEASED, STATUS_CHANGED, JOB_FAILURE_ATTRIBUTED

The nodeHealth central-config block keeps one field, maxEvictionRestarts. The penalties, thresholds, failure-tally weights, policy mode and evictOnFaultClasses it used to carry are ignored – a stored config that still sets them keeps loading, so an upgrade does not need the file edited first.

On the API, the retired fields stay on the wire and are never populated: reliability_score, health_state, conditions and score_explanation on a node; the score, condition, threshold and policy fields on an event. A client reading them sees the zero value rather than a stale judgement.

fuzzball node drain is deprecated in favour of fuzzball node cordon, and fuzzball node evict in place of --active. It keeps working, but --active has changed: it now honours what each step declared rather than requeueing everything on the node. A step whose policy.retry.on includes node is requeued; anything else is failed. Earlier releases restarted steps whose authors had said they must never be restarted.