Node Health Monitoring
Fuzzball publishes facts about its compute nodes. It does not judge them.
Each node sends a periodic heartbeat, which is how the control plane tells a live node from a silent one, and exports hardware counters for a monitoring stack to scrape. Which counters matter, how long a fault must persist, and what severity it warrants are decided by Prometheus alerting rules – reviewable in git, tunable per site – not by thresholds inside the product.
When the monitoring stack concludes a node should stop taking work, it says so by placing a hold. That is the hook, and it is the only thing that takes a node out of service.
This split is new. Earlier releases scored each node 0-100, derived a health state from that score, and shipped an opt-in policy engine that could cordon, drain or replace a node on its own. The score, the health state, the condition vocabulary, the policy engine and the Fuzzball-native notification webhook are all retired – see What Was Retired.
Reading orchestrate service logs requires access to the Fuzzball control plane
namespace. Changing the heartbeat miss threshold requires edit access to the
FuzzballOrchestrate resource in the fuzzball-system namespace.
Four things, and only these:
| Signal | Where it appears |
|---|---|
Node status – unknown, not ready, ready, offline | fuzzball node list, fuzzball node show, the API |
| When the node last reported | fuzzball node show, fuzzball_node_last_health_report_timestamp_seconds |
| Job failures attributed to the node | fuzzball_node_attributed_job_failures |
| The state of each device plugin on the node | fuzzball node show, fuzzball_node_device_plugin |
Every change to a node’s status is recorded as a STATUS_CHANGED event
carrying both the previous and the new status, so a node’s history can be
rebuilt without parsing the event’s message. Read them with
fuzzball node events NODE --type status_changed.
Status reports what Fuzzball has observed about a node, and nothing else:
| Status | Means |
|---|---|
unknown | Fuzzball cannot see the node: no report yet, or reports have gone stale |
not ready | The node is reporting, but no provisioner definition has claimed it |
ready | Reporting, claimed, and able to accept work |
offline | The node is gone. Work placed there is released, and the record is removed after 30 minutes – unless it still carries an administrator or monitoring hold, in which case it stays, held, until it is removed or comes back. A monitoring hold usually goes when its alert resolves, which happens once the node stops being scraped, unless the rule is marked fb_release: manual. See Retiring a Node |
A cordoned node is not a status. It reads ready and carries a cordon
hold, because “nothing is wrong with this machine” and “nobody wants new work
on it” are different statements. See
Node Cordoning and Uncordoning.
The attributed-failure counter is the one node signal Fuzzball can see that a
monitoring stack cannot: it knows which node a job was running on when it
stopped responding. It is exported raw, for rules to alert on. Only a job still
meant to be running counts: one that was stopped with fuzzball workflow stop,
had already finished or reported its exit, or was being moved off the node is
going away for its own reasons, and is not counted against the node.
The orchestrate extension needs a minimum version of Fuzzball Substrate on the node, currently v2.7.0, to pull and build container images. Each node’s extension reports both the version substrate states and the minimum that extension requires. When a node runs an older substrate than its extension requires, or runs an extension too old to report a minimum, the extension keeps running and reporting, and the control plane places a cordon hold on it:
| Hold id | Source | Effect | Placed when |
|---|---|---|---|
substrate-version | provisioner | cordon | The substrate on the node states a version older than the extension requires, or none at all, or the extension reports no minimum |
For an unsupported substrate, the hold’s reason names both versions. While it is held the node refuses image pulls, so workflow stages that need an image fail on it with the same message, and the scheduler places no new work there. Work already running is left alone – the hold cordons, it does not evict: draining would move work that is running fine, and replacing a provisioned node would bring back the same substrate.
An extension that reports no minimum predates v4.4, which converts images to EROFS. It reads every
image as a SIF, while current nodes convert images into the image store the nodes share, and an
image is pulled separately from the job that uses it. A job placed on such a node could fail on an
image another node converted, so once Orchestrate is upgraded the node takes no new work until
fuzzball-substrate-orchestrate is updated to match. Dynamically provisioned nodes finish the
work they are running and are torn down when idle. Static nodes stay held until their packages are
updated, so update them together with Orchestrate, as Updating Fuzzball
describes.
This is the one hold Fuzzball places on its own judgement rather than an operator’s or a monitoring rule’s, and it is the exception for a specific reason: whether the extension can serve a node’s image pulls is Fuzzball’s own knowledge, not something a rule reading metrics could determine.
The fix is to upgrade fuzzball-substrate on the node, or fuzzball-substrate-orchestrate when the
reason names the extension. The hold is released on the node’s next
health report – within 30 seconds by default – and it takes work again. Releasing it by hand with
fuzzball node hold release succeeds but achieves nothing: the next report places it again, because
the condition it describes is still true.
Fuzzball Substrate serves each device kind on a node, such as a GPU, through a
device plugin. When a plugin fails to initialize, the node reports every other
resource normally and the device kind is simply missing, so the scheduler
passes the node over for every job that requests it while the node stays
ready and keeps taking work that needs no device. Nothing else about the node
reads wrong, which is why this is one of the four things Fuzzball reports.
Substrate records every plugin it loaded or tried to load, with its state and the error for one that failed, and the node’s report carries the list. From it, Fuzzball publishes:
fuzzball node showprints aDevice plugins:block naming each plugin, its state and its error.fuzzball_node_device_plugin{node, node_id, plugin, state}is 1 for the state the plugin is in and 0 for the other, so a repaired plugin reads as a transition. The bundled ruleFuzzballNodeDevicePluginFailedfires on it.- A job that too few nodes can hold, where the nodes that report a failed
plugin and lack nothing for the job but a device kind they have none of would
make up the difference, gets a
scheduling_blockedevent whosereasonisdevice_plugin_failed, andfuzzball workflow whysays, for example,1 without the requested device type and with a failed device plugin (nvidia). A job the pool could not hold even when empty is refused at submission instead, and for such nodes that message carries the same note:1 node(s) report a failed device plugin (nvidia). Substrate reports which plugin failed but not yet which device kind it serves, so both messages state the two facts side by side.
The node is not held: a cordon would take healthy CPU capacity out of service
because its GPU plugin is broken, and draining or replacing the node would not
repair the plugin. The fix is on the node – repair the plugin, for example by
installing the missing driver, and restart fuzzball-substrate – after which
the next report lists the plugin ready, the device kind reappears, and placement
resumes.
Reporting plugin state requires Fuzzball Substrate v2.7.1 or later on the node.
An older substrate sends no plugin list, so fuzzball node show omits the block
and no series is published for that node.
Each compute node reports every 30 seconds by default. When a node misses the
configured number of consecutive reports, three by default, the control plane
moves it to the unknown status, records a STATUS_CHANGED event, and logs a
warning naming the node and the time of its last report.
Heartbeat tracking does not depend on detecting a network disconnect. A node
whose hardware has hung, or whose substrate extension has stopped while the
host is still reachable, stops sending heartbeats and goes unknown on the
same timer as any other silent node.
A node that starts reporting again returns to ready when a provisioner
definition has already claimed it, and to not ready when none has – the
status it would have had if it had never gone silent. The health report itself
does not know whether the node is claimed; that is read from the node’s last
substrate resource report.
Only a health report brings a node out of unknown. A substrate resource
report for a node whose health reports have stopped does not return it to
service, because the two come from different places: substrate publishes the
resource, and the orchestrate extension reports the health. A node whose
extension has stopped while substrate keeps running would otherwise be handed
work again on its next capacity change, and taken back out on the next
liveness sweep.
A node that goes offline and re-registers is a different case: its last-report time is cleared with the status, so it comes back as a node that has never reported and is exempt from the sweep until it reports again. Without that, a rebooted node would be condemned on a timestamp from before it left.
A node whose health reporting is switched off after it has reported at least once staysunknown, and out of the placement set, until reporting resumes. Switch health reporting off before the node first reports, or leave it on: a node that has never reported is exempt from the liveness sweep, but one that stops is treated as unreachable.
On the API, that status is NODE_STATUS_UNKNOWN. It is a value of its own
rather than NODE_STATUS_UNSPECIFIED, so a client can tell “Fuzzball cannot
see this node” from “the server did not set this field”.
Set the number of consecutive reports a node can miss in the cluster’s
FuzzballOrchestrate custom resource:
apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
name: fuzzball-orchestrate
spec:
nodeHealth:
heartbeatMissThreshold: 3
See Node Health Configuration in the CRD reference for the full field description.
The threshold is a count, not a duration, and each node is measured against
its own reporting interval. Nodes that report at different intervals therefore
need no per-node tuning. With the defaults, a node goes unknown roughly 90
seconds after its last report.
A lower threshold detects a failed node sooner, at the cost of more false positives on a congested or high-latency network. A higher threshold reduces false positives but delays detection.
The reporting cadence is set per node, in the substrate orchestrate extension
configuration rather than on the FuzzballOrchestrate resource. This applies
to nodes whose extension configuration you manage directly; nodes provisioned
by the Fuzzball operator use the defaults below and do not expose these
settings.
binary: fuzzball-substrate-orchestrate
enabled: true
config:
health:
disabled: false
report-interval-seconds: 30
| Parameter | Type | Default | Description |
|---|---|---|---|
disabled | boolean | false | Turn off health reporting on this node entirely |
report-interval-seconds | integer | 30 | Seconds between reports. Values below 5 fall back to the default |
Turning health reporting off also removes the node from missed-heartbeat detection. A node that has never sent a health report never goesunknownfor missing one, because it has missed no heartbeat it promised – the same applies to a node running an extension too old to report health. Nodes that reported and then had reporting disabled are still detected.
The control plane puts a floor under the window it derives from these values, and a ceiling on the interval it derives it from. It waits at least 15 seconds after a node’s last report before recording the node as silent, however short the interval. It also treats any reported interval longer than 24 hours as 24 hours, so a node configured to report very rarely is still recorded as silent after the threshold multiplied by 24 hours – three days at the default threshold.
The same extension configuration carries the settings of the node’s image provider, which converts container images to EROFS and serves them to jobs:
binary: fuzzball-substrate-orchestrate
enabled: true
config:
image-provider:
mount-mode: auto
prune-interval: 6h
| Parameter | Type | Default | Description |
|---|---|---|---|
mount-mode | string | auto | How converted images are mounted for containers: fuse serves them through a FUSE daemon, loop through loop devices, and auto picks what the node’s kernel serves best. auto picks loop only where the kernel can also map file ownership on it (Linux 5.19 or newer), and serves any container whose mount the kernel still refuses to map through FUSE, so containers see the same file ownership on every node. With loop set explicitly, such a container fails to start instead |
prune-interval | duration | 6h | How often orphaned layers are removed from the node’s image store, as a Go duration such as 6h or 90m. Nodes sharing a store take a cluster-wide lock, so one prune runs at a time; the store records when it last ran and a restarting node prunes at once if that is older than the interval |
Status and holds appear in fuzzball node list:
fuzzball node listID | CLUSTER | HOSTNAME | STATUS | HOLDS | RUNNING
node-01 | default | compute-1 | Ready | - | 3
node-02 | default | compute-2 | Ready | cordon | 1
node-03 | default | compute-3 | Unknown | - | 0
node-03 reading Unknown means Fuzzball cannot see it – it has not reported
yet, or its reports have stopped. That is not evidence its hardware is faulty.
node-02 reading Ready with a cordon hold is not a contradiction: the
hardware is fine and somebody has asked that no new work be placed there.
fuzzball node show adds when the node last reported, the failures attributed
to it, and each hold with its source and reason:
fuzzball node show node-02ID: 10.0.0.42/9100
Hostname: compute-2
CPU: x86_64
Status: Ready
Reported: 2026-09-17T14:05:00Z
Attributed job failures: 2 (last 2026-09-17T11:20:14Z)
Holds: 1
7f3c1a90-2b11-4e6d-9a7f-2f0c5d8e1b42 cordon monitoring "4 correctable ECC errors in 24h"
since 2026-09-17T11:22:03Z
Node metrics come from two places, joined on a shared node label: the
cluster endpoint, for facts only the control plane knows, and the
node exporter, for signals only the node can see. Each node also runs the
Prometheus node_exporter, bundled with the
compute-node package, for standard host metrics such as CPU, memory, disk and
network.
node is the node’s host name, falling back to its id when it has not reported
one. Both endpoints label their series with it, so a rule can join across them
with on(node).
Compute nodes are not Kubernetes objects, so no Kubernetes service discovery can find them, and on a cluster that provisions them per workflow a static target list is stale before you finish writing it. Fuzzball therefore serves its own service discovery, and can relay scrapes to nodes a monitoring stack cannot reach – see Finding Nodes to Scrape.
| Metric | Labels | Meaning |
|---|---|---|
fuzzball_node_info | node, node_id, definition, cluster, dynamic, region, zone, segment | Constant 1 carrying the node’s identity |
fuzzball_node_status | node, node_id, status | 1 for the node’s current status, 0 for the rest. status is one of unknown, not_ready, ready, offline – underscored, unlike the not ready the CLI prints |
fuzzball_node_hold | node, node_id, effect, source | Holds on the node, counted by effect and source |
fuzzball_node_running_allocations | node, node_id | Allocations currently running there |
fuzzball_node_attributed_job_failures | node, node_id | Cumulative jobs that stopped responding on the node |
fuzzball_node_last_health_report_timestamp_seconds | node, node_id | Unix timestamp of the node’s last health report |
fuzzball_node_device_plugin | node, node_id, plugin, state | 1 for the state a device plugin is in, 0 for the other. state is ready or failed. Absent for nodes running a substrate older than v2.7.1 |
fuzzball_node_monitoring_holds_capped_total | definition | Counter of monitoring holds refused because the provisioner definition was at its ceiling. Each one is a node an alert called faulty that is still in service. definition is empty for nodes no definition owns. What FuzzballNodeHoldCeilingReached fires on |
Every series but the last carries both node and node_id; the capped-holds
counter is about a pool, not a node. node is the host name and is
what joins to an exporter; node_id is the node’s address and is what makes
the series unique. A host name is not unique – a substrate that moves to a new
address leaves its old record behind until a ghost sweep reaps it, and one host
can run more than one substrate – so aggregate with max by (node) when you
want one answer per machine, and keep node_id when you need to tell two
records apart.
fuzzball_node_info carries only slow-changing identity. Anything that moves
while a node is alive gets its own series with a value, so a transition changes
a number rather than minting a new series and leaving the old one stranded at
its last reading. That is why status and holds are separate series rather than
more labels on the info one.
Join exporter series to Fuzzball’s view through node. Which side to collapse
depends on what the alert should name.
To name each substrate – what a rule that places holds wants, so the receiver
holds exactly the right records – keep node_id on the exporter series (the
scrape job promotes it) and join on both labels:
(
max by (node, node_id) (increase(fuzzball_node_memory_correctable_errors_total[24h]))
or
sum by (node, node_id) (increase(node_edac_correctable_errors_total[24h]))
)
* on(node, node_id) group_left(definition)
max by (node, node_id, definition) (fuzzball_node_info)
> 1000
That is the shipped FuzzballNodeMemoryErrors rule. It raises one alert per
substrate on the host, each carrying node_id, because a host-wide fault
affects every substrate on it. Aggregate by (node, node_id), never by node
alone: a host running two substrates is scraped once per substrate, so its
counters arrive twice, and a sum by host would double them.
The count comes from Fuzzball’s own exporter, which reads the same EDAC
counters as node_exporter and publishes the host’s total. or falls back to
node_exporter’s per-controller counters, summed, only for a substrate whose
Fuzzball exporter is turned off; never add the two, or every error counts
twice. max by (node, node_id, definition) collapses the copies both leaders publish during a failover, and
keeps their differing pod labels out of the alert, where a leader change would
otherwise change the alert’s identity.
To attach a label to a per-host series instead, use group_left with
topk by (node) (1, fuzzball_node_info) on the right. group_left needs
exactly one series per node there, and there can be more – both leaders
during a failover, or two records sharing a host name – so without collapsing
the rule fails outright with many-to-many matching not allowed. topk picks
one series per node whatever its labels disagree on; max by (node, definition)
only collapses duplicates that agree on definition. Do not carry node_id
this way: fuzzball_node_info is always 1, so topk chooses between ties with
no stable order, and the node_id it carries can change between evaluations.
fuzzball_node_status and fuzzball_node_hold publish zeroes for the values
that do not currently apply, so a rule can say == 0 rather than reasoning
about an absent series, and a release or a transition is visible to a range
query.
fuzzball_node_last_health_report_timestamp_seconds is what a silent-node
alert reads. A timestamp rather than an age, so the value stays true without
being republished every scrape; compute the age in the rule:
time() - max by (node, node_id) (fuzzball_node_last_health_report_timestamp_seconds) > 600
It is absent until a node reports once, so pair a staleness rule with an
absent() check rather than treating a missing series as a node that never
reported. Publishing zero instead would read as “last reported in 1970” and
alert on every newly registered node.
Only the leader replica publishes these, and it drops them when it stops being
leader, so aggregate with max rather than sum: during a failover both the
old and new leader can be scraped, and summing would double count.
Group on (node, node_id) for a staleness rule, not on node alone. max
takes the freshest timestamp in the group, so grouping on the host name only
would let a live substrate cover for a dead one on a host running two – the
alert would never fire for the dead one. Grouping on node_id still dedupes
two leaders publishing the same node, because both publish the same id.
That costs nothing in false alerts from ghost records. A substrate that moves address leaves its old record behind, but that record gets a delete event, which marks it offline and clears its last-report timestamp – so its series is dropped rather than left to go stale. Only the liveness sweep leaves a stale timestamp, and that is the case a staleness rule is for.
fuzzball_scheduler_node_attributed_job_failuresis still published, with the same value and its originalnode_idlabel, so existing dashboards and rules keep working. It is deprecated in favour offuzzball_node_attributed_job_failures.
Each compute node serves its own metrics on port 9101. Deliberately narrow: it publishes only what node_exporter – which Fuzzball runs beside it on port 9100 – and DCGM Exporter cannot see for themselves.
| Metric | Labels | Meaning |
|---|---|---|
fuzzball_node_machine_check_errors_total | node | Machine check records seen since the extension started |
fuzzball_node_machine_check_fatal_errors_total | node | Fatal machine check records |
fuzzball_node_gpu_xid_errors_total | node, gpu, uuid, pci_bus_id | Xid faults attributed to a GPU, per device |
fuzzball_node_gpu_xid_errors_node_total | node | Xid faults seen anywhere on the node, including any that matched no device |
fuzzball_node_gpu_hardware_xid | node, gpu, uuid, pci_bus_id | 1 when an Xid indicating a faulty card has been seen on this device |
fuzzball_node_gpu_hardware_xid_node | node | 1 when such an Xid has been seen anywhere on the node |
fuzzball_node_disk_smart_failing | node, device | 1 when SMART reports the device failing. Absent when SMART could not be read |
fuzzball_node_disk_smart_status | node, device | Whether SMART could be read for this device: 1 ok, 0 not available, -1 the probe failed |
fuzzball_node_disk_io_errors_total | node, device | Counter of block-layer I/O errors the kernel logged against the disk since the extension started, mounted or not |
fuzzball_node_disk_scsi_ioerr_cnt | node, device | The raw SCSI ioerr_cnt for the device. Not a fault signal on its own |
fuzzball_node_disk_read_only | node, device | 1 when the device has been marked read-only while it backs a mounted filesystem or swap |
fuzzball_node_memory_correctable_errors_total | node | Correctable ECC errors from the EDAC controllers |
fuzzball_node_memory_uncorrectable_errors_total | node | Uncorrectable ECC errors |
fuzzball_node_thermal_celsius | node, zone, zone_id | Current temperature of a thermal zone |
fuzzball_node_thermal_critical_celsius | node, zone, zone_id | The zone’s critical trip point, absent when the hardware declares none |
fuzzball_node_thermal_throttled | node | 1 when a zone is at or above its trip point |
fuzzball_node_substrate_ready | node | 1 when the local substrate runtime is ready |
fuzzball_node_run_queue_15s | node | Runnable tasks, exponentially averaged over 15 seconds from one-second samples of /proc/loadavg (LSF r15s). Excludes the sampler itself and tasks in uninterruptible sleep, so an idle node reads 0 |
fuzzball_node_login_sessions | node | User sessions in utmp (LSF ls): console and terminal logins, including ssh with a terminal. Batch jobs and ssh commands without a terminal do not count |
fuzzball_node_idle_seconds | node | Seconds since the last input on any logged-in terminal (LSF it). With no session that has a terminal (nobody logged in, or only graphical logins), since the last terminal input the exporter observed, or since boot if it has observed none |
fuzzball_node_signal_status | node, signal | Whether a collector produced a usable reading. signal is memory, machine_check, disk, thermal, gpu, agent, load or sessions |
fuzzball_node_collector_failures | node, signal | How many of a signal’s collectors failed while others succeeded |
fuzzball_node_signals_age_seconds | node | Seconds since these readings were collected |
fuzzball_node_exporter_up | node | 1 when the exporter answered the scrape |
The node-wide GPU series are separate names rather than the per-device ones
with empty labels, so sum by (node) over the per-device series cannot double
count by adding a total to its own components. Use
fuzzball_node_gpu_xid_errors_node_total for a node total: a fault whose PCI
address matched no device the collector could see still happened, so it can
exceed the sum of the per-device series.
zone is the kernel’s name for a thermal zone and is not unique – a
two-socket host has two zones both named x86_pkg_temp – so zone_id carries
the sysfs directory (thermal_zone0) to tell them apart. Aggregate on
zone_id where you need one series per zone.
Machine checks and GPU Xid faults are the reason this exporter exists: both are
written only to the kernel ring buffer, which no other exporter reads. SMART
needs smartctl, and whether the local substrate runtime is up is Fuzzball’s
own knowledge.
fuzzball_node_signal_status distinguishes three things a rule should not
conflate:
| Value | Means |
|---|---|
1 | The collector ran and produced a reading |
0 | The hardware or tooling is not present on this machine – expected, not a fault |
-1 | The collector ran and failed – the node is not being fully checked |
A signal that could not be read is never reported as passing. Alert on
fuzzball_node_signal_status == -1 to catch a node whose telemetry has
silently stopped being collected.
Two cases the signal-wide status cannot express, because a signal reports 1
as soon as any of its collectors succeeded:
- A disk whose SMART probe failed.
fuzzball_node_disk_smart_failingis absent for that device rather than0, andfuzzball_node_disk_smart_statussays why. - A mixed-vendor GPU host where one vendor’s probe failed. The cards that did
report keep their series, and
fuzzball_node_collector_failuresis non-zero.
Alert on fuzzball_node_collector_failures > 0 as well as on the signal
status, or a partly unchecked node reads as fully checked.
The run queue, login session and idle time series are absent unless their
signal (load or sessions) reads 1 – before the first sample, or with
/proc/loadavg or utmp unreadable – rather than publishing a 0 that would
read as an idle, empty node.
Most series are the readings the node’s reporting loop last collected, so they
are at most one report interval old – 30 seconds by default. Collecting per
scrape would fork smartctl for every disk and nvidia-smi on the request
path of an unauthenticated endpoint, worst on exactly the failing hardware that
makes those tools hang. fuzzball_node_signals_age_seconds carries the age: a
climbing age with unchanging values means collection has stalled, which would
otherwise look like healthy, unchanging hardware. The run queue, login
sessions, idle time and disk I/O error counts are the exception: they are kept
in memory or cheap to read, so each scrape reads them live. A scrape does not
count as a heartbeat.
Turn the endpoints off, or move them, in the substrate orchestrate extension configuration. These settings govern both Fuzzball’s exporter and the bundled node_exporter:
binary: fuzzball-substrate-orchestrate
enabled: true
config:
health:
metrics-disabled: false
metrics-port: 9101
metrics-address: ""
node-exporter-disabled: false
| Parameter | Type | Default | Description |
|---|---|---|---|
metrics-disabled | boolean | false | Turn off both Prometheus endpoints – Fuzzball’s exporter and the bundled node_exporter – leaving the heartbeat running |
metrics-port | integer | 9101 | Port Fuzzball’s exporter listens on. node_exporter always uses 9100 |
metrics-address | string | all interfaces | Interface both endpoints bind. A value that cannot be read as a host, such as [] or a lone bracket, keeps both endpoints closed and is logged as an error |
node-exporter-disabled | boolean | false | Do not run the bundled node_exporter, leaving Fuzzball’s exporter running. Set it on a node that runs its own node_exporter |
On upgrade, both endpoints are on by default, unauthenticated, and bound to every interface – port 9101 for Fuzzball’s exporter and port 9100 for node_exporter. Both are new to a node upgrading from a release before they shipped. Fuzzball’s serves the node’s host name, GPU UUIDs, PCI bus IDs and disk device names; node_exporter serves the host’s full inventory of CPUs, memory, filesystems, mount points and network interfaces. That default exists because a monitoring stack scrapes them from off the node, so binding loopback only would make them useless for the purpose they have – but it is worth a decision rather than a default.
Three knobs:
metrics-addressnarrows both binds. Set it to one interface (metrics-address: "10.0.0.42") on a node reachable from beyond the cluster network, or to127.0.0.1to serve only local scrapes.node-exporter-disabled: truekeeps the bundled node_exporter off port 9100.metrics-disabled: truecloses both of Fuzzball’s endpoints.These settings govern only what Fuzzball runs. A node_exporter the site runs itself on 9100 is unaffected by either one, so closing that port is the site’s own configuration.
With proxy-mode service discovery,
metrics-address: "127.0.0.1"is the tightest useful setting: the scrape relay runs on the node and reaches both exporters over loopback, so they stay reachable through the authenticated API and nowhere else. Narrowing it to a single non-loopback interface instead puts both out of the relay’s reach – proxied scrapes offuzzball-nodeandnodethen fail with a connection error naming the port, whiledcgmkeeps working because DCGM Exporter binds every interface by convention. Use direct mode if you need that arrangement, and setmetrics-addressto the address the node registered with (the address in its node ID): that is the address direct mode connects to.Also check for a port clash. 9101 is the HAProxy exporter’s conventional port, and a clash is not fatal here: whichever service starts second logs a warning and loses the port. Grep the extension’s log for
node metrics endpoint unavailable– it names the address it could not bind.
metrics-disabledis separate fromdisabledbecause they answer different questions. Turning off collection makes the node invisible to Fuzzball; turning off the endpoints only stops a monitoring stack scraping them, which is what a site with no Prometheus, or one that objects to the open ports, actually wants.disabledleaves node_exporter running: it serves the host’s metrics, not Fuzzball’s telemetry.
fuzzball_node_disk_io_errors_total counts the failed requests the kernel logs for a
disk, read from the same ring buffer as machine checks. A record is counted
when it reports one of the following:
- A failed request, such as
I/O error, dev sda, sector ...orcritical medium error, dev sda, sector .... Every kernel from 4.18 (RHEL 8) to 6.x prints this record, with or without aprint_req_error:orblk_update_request:prefix. - A failed filesystem block read or write, such as
Buffer I/O error on dev sda1, .... A partition’s records count against its disk.
The count is a floor rather than a complete history. It starts at zero when the extension starts, the kernel rate-limits these lines, and one failed request can log more than one record. Treat any increase as the signal:
increase(fuzzball_node_disk_io_errors_total[15m]) > 0
Errors are counted whether or not the disk is mounted, because the kernel logs them only for requests that actually failed. A disk that fails badly enough to drop off the bus keeps its series. Loop, optical, floppy, device-mapper and md devices are not counted: for device-mapper and md, the physical disk underneath logs its own record for the same failure, and a virtual machine’s empty floppy logs errors whenever it is probed.
fuzzball_node_signal_status{signal="machine_check"} reports whether the ring
buffer is being read. While it is not 1, a disk reading 0 has not been
checked.
fuzzball_node_disk_read_only reports a read-only device only while it backs a
mounted filesystem or swap. Devices that are read-only by design are excluded,
such as config drives, images and virtual CD-ROMs. A device mounted through an
alias is recognised by its device number, such as /dev/disk/by-uuid/...,
/dev/disk/by-id/... or the /dev/root some images report. A disk under LVM,
LUKS or software RAID is mounted through the mapping, not by itself, so it
reads as unmounted. A btrfs filesystem mounted through an alias also reads as
unmounted, because the mount table reports an anonymous device number for it.
A btrfs filesystem mounted by its kernel device name, such as /dev/sdb1,
still matches.
The fuzzball-substrate-orchestrate package, and the orchestrate substrate
container images, ship the Prometheus
node_exporter – version
v1.12.1, pinned and checksum-verified at build time, so nodes never download
it. It is installed at /usr/libexec/fuzzball-substrate/node_exporter, with
its license and notice under
/usr/share/doc/fuzzball-substrate-orchestrate/node_exporter/.
The extension runs it as root on port 9100, on the interface
metrics-address names, restarts it with backoff if it exits, and stops it
when the extension stops. This is what the node exporter in the
discovery inventory and the bundled stack’s node scrape
job read, so on an upgraded node that target now answers rather than reading
down.
It does not run where it cannot, or should not:
- Port 9100 already taken. The bundled one stands by rather than fighting
for the port: the extension logs
port in use; standing by -- the bundled node_exporter starts if it frees, checks again every two minutes, and starts it once the port is free. Whatever holds the port is what gets scraped, so check what it is. An address that cannot be bound at all is handled the same way, logged as a warning. - Binary absent. A node on an older package, or an image built without it,
runs none, and its
nodetarget reads down as before. - Switched off.
node-exporter-disabledormetrics-disabled– see the configuration above.
A node that runs its own node_exporter should setnode-exporter-disabled: true. Nothing orders the substrate service against a site’snode_exporterservice at boot, so whichever starts first holds port 9100. If the bundled one wins, the site’s fails to bind – and the bundled one runs with the collector choices below, without the site’s textfile metrics. Standing by only helps when the site’s exporter is already running.
It runs with node_exporter’s default collectors, except where a Fuzzball compute node would make them churn:
- The textfile collector is off.
- Mounts beneath
/var/lib/fuzzball/and/run/fuzzball-substrate/– the per-job container, image and volume mounts – and beneath anyerofsmount-<id>directory, where Fuzzball’s image server mounts, are excluded. A filesystem mounted at/var/lib/fuzzballis still reported. - The
loandveth*interfaces are excluded. Thefb0bridge, which carries all container traffic in aggregate, is kept.
node_exporter itself logs only warnings and errors. The extension relays each line to its own log at the same level – error for an error, warning otherwise – so a site filtering the extension’s log at warning still sees them.
How busy a node is, as opposed to whether it is healthy. Each item an LSF
lsload user expects maps to a series and a panel on the Fuzzball Node
Resources dashboard:
| Item | Series | Source | Panel |
|---|---|---|---|
| Fuzzball status | fuzzball_node_status, fuzzball_node_hold | Cluster endpoint | State, Fuzzball status, Holds |
CPU utilization (ut) | node_cpu_seconds_total | node_exporter | CPU utilization |
15-second run queue (r15s) | fuzzball_node_run_queue_15s | Fuzzball exporter | Run queue (15 s) |
1- and 15-minute load average (r1m, r15m) | node_load1, node_load5, node_load15 | node_exporter | Load average |
Page rate (pg) | node_vmstat_pgpgin, node_vmstat_pgpgout, node_vmstat_pswpin, node_vmstat_pswpout, node_vmstat_pgmajfault | node_exporter | Paging |
Free memory (mem) | node_memory_MemAvailable_bytes, node_memory_MemFree_bytes | node_exporter | Memory |
Filesystem utilization (tmp, for every mount) | node_filesystem_avail_bytes, node_filesystem_size_bytes | node_exporter | Filesystem utilization |
Swap utilization (swp) | node_memory_SwapFree_bytes, node_memory_SwapTotal_bytes | node_exporter | Swap utilization |
Idle time (it) | fuzzball_node_idle_seconds | Fuzzball exporter | Idle time |
Login sessions (ls) | fuzzball_node_login_sessions | Fuzzball exporter | Login sessions |
| Network I/O per interface | node_network_receive_bytes_total, node_network_transmit_bytes_total | node_exporter | Network I/O |
| Number of GPUs | fuzzball_node_gpu_hardware_xid, one series per GPU | Fuzzball exporter | GPUs |
The State panel folds status and holds into one LSF-style host state. The most severe applies:
| State | When | LSF |
|---|---|---|
| Down | Status offline | unavail |
| Unknown | Status unknown | unreach |
| Not ready | Status not ready | closed |
| Locked | Any cordon, evict or deprovision hold | lockU |
| OK | None of the above | ok |
A held node that is also offline reads Down, because that is what needs acting on first; the Holds panel still lists the hold.
Two of these are not what their names might suggest:
- The 15-second figure is a run-queue average, not a Linux load average.
It follows LSF’s
r15s: runnable tasks only, so a node blocked on a dead NFS server reads idle here and busy innode_load1. Compare it against the core count, which the panel draws beside it. - Idle time is terminal input. With no session that has a terminal – a headless node nobody logs into, or only graphical logins – it reads as time since the last terminal input the exporter observed, or since boot if it has observed none, as after an extension restart. It answers “is a person at this machine”, not “is the machine doing work”.
A compute node is not a Kubernetes object, so no Kubernetes service discovery can see it. On a cluster that provisions nodes for individual workflows, a static target list is also useless – a node may exist for two minutes.
Fuzzball serves the target list itself, in Prometheus
http_sd_config
form:
GET /v4/nodes/-/sd_config?exporter=nodeexporter is required, and names one exporter from the cluster’s inventory.
Run one Prometheus job per exporter: a single job covering all three would
stamp one job label on series from three different exporters, and anything
selecting by job would then be selecting by nothing.
The response is a Prometheus target list:
[
{
"targets": ["orchestrate.example.com"],
"labels": {
"instance": "cn-042:9100",
"node": "cn-042",
"__param_node": "cn-042",
"__param_exporter": "node",
"__meta_fuzzball_node_id": "10.0.0.42/2375",
"__meta_fuzzball_definition": "gpu-a100",
"__meta_fuzzball_cluster": "lab1",
"__meta_fuzzball_dynamic": "true",
"__meta_fuzzball_exporter": "node",
"__meta_fuzzball_sd_mode": "proxy"
}
}
]
The __meta_fuzzball_* labels mirror fuzzball_node_info field for field, so
a relabel rule can promote any of them and the result joins Fuzzball’s own
series without a translation table. They are discovery labels: Prometheus drops
them from the final series unless a rule keeps one, so none of them inflate
cardinality by default.
Nodes the control plane has written off as offline are omitted. A failed
scrape is a problem worth alerting on, and a node already known to be gone is
not news – fuzzball_node_status reports that with no scrape involved.
Only your site knows whether Prometheus can open a connection to the compute network at all, so the addressing is configuration:
nodeMetrics:
serviceDiscovery:
mode: proxy # proxy (default) | direct
exporters:
- name: node
port: 9100
- name: dcgm
port: 9400
requires: nvidia-gpu
- name: fuzzball-node
port: 9101
In proxy mode every target is the Fuzzball API, and the node and exporter
travel as scrape parameters. Prometheus turns the __param_* labels above into
a query string, and the API relays the scrape to the node over the connection
the node already holds – so nothing needs a route into the compute network,
and no node port needs to be reachable from off the node. Pair it with
metrics-address: "127.0.0.1" to keep the ports Fuzzball opens, 9100 and 9101,
off the network entirely.
In direct mode the target is the node’s own address – the IP address it registered with, as shown in its node ID – and Prometheus connects to it. Compute host names do not need to resolve from where Prometheus runs. Lower latency and no fan-in through the control plane, but it requires a route to that address.
instance is the node’s host name and exporter port in both modes. That
is deliberate: instance is the identity of a target in every dashboard and
recording rule you have, and pointing it at the proxy would make every series
in the fleet share one instance – then changing the mode back would relabel
all of them again.
Configuring exporters replaces the defaults rather than adding to them,
so a site that does not run DCGM can say so by listing only what it does run.
requires: nvidia-gpu omits nodes with no NVIDIA GPU from that exporter’s
target list entirely, which is what lets a rule read an absent dcgm target as
absence rather than as failure.
For the fuzzball-node exporter the node’s reported port wins over the
configured one. It is the only port Fuzzball actually knows: the extension may
have been configured elsewhere, failed to bind the port it was given – 9101 is
also the HAProxy exporter’s conventional port – or been switched off. A node
that has reported no port is omitted, because
publishing the configured number would produce a target that fails every scrape
and reads as broken hardware rather than as a disabled endpoint.
| Parameter | Type | Default | Description |
|---|---|---|---|
serviceDiscovery.mode | string | proxy | proxy to scrape through the API, direct to scrape nodes |
exporters[].name | string | – | Exporter name, used as the ?exporter= value |
exporters[].port | integer | – | Port the exporter listens on |
exporters[].requires | string | – | nvidia-gpu to restrict the exporter to GPU nodes |
In proxy mode Prometheus scrapes:
GET /v4/metrics/node?node=cn-042&exporter=nodeThe API resolves exporter to a port from the inventory above – never from the
request, so the endpoint cannot be used to reach an arbitrary port – and asks
the node for that exporter’s output over its existing connection. The node
scrapes its own exporter over loopback and compresses the response before
sending it back, which is what keeps a large node_exporter payload inside the
message limit on that hop.
node is the node’s host name, which is what the discovery response puts in
__param_node; the endpoint accepts a node id as well when you call it by
hand.
One caveat. A host name does not uniquely identify a node – one host can run
more than one substrate, and a moved substrate leaves its old record behind
until a ghost sweep reaps it. Where two records share a host name, the relay
picks one deterministically: a record that is not offline, then the lowest node
id. So a job that labels its series with node_id must also route by it, or
both substrates’ targets reach the same record and one of them is labelled
with the other’s data. The bundled jobs, and the
example job, set __param_node from
__meta_fuzzball_node_id for exactly this.
Compute nodes share a single JetStream credential, so the relay trusts that nodes are not adversarial to one another: a node holding that credential could answer a scrape, or a debug request, addressed to another node. The exchange goes through the cluster’s own NATS server, so separating nodes on the network does not prevent it. Add a machine as a compute node only if you would trust it to report for the others.
Proxy mode always relays an exporter’s plain-text exposition format. The
scrape’s Accept header is not forwarded to the node, so OpenMetrics and
protobuf exposition – and with them exemplars and native histograms – are
available only in direct mode.
Upgrade compute nodes before relying on proxy mode. The scrape relay runs
in the node’s fuzzball-substrate-orchestrate. A node still on an older
version is listed by service discovery but cannot answer a proxied scrape, and
its node and dcgm targets fail with an error saying nothing on the node is
serving scrape requests. That is expected while a fleet upgrades: those
targets recover as each node is upgraded.
Each node’s scrape succeeds or fails on its own: a node that cannot be reached,
or whose exporter answered an error, produces a non-200 for that target. This
matters more than it sounds. Every proxy-mode target is the same address, so if
a missing node answered 200 with an empty body, up would read 1 for every
node in the fleet forever and every absence rule would be reading a constant.
Both the discovery poll and the scrape are authenticated API calls, so both need a bearer token.
The token’s identity needs the cluster
scrape_node_metricspermission. Service discovery is open to any authenticated user, but the scrape proxy is not: it relays live exporter output for nodes other tenants may be running on.Cluster administrators hold the permission, so a cluster administrator user’s credential works. There is currently no API to grant
scrape_node_metricsto another identity: the bundled stack’s own scoped identity is provisioned automatically, but a Prometheus you run yourself has to authenticate as a cluster administrator. Refresh that credential the way the warning below describes.
Set refresh_interval explicitly – Prometheus defaults http_sd_config to
60 seconds, and a node that lives two minutes would be scraped once, or not at
all. The example below is a proxy-mode job; see the warning after it for
what changes in direct mode:
scrape_configs:
- job_name: fuzzball-node-node
scrape_timeout: 8s
authorization:
credentials_file: /var/run/fuzzball-api/token
tls_config:
ca_file: /etc/fuzzball/ca/ca-certificates.crt
http_sd_configs:
- url: https://orchestrate.example.com/v4/nodes/-/sd_config?exporter=node
refresh_interval: 20s
authorization:
credentials_file: /var/run/fuzzball-api/token
tls_config:
ca_file: /etc/fuzzball/ca/ca-certificates.crt
relabel_configs:
# Scrape only targets discovery built for this job's mode. Without it,
# a job carrying the credential will happily scrape direct-mode targets
# -- see the warning below.
- source_labels: [__meta_fuzzball_sd_mode]
regex: proxy
action: keep
# Name the substrate, not just the host -- see below.
- source_labels: [__meta_fuzzball_node_id]
target_label: node_id
# Proxy mode only: route the scrape to that same substrate.
- source_labels: [__meta_fuzzball_node_id]
target_label: __param_node
Note what the job does not set. Service discovery emits __scheme__ and
__metrics_path__ on every target, and Prometheus honours those over the
job’s own settings, so the URL is always right for the mode in force. Stating
them yourself is how you get a job that is correct for one mode and wrong for
the other: proxy mode addresses this API over HTTPS at a versioned path,
direct mode addresses a node’s own exporter over plain HTTP at /metrics.
The credential is the part that does not follow the mode, which is why the
keep rule above matters. Prometheus has no per-target credential, so
authorization is a property of the job; if discovery is serving direct-mode
targets and the job still carries the token, the URL will be correct and the
token will go to a node exporter anyway.
ca_file is needed wherever the API’s certificate comes from a private
issuer, which is the usual Fuzzball arrangement; drop it if your ingress
certificate is publicly trusted.
In direct mode, do not putauthorizationon the scrape – only on thehttp_sd_configsentry – and set thekeeprule’sregextodirect. Prometheus has no per-target credential, so a job-level one is sent to every target, and in direct mode those targets are node exporters. That would hand a Fuzzball control-plane token to every compute node in the fleet, over plain HTTP, for an endpoint that ignores it – and because it ignores it, every scrape would succeed and nothing would report the leak.
Service discovery emits instance and node as target labels; the second
relabel rule adds node_id. node is the host name, and a host can run more
than one substrate, so without node_id an alert over an exporter series names
a host and leaves the hold receiver to pick one of its substrates. With it,
each series names its substrate. The shipped rules rely on it. A host running
two substrates is then scraped once per substrate, so aggregate host-level
counters by (node, node_id) or take the max across node_id – never a sum by
node alone. Promote no other __meta_fuzzball_* label unless a rule selects
on it: the shipped rules reach definition by joining to fuzzball_node_info.
For the fuzzball-node exporter’s job only, set honor_labels: true.
Fuzzball’s own exporter already labels every series with node, and service
discovery attaches node as a target label too. With Prometheus’s default
honor_labels: false, a scraped label that collides with a target label is
renamed exported_node – even when the two values are identical – so every
series would carry a redundant exported_node. The node and dcgm exporters
do not label by node, so their jobs leave it unset.
scrape_timeout is below Prometheus’s 10-second default because a proxied
scrape is a round trip to the node on top of the API request, and a sample that
arrives after the deadline is discarded anyway.
Tokens expire, so something has to refresh the credentials file. Prometheus re-reads
credentials_fileon every request, so a process that writes a fresh token beside it is enough – it needs no reload. Write through a temporary file and rename it into place; a half-written file read mid-write sends a truncated token and fails an otherwise healthy scrape.
fuzzball context tokenprints the active context’s token for exactly this. The bundled stack does this for you – see Bundled Monitoring Stack.
Fuzzball deploys Prometheus, Alertmanager and Grafana alongside the cluster by default, preconfigured against the cluster’s own metrics and every compute node’s exporters. It ships with the rule pack, the Alertmanager routes that turn alerts into node holds, and three dashboards – the same configuration Fuzzball publishes for sites that run their own monitoring.
The stack is what closes the node-health loop: Fuzzball publishes facts about its nodes, Prometheus judges them against the rules, and Alertmanager calls back to place and release holds. A cluster without it publishes facts nobody acts on, unless a site’s own monitoring does the judging.
Upgrading. The stack used to be opt-in. A cluster upgraded to this release that never set
monitoring.enablednow gets the stack, in thefuzzball-monitoringnamespace, with a 20Gi Prometheus volume and a small Alertmanager volume. To keep the old behaviour, setmonitoring.enabled: falsein theFuzzballOrchestrateresource before you upgrade the operator.
- Storage. The volumes use
monitoring.storageClassName, or the cluster’s default storage class when that is unset. On a cluster with neither, a resource that never setmonitoring.enabledormonitoring.storageClassNamedeploys without the stack, and the operator records aMonitoringNotDeployedwarning event; setmonitoring.storageClassName, ormonitoring.storageSize: ""to run the stack onemptyDir, to bring it in. A resource that does ask for the stack –monitoring.enabled: true, or a named class that does not exist – is refused with a configuration error namingspec.monitoring.storageClassName.- Images. The stack’s images follow the cluster’s
imagesettings like every other component, so withimage.exclusivethey are pulled from your own repository, under their last path element. Mirror them before upgrading:quay.io/prometheus/prometheus:v2.45.0,quay.io/prometheus-operator/prometheus-config-reloader:v0.66.0,quay.io/prometheus/alertmanager:v0.25.0anddocker.io/grafana/grafana:10.0.1.- Existing prometheus-operator CRDs. Earlier releases installed the
monitoring.coreos.comCRDs with the stack. This release does not, and leaves any it installed in place rather than deleting them – and with them every ServiceMonitor or PrometheusRule in the cluster.- Alertmanager’s volume. On a cluster that already ran the stack, setting
monitoring.storageClassNameormonitoring.storageSize: ""changes Alertmanager’s volume claim template, which Kubernetes does not allow on an existing StatefulSet. The operator deletes thefuzzball-prometheus-alertmanagerStatefulSet first, leaving its pod and volume in place, and the upgrade recreates it. An existing volume keeps the storage class it was created with.
A site that runs its own monitoring turns the stack off and adopts the published configuration instead:
spec:
monitoring:
enabled: false
The stack can also run beside an existing kube-prometheus-stack. It installs no CRDs, no DaemonSet, and no host ports, and everything it creates is named after its own Helm releases in its own namespace.
Nothing in the stack is exposed outside the cluster. Reach each UI with a port forward:
$ kubectl -n fuzzball-monitoring port-forward svc/fuzzball-grafana 3000:80
$ kubectl -n fuzzball-monitoring port-forward svc/fuzzball-prometheus-server 9090:80
$ kubectl -n fuzzball-monitoring port-forward svc/fuzzball-prometheus-alertmanager 9093:9093
Unless you set monitoring.grafana.adminSecretRef, Grafana’s admin user is
admin and its generated password is stored in the admin-password key of the
fuzzball-grafana Secret:
$ kubectl -n fuzzball-monitoring get secret fuzzball-grafana \
-o jsonpath='{.data.admin-password}' | base64 -d
To choose the credentials yourself, create a Secret in the monitoring namespace
holding the user name under admin-user and the password under a key you name,
and point monitoring.grafana.adminSecretRef at it. The Secret must be in the
monitoring namespace, where Grafana reads it; one named elsewhere is refused.
To publish Grafana under a hostname, enable its ingress through
monitoring.grafana.valuesOverride – see the example in the
CRD reference.
Retention, storage, the alert webhook and the Grafana admin Secret have their
own fields. Anything else either chart supports is set through a
valuesOverride per component – monitoring.prometheus,
monitoring.alerting and monitoring.grafana – which is merged over the
generated values the way Helm layers a values file. The
CRD reference
lists every field.
The rule pack keeps the group name fuzzball-node-health, so a site that has
already silenced or routed that group keeps working. Three of its rules carry
an fb_action label, so on a default install they place holds; see
Placing Holds From Alerts for what each label does.
| Alert | Fires when | Action |
|---|---|---|
FuzzballNodeUnknown | A node has not reported for 10 minutes | Advisory |
FuzzballNodeDevicePluginFailed | A device plugin on a node has been in the failed state for 5 minutes | Advisory |
FuzzballNodeAttributedJobFailures | Five or more jobs have stopped responding on one node in a day | Cordon |
FuzzballNodeHealthMetricsAbsent | No node series at all – no leader, or scraping is broken | Advisory |
FuzzballNodeMemoryErrors | A node logs more than 1,000 corrected memory errors in a day | Cordon |
FuzzballNodeGPUUncorrectableECC | A GPU reports an uncorrectable ECC error | Evict, released by hand |
FuzzballNodeHoldCeilingReached | Fuzzball refused a monitoring hold because the node’s pool is at its ceiling | Advisory |
FuzzballNodeMemoryErrors reads the EDAC counters Fuzzball’s node exporter
publishes, so it works on a default install; on a node without ECC memory, or
with the EDAC modules unloaded, there is nothing to read and it stays quiet.
FuzzballNodeGPUUncorrectableECC reads DCGM, which Fuzzball does not deploy. On
a cluster without it the rule never fires, and that is absence rather than
failure.
FuzzballNodeUnknown and FuzzballNodeDevicePluginFailed ship with the
product rather than being left to a site’s own rules, because what they report
is only visible to Fuzzball.
A failed scrape is not the same signal. The stack does scrape each node’s
exporters, so a silent host does produce an up series that drops – but a
failed scrape says only that a target could not be read, which a network
fault, the proxy, or an expired credential all produce too. FuzzballNodeUnknown
says the specific thing: Fuzzball stopped hearing from the node. In direct mode
the two come apart entirely, since the node’s exporter answers Prometheus
directly and keeps reporting up while the extension behind it is gone.
Nothing notices a failed device plugin at all. A node with one is otherwise reporting normally, so no exporter has anything to say about it. The plugin alert names the plugin in its labels, so two broken plugins on one node are two alerts.
FuzzballNodeHealthMetricsAbsent exists because every other rule goes quiet
when the control plane publishes nothing, and an absent series must never read
as a healthy fleet.
Grafana is provisioned with three dashboards in a Fuzzball folder, linked to each other:
- Fuzzball Fleet Health – nodes by status and by hold effect, who placed the holds in force, hold churn over time, the node-health alerts firing, the nodes that have gone silent, and the nodes with the most attributed job failures or a failed device plugin. It keeps the address of the earlier Fuzzball Node Reliability dashboard, so existing links land here.
- Fuzzball Node Health – one node, chosen from a list: its status, its holds and their history, the alerts firing on it, its device plugins, and the exporter series behind the hardware alerts – memory and GPU errors, temperature, machine checks, disk I/O errors, disk faults (read-only or failing SMART), whether each exporter can be scraped, and signal coverage: whether each of Fuzzball’s collectors, and SMART on each disk, produced a reading, so a zero on an error panel can be told from an unchecked one.
- Fuzzball Node Resources – how busy nodes are, as opposed to whether they are healthy. A fleet row (nodes by state, nodes not OK, busiest by CPU, longest run queues) and a per-node row: state, Fuzzball status, holds, running allocations, GPUs, idle time, login sessions, CPU utilization, run queue, load average, paging, memory, filesystem and swap utilization, and network I/O. See Node Resource Metrics for what each panel reads and how State is derived. The panels that read node_exporter need it scraped – the bundled one is, by default.
A hold’s id and reason are not metrics. Read them with
fuzzball node hold list and fuzzball node events.
The bundled stack scrapes both endpoints: Fuzzball’s own pods, and one job per compute-node exporter using the service discovery above. You do not have to configure either.
If you set nodeMetrics.serviceDiscovery.mode to direct, set
monitoring.nodeMetricsMode: direct in the operator configuration to match.
The operator cannot read central config, and the two modes need different
scrape jobs – direct mode’s targets are unauthenticated node exporters, so
the Fuzzball credential is left off the scrape.
A mismatch is caught rather than tolerated: each job keeps only the targets
advertising the mode it was built for, so a disagreement leaves the job with
no targets – visible immediately on Prometheus’s target page. That
interlock matters most in one direction. Left at proxy while central config
serves direct, the job would otherwise carry the Fuzzball bearer to every
node’s own plaintext exporter, which ignores the header – so every scrape
would appear to succeed while a control-plane credential went out in the clear
across the compute network.
It handles its own credential. A sidecar beside Prometheus exchanges its projected Kubernetes ServiceAccount token for a Fuzzball token every half hour and writes it where Prometheus reads it, so there is no password anywhere in the stack’s configuration and nothing to rotate by hand.
That token belongs to a scoped fuzzball-monitoring identity, not to a
cluster administrator. It holds one permission, scrape_node_metrics, to
scrape compute nodes through the control plane, and carries no password, so the
ServiceAccount exchange is the only way to obtain it. It cannot place holds:
Alertmanager reaches the hold receiver with a client certificate instead – see
What the Bundled Stack Configures. On a fresh install the
identity is created shortly after the local organization is, so the stack’s
first scrapes may be refused for a minute or two before they start succeeding.
The stack validates the Fuzzball API’s certificate against the cluster’s CA
bundle only where the operator creates that bundle: when it deploys
trust-manager and issues internal certificates itself. Without
trust-manager, or with an external internal issuer
(tls.internalIssuer.external), there is no bundle for the stack to read, and
it trusts public certificate authorities only – so the API’s ingress
certificate must be publicly trusted. With a privately issued certificate in
that configuration, scrapes fail with a certificate error.
The bundled jobs cover the three built-in exporters. If you replace the inventory withnodeMetrics.exporters, the operator cannot see that – it does not read central config – so the shipped jobs will still name the built-ins, discovery will refuse the ones you removed, and the ones you added will not be scraped. Add jobs for your own exporters throughextraScrapeConfigsinmonitoring.prometheus.valuesOverride– a string holding the job list, see the example in the CRD reference – or keep the built-in names.
The ServiceAccount is named fuzzball-monitoring in the monitoring namespace.
The control plane accepts its projected token alongside the one Fuzzball’s own
in-cluster jobs use, and no others – and only on a cluster that runs the
bundled stack, so a cluster that opted out grants it nothing.
The rule pack, the Alertmanager routing and the dashboards are published as files, for a site that runs its own Prometheus, Alertmanager and Grafana. Each file is exactly what the bundled stack deploys in this release:
| File | What it is | Adjust before use |
|---|---|---|
| fuzzball-node-health-rules.yaml | Prometheus rule file | Nothing. Load it with rule_files |
| fuzzball-alertmanager.yaml | Alertmanager configuration: the hold routes and receivers, and a catch-all route to your own receiver | Replace alertmanager-receiver.example.com:9443 with the external receiver’s hostname and port and https://alerts.example.com/hook with your receiver. Point cert_file, key_file and ca_file at the client certificate, key and receiver CA the operator issues – see Using Your Own Alertmanager |
| fuzzball-fleet-health.json | Grafana fleet dashboard | Nothing. Import it and pick your Prometheus datasource |
| fuzzball-node-health.json | Grafana node dashboard | Nothing. Import it and pick your Prometheus datasource |
| fuzzball-node-resources.json | Grafana node resources dashboard | Nothing. Import it and pick your Prometheus datasource |
The rules need the scrape jobs above in place, and the hardware rules need the
exporters they read. If you merge the Alertmanager configuration into an
existing one, keep its hold routes ahead of your own and keep
continue: true on them, or an alert that places a hold stops notifying you.
Fuzzball accepts Alertmanager notifications and turns them into node holds. Prometheus decides what a node’s metrics mean; Fuzzball provides the facts and the hooks, and the judgement stays in rules you can read and change.
POST https://<receiver>:9443/v4/receivers/alertmanager
The endpoint takes Alertmanager’s webhook payload, version 4. It authenticates
by mutual TLS: a caller must present a client certificate issued by one of
the cluster’s monitoring client CAs, which the operator creates for this purpose
and nothing else. There is one in each namespace an Alertmanager’s credential is
issued into – the bundled stack’s namespace, and the Fuzzball namespace when you
enable an external receiver – each behind a
cert-manager Issuer that signs only for certificates in its own namespace, so
creating a certificate anywhere else grants nothing. A connection without a
client certificate – or with one from any other CA, including the one that
signs the cluster’s server certificates – is refused during the TLS handshake,
before any request is read. There is no bearer-token
route: the public API answers this path with a 403 naming where the receiver is
served, so a cluster-admin token cannot place a monitoring hold.
The receiver is served by the orchestrator on its own port, 9443, not through
the cluster ingress: the ingress terminates TLS, so a client certificate would
never reach the receiver. Inside the cluster it is
fuzzball-orchestrator.fuzzball.svc:9443; outside, it is published only when you
ask for it – see Using Your Own Alertmanager.
Holds it writes carry the monitoring source, which no other entry point can
write and which this one cannot avoid writing – the source comes from the
endpoint, not from the request, so a certificate issued to your monitoring stack
cannot place a hold that reads as an operator’s.
A rule controls what happens to a node through two labels:
| Label | Effect |
|---|---|
fb_action: cordon | Place a cordon hold: no new work is placed on the node |
fb_action: evict | Place an evict hold: no new work, and what is running is taken off |
fb_action absent | Advisory. The alert notifies and holds nothing |
fb_release: manual | The hold is not released when the alert resolves |
The alert names the node with either node_id or node, and its summary
annotation becomes the hold’s reason – which is then recorded against any work
the hold displaces, so a user whose job moved is told what was wrong with the
node.
| Label | Names |
|---|---|
node_id | The substrate, uniquely. Preferred when both are present |
node | The host name. Used when node_id is absent |
Prefer node_id where your exporter can supply it. A host can run more than
one substrate, and they share a host name – so an alert carrying only node
on such a host names two records, and the hold lands on one of them. Fuzzball
resolves that consistently, so a repeat and its resolve reach the same record,
but it is still a node chosen for you rather than the one that is faulty. The
rules Fuzzball ships carry node_id for this reason – including the two over
third-party exporters, because the bundled scrape jobs promote it onto every
exporter series. If you run your own Prometheus, add the same relabel rule; see
the scrape job example. A host-wide fault, such as a
failing DIMM or GPU, then holds each substrate on that host.
Alertmanager’s own alert fingerprint becomes the hold id, so the notification that resolves an alert releases exactly the hold that the firing notification placed. A firing alert repeats for as long as it fires; each repeat refreshes the hold rather than stacking a second one – on the same record it was placed on, even if a node named by host name has since gone offline, and a resolve releases it there.
A hold id belongs to the source that placed it. Placing a hold through the
admin API with an explicit hold_id that a firing alert already uses is
refused for that node, so choose another id – fuzzball node hold add
generates its own, so it never collides. And a resolved alert only ever
releases the monitoring hold it placed. A summary longer than 1,024 bytes is truncated, ending in ...,
rather than refused, so a long description never stops a hold being placed.
fb_action: deprovisionis refused, and reported as refused rather than as an unrecognised value. A deprovision hold ends with the instance destroyed, and a rule firing on the wrong series would take the hardware with it. Deprovisioning a node stays an operator’s decision – you can still do it by hand after reading the alert.
Nothing is held until a rule says so, and three of the seven shipped rules do
say so. FuzzballNodeAttributedJobFailures and FuzzballNodeMemoryErrors carry
fb_action: cordon and read series Fuzzball publishes itself, so on a default
install they cordon a node that accumulates five job failures in a day, or more
than 1,000 corrected memory errors. FuzzballNodeGPUUncorrectableECC also
carries an fb_action, but reads DCGM series that Fuzzball does not deploy, so
it stays inert until you install DCGM Exporter.
You opt a rule out by dropping its fb_action label, which leaves it
notifying and holding nothing.
A rule labelled fb_release: manual routes to a second receiver that sends no
resolved notification. Fuzzball never hears the alert resolve, so the hold it
placed stays until an administrator releases it with
fuzzball node hold release <fingerprint> --source monitoring. The --source
flag is required: a release defaults to administrator holds only, so without it
the command reports the hold as not released and the node stays cordoned.
Use it where recovery of the metric does not mean recovery of the node – a GPU that threw an uncorrectable ECC error, say. The error stops being reported; the GPU is still suspect.
If you deploy the bundled stack, this is already wired. The operator configures
two Alertmanager receivers pointed at the receiver’s in-cluster address,
https://fuzzball-orchestrator.fuzzball.svc:9443/v4/receivers/alertmanager, and
routes alerts carrying fb_action to them. Alertmanager authenticates with a
client certificate cert-manager issues into the monitoring namespace
(fuzzball-alertmanager-client-tls) from that namespace’s own monitoring client
CA, and verifies the receiver against the CA in
the fuzzball-alertmanager-receiver-ca ConfigMap beside it. cert-manager renews
the certificate before it expires and Alertmanager picks the renewal up without a
restart, so there is nothing to rotate by hand. Alertmanager holds no Fuzzball
token.
Alerts still reach whatever destination you already route to, because a final catch-all route follows the hold routes – taking a node out of service is not a reason to stop telling anyone about it.
Setting monitoring.alerting.configOverride replaces the whole routing
configuration, hold receivers included. If you override it, wire the receivers
yourself, keeping the tls_config shown in the
published configuration: the certificate and CA are
still mounted at /etc/fuzzball/alertmanager-client/ and
/etc/fuzzball/alertmanager-receiver-ca/.
An Alertmanager the operator does not deploy reaches the receiver through an
external endpoint the operator publishes on request, with a client certificate
the operator issues for it. Enable it in the FuzzballOrchestrate resource –
it works whether or not the bundled stack is enabled:
spec:
monitoring:
enabled: false # optional; the bundled stack may stay on as well
alerting:
externalReceiver:
enabled: true
hostname: alertmanager-receiver.example.com
# serviceType: NodePort # default LoadBalancer
# annotations: {} # e.g. request an internal load balancer
The operator then creates, in the Fuzzball namespace:
| Resource | What it is |
|---|---|
Service fuzzball-alertmanager-receiver | Publishes the receiver on port 9443. A LoadBalancer by default. Point hostname’s DNS at its address |
Secret fuzzball-external-alertmanager-client-tls | The client certificate (tls.crt) and key (tls.key) your Alertmanager presents |
ConfigMap fuzzball-alertmanager-receiver-ca | The CA (ca.crt) that signs the receiver’s certificate, for your Alertmanager’s ca_file |
The client certificate is issued by the Fuzzball namespace’s own monitoring
client CA, whose key is in the Secret fuzzball-monitoring-client-ca beside it.
Anyone who can read Secrets or create cert-manager Certificate resources in
the Fuzzball namespace can therefore obtain a receiver credential; restrict that
access as you would the client certificate itself.
hostname defaults to alertmanager-receiver.<domain>; the receiver’s
certificate is issued for it, so your Alertmanager must use that name. Copy the
credentials to your Alertmanager:
kubectl -n fuzzball get secret fuzzball-external-alertmanager-client-tls \
-o jsonpath='{.data.tls\.crt}' | base64 -d > fuzzball-client.crt
kubectl -n fuzzball get secret fuzzball-external-alertmanager-client-tls \
-o jsonpath='{.data.tls\.key}' | base64 -d > fuzzball-client.key
kubectl -n fuzzball get configmap fuzzball-alertmanager-receiver-ca \
-o jsonpath='{.data.ca\.crt}' > fuzzball-receiver-ca.crt
The client certificate is valid for a year and cert-manager renews it 30 days before it expires. Copy the renewed Secret to your Alertmanager when it does – or sync it with your own tooling. Alertmanager re-reads the files when they change, so no restart is needed.
Then merge the routes and receivers entries below into your existing routing
tree, keeping your own root receiver:
route:
# Your existing default receiver. Alertmanager refuses to load a
# configuration whose root route has none.
receiver: your-existing-receiver
routes:
- matchers: [ 'fb_action =~ ".+"', 'fb_release = "manual"' ]
receiver: fuzzball-holds-manual
group_by: ['alertname', 'node', 'node_id']
repeat_interval: 15m
continue: true
- matchers: [ 'fb_action =~ ".+"', 'fb_release != "manual"' ]
receiver: fuzzball-holds
group_by: ['alertname', 'node', 'node_id']
repeat_interval: 15m
continue: true
# Last, and without continue. This is what keeps your own notifications
# working -- see below.
- receiver: your-existing-receiver
receivers:
- name: fuzzball-holds
webhook_configs:
- url: https://alertmanager-receiver.example.com:9443/v4/receivers/alertmanager
send_resolved: true
http_config: &fuzzball_tls
tls_config:
cert_file: /etc/alertmanager/fuzzball/fuzzball-client.crt
key_file: /etc/alertmanager/fuzzball/fuzzball-client.key
ca_file: /etc/alertmanager/fuzzball/fuzzball-receiver-ca.crt
- name: fuzzball-holds-manual
webhook_configs:
- url: https://alertmanager-receiver.example.com:9443/v4/receivers/alertmanager
send_resolved: false
http_config: *fuzzball_tls
If you reach the receiver by an address other than hostname – a node IP with
serviceType: NodePort, say – add server_name: <hostname> to the
tls_config so the receiver’s certificate still verifies.
Five details in that configuration are load-bearing:
- The two hold routes are mutually exclusive rather than ordered, so both
can carry
continue: true.fb_release != "manual"matches an alert that omits the label entirely, which is the common case. Ordering alone would not work: withcontinue: trueon the first route, the second is still evaluated. - The trailing catch-all route is what keeps your existing notifications
working, and it is not the same thing as the root route’s
receiver. Alertmanager falls back to a route’s own receiver only when none of its children matched, andcontinue: truemeans “keep evaluating siblings” – not “also deliver to the parent”. Without that last route, an alert carryingfb_actionmatches a hold route and is delivered there only, so you would silently stop being told about exactly the alerts that took a node out of service. send_resolvedis set explicitly on both hold receivers, because it is the only difference between them.- Grouping is by
nodeandnode_id, notinstance.instanceis per exporter, so a node running three exporters would otherwise be held three times for one fault. Both node labels are needed because the shipped rules carry different ones:FuzzballNodeAttributedJobFailurescarries onlynode_id, and grouping onnodealone would put every node’s alert from it into one fleet-wide group. repeat_intervalon the hold routes is deliberately short. A hold’s refresh timestamp only advances when a notification arrives, and that timestamp is what tells you a still-firing fault from an integration that has stopped talking. Inheriting a typical 12-hour notification cadence would make every live hold look abandoned.
If notifications fail their handshake, the orchestrator logs each refused
connection (TLS handshake error) – usually a client certificate that was not
copied, has expired, or was issued for another cluster.
The receiver is per-cluster. A hold is written to the node store of the cluster the node belongs to, and two clusters can name a node the same thing, so point each member cluster’s Alertmanager at that cluster’s own receiver. A federate serves no receiver.
Holds stay until the alert that placed each one resolves, so a rule that
misfires across a whole pool – a bad threshold, an exporter reporting nonsense
after an upgrade – does not heal itself when the rule is fixed. To keep that
from taking a pool out of service, monitoring holds may cover at most 20% of a
provisioner definition’s nodes at once. Past that, Fuzzball refuses the next
monitoring hold in the pool and raises FuzzballNodeHoldCeilingReached, naming
the definition.
- Nodes are counted, not holds. A firing alert’s repeat, a second alert on a node monitoring already holds, and a hold on an offline node are never refused, since none of them takes another node out of service.
- Offline nodes are left out of the count entirely.
- One node is always allowed, so a pool of four or fewer can still hold its one bad node.
- Administrator holds are never refused and never counted: cordoning half a pool by hand is a decision you have made.
- A refused alert keeps re-notifying while it fires, so its hold lands on its own once another hold in the pool is released.
When the alert fires, either that many nodes really are failing or a rule is
misfiring. fuzzball node hold list shows which alerts placed the holds. Hold
the remaining faulty nodes by hand, or raise the ceiling with
nodeHealth.maxMonitoringHeldPercent – cluster-wide, or per definition; see
the configuration reference.
100 removes it.
The response reports one outcome per alert, which answers “the alert fired, so
why is the node not held” without reading control-plane logs. Each outcome’s
result is ALERTMANAGER_ALERT_OUTCOME_RESULT_ followed by one of:
| Outcome | Meaning |
|---|---|
PLACED | A new hold was written |
REFRESHED | The hold already existed; its reason and timestamp were updated |
RELEASED | A resolved alert removed the hold it had placed |
ADVISORY | The alert carried no fb_action, so it held nothing |
IGNORED | Understood but changed nothing – an unknown node, an unknown or refused action, a hold id another source already holds, a node already carrying 64 holds, a pool at its ceiling, or a release for a hold already gone |
A write that genuinely failed is answered with an error instead of this list, so Alertmanager retries. Retrying is safe: placing a hold twice refreshes it, and releasing one twice finds nothing the second time.
Because Alertmanager discards the body of a successful response, these outcomes
are also written to the control plane’s log – an IGNORED alert is logged as a
warning naming the alert and the reason. That log is where to look when a rule
fires and nothing is held; a request made by hand (curl) is where to read the
list above directly.
A resolved notification that never reaches Fuzzball – because Alertmanager was
down, or the route was misconfigured – leaves a hold in place. Holds do not
expire, so check the hold’s refresh timestamp with fuzzball node hold list to tell a
fault that is still firing from an integration that stopped talking – see
Spotting an Abandoned Hold.
When a node disappears – power loss, a killed agent, an instance terminated underneath Fuzzball – the work placed on it is already gone. Fuzzball releases it immediately rather than waiting for each lease to time out one at a time:
- Steps with
policy.retry.on: nodeare requeued and run again elsewhere. - Everything else fails, with the node named.
There is no judgement to make here, so this is not configurable: the node is gone and the work died with it. The only question is whether you are told now or after a timeout. A multinode job in particular no longer holds its remaining nodes for the length of that wait.
The failure names no cause. A node disappearing says nothing about its hardware – a rack losing power and an agent being killed look the same from the control plane – so Fuzzball reports what it knows and no more:
node 10.0.0.42/9100 stopped hosting this work; job failed. Set policy.retry.on to node for work that is safe to run again from the beginning
A step that declared policy.retry.on: node and has already been moved
maxEvictionRestarts times is failed with the count rather than that advice:
node 10.0.0.42/9100 stopped hosting this work; job failed after 3 node restart(s), the limit for this step
When a job ends because its node stopped responding, the error names the node:
job stopped responding on node 10.0.0.42/9100
If that node is held at the time, the error names the hold’s reason as well, so a user can tell immediately that the failure was not theirs:
node 10.0.0.42/9100 is held: GPU 2 uncorrectable ECC; job stopped responding on this node
The reason comes from the hold rather than from a diagnosis Fuzzball made: a person or an alerting rule wrote that sentence about this node deliberately. A node with nothing held against it is still named, because knowing which machine a job died on is useful on its own, and claims nothing.
The failure is also counted against the node’s attributed-failure total.
The control plane keeps the latest resource report for every attached node in a
JetStream key-value bucket named substrate-nodes. Bucket usage grows with the
number of nodes and with the size of each report, and a report grows with the
node’s core count, its devices and its annotations.
Size the bucket from the node count multiplied by the size of one report. A 192-core node with 8 GPUs and 10 annotations produces a report of roughly 4 KiB, so 3,000 such nodes need about 12 MiB. The bucket defaults to 64 MiB, which holds roughly 16,000 reports of that size.
A full bucket fails quietly – no error surfaces in the CLI, the API or a node’s status. JetStream stops accepting reports from nodes the bucket has not seen before, and stops accepting any report larger than the one it replaces. The cluster stops registering new nodes, and the scheduler keeps planning against the last resources it recorded for the nodes it already has. The only signals are the substrate bridge log, which reports a rejection at error level at most once every 30 seconds, and the metrics in Bucket Metrics.
Set the bucket’s size cap with nodesBucketMaxBytes in the cluster’s
FuzzballOrchestrate custom resource:
apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
name: fuzzball-orchestrate
spec:
fuzzball:
substrate:
nodesBucketMaxBytes: 1073741824
See Substrate Configuration in the CRD reference for the full field description.
When the substrate bridge restarts, it applies the new cap, resizing the existing bucket in place. Leaving the field unset changes nothing: the 64 MiB default is the size clusters ran with before the field existed.
Checkfuzzball_substrate_nodes_bucket_bytes_usedbefore lowering a cap. Any cap below what the bucket already stores produces the same failure as a full bucket. The field’s 64 MiB minimum only rejects caps below 64 MiB; it does not compare the new cap against what the bucket currently stores, so a bucket already holding more than 64 MiB can still be capped below its own contents.
The cap is a reservation, not a limit on actual use. JetStream subtracts it from the NATS server’s on-disk allowance of 10 GiB whether or not the bucket holds that much, and every other Fuzzball stream on that server draws on the same allowance – including the node event history (1 GiB) and the node allocation history (512 MiB).
That allowance is per server. A high-availability deployment replicates the bucket across three NATS servers, and each server reserves the full cap against its own 10 GiB. A 1 GiB cap therefore costs 1 GiB on every server, not 3 GiB on any one of them.
An oversized reservation stops the substrate bridge from starting rather than
degrading quietly, so the cap is bounded at 2 GiB. The Kubernetes API server
rejects a nodesBucketMaxBytes value below 64 MiB or above 2 GiB when the
custom resource is applied, so a mistyped size fails at apply time. A value that
reaches the orchestrate service configuration by another path is clamped into
that range rather than preventing the bridge from starting. The bridge logs the cap
in force at startup, and logs the configured value alongside it only when that
value had to be clamped. Raise nodesBucketMaxBytes as the node count grows; do
not set a large value up front.
Three series published by the substrate bridge cover the bucket:
| Metric | Type | Labels | Description |
|---|---|---|---|
fuzzball_substrate_nodes_bucket_bytes_used | gauge | - | Bytes stored in the substrate-nodes bucket |
fuzzball_substrate_nodes_bucket_bytes_max | gauge | - | Configured cap on the bucket |
fuzzball_substrate_nodes_bucket_put_failures_total | counter | - | Resource reports the bucket rejected |
The two gauges are sampled once a minute, regardless of how many nodes are
attached. Alert when fuzzball_substrate_nodes_bucket_bytes_used exceeds 80% of
fuzzball_substrate_nodes_bucket_bytes_max. That leaves room to raise the cap
and restart the substrate bridge before the bucket fills.
Both gauges are absent until the first successful sample, so absence means usage is unknown rather than zero. Pair a headroom alert with anabsent()check rather than treating a missing series as an empty bucket.
Events are retained for 90 days and can be read back per node through the
ListNodeEvents API, filtered by time and by event type and returned oldest
first in pages. Allocation events (ALLOCATION_ADDED, ALLOCATION_REMOVED) are
kept in a separate 512 MiB history, so on a busy cluster the oldest of them can
age out before 90 days; hold and status history is unaffected by allocation
volume.
Read a node’s history with fuzzball node events, oldest first:
fuzzball node events 10.0.0.42/9100 --since 24hTIME | TYPE | DETAIL
2026-09-17T09:14:02Z | HOLD_PLACED | cordon by monitoring: 4 correctable ECC errors in 24h
2026-09-17T11:37:41Z | JOB_FAILURE_ATTRIBUTED | job alloc-93f2 failed on this node
2026-09-17T11:52:10Z | STATUS_CHANGED | status ready -> unknown: no health report for 3 consecutive intervals
Filter by type with --type, repeatable, and use -o json to pull the series
out for plotting.
| Type | Recorded when | Carries |
|---|---|---|
NODE_PROVISIONED | The cluster first sees the node, so its history has a beginning | – |
STATUS_CHANGED | The node’s status changes | The previous and new status |
HOLD_PLACED, HOLD_RELEASED | A hold is placed on or released from the node | The hold: id, source, effect, reason |
NODE_EVICTED | The node’s work was taken off it because of an evict hold | The hold that caused it, copied onto the event |
NODE_DRAINED | The node is drained, by an operator or automatically | Whether running work is left to finish (passive) or taken off (active), and the reason |
ALLOCATION_ADDED, ALLOCATION_REMOVED | An allocation’s rank starts or stops running on the node | allocation_id and rank |
JOB_FAILURE_ATTRIBUTED | A job stopped responding on this node | allocation_id |
A hold is copied onto its events rather than referenced, because by the time
the history is read the hold has usually been released. allocation_id is the
join into the workflow’s own events, which say what then happened to that
work – see fuzzball workflow events.
STATUS_CHANGED is the event to watch for a node going silent. It replaces the
two events that used to describe it – a raised HEARTBEAT_MISSED condition and
a health-state transition into UNKNOWN – which said the same thing twice in a
vocabulary that no longer exists.
If you are upgrading, these are gone. Nothing needs to be done to remove them; they simply stop being produced.
| Retired | Replaced by |
|---|---|
| The 0-100 reliability score, its penalties, thresholds and explanation | Prometheus rules over the raw counters |
DEGRADED as a health state, and the closed condition-type list | Whatever your rules match on |
The policy engine and its observe / cordon / drain / replace modes | Which rules a site enables, and what action each carries |
evictOnFaultClasses | A rule with an evict action |
| The score-based placement gate | Holds |
| Automated node replacement and its churn ceiling | A deprovision hold; the provisioner supplies capacity again when work needs it |
| The manual-uncordon suppression window | Hold source – an administrator hold is one automation may never clear |
| The Fuzzball-native signed notification webhook | Alertmanager receivers |
| Score, condition, health-state and policy node events | HOLD_PLACED, HOLD_RELEASED, STATUS_CHANGED, JOB_FAILURE_ATTRIBUTED |
The nodeHealth central-config block keeps one field, maxEvictionRestarts.
The penalties, thresholds, failure-tally weights, policy mode and
evictOnFaultClasses it used to carry are ignored – a stored config that still
sets them keeps loading, so an upgrade does not need the file edited first.
On the API, the retired fields stay on the wire and are never populated:
reliability_score, health_state, conditions and score_explanation on a
node; the score, condition, threshold and policy fields on an event. A client
reading them sees the zero value rather than a stale judgement.
fuzzball node drainis deprecated in favour offuzzball node cordon, andfuzzball node evictin place of--active. It keeps working, but--activehas changed: it now honours what each step declared rather than requeueing everything on the node. A step whosepolicy.retry.onincludesnodeis requeued; anything else is failed. Earlier releases restarted steps whose authors had said they must never be restarted.