# Shipped with Fuzzball's bundled monitoring stack, published for sites that run # their own. Generated from the operator's copy -- edit that, not this file. groups: - name: fuzzball-node-health rules: - alert: FuzzballNodeAttributedJobFailures expr: >- delta((max by (node_id) (fuzzball_scheduler_node_attributed_job_failures))[24h:5m]) >= 5 for: 15m labels: severity: warning component: fuzzball-node fb_action: cordon annotations: summary: >- {{ $value | printf "%.0f" }} job failures attributed to node {{ $labels.node_id }} in 24h description: >- Five or more jobs have stopped responding on this node. That is not proof the hardware is at fault -- jobs die for their own reasons -- but a node accumulating them while its neighbours do not is worth looking at, and is the one node signal Fuzzball can see that a monitoring stack cannot. Five failures in the last day, not five ever. The tally only counts up -- nothing resets it, and the decay it used to have lived in the retired score -- so alerting on the level meant a node fired forever once it crossed five, including after it was repaired. That is the kind of alert that gets silenced, and this is the only per-node rule that ships. delta() rather than increase(): the series is a gauge restored from the node record when the control plane restarts, not a counter, and increase() would read a leader failover as a burst of new failures. delta() over a gauge is the honest reading. The max across scheduler pods is taken before the delta, as a subquery. Only the leader publishes the series, so a failover or a rollout moves it to a different pod -- a different series -- and a delta taken per series would split the day at that point: three failures before an upgrade and three after would read as three, and the rule would never fire. The max is one series per node whatever pod is leading. - alert: FuzzballNodeUnknown expr: >- time() - max by (node, node_id) (fuzzball_node_last_health_report_timestamp_seconds) > 600 for: 0m labels: severity: warning component: fuzzball-node annotations: summary: >- Node {{ $labels.node }} ({{ $labels.node_id }}) has not reported for 10 minutes description: >- Fuzzball cannot see this node. Its extension has stopped reporting, which means the host is down, hung, or the extension itself has died -- the control plane cannot tell which from here. The node is already out of the placement set; this says somebody should go and find out why. The one alert that has to ship with the product. Everything else about a node's condition is a monitoring rule's business now, but a node going silent is only something Fuzzball can say. The node's exporter up series drops too, but a failed scrape says only that a target could not be read, which a network fault, the proxy or an expired credential also produce. "for: 0m" because the expression carries its own ten minutes. The default report interval is 30 seconds and the liveness sweep gives up after three misses, so by ten minutes this is not jitter. Aggregated by (node, node_id), not by node alone. max takes the freshest timestamp in the group, so grouping on the host name only would let a live substrate cover for a dead one on a host running two -- the alert would never fire for the dead one. node_id is per substrate, so each is judged on its own, and the summary names it because two substrates on one host share a host name. This costs nothing in false alerts. The other reason to group on the host name -- suppressing a ghost record left behind when a substrate moves address -- is already handled upstream: that record gets a delete event, which sets it offline and clears its last-report timestamp, so its series is dropped rather than left to go stale. Only the liveness sweep leaves a stale timestamp, and that is exactly the case this alert is for. node_id still dedupes two leaders publishing the same node during a failover, since both publish the same id. - alert: FuzzballNodeDevicePluginFailed expr: >- max by (node, node_id, plugin) (fuzzball_node_device_plugin{state="failed"}) == 1 for: 5m labels: severity: warning component: fuzzball-node annotations: summary: >- Device plugin {{ $labels.plugin }} failed on node {{ $labels.node }} ({{ $labels.node_id }}) description: >- The plugin failed to initialize when fuzzball-substrate started, so the device kind it serves is missing from the node's report. The node stays ready and keeps taking work that needs no device, but the scheduler passes it over for every job requesting that kind, and a job with nowhere else to go reports a failed device plugin as its blocked reason. Only Fuzzball sees this: the node reports every other resource normally, and no exporter notices a plugin that never came up. fuzzball node show names the plugin and its error. The fix is on the node -- repair the plugin, then restart fuzzball-substrate -- and the series flips to ready on the next report. Five minutes rather than a single scrape, so a substrate that is restarting while a repair lands does not page before its first fresh report arrives. - alert: FuzzballNodeHealthMetricsAbsent expr: absent(fuzzball_scheduler_node_attributed_job_failures) for: 15m labels: severity: critical component: fuzzball-control-plane annotations: summary: "Fuzzball is publishing no node metrics" description: >- No node series at all. Either no scheduler replica holds leadership, or scraping is broken. Every other rule here goes quiet in this state, so this one has to exist -- an absent series must never read as a healthy fleet. - alert: FuzzballNodeHoldCeilingReached expr: >- max by (definition) (increase(fuzzball_node_monitoring_holds_capped_total[30m])) > 0 for: 0m labels: severity: critical component: fuzzball-control-plane annotations: summary: >- Monitoring holds refused in {{ if $labels.definition }}definition {{ $labels.definition }}{{ else }}nodes with no provisioner definition{{ end }}: the pool is at its ceiling description: >- Fuzzball refused to hold a node an alert called faulty, because nodeHealth.maxMonitoringHeldPercent of the pool already carries monitoring holds. Either that many nodes really are failing, or a rule is misfiring across the pool -- which is what the ceiling is for, since holds stay until each alert resolves. Look at which alerts placed the holds (fuzzball node hold list), then hold the remaining faulty nodes by hand or raise the ceiling. No fb_action: this alert is about a pool, not a node, and must never place a hold. Fires on any refusal in the last 30m; a refused alert re-notifies every 15m on the hold route, so this stays firing for as long as refusals continue. - alert: FuzzballNodeMemoryErrors expr: >- ( max by (node, node_id) (increase(fuzzball_node_memory_correctable_errors_total[24h])) or sum by (node, node_id) (increase(node_edac_correctable_errors_total[24h])) ) * on(node, node_id) group_left(definition) max by (node, node_id, definition) (fuzzball_node_info) > 1000 for: 30m labels: severity: warning component: fuzzball-node fb_action: cordon annotations: summary: >- {{ $labels.node }} ({{ $labels.node_id }}): {{ $value | printf "%.0f" }} correctable ECC errors in 24h description: >- Correctable errors are corrected, so nothing has failed yet -- but a DIMM producing them at this rate is the usual precursor to one that produces uncorrectable ones, and those kill a job. Cordon rather than evict: the work already running is not at risk from errors the hardware is still correcting. One alert per substrate on the host, each carrying its node_id. The EDAC counters belong to the host's memory, so every substrate on a host with a failing DIMM is affected, and node_id lets the receiver hold exactly those records. node_id comes from the scrape job, which promotes it onto every compute-node series. A host running two substrates is scraped once per substrate, so its counters arrive twice, differing only in node_id -- which is why the sum is by (node, node_id): by node alone it would double the count on exactly those hosts. The join is one-to-one on (node, node_id). The right side is max by (node, node_id, definition) rather than the bare series because both leaders publish fuzzball_node_info during a failover, and their copies differ in pod labels; the max collapses them, and keeps those pod labels out of the alert, where a leader change would otherwise change the alert's identity. Two sources, one count. Fuzzball's node exporter reads the same EDAC counters and publishes the host's total, so the rule works on a node with no node_exporter; it is preferred where present, and "or" falls back to node_exporter's reading only for a node_id that Fuzzball's exporter does not report -- one with health.metrics-disabled set. Never both: adding them would double the count. node_exporter's counters are per memory controller, so they are summed before the comparison. Without the sum the threshold applied per controller while the summary read as a node total: a two-controller node at 700 each would stay quiet at 1400 node-wide, and one crossing on both controllers would fire twice and place two holds for one fault. Fuzzball's series is already the sum across controllers, and the max only drops scrape labels. A node with neither exporter reading EDAC -- no ECC memory, or the edac modules not loaded -- has no series, and this rule is inert there; that is absence rather than failure. - alert: FuzzballNodeGPUUncorrectableECC expr: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[1h]) > 0 for: 0m labels: severity: critical component: fuzzball-node fb_action: evict fb_release: manual annotations: summary: >- {{ $labels.node }} GPU {{ $labels.gpu }}: uncorrectable ECC error description: >- An uncorrectable double-bit error. Whatever was computing on that GPU has already read wrong data, so unlike the correctable case this evicts rather than cordons: the work running on it is the thing at risk. "for: 0m" because one is enough. A double-bit error is not a rate to be tolerated, and increase() over an hour only establishes that it happened in this window. fb_release: manual, so the hold outlives the alert. The counter stops incrementing as soon as nothing is scheduled on the GPU -- which is exactly what the eviction causes -- so an automatic release would resolve the alert and hand the suspect card straight back out. An administrator clears this one. Reads DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, which Fuzzball does not ship. A cluster with no GPUs has no dcgm job, and that absence is not a failure.