Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Explain metric alert thresholds without mistaking Off for recovery
pulse-triage[bot]07afe19ff8a1
diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.mdindex 1aeeb84aae..7ffcf6ffdb 100644--- a/docs/CONFIGURATION.md+++ b/docs/CONFIGURATION.md@@ -484,9 +484,50 @@ For remote scraping with `PULSE_METRICS_TOKEN`, prefer a local scraper, tunnel,## 🔔 Alerts (`alerts.json`)-Pulse uses a powerful alerting engine with hysteresis (separate trigger/clear thresholds) to prevent flapping.+Manage alert rules in **Alerts → Thresholds**. Choose the platform and resource+before editing; a group default can affect many resources, while a custom+resource override takes precedence. Finish the edit and use **Save Changes**;+an unsaved value is not the running policy. Reload after a successful save to+check that the intended value persisted.-**Managed via UI**: Alerts → Thresholds+### Metric thresholds, Off and inheritance++For numeric metric rules such as CPU, memory and disk usage:++| Setting | Meaning |+| --- | --- |+| Positive metric threshold | Enables that metric rule at the entered value, subject to other alert policies. Read the column's unit: percentages, temperatures and throughput are not interchangeable. |+| Metric **Off** | Disables that metric rule, not collection of its readings. Use the On/Off control rather than an empty input to disable it. |+| Blank per-resource metric value | Inherits the group default; it does **not** mean Off. An inherited Off default remains Off. |+| Saved numeric metric trigger `0` or a negative value | Disables that metric rule. Zero does **not** mean “alert on any usage”. Older saved rules may use `0`; the current metric Off control writes `-1`. |++**Zero has different meanings in different fields.** A zero *metric trigger*+disables the rule, but zero *powered-off tolerance* below means immediate+eligibility on an authoritative stopped observation. Do not copy one field's+meaning into another. Platform/resource alert switches, offline alerts and+notification delivery have their own controls.++Usage rules have separate trigger and clear values (hysteresis), so a small+drop below the trigger does not repeatedly close and reopen the alert. In the+manual CPU example below, `trigger: 90` and `clear: 80` mean:++- A fresh evaluated value of **90% or higher** is eligible to activate the alert.+- An already active alert stays active **above 80%**, even at 85%.+- A fresh evaluated value of **80% or lower** is eligible to clear it.++Configured evaluation windows and activation/recovery delays still apply;+these are not promises of immediate notifications. The excerpt illustrates a+saved rule, not the clear value produced by every UI edit.++Disabling a metric can close its existing alert even while usage remains high.+That disappearance is a policy change, **not measured recovery**. Missing or+stale readings are not a healthy zero either. Check fresh readings and the+workload before judging recovery; do not lower thresholds, create load or stop+a workload just to test the rule. To quiet notifications without disabling a+metric rule, review [Quiet hours](#quiet-hours-and-notification-holds) and their+critical-alert exceptions. Check [Recent delivery activity](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing)+separately: a saved threshold or a successful destination Test does not prove+ordinary alert delivery.### VM and container powered-off tolerancediff --git a/frontend-modern/public/docs/CONFIGURATION.md b/frontend-modern/public/docs/CONFIGURATION.mdindex 1aeeb84aae..7ffcf6ffdb 100644--- a/frontend-modern/public/docs/CONFIGURATION.md+++ b/frontend-modern/public/docs/CONFIGURATION.md@@ -484,9 +484,50 @@ For remote scraping with `PULSE_METRICS_TOKEN`, prefer a local scraper, tunnel,## 🔔 Alerts (`alerts.json`)-Pulse uses a powerful alerting engine with hysteresis (separate trigger/clear thresholds) to prevent flapping.+Manage alert rules in **Alerts → Thresholds**. Choose the platform and resource+before editing; a group default can affect many resources, while a custom+resource override takes precedence. Finish the edit and use **Save Changes**;+an unsaved value is not the running policy. Reload after a successful save to+check that the intended value persisted.-**Managed via UI**: Alerts → Thresholds+### Metric thresholds, Off and inheritance++For numeric metric rules such as CPU, memory and disk usage:++| Setting | Meaning |+| --- | --- |+| Positive metric threshold | Enables that metric rule at the entered value, subject to other alert policies. Read the column's unit: percentages, temperatures and throughput are not interchangeable. |+| Metric **Off** | Disables that metric rule, not collection of its readings. Use the On/Off control rather than an empty input to disable it. |+| Blank per-resource metric value | Inherits the group default; it does **not** mean Off. An inherited Off default remains Off. |+| Saved numeric metric trigger `0` or a negative value | Disables that metric rule. Zero does **not** mean “alert on any usage”. Older saved rules may use `0`; the current metric Off control writes `-1`. |++**Zero has different meanings in different fields.** A zero *metric trigger*+disables the rule, but zero *powered-off tolerance* below means immediate+eligibility on an authoritative stopped observation. Do not copy one field's+meaning into another. Platform/resource alert switches, offline alerts and+notification delivery have their own controls.++Usage rules have separate trigger and clear values (hysteresis), so a small+drop below the trigger does not repeatedly close and reopen the alert. In the+manual CPU example below, `trigger: 90` and `clear: 80` mean:++- A fresh evaluated value of **90% or higher** is eligible to activate the alert.+- An already active alert stays active **above 80%**, even at 85%.+- A fresh evaluated value of **80% or lower** is eligible to clear it.++Configured evaluation windows and activation/recovery delays still apply;+these are not promises of immediate notifications. The excerpt illustrates a+saved rule, not the clear value produced by every UI edit.++Disabling a metric can close its existing alert even while usage remains high.+That disappearance is a policy change, **not measured recovery**. Missing or+stale readings are not a healthy zero either. Check fresh readings and the+workload before judging recovery; do not lower thresholds, create load or stop+a workload just to test the rule. To quiet notifications without disabling a+metric rule, review [Quiet hours](#quiet-hours-and-notification-holds) and their+critical-alert exceptions. Check [Recent delivery activity](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing)+separately: a saved threshold or a successful destination Test does not prove+ordinary alert delivery.### VM and container powered-off tolerancediff --git a/frontend-modern/src/utils/__tests__/metricThresholdHelp.test.ts b/frontend-modern/src/utils/__tests__/metricThresholdHelp.test.tsnew file mode 100644index 0000000000..828ac006cd--- /dev/null+++ b/frontend-modern/src/utils/__tests__/metricThresholdHelp.test.ts@@ -0,0 +1,82 @@+import { describe, expect, it } from 'vitest';+import { readFileSync } from 'node:fs';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { renderDocMarkdown } from '@/features/docs/docMarkdown';+import {+ getAlertResourceMetricDisplayValue,+ isAlertResourceMetricOff,+} from '@/components/Alerts/alertResourceTableModel';++const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');+const source = readFileSync(path.join(repoRoot, 'docs/CONFIGURATION.md'), 'utf8');++function article() {+ const section = source+ .split('### Metric thresholds, Off and inheritance')[1]+ ?.split('### VM and container powered-off tolerance')[0];+ expect(section, 'metric threshold help must be present').toBeDefined();+ const element = document.createElement('article');+ element.innerHTML = renderDocMarkdown(section!, 'CONFIGURATION');+ return element;+}++describe('existing metric threshold help', () => {+ it('renders Off and inheritance separately, matching the live editor model', () => {+ const rows = [...article().querySelectorAll('tbody tr')].map((row) =>+ row.textContent?.replace(/\s+/g, ' '),+ );+ expect(rows).toHaveLength(4);+ expect(rows[1]).toContain('Disables that metric rule, not collection');+ expect(rows[2]).toContain('Inherits the group default; it does not mean Off');+ expect(rows[3]).toContain('Zero does not mean “alert on any usage”');+ for (const off of [0, -1]) {+ expect(isAlertResourceMetricOff(off)).toBe(true);+ }+ const resource = {+ id: 'help-vm',+ name: 'Help VM',+ thresholds: { cpu: 95 },+ defaults: { cpu: 80 },+ };+ expect(getAlertResourceMetricDisplayValue(resource, 'cpu', { cpu: undefined }, true)).toBe(80);+ expect(+ isAlertResourceMetricOff(+ getAlertResourceMetricDisplayValue(+ { ...resource, defaults: { cpu: 0 } },+ 'cpu',+ { cpu: undefined },+ true,+ ),+ ),+ ).toBe(true);+ });++ it('renders inclusive hysteresis boundaries without promising immediate delivery', () => {+ const element = article();+ const text = element.textContent?.replace(/\s+/g, ' ');+ expect(text).toContain('90% or higher');+ expect(text).toContain('above 80%, even at 85%');+ expect(text).toContain('80% or lower');+ expect(text).toContain('evaluation windows and activation/recovery delays still apply');+ expect(text).toContain('not measured recovery');+ expect(text).toContain('Missing or stale readings are not a healthy zero');+ expect(text).toContain('do not lower thresholds, create load or stop a workload');+ expect(text).toContain('zero powered-off tolerance below means immediate eligibility');+ expect(element.querySelector('pre')).toBeNull();+ });++ it('keeps save readback and both safe delivery alternatives usable in bundled help', () => {+ const alerts = source.split('## 🔔 Alerts (`alerts.json`)')[1];+ expect(alerts).toContain('use **Save Changes**');+ expect(alerts).toContain('Reload after a successful save');+ const element = article();+ expect(element.querySelector('a[href="#quiet-hours-and-notification-holds"]')).not.toBeNull();+ expect(+ element.querySelector('a[href="/docs/TROUBLESHOOTING#test-succeeds-but-real-alerts-are-missing"]'),+ ).not.toBeNull();+ expect(readFileSync(path.join(repoRoot, 'frontend-modern/public/docs/CONFIGURATION.md'), 'utf8')).toBe(+ source,+ );+ });+});diff --git a/internal/alerts/configuration_help_example_test.go b/internal/alerts/configuration_help_example_test.gonew file mode 100644index 0000000000..b1ea17f9cc--- /dev/null+++ b/internal/alerts/configuration_help_example_test.go@@ -0,0 +1,74 @@+package alerts++import (+ "encoding/json"+ "os"+ "strings"+ "testing"+)++// Exercise the actual copied example through the metric runtime, not a second+// implementation of the comparisons in the guide. Delays/windows are disabled+// here to isolate eligibility; the guide explicitly retains those conditions.+func TestConfigurationHelpMetricExample(t *testing.T) {+ doc, err := os.ReadFile("../../docs/CONFIGURATION.md")+ if err != nil {+ t.Fatal(err)+ }+ _, alertsSection, ok := strings.Cut(string(doc), "## 🔔 Alerts (`alerts.json`)")+ if !ok {+ t.Fatal("missing alert help")+ }+ _, example, ok := strings.Cut(alertsSection, "```json\n")+ if !ok {+ t.Fatal("missing alert JSON example")+ }+ example, _, ok = strings.Cut(example, "\n```")+ if !ok {+ t.Fatal("unterminated alert JSON example")+ }+ var cfg AlertConfig+ if err := json.Unmarshal([]byte(example), &cfg); err != nil {+ t.Fatal(err)+ }+ threshold := cfg.GuestDefaults.CPU+ if threshold == nil || threshold.Trigger != 90 || threshold.Clear != 80 {+ t.Fatalf("copied CPU rule = %+v, update the documented boundary cases together", threshold)+ }++ for _, scenario := range []struct {+ name string+ values []float64+ active []bool+ }{+ {"trigger equality, hold and clear equality", []float64{89.9, 90, 85, 80.1, 80}, []bool{false, true, true, true, false}},+ {"below clear", []float64{95, 79.9}, []bool{true, false}},+ } {+ t.Run(scenario.name, func(t *testing.T) {+ m := newTestManager(t)+ configureUnifiedEvalManager(t, m, unifiedEvalBaseConfig())+ for i, value := range scenario.values {+ m.checkMetric("help-vm", "Help VM", "help-node", "help-instance", "guest", "cpu", value, threshold, nil)+ id := canonicalMetricStateID("help-vm", "cpu")+ if got := testHasActiveAlert(t, m, id); got != scenario.active[i] {+ t.Fatalf("step %d value %g: active=%v, want %v", i, value, got, scenario.active[i])+ }+ }+ })+ }++ for _, off := range []struct {+ name string+ value float64+ }{{"legacy zero is Off without recovery", 0}, {"current negative sentinel is Off without recovery", -1}} {+ t.Run(off.name, func(t *testing.T) {+ m := newTestManager(t)+ configureUnifiedEvalManager(t, m, unifiedEvalBaseConfig())+ id := canonicalMetricStateID("help-vm", "cpu")+ m.checkMetric("help-vm", "Help VM", "help-node", "help-instance", "guest", "cpu", 95, threshold, nil)+ assertAlertPresent(t, m, id)+ m.checkMetric("help-vm", "Help VM", "help-node", "help-instance", "guest", "cpu", 95, &HysteresisThreshold{Trigger: off.value}, nil)+ assertAlertMissing(t, m, id)+ })+ }+}