Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Explain metric alert thresholds without mistaking Off for recovery

pulse-triage[bot]07afe19ff8a1
diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md
index 1aeeb84aae..7ffcf6ffdb 100644
--- a/docs/CONFIGURATION.md
+++ b/docs/CONFIGURATION.md
@@ -484,9 +484,50 @@ For remote scraping with `PULSE_METRICS_TOKEN`, prefer a local scraper, tunnel,
## 🔔 Alerts (`alerts.json`)
-Pulse uses a powerful alerting engine with hysteresis (separate trigger/clear thresholds) to prevent flapping.
+Manage alert rules in **Alerts → Thresholds**. Choose the platform and resource
+before editing; a group default can affect many resources, while a custom
+resource override takes precedence. Finish the edit and use **Save Changes**;
+an unsaved value is not the running policy. Reload after a successful save to
+check that the intended value persisted.
-**Managed via UI**: Alerts → Thresholds
+### Metric thresholds, Off and inheritance
+
+For numeric metric rules such as CPU, memory and disk usage:
+
+| Setting | Meaning |
+| --- | --- |
+| Positive metric threshold | Enables that metric rule at the entered value, subject to other alert policies. Read the column's unit: percentages, temperatures and throughput are not interchangeable. |
+| Metric **Off** | Disables that metric rule, not collection of its readings. Use the On/Off control rather than an empty input to disable it. |
+| Blank per-resource metric value | Inherits the group default; it does **not** mean Off. An inherited Off default remains Off. |
+| Saved numeric metric trigger `0` or a negative value | Disables that metric rule. Zero does **not** mean “alert on any usage”. Older saved rules may use `0`; the current metric Off control writes `-1`. |
+
+**Zero has different meanings in different fields.** A zero *metric trigger*
+disables the rule, but zero *powered-off tolerance* below means immediate
+eligibility on an authoritative stopped observation. Do not copy one field's
+meaning into another. Platform/resource alert switches, offline alerts and
+notification delivery have their own controls.
+
+Usage rules have separate trigger and clear values (hysteresis), so a small
+drop below the trigger does not repeatedly close and reopen the alert. In the
+manual CPU example below, `trigger: 90` and `clear: 80` mean:
+
+- A fresh evaluated value of **90% or higher** is eligible to activate the alert.
+- An already active alert stays active **above 80%**, even at 85%.
+- A fresh evaluated value of **80% or lower** is eligible to clear it.
+
+Configured evaluation windows and activation/recovery delays still apply;
+these are not promises of immediate notifications. The excerpt illustrates a
+saved rule, not the clear value produced by every UI edit.
+
+Disabling a metric can close its existing alert even while usage remains high.
+That disappearance is a policy change, **not measured recovery**. Missing or
+stale readings are not a healthy zero either. Check fresh readings and the
+workload before judging recovery; do not lower thresholds, create load or stop
+a workload just to test the rule. To quiet notifications without disabling a
+metric rule, review [Quiet hours](#quiet-hours-and-notification-holds) and their
+critical-alert exceptions. Check [Recent delivery activity](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing)
+separately: a saved threshold or a successful destination Test does not prove
+ordinary alert delivery.
### VM and container powered-off tolerance
diff --git a/frontend-modern/public/docs/CONFIGURATION.md b/frontend-modern/public/docs/CONFIGURATION.md
index 1aeeb84aae..7ffcf6ffdb 100644
--- a/frontend-modern/public/docs/CONFIGURATION.md
+++ b/frontend-modern/public/docs/CONFIGURATION.md
@@ -484,9 +484,50 @@ For remote scraping with `PULSE_METRICS_TOKEN`, prefer a local scraper, tunnel,
## 🔔 Alerts (`alerts.json`)
-Pulse uses a powerful alerting engine with hysteresis (separate trigger/clear thresholds) to prevent flapping.
+Manage alert rules in **Alerts → Thresholds**. Choose the platform and resource
+before editing; a group default can affect many resources, while a custom
+resource override takes precedence. Finish the edit and use **Save Changes**;
+an unsaved value is not the running policy. Reload after a successful save to
+check that the intended value persisted.
-**Managed via UI**: Alerts → Thresholds
+### Metric thresholds, Off and inheritance
+
+For numeric metric rules such as CPU, memory and disk usage:
+
+| Setting | Meaning |
+| --- | --- |
+| Positive metric threshold | Enables that metric rule at the entered value, subject to other alert policies. Read the column's unit: percentages, temperatures and throughput are not interchangeable. |
+| Metric **Off** | Disables that metric rule, not collection of its readings. Use the On/Off control rather than an empty input to disable it. |
+| Blank per-resource metric value | Inherits the group default; it does **not** mean Off. An inherited Off default remains Off. |
+| Saved numeric metric trigger `0` or a negative value | Disables that metric rule. Zero does **not** mean “alert on any usage”. Older saved rules may use `0`; the current metric Off control writes `-1`. |
+
+**Zero has different meanings in different fields.** A zero *metric trigger*
+disables the rule, but zero *powered-off tolerance* below means immediate
+eligibility on an authoritative stopped observation. Do not copy one field's
+meaning into another. Platform/resource alert switches, offline alerts and
+notification delivery have their own controls.
+
+Usage rules have separate trigger and clear values (hysteresis), so a small
+drop below the trigger does not repeatedly close and reopen the alert. In the
+manual CPU example below, `trigger: 90` and `clear: 80` mean:
+
+- A fresh evaluated value of **90% or higher** is eligible to activate the alert.
+- An already active alert stays active **above 80%**, even at 85%.
+- A fresh evaluated value of **80% or lower** is eligible to clear it.
+
+Configured evaluation windows and activation/recovery delays still apply;
+these are not promises of immediate notifications. The excerpt illustrates a
+saved rule, not the clear value produced by every UI edit.
+
+Disabling a metric can close its existing alert even while usage remains high.
+That disappearance is a policy change, **not measured recovery**. Missing or
+stale readings are not a healthy zero either. Check fresh readings and the
+workload before judging recovery; do not lower thresholds, create load or stop
+a workload just to test the rule. To quiet notifications without disabling a
+metric rule, review [Quiet hours](#quiet-hours-and-notification-holds) and their
+critical-alert exceptions. Check [Recent delivery activity](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing)
+separately: a saved threshold or a successful destination Test does not prove
+ordinary alert delivery.
### VM and container powered-off tolerance
diff --git a/frontend-modern/src/utils/__tests__/metricThresholdHelp.test.ts b/frontend-modern/src/utils/__tests__/metricThresholdHelp.test.ts
new file mode 100644
index 0000000000..828ac006cd
--- /dev/null
+++ b/frontend-modern/src/utils/__tests__/metricThresholdHelp.test.ts
@@ -0,0 +1,82 @@
+import { describe, expect, it } from 'vitest';
+import { readFileSync } from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { renderDocMarkdown } from '@/features/docs/docMarkdown';
+import {
+ getAlertResourceMetricDisplayValue,
+ isAlertResourceMetricOff,
+} from '@/components/Alerts/alertResourceTableModel';
+
+const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');
+const source = readFileSync(path.join(repoRoot, 'docs/CONFIGURATION.md'), 'utf8');
+
+function article() {
+ const section = source
+ .split('### Metric thresholds, Off and inheritance')[1]
+ ?.split('### VM and container powered-off tolerance')[0];
+ expect(section, 'metric threshold help must be present').toBeDefined();
+ const element = document.createElement('article');
+ element.innerHTML = renderDocMarkdown(section!, 'CONFIGURATION');
+ return element;
+}
+
+describe('existing metric threshold help', () => {
+ it('renders Off and inheritance separately, matching the live editor model', () => {
+ const rows = [...article().querySelectorAll('tbody tr')].map((row) =>
+ row.textContent?.replace(/\s+/g, ' '),
+ );
+ expect(rows).toHaveLength(4);
+ expect(rows[1]).toContain('Disables that metric rule, not collection');
+ expect(rows[2]).toContain('Inherits the group default; it does not mean Off');
+ expect(rows[3]).toContain('Zero does not mean “alert on any usage”');
+ for (const off of [0, -1]) {
+ expect(isAlertResourceMetricOff(off)).toBe(true);
+ }
+ const resource = {
+ id: 'help-vm',
+ name: 'Help VM',
+ thresholds: { cpu: 95 },
+ defaults: { cpu: 80 },
+ };
+ expect(getAlertResourceMetricDisplayValue(resource, 'cpu', { cpu: undefined }, true)).toBe(80);
+ expect(
+ isAlertResourceMetricOff(
+ getAlertResourceMetricDisplayValue(
+ { ...resource, defaults: { cpu: 0 } },
+ 'cpu',
+ { cpu: undefined },
+ true,
+ ),
+ ),
+ ).toBe(true);
+ });
+
+ it('renders inclusive hysteresis boundaries without promising immediate delivery', () => {
+ const element = article();
+ const text = element.textContent?.replace(/\s+/g, ' ');
+ expect(text).toContain('90% or higher');
+ expect(text).toContain('above 80%, even at 85%');
+ expect(text).toContain('80% or lower');
+ expect(text).toContain('evaluation windows and activation/recovery delays still apply');
+ expect(text).toContain('not measured recovery');
+ expect(text).toContain('Missing or stale readings are not a healthy zero');
+ expect(text).toContain('do not lower thresholds, create load or stop a workload');
+ expect(text).toContain('zero powered-off tolerance below means immediate eligibility');
+ expect(element.querySelector('pre')).toBeNull();
+ });
+
+ it('keeps save readback and both safe delivery alternatives usable in bundled help', () => {
+ const alerts = source.split('## 🔔 Alerts (`alerts.json`)')[1];
+ expect(alerts).toContain('use **Save Changes**');
+ expect(alerts).toContain('Reload after a successful save');
+ const element = article();
+ expect(element.querySelector('a[href="#quiet-hours-and-notification-holds"]')).not.toBeNull();
+ expect(
+ element.querySelector('a[href="/docs/TROUBLESHOOTING#test-succeeds-but-real-alerts-are-missing"]'),
+ ).not.toBeNull();
+ expect(readFileSync(path.join(repoRoot, 'frontend-modern/public/docs/CONFIGURATION.md'), 'utf8')).toBe(
+ source,
+ );
+ });
+});
diff --git a/internal/alerts/configuration_help_example_test.go b/internal/alerts/configuration_help_example_test.go
new file mode 100644
index 0000000000..b1ea17f9cc
--- /dev/null
+++ b/internal/alerts/configuration_help_example_test.go
@@ -0,0 +1,74 @@
+package alerts
+
+import (
+ "encoding/json"
+ "os"
+ "strings"
+ "testing"
+)
+
+// Exercise the actual copied example through the metric runtime, not a second
+// implementation of the comparisons in the guide. Delays/windows are disabled
+// here to isolate eligibility; the guide explicitly retains those conditions.
+func TestConfigurationHelpMetricExample(t *testing.T) {
+ doc, err := os.ReadFile("../../docs/CONFIGURATION.md")
+ if err != nil {
+ t.Fatal(err)
+ }
+ _, alertsSection, ok := strings.Cut(string(doc), "## 🔔 Alerts (`alerts.json`)")
+ if !ok {
+ t.Fatal("missing alert help")
+ }
+ _, example, ok := strings.Cut(alertsSection, "```json\n")
+ if !ok {
+ t.Fatal("missing alert JSON example")
+ }
+ example, _, ok = strings.Cut(example, "\n```")
+ if !ok {
+ t.Fatal("unterminated alert JSON example")
+ }
+ var cfg AlertConfig
+ if err := json.Unmarshal([]byte(example), &cfg); err != nil {
+ t.Fatal(err)
+ }
+ threshold := cfg.GuestDefaults.CPU
+ if threshold == nil || threshold.Trigger != 90 || threshold.Clear != 80 {
+ t.Fatalf("copied CPU rule = %+v, update the documented boundary cases together", threshold)
+ }
+
+ for _, scenario := range []struct {
+ name string
+ values []float64
+ active []bool
+ }{
+ {"trigger equality, hold and clear equality", []float64{89.9, 90, 85, 80.1, 80}, []bool{false, true, true, true, false}},
+ {"below clear", []float64{95, 79.9}, []bool{true, false}},
+ } {
+ t.Run(scenario.name, func(t *testing.T) {
+ m := newTestManager(t)
+ configureUnifiedEvalManager(t, m, unifiedEvalBaseConfig())
+ for i, value := range scenario.values {
+ m.checkMetric("help-vm", "Help VM", "help-node", "help-instance", "guest", "cpu", value, threshold, nil)
+ id := canonicalMetricStateID("help-vm", "cpu")
+ if got := testHasActiveAlert(t, m, id); got != scenario.active[i] {
+ t.Fatalf("step %d value %g: active=%v, want %v", i, value, got, scenario.active[i])
+ }
+ }
+ })
+ }
+
+ for _, off := range []struct {
+ name string
+ value float64
+ }{{"legacy zero is Off without recovery", 0}, {"current negative sentinel is Off without recovery", -1}} {
+ t.Run(off.name, func(t *testing.T) {
+ m := newTestManager(t)
+ configureUnifiedEvalManager(t, m, unifiedEvalBaseConfig())
+ id := canonicalMetricStateID("help-vm", "cpu")
+ m.checkMetric("help-vm", "Help VM", "help-node", "help-instance", "guest", "cpu", 95, threshold, nil)
+ assertAlertPresent(t, m, id)
+ m.checkMetric("help-vm", "Help VM", "help-node", "help-instance", "guest", "cpu", 95, &HysteresisThreshold{Trigger: off.value}, nil)
+ assertAlertMissing(t, m, id)
+ })
+ }
+}