Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Explain why a usage alert stays open after readings fall
pulse-triage[bot]a071af3d6789
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.mdindex da08657d7a..9152abe354 100644--- a/docs/TROUBLESHOOTING.md+++ b/docs/TROUBLESHOOTING.md@@ -62,6 +62,9 @@ establish safe recovery.[memory baseline and growth checks](TROUBLESHOOTING.md#memory-use-keeps-growing)or [CPU and write-rate checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth).Guest usage and database size are different measurements.+- **Usage fell but an alert remains open:** compare the+ [evaluated reading, clear level and recovery progress](#usage-alert-stays-open-after-the-reading-falls).+ A previous breach, an active incident and a retained notification are different evidence.- **Notification Test works but alerts do not arrive:** follow[ordinary delivery checks](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing).Do not trigger an alert or replay the queue just to complete a report.@@ -706,6 +709,45 @@ planned maintenance window, preserving the deployment's settings and data.pre-pause states are not permission to start them. **Pulse monitoring andalerts are unavailable while stopped**; arrange independent outage coverage.+#### Usage alert stays open after the reading falls++Start in **Alerts → Overview** with the affected incident, not an old email or+**Recent delivery activity**. Match the resource and metric; for guest disks,+match the filesystem too. Freeing space on one filesystem does not establish+recovery of another, a host storage pool or a separate backup-age alert.++Where the card supplies live metric details, read its evaluated value and+clear rule together:++- **Alert still open** means the evaluated value has fallen below the trigger+ but has not completed the clear rule. Usage rules have separate trigger and+ clear values: with a 90% trigger and 85% clear, 88% can still hold the alert.+ This is an example, not the saved rule for every resource. Check+ [threshold defaults and resource overrides](CONFIGURATION.md#metric-thresholds-off-and-inheritance).+- **Recovering** means the reading meets the clear level but the recovery delay+ is still being evaluated. Use the reported progress; elapsed wall-clock time+ alone does not prove that fresh eligible observations continued throughout.+ If the card says **averaged**, compare that evaluated value with the clear+ level, not only its **latest** sample or a different tool's instantaneous value.+- **No recent reading** is retained evidence, not a healthy zero or a completed+ recovery. The separate **Last reading at or above** value describes the+ previous breach, not current usage. Older versions or a restored incident+ may lack live detail; missing detail does not establish a current breach.++Acknowledgement and snooze affect attention or notification delivery; they are+not required for a threshold alert to recover. A retained failed notification+can outlive its incident, so use [delivery recovery](#recover-retained-delivery-failures)+for that warning rather than changing the metric rule.++If fresh readings for the same resource remain within the displayed clear rule+but the incident does not recover, keep the original times, units, evaluated+value, clear rule and any reported progress with a locally reviewed screenshot+or excerpt. Unknown details are valid. Add only missing evidence to the+[existing report](#-getting-help); do not rerun a backup, create load, restart,+change thresholds, disable monitoring or retry notifications just to clear it.+Do not delete alert or queue data. A disappearing alert after a policy change+is not proof that the workload recovered.+#### Backup-age alerts were not evaluatedThe **Pulse** warning “Backup-age alerts were not evaluated because recoverydiff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.mdindex da08657d7a..9152abe354 100644--- a/frontend-modern/public/docs/TROUBLESHOOTING.md+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md@@ -62,6 +62,9 @@ establish safe recovery.[memory baseline and growth checks](TROUBLESHOOTING.md#memory-use-keeps-growing)or [CPU and write-rate checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth).Guest usage and database size are different measurements.+- **Usage fell but an alert remains open:** compare the+ [evaluated reading, clear level and recovery progress](#usage-alert-stays-open-after-the-reading-falls).+ A previous breach, an active incident and a retained notification are different evidence.- **Notification Test works but alerts do not arrive:** follow[ordinary delivery checks](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing).Do not trigger an alert or replay the queue just to complete a report.@@ -706,6 +709,45 @@ planned maintenance window, preserving the deployment's settings and data.pre-pause states are not permission to start them. **Pulse monitoring andalerts are unavailable while stopped**; arrange independent outage coverage.+#### Usage alert stays open after the reading falls++Start in **Alerts → Overview** with the affected incident, not an old email or+**Recent delivery activity**. Match the resource and metric; for guest disks,+match the filesystem too. Freeing space on one filesystem does not establish+recovery of another, a host storage pool or a separate backup-age alert.++Where the card supplies live metric details, read its evaluated value and+clear rule together:++- **Alert still open** means the evaluated value has fallen below the trigger+ but has not completed the clear rule. Usage rules have separate trigger and+ clear values: with a 90% trigger and 85% clear, 88% can still hold the alert.+ This is an example, not the saved rule for every resource. Check+ [threshold defaults and resource overrides](CONFIGURATION.md#metric-thresholds-off-and-inheritance).+- **Recovering** means the reading meets the clear level but the recovery delay+ is still being evaluated. Use the reported progress; elapsed wall-clock time+ alone does not prove that fresh eligible observations continued throughout.+ If the card says **averaged**, compare that evaluated value with the clear+ level, not only its **latest** sample or a different tool's instantaneous value.+- **No recent reading** is retained evidence, not a healthy zero or a completed+ recovery. The separate **Last reading at or above** value describes the+ previous breach, not current usage. Older versions or a restored incident+ may lack live detail; missing detail does not establish a current breach.++Acknowledgement and snooze affect attention or notification delivery; they are+not required for a threshold alert to recover. A retained failed notification+can outlive its incident, so use [delivery recovery](#recover-retained-delivery-failures)+for that warning rather than changing the metric rule.++If fresh readings for the same resource remain within the displayed clear rule+but the incident does not recover, keep the original times, units, evaluated+value, clear rule and any reported progress with a locally reviewed screenshot+or excerpt. Unknown details are valid. Add only missing evidence to the+[existing report](#-getting-help); do not rerun a backup, create load, restart,+change thresholds, disable monitoring or retry notifications just to clear it.+Do not delete alert or queue data. A disappearing alert after a policy change+is not proof that the workload recovered.+#### Backup-age alerts were not evaluatedThe **Pulse** warning “Backup-age alerts were not evaluated because recoverydiff --git a/scripts/tests/test_metric_alert_help_docs.py b/scripts/tests/test_metric_alert_help_docs.pynew file mode 100644index 0000000000..e2a3135f0b--- /dev/null+++ b/scripts/tests/test_metric_alert_help_docs.py@@ -0,0 +1,73 @@+#!/usr/bin/env python3+"""Keep lingering-alert help tied to live evaluation, not a retained breach."""++from pathlib import Path+import unittest+++ROOT = Path(__file__).resolve().parents[2]+HEADING = '#### Usage alert stays open after the reading falls\n'+ANCHOR = '#usage-alert-stays-open-after-the-reading-falls'+++class MetricAlertHelpDocsTest(unittest.TestCase):+ def setUp(self):+ self.source = (ROOT / 'docs/TROUBLESHOOTING.md').read_text()+ self.guide = ' '.join(self.source.split(HEADING, 1)[1].split('\n#### ', 1)[0].split())+ self.presentation = (ROOT / 'frontend-modern/src/features/alerts/metricAlertPresentation.ts').read_text()++ def test_workflow_entry_reaches_one_section_in_the_shipped_copy(self):+ self.assertEqual(self.source.count(HEADING), 1)+ self.assertIn(ANCHOR, self.source.split('## ⚡ Quick Fixes', 1)[0])+ self.assertEqual((ROOT / 'docs/TROUBLESHOOTING.md').read_bytes(),+ (ROOT / 'frontend-modern/public/docs/TROUBLESHOOTING.md').read_bytes())++ def test_named_states_and_units_follow_the_actual_card_presenter(self):+ for label in ('Alert still open', 'Recovering', 'No recent reading'):+ with self.subTest(label=label):+ self.assertIn(f"phaseLabel = '{label}'", self.presentation)+ self.assertIn(f'**{label}**', self.guide)+ for label in ('averaged', 'latest', 'Last reading at or above'):+ with self.subTest(label=label):+ self.assertIn(label, self.presentation)+ self.assertIn(f'**{label}**', self.guide)+ card = (ROOT / 'frontend-modern/src/features/alerts/AlertOverviewAlertCard.tsx').read_text()+ self.assertIn('getMetricAlertPresentation(props.alert', card)+ self.assertIn('not current usage', self.guide)+ self.assertIn('match the filesystem too', self.guide)++ def test_clear_rule_example_is_not_a_universal_or_immediate_promise(self):+ self.assertIn('90% trigger and 85% clear, 88% can still hold', self.guide)+ self.assertIn('not the saved rule for every resource', self.guide)+ self.assertIn('elapsed wall-clock time alone does not prove', self.guide)+ self.assertIn('fresh eligible observations', self.guide)+ anchor = 'CONFIGURATION.md#metric-thresholds-off-and-inheritance'+ self.assertIn(anchor, self.guide)+ self.assertIn('### Metric thresholds, Off and inheritance',+ (ROOT / 'docs/CONFIGURATION.md').read_text())+ # The producer retains breach fields during a hold; only live status moves.+ producer = (ROOT / 'internal/alerts/canonical_metric.go').read_text()+ hold = producer.split('if !triggered && primary == "" {', 1)[1].split('return true', 1)[0]+ self.assertIn('existingAlert.MetricStatus = metricStatus', hold)+ self.assertNotIn('existingAlert.Value =', hold)+ self.assertNotIn('existingAlert.LastSeen =', hold)++ def test_missing_data_attention_and_delivery_are_not_measured_recovery(self):+ for boundary in (+ 'retained evidence, not a healthy zero',+ 'missing detail does not establish a current breach',+ 'not required for a threshold alert to recover',+ 'retained failed notification can outlive its incident',+ '#recover-retained-delivery-failures',+ 'Unknown details are valid', 'Add only missing evidence',+ 'do not rerun a backup, create load, restart, change thresholds',+ 'disable monitoring or retry notifications', 'Do not delete alert or queue data',+ 'not proof that the workload recovered',+ ):+ with self.subTest(boundary=boundary):+ self.assertIn(boundary, self.guide)+ self.assertNotIn('```', self.guide)+++if __name__ == '__main__':+ unittest.main()