Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Explain why a usage alert stays open after readings fall

pulse-triage[bot]a071af3d6789
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md
index da08657d7a..9152abe354 100644
--- a/docs/TROUBLESHOOTING.md
+++ b/docs/TROUBLESHOOTING.md
@@ -62,6 +62,9 @@ establish safe recovery.
[memory baseline and growth checks](TROUBLESHOOTING.md#memory-use-keeps-growing)
or [CPU and write-rate checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth).
Guest usage and database size are different measurements.
+- **Usage fell but an alert remains open:** compare the
+ [evaluated reading, clear level and recovery progress](#usage-alert-stays-open-after-the-reading-falls).
+ A previous breach, an active incident and a retained notification are different evidence.
- **Notification Test works but alerts do not arrive:** follow
[ordinary delivery checks](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing).
Do not trigger an alert or replay the queue just to complete a report.
@@ -706,6 +709,45 @@ planned maintenance window, preserving the deployment's settings and data.
pre-pause states are not permission to start them. **Pulse monitoring and
alerts are unavailable while stopped**; arrange independent outage coverage.
+#### Usage alert stays open after the reading falls
+
+Start in **Alerts → Overview** with the affected incident, not an old email or
+**Recent delivery activity**. Match the resource and metric; for guest disks,
+match the filesystem too. Freeing space on one filesystem does not establish
+recovery of another, a host storage pool or a separate backup-age alert.
+
+Where the card supplies live metric details, read its evaluated value and
+clear rule together:
+
+- **Alert still open** means the evaluated value has fallen below the trigger
+ but has not completed the clear rule. Usage rules have separate trigger and
+ clear values: with a 90% trigger and 85% clear, 88% can still hold the alert.
+ This is an example, not the saved rule for every resource. Check
+ [threshold defaults and resource overrides](CONFIGURATION.md#metric-thresholds-off-and-inheritance).
+- **Recovering** means the reading meets the clear level but the recovery delay
+ is still being evaluated. Use the reported progress; elapsed wall-clock time
+ alone does not prove that fresh eligible observations continued throughout.
+ If the card says **averaged**, compare that evaluated value with the clear
+ level, not only its **latest** sample or a different tool's instantaneous value.
+- **No recent reading** is retained evidence, not a healthy zero or a completed
+ recovery. The separate **Last reading at or above** value describes the
+ previous breach, not current usage. Older versions or a restored incident
+ may lack live detail; missing detail does not establish a current breach.
+
+Acknowledgement and snooze affect attention or notification delivery; they are
+not required for a threshold alert to recover. A retained failed notification
+can outlive its incident, so use [delivery recovery](#recover-retained-delivery-failures)
+for that warning rather than changing the metric rule.
+
+If fresh readings for the same resource remain within the displayed clear rule
+but the incident does not recover, keep the original times, units, evaluated
+value, clear rule and any reported progress with a locally reviewed screenshot
+or excerpt. Unknown details are valid. Add only missing evidence to the
+[existing report](#-getting-help); do not rerun a backup, create load, restart,
+change thresholds, disable monitoring or retry notifications just to clear it.
+Do not delete alert or queue data. A disappearing alert after a policy change
+is not proof that the workload recovered.
+
#### Backup-age alerts were not evaluated
The **Pulse** warning “Backup-age alerts were not evaluated because recovery
diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.md
index da08657d7a..9152abe354 100644
--- a/frontend-modern/public/docs/TROUBLESHOOTING.md
+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md
@@ -62,6 +62,9 @@ establish safe recovery.
[memory baseline and growth checks](TROUBLESHOOTING.md#memory-use-keeps-growing)
or [CPU and write-rate checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth).
Guest usage and database size are different measurements.
+- **Usage fell but an alert remains open:** compare the
+ [evaluated reading, clear level and recovery progress](#usage-alert-stays-open-after-the-reading-falls).
+ A previous breach, an active incident and a retained notification are different evidence.
- **Notification Test works but alerts do not arrive:** follow
[ordinary delivery checks](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing).
Do not trigger an alert or replay the queue just to complete a report.
@@ -706,6 +709,45 @@ planned maintenance window, preserving the deployment's settings and data.
pre-pause states are not permission to start them. **Pulse monitoring and
alerts are unavailable while stopped**; arrange independent outage coverage.
+#### Usage alert stays open after the reading falls
+
+Start in **Alerts → Overview** with the affected incident, not an old email or
+**Recent delivery activity**. Match the resource and metric; for guest disks,
+match the filesystem too. Freeing space on one filesystem does not establish
+recovery of another, a host storage pool or a separate backup-age alert.
+
+Where the card supplies live metric details, read its evaluated value and
+clear rule together:
+
+- **Alert still open** means the evaluated value has fallen below the trigger
+ but has not completed the clear rule. Usage rules have separate trigger and
+ clear values: with a 90% trigger and 85% clear, 88% can still hold the alert.
+ This is an example, not the saved rule for every resource. Check
+ [threshold defaults and resource overrides](CONFIGURATION.md#metric-thresholds-off-and-inheritance).
+- **Recovering** means the reading meets the clear level but the recovery delay
+ is still being evaluated. Use the reported progress; elapsed wall-clock time
+ alone does not prove that fresh eligible observations continued throughout.
+ If the card says **averaged**, compare that evaluated value with the clear
+ level, not only its **latest** sample or a different tool's instantaneous value.
+- **No recent reading** is retained evidence, not a healthy zero or a completed
+ recovery. The separate **Last reading at or above** value describes the
+ previous breach, not current usage. Older versions or a restored incident
+ may lack live detail; missing detail does not establish a current breach.
+
+Acknowledgement and snooze affect attention or notification delivery; they are
+not required for a threshold alert to recover. A retained failed notification
+can outlive its incident, so use [delivery recovery](#recover-retained-delivery-failures)
+for that warning rather than changing the metric rule.
+
+If fresh readings for the same resource remain within the displayed clear rule
+but the incident does not recover, keep the original times, units, evaluated
+value, clear rule and any reported progress with a locally reviewed screenshot
+or excerpt. Unknown details are valid. Add only missing evidence to the
+[existing report](#-getting-help); do not rerun a backup, create load, restart,
+change thresholds, disable monitoring or retry notifications just to clear it.
+Do not delete alert or queue data. A disappearing alert after a policy change
+is not proof that the workload recovered.
+
#### Backup-age alerts were not evaluated
The **Pulse** warning “Backup-age alerts were not evaluated because recovery
diff --git a/scripts/tests/test_metric_alert_help_docs.py b/scripts/tests/test_metric_alert_help_docs.py
new file mode 100644
index 0000000000..e2a3135f0b
--- /dev/null
+++ b/scripts/tests/test_metric_alert_help_docs.py
@@ -0,0 +1,73 @@
+#!/usr/bin/env python3
+"""Keep lingering-alert help tied to live evaluation, not a retained breach."""
+
+from pathlib import Path
+import unittest
+
+
+ROOT = Path(__file__).resolve().parents[2]
+HEADING = '#### Usage alert stays open after the reading falls\n'
+ANCHOR = '#usage-alert-stays-open-after-the-reading-falls'
+
+
+class MetricAlertHelpDocsTest(unittest.TestCase):
+ def setUp(self):
+ self.source = (ROOT / 'docs/TROUBLESHOOTING.md').read_text()
+ self.guide = ' '.join(self.source.split(HEADING, 1)[1].split('\n#### ', 1)[0].split())
+ self.presentation = (ROOT / 'frontend-modern/src/features/alerts/metricAlertPresentation.ts').read_text()
+
+ def test_workflow_entry_reaches_one_section_in_the_shipped_copy(self):
+ self.assertEqual(self.source.count(HEADING), 1)
+ self.assertIn(ANCHOR, self.source.split('## ⚡ Quick Fixes', 1)[0])
+ self.assertEqual((ROOT / 'docs/TROUBLESHOOTING.md').read_bytes(),
+ (ROOT / 'frontend-modern/public/docs/TROUBLESHOOTING.md').read_bytes())
+
+ def test_named_states_and_units_follow_the_actual_card_presenter(self):
+ for label in ('Alert still open', 'Recovering', 'No recent reading'):
+ with self.subTest(label=label):
+ self.assertIn(f"phaseLabel = '{label}'", self.presentation)
+ self.assertIn(f'**{label}**', self.guide)
+ for label in ('averaged', 'latest', 'Last reading at or above'):
+ with self.subTest(label=label):
+ self.assertIn(label, self.presentation)
+ self.assertIn(f'**{label}**', self.guide)
+ card = (ROOT / 'frontend-modern/src/features/alerts/AlertOverviewAlertCard.tsx').read_text()
+ self.assertIn('getMetricAlertPresentation(props.alert', card)
+ self.assertIn('not current usage', self.guide)
+ self.assertIn('match the filesystem too', self.guide)
+
+ def test_clear_rule_example_is_not_a_universal_or_immediate_promise(self):
+ self.assertIn('90% trigger and 85% clear, 88% can still hold', self.guide)
+ self.assertIn('not the saved rule for every resource', self.guide)
+ self.assertIn('elapsed wall-clock time alone does not prove', self.guide)
+ self.assertIn('fresh eligible observations', self.guide)
+ anchor = 'CONFIGURATION.md#metric-thresholds-off-and-inheritance'
+ self.assertIn(anchor, self.guide)
+ self.assertIn('### Metric thresholds, Off and inheritance',
+ (ROOT / 'docs/CONFIGURATION.md').read_text())
+ # The producer retains breach fields during a hold; only live status moves.
+ producer = (ROOT / 'internal/alerts/canonical_metric.go').read_text()
+ hold = producer.split('if !triggered && primary == "" {', 1)[1].split('return true', 1)[0]
+ self.assertIn('existingAlert.MetricStatus = metricStatus', hold)
+ self.assertNotIn('existingAlert.Value =', hold)
+ self.assertNotIn('existingAlert.LastSeen =', hold)
+
+ def test_missing_data_attention_and_delivery_are_not_measured_recovery(self):
+ for boundary in (
+ 'retained evidence, not a healthy zero',
+ 'missing detail does not establish a current breach',
+ 'not required for a threshold alert to recover',
+ 'retained failed notification can outlive its incident',
+ '#recover-retained-delivery-failures',
+ 'Unknown details are valid', 'Add only missing evidence',
+ 'do not rerun a backup, create load, restart, change thresholds',
+ 'disable monitoring or retry notifications', 'Do not delete alert or queue data',
+ 'not proof that the workload recovered',
+ ):
+ with self.subTest(boundary=boundary):
+ self.assertIn(boundary, self.guide)
+ self.assertNotIn('```', self.guide)
+
+
+if __name__ == '__main__':
+ unittest.main()