Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Explain alert reminders and destination-qualified recovery
pulse-triage[bot]96648dab049a
diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.mdindex 2e4a1ee2b6..472a0a4eb2 100644--- a/docs/CONFIGURATION.md+++ b/docs/CONFIGURATION.md@@ -549,6 +549,45 @@ This tolerance delays activation; it does not disable powered-off monitoring.Use the existing guest offline-alert toggle when a guest should never producepowered-off alerts.+### Alert reminders and recovery notifications++Open **Alerts → Schedule**. These controls govern notification eligibility, not+when a threshold first activates or when fresh readings clear an incident.+Save changes and reload to confirm the running settings.++| Control | Effect |+| --- | --- |+| **Alert cooldown** on | **Cooldown period** is the minimum interval, in minutes, before an existing alert can send an ordinary reminder after its last notification. It does not delay the first eligible notification. |+| **Alert cooldown** off | Stops ordinary reminders for the same alert occurrence after its first notification. Off does **not** mean “send on every poll” or “no rate limit”, and it does not turn off monitoring or initial delivery. |+| **Max alerts / hour** | Limits ordinary notifications per workload/metric combination over a rolling hour, not across the whole installation. Saving cooldown as off retains this separate limit; it does not remove it. |+| **Recovery notifications** on | Enables eligible recovery messages for destinations with a recorded successful firing delivery for that same alert occurrence. It does not send an all-clear to every configured destination. |+| **Recovery notifications** off | Stops recovery messages, not detection of recovery or closure of the incident in Pulse. |++For example, with a **30-minute** cooldown, an alert last notified at **10:00**+is not eligible for an ordinary reminder before **10:30**. That is a minimum+interval, not a promise of a message at 10:30: the alert must still be active+and its other policies must permit delivery. Severity changes and configured+escalation are handled separately; cooldown off is not a way to silence them.+The hourly limit is not a count of provider requests: delivery retries and+multiple recipients are separate from alert eligibility.++Recovery follows the configured grouping window too, so it need not send+immediately when the incident clears. A destination that never successfully+received that occurrence's firing notification is not eligible for its recovery.+A successful delivery record means provider acceptance, **not that a person+read the message**. Acknowledgement, snooze, quiet hours, destination settings+and provider failures can also prevent recovery delivery; enabling the toggle+does not bypass those controls.++**A missing reminder or recovery message is not a workload health check.** Check+fresh readings and the incident's state in **Alerts → Overview** or **History**,+then correlate [Recent delivery activity](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing).+Keep these policies distinct from [acknowledgement and snooze](#acknowledge-and-snooze-existing-alerts),+[quiet hours](#quiet-hours-and-notification-holds) and+[retrying retained failures](TROUBLESHOOTING.md#recover-retained-delivery-failures).+Do not lower thresholds, create load or stop a workload to test reminders or+recovery messages; inspect the next normally occurring incident instead.+### Acknowledge and snooze existing alertsIn **Alerts → Overview**, these actions apply to an existing incident, not itsdiff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.mdindex 45ff33545c..05c5b64e7e 100644--- a/docs/TROUBLESHOOTING.md+++ b/docs/TROUBLESHOOTING.md@@ -712,6 +712,11 @@ alert was generated, routed or delivered to the intended recipient.plus any mute or maintenance policy, before treating an absent attempt as atransport failure. In **Alerts → Overview**, use **Show acknowledged** to findhidden incidents; **Resume** ends a snooze but does not remove acknowledgement.+- For a missing repeat or all-clear, review [alert reminders and recovery notifications](CONFIGURATION.md#alert-reminders-and-recovery-notifications).+ Cooldown off stops ordinary reminders for the same occurrence; it does not+ mean unlimited repeats. Recovery messages require eligible prior firing+ delivery to that destination. Their absence does not establish whether the+ workload recovered: check fresh readings and the incident state separately.- Use **Recent delivery activity** to correlate the original alert, destinationand absolute timestamp, including held-notification reasons. An empty windowis not proof of healthy delivery; an **unavailable** read is not an empty log.diff --git a/frontend-modern/public/docs/CONFIGURATION.md b/frontend-modern/public/docs/CONFIGURATION.mdindex 2e4a1ee2b6..472a0a4eb2 100644--- a/frontend-modern/public/docs/CONFIGURATION.md+++ b/frontend-modern/public/docs/CONFIGURATION.md@@ -549,6 +549,45 @@ This tolerance delays activation; it does not disable powered-off monitoring.Use the existing guest offline-alert toggle when a guest should never producepowered-off alerts.+### Alert reminders and recovery notifications++Open **Alerts → Schedule**. These controls govern notification eligibility, not+when a threshold first activates or when fresh readings clear an incident.+Save changes and reload to confirm the running settings.++| Control | Effect |+| --- | --- |+| **Alert cooldown** on | **Cooldown period** is the minimum interval, in minutes, before an existing alert can send an ordinary reminder after its last notification. It does not delay the first eligible notification. |+| **Alert cooldown** off | Stops ordinary reminders for the same alert occurrence after its first notification. Off does **not** mean “send on every poll” or “no rate limit”, and it does not turn off monitoring or initial delivery. |+| **Max alerts / hour** | Limits ordinary notifications per workload/metric combination over a rolling hour, not across the whole installation. Saving cooldown as off retains this separate limit; it does not remove it. |+| **Recovery notifications** on | Enables eligible recovery messages for destinations with a recorded successful firing delivery for that same alert occurrence. It does not send an all-clear to every configured destination. |+| **Recovery notifications** off | Stops recovery messages, not detection of recovery or closure of the incident in Pulse. |++For example, with a **30-minute** cooldown, an alert last notified at **10:00**+is not eligible for an ordinary reminder before **10:30**. That is a minimum+interval, not a promise of a message at 10:30: the alert must still be active+and its other policies must permit delivery. Severity changes and configured+escalation are handled separately; cooldown off is not a way to silence them.+The hourly limit is not a count of provider requests: delivery retries and+multiple recipients are separate from alert eligibility.++Recovery follows the configured grouping window too, so it need not send+immediately when the incident clears. A destination that never successfully+received that occurrence's firing notification is not eligible for its recovery.+A successful delivery record means provider acceptance, **not that a person+read the message**. Acknowledgement, snooze, quiet hours, destination settings+and provider failures can also prevent recovery delivery; enabling the toggle+does not bypass those controls.++**A missing reminder or recovery message is not a workload health check.** Check+fresh readings and the incident's state in **Alerts → Overview** or **History**,+then correlate [Recent delivery activity](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing).+Keep these policies distinct from [acknowledgement and snooze](#acknowledge-and-snooze-existing-alerts),+[quiet hours](#quiet-hours-and-notification-holds) and+[retrying retained failures](TROUBLESHOOTING.md#recover-retained-delivery-failures).+Do not lower thresholds, create load or stop a workload to test reminders or+recovery messages; inspect the next normally occurring incident instead.+### Acknowledge and snooze existing alertsIn **Alerts → Overview**, these actions apply to an existing incident, not itsdiff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.mdindex 45ff33545c..05c5b64e7e 100644--- a/frontend-modern/public/docs/TROUBLESHOOTING.md+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md@@ -712,6 +712,11 @@ alert was generated, routed or delivered to the intended recipient.plus any mute or maintenance policy, before treating an absent attempt as atransport failure. In **Alerts → Overview**, use **Show acknowledged** to findhidden incidents; **Resume** ends a snooze but does not remove acknowledgement.+- For a missing repeat or all-clear, review [alert reminders and recovery notifications](CONFIGURATION.md#alert-reminders-and-recovery-notifications).+ Cooldown off stops ordinary reminders for the same occurrence; it does not+ mean unlimited repeats. Recovery messages require eligible prior firing+ delivery to that destination. Their absence does not establish whether the+ workload recovered: check fresh readings and the incident state separately.- Use **Recent delivery activity** to correlate the original alert, destinationand absolute timestamp, including held-notification reasons. An empty windowis not proof of healthy delivery; an **unavailable** read is not an empty log.diff --git a/frontend-modern/src/utils/__tests__/alertReminderHelp.test.ts b/frontend-modern/src/utils/__tests__/alertReminderHelp.test.tsnew file mode 100644index 0000000000..7d5df53730--- /dev/null+++ b/frontend-modern/src/utils/__tests__/alertReminderHelp.test.ts@@ -0,0 +1,131 @@+import { readFileSync } from 'node:fs';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { describe, expect, it } from 'vitest';++import { renderDocMarkdown } from '@/features/docs/docMarkdown';+import {+ buildAlertsConfigurationPayload,+ createDefaultAlertsConfigurationSnapshot,+ readAlertsConfigurationSnapshot,+} from '@/features/alerts/alertsConfigurationModel';+import {+ ALERT_CONFIG_COOLDOWN_TITLE,+ ALERT_CONFIG_COOLDOWN_PERIOD_LABEL,+ ALERT_CONFIG_COOLDOWN_MAX_ALERTS_LABEL,+ ALERT_CONFIG_RECOVERY_TITLE,+} from '@/utils/alertConfigPresentation';++const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');+const read = (name: string): string => readFileSync(path.join(root, name), 'utf8');++function section(name: string, heading: string) {+ const source = read(`docs/${name}.md`)+ .split(heading)[1]+ ?.split(/\n#{1,6} /)[0];+ expect(source, `missing ${heading}`).toBeDefined();+ const element = document.createElement('article');+ element.innerHTML = renderDocMarkdown(heading + source!, name);+ return element;+}++const reminderHelp = () =>+ section('CONFIGURATION', '### Alert reminders and recovery notifications');+const text = (element: HTMLElement) => element.textContent!.replace(/\s+/g, ' ');++describe('existing alert reminders and recovery help', () => {+ it('distinguishes first delivery, minimum reminder intervals and once-per-occurrence off', () => {+ const guide = reminderHelp();+ const rows = [...guide.querySelectorAll('tbody tr')].map((row) => text(row as HTMLElement));+ expect(rows).toHaveLength(5);+ expect(rows[0]).toContain(`${ALERT_CONFIG_COOLDOWN_TITLE} on`);+ expect(rows[0]).toContain(ALERT_CONFIG_COOLDOWN_PERIOD_LABEL);+ expect(rows[0]).toContain('It does not delay the first eligible notification');+ expect(rows[1]).toContain('Stops ordinary reminders for the same alert occurrence');+ expect(rows[1]).toContain('Off does not mean “send on every poll” or “no rate limit”');+ expect(rows[1]).toContain('does not turn off monitoring or initial delivery');+ expect(text(guide)).toContain('not eligible for an ordinary reminder before 10:30');+ expect(text(guide)).toContain('not a promise of a message at 10:30');+ expect(text(guide)).toContain('cooldown off is not a way to silence them');+ });++ it('binds off and the independent hourly limit to the actual saved and reloaded payload', () => {+ const snapshot = createDefaultAlertsConfigurationSnapshot();+ snapshot.scheduleCooldown = { enabled: false, minutes: 30, maxAlerts: 3 };+ const saved = buildAlertsConfigurationPayload({+ snapshot,+ rawOverridesConfig: {},+ alertsActivationState: 'active',+ alertsActivationConfig: { enabled: true },+ });+ expect(saved.alertConfig?.schedule?.cooldown).toBe(0);+ expect(saved.alertConfig?.schedule?.maxAlertsHour).toBe(3);+ const loaded = readAlertsConfigurationSnapshot(saved.alertConfig!);+ expect(loaded.scheduleCooldown.enabled).toBe(false);+ expect(loaded.scheduleCooldown.maxAlerts).toBe(3);+ const guide = text(reminderHelp());+ expect(guide).toContain(ALERT_CONFIG_COOLDOWN_MAX_ALERTS_LABEL);+ expect(guide).toContain('over a rolling hour, not across the whole installation');+ expect(guide).toContain('retains this separate limit; it does not remove it');+ expect(guide).toContain('hourly limit is not a count of provider requests');+ });++ it('keeps recovery opt-in separate from occurrence, destination and human receipt evidence', () => {+ const guide = text(reminderHelp());+ expect(guide).toContain(`${ALERT_CONFIG_RECOVERY_TITLE} on`);+ expect(guide).toContain('recorded successful firing delivery for that same alert occurrence');+ expect(guide).toContain('does not send an all-clear to every configured destination');+ expect(guide).toContain('Stops recovery messages, not detection of recovery');+ expect(guide).toContain('Recovery follows the configured grouping window too');+ expect(guide).toContain('provider acceptance, not that a person read the message');+ expect(guide).toContain('enabling the toggle does not bypass those controls');+ expect(guide).toContain(+ 'A missing reminder or recovery message is not a workload health check',+ );+ expect(guide).toContain('Do not lower thresholds, create load or stop a workload');+ });++ it('ships exact mirrors and navigable diagnosis, holds and retained-failure links', () => {+ for (const name of ['CONFIGURATION', 'TROUBLESHOOTING']) {+ expect(read(`frontend-modern/public/docs/${name}.md`)).toBe(read(`docs/${name}.md`));+ }+ // Only render the linked sections: unrelated large recipes are not part of+ // these link assertions, and repeated whole-guide parsing obscures failures.+ const diagnosis = section('TROUBLESHOOTING', '#### Test succeeds but real alerts are missing');+ const reminder = reminderHelp();+ const backlink = diagnosis.querySelector(+ 'a[href="/docs/CONFIGURATION#alert-reminders-and-recovery-notifications"]',+ );+ expect(backlink?.textContent).toBe('alert reminders and recovery notifications');+ expect(backlink?.hasAttribute('data-doc-link')).toBe(true);+ expect(reminder.querySelector('#alert-reminders-and-recovery-notifications')).not.toBeNull();+ expect(text(diagnosis)).toContain(+ 'Their absence does not establish whether the workload recovered',+ );+ for (const [href, target, heading] of [+ [+ '#acknowledge-and-snooze-existing-alerts',+ 'CONFIGURATION',+ '### Acknowledge and snooze existing alerts',+ ],+ [+ '#quiet-hours-and-notification-holds',+ 'CONFIGURATION',+ '### Quiet hours and notification holds',+ ],+ [+ '/docs/TROUBLESHOOTING#test-succeeds-but-real-alerts-are-missing',+ 'TROUBLESHOOTING',+ '#### Test succeeds but real alerts are missing',+ ],+ [+ '/docs/TROUBLESHOOTING#recover-retained-delivery-failures',+ 'TROUBLESHOOTING',+ '#### Recover retained delivery failures',+ ],+ ]) {+ expect(reminder.querySelector(`a[href="${href}"]`)).not.toBeNull();+ expect(section(target, heading).querySelector(href.slice(href.indexOf('#')))).not.toBeNull();+ }+ });+});