Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Explain alert reminders and destination-qualified recovery

pulse-triage[bot]96648dab049a
diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md
index 2e4a1ee2b6..472a0a4eb2 100644
--- a/docs/CONFIGURATION.md
+++ b/docs/CONFIGURATION.md
@@ -549,6 +549,45 @@ This tolerance delays activation; it does not disable powered-off monitoring.
Use the existing guest offline-alert toggle when a guest should never produce
powered-off alerts.
+### Alert reminders and recovery notifications
+
+Open **Alerts → Schedule**. These controls govern notification eligibility, not
+when a threshold first activates or when fresh readings clear an incident.
+Save changes and reload to confirm the running settings.
+
+| Control | Effect |
+| --- | --- |
+| **Alert cooldown** on | **Cooldown period** is the minimum interval, in minutes, before an existing alert can send an ordinary reminder after its last notification. It does not delay the first eligible notification. |
+| **Alert cooldown** off | Stops ordinary reminders for the same alert occurrence after its first notification. Off does **not** mean “send on every poll” or “no rate limit”, and it does not turn off monitoring or initial delivery. |
+| **Max alerts / hour** | Limits ordinary notifications per workload/metric combination over a rolling hour, not across the whole installation. Saving cooldown as off retains this separate limit; it does not remove it. |
+| **Recovery notifications** on | Enables eligible recovery messages for destinations with a recorded successful firing delivery for that same alert occurrence. It does not send an all-clear to every configured destination. |
+| **Recovery notifications** off | Stops recovery messages, not detection of recovery or closure of the incident in Pulse. |
+
+For example, with a **30-minute** cooldown, an alert last notified at **10:00**
+is not eligible for an ordinary reminder before **10:30**. That is a minimum
+interval, not a promise of a message at 10:30: the alert must still be active
+and its other policies must permit delivery. Severity changes and configured
+escalation are handled separately; cooldown off is not a way to silence them.
+The hourly limit is not a count of provider requests: delivery retries and
+multiple recipients are separate from alert eligibility.
+
+Recovery follows the configured grouping window too, so it need not send
+immediately when the incident clears. A destination that never successfully
+received that occurrence's firing notification is not eligible for its recovery.
+A successful delivery record means provider acceptance, **not that a person
+read the message**. Acknowledgement, snooze, quiet hours, destination settings
+and provider failures can also prevent recovery delivery; enabling the toggle
+does not bypass those controls.
+
+**A missing reminder or recovery message is not a workload health check.** Check
+fresh readings and the incident's state in **Alerts → Overview** or **History**,
+then correlate [Recent delivery activity](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing).
+Keep these policies distinct from [acknowledgement and snooze](#acknowledge-and-snooze-existing-alerts),
+[quiet hours](#quiet-hours-and-notification-holds) and
+[retrying retained failures](TROUBLESHOOTING.md#recover-retained-delivery-failures).
+Do not lower thresholds, create load or stop a workload to test reminders or
+recovery messages; inspect the next normally occurring incident instead.
+
### Acknowledge and snooze existing alerts
In **Alerts → Overview**, these actions apply to an existing incident, not its
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md
index 45ff33545c..05c5b64e7e 100644
--- a/docs/TROUBLESHOOTING.md
+++ b/docs/TROUBLESHOOTING.md
@@ -712,6 +712,11 @@ alert was generated, routed or delivered to the intended recipient.
plus any mute or maintenance policy, before treating an absent attempt as a
transport failure. In **Alerts → Overview**, use **Show acknowledged** to find
hidden incidents; **Resume** ends a snooze but does not remove acknowledgement.
+- For a missing repeat or all-clear, review [alert reminders and recovery notifications](CONFIGURATION.md#alert-reminders-and-recovery-notifications).
+ Cooldown off stops ordinary reminders for the same occurrence; it does not
+ mean unlimited repeats. Recovery messages require eligible prior firing
+ delivery to that destination. Their absence does not establish whether the
+ workload recovered: check fresh readings and the incident state separately.
- Use **Recent delivery activity** to correlate the original alert, destination
and absolute timestamp, including held-notification reasons. An empty window
is not proof of healthy delivery; an **unavailable** read is not an empty log.
diff --git a/frontend-modern/public/docs/CONFIGURATION.md b/frontend-modern/public/docs/CONFIGURATION.md
index 2e4a1ee2b6..472a0a4eb2 100644
--- a/frontend-modern/public/docs/CONFIGURATION.md
+++ b/frontend-modern/public/docs/CONFIGURATION.md
@@ -549,6 +549,45 @@ This tolerance delays activation; it does not disable powered-off monitoring.
Use the existing guest offline-alert toggle when a guest should never produce
powered-off alerts.
+### Alert reminders and recovery notifications
+
+Open **Alerts → Schedule**. These controls govern notification eligibility, not
+when a threshold first activates or when fresh readings clear an incident.
+Save changes and reload to confirm the running settings.
+
+| Control | Effect |
+| --- | --- |
+| **Alert cooldown** on | **Cooldown period** is the minimum interval, in minutes, before an existing alert can send an ordinary reminder after its last notification. It does not delay the first eligible notification. |
+| **Alert cooldown** off | Stops ordinary reminders for the same alert occurrence after its first notification. Off does **not** mean “send on every poll” or “no rate limit”, and it does not turn off monitoring or initial delivery. |
+| **Max alerts / hour** | Limits ordinary notifications per workload/metric combination over a rolling hour, not across the whole installation. Saving cooldown as off retains this separate limit; it does not remove it. |
+| **Recovery notifications** on | Enables eligible recovery messages for destinations with a recorded successful firing delivery for that same alert occurrence. It does not send an all-clear to every configured destination. |
+| **Recovery notifications** off | Stops recovery messages, not detection of recovery or closure of the incident in Pulse. |
+
+For example, with a **30-minute** cooldown, an alert last notified at **10:00**
+is not eligible for an ordinary reminder before **10:30**. That is a minimum
+interval, not a promise of a message at 10:30: the alert must still be active
+and its other policies must permit delivery. Severity changes and configured
+escalation are handled separately; cooldown off is not a way to silence them.
+The hourly limit is not a count of provider requests: delivery retries and
+multiple recipients are separate from alert eligibility.
+
+Recovery follows the configured grouping window too, so it need not send
+immediately when the incident clears. A destination that never successfully
+received that occurrence's firing notification is not eligible for its recovery.
+A successful delivery record means provider acceptance, **not that a person
+read the message**. Acknowledgement, snooze, quiet hours, destination settings
+and provider failures can also prevent recovery delivery; enabling the toggle
+does not bypass those controls.
+
+**A missing reminder or recovery message is not a workload health check.** Check
+fresh readings and the incident's state in **Alerts → Overview** or **History**,
+then correlate [Recent delivery activity](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing).
+Keep these policies distinct from [acknowledgement and snooze](#acknowledge-and-snooze-existing-alerts),
+[quiet hours](#quiet-hours-and-notification-holds) and
+[retrying retained failures](TROUBLESHOOTING.md#recover-retained-delivery-failures).
+Do not lower thresholds, create load or stop a workload to test reminders or
+recovery messages; inspect the next normally occurring incident instead.
+
### Acknowledge and snooze existing alerts
In **Alerts → Overview**, these actions apply to an existing incident, not its
diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.md
index 45ff33545c..05c5b64e7e 100644
--- a/frontend-modern/public/docs/TROUBLESHOOTING.md
+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md
@@ -712,6 +712,11 @@ alert was generated, routed or delivered to the intended recipient.
plus any mute or maintenance policy, before treating an absent attempt as a
transport failure. In **Alerts → Overview**, use **Show acknowledged** to find
hidden incidents; **Resume** ends a snooze but does not remove acknowledgement.
+- For a missing repeat or all-clear, review [alert reminders and recovery notifications](CONFIGURATION.md#alert-reminders-and-recovery-notifications).
+ Cooldown off stops ordinary reminders for the same occurrence; it does not
+ mean unlimited repeats. Recovery messages require eligible prior firing
+ delivery to that destination. Their absence does not establish whether the
+ workload recovered: check fresh readings and the incident state separately.
- Use **Recent delivery activity** to correlate the original alert, destination
and absolute timestamp, including held-notification reasons. An empty window
is not proof of healthy delivery; an **unavailable** read is not an empty log.
diff --git a/frontend-modern/src/utils/__tests__/alertReminderHelp.test.ts b/frontend-modern/src/utils/__tests__/alertReminderHelp.test.ts
new file mode 100644
index 0000000000..7d5df53730
--- /dev/null
+++ b/frontend-modern/src/utils/__tests__/alertReminderHelp.test.ts
@@ -0,0 +1,131 @@
+import { readFileSync } from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { describe, expect, it } from 'vitest';
+
+import { renderDocMarkdown } from '@/features/docs/docMarkdown';
+import {
+ buildAlertsConfigurationPayload,
+ createDefaultAlertsConfigurationSnapshot,
+ readAlertsConfigurationSnapshot,
+} from '@/features/alerts/alertsConfigurationModel';
+import {
+ ALERT_CONFIG_COOLDOWN_TITLE,
+ ALERT_CONFIG_COOLDOWN_PERIOD_LABEL,
+ ALERT_CONFIG_COOLDOWN_MAX_ALERTS_LABEL,
+ ALERT_CONFIG_RECOVERY_TITLE,
+} from '@/utils/alertConfigPresentation';
+
+const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');
+const read = (name: string): string => readFileSync(path.join(root, name), 'utf8');
+
+function section(name: string, heading: string) {
+ const source = read(`docs/${name}.md`)
+ .split(heading)[1]
+ ?.split(/\n#{1,6} /)[0];
+ expect(source, `missing ${heading}`).toBeDefined();
+ const element = document.createElement('article');
+ element.innerHTML = renderDocMarkdown(heading + source!, name);
+ return element;
+}
+
+const reminderHelp = () =>
+ section('CONFIGURATION', '### Alert reminders and recovery notifications');
+const text = (element: HTMLElement) => element.textContent!.replace(/\s+/g, ' ');
+
+describe('existing alert reminders and recovery help', () => {
+ it('distinguishes first delivery, minimum reminder intervals and once-per-occurrence off', () => {
+ const guide = reminderHelp();
+ const rows = [...guide.querySelectorAll('tbody tr')].map((row) => text(row as HTMLElement));
+ expect(rows).toHaveLength(5);
+ expect(rows[0]).toContain(`${ALERT_CONFIG_COOLDOWN_TITLE} on`);
+ expect(rows[0]).toContain(ALERT_CONFIG_COOLDOWN_PERIOD_LABEL);
+ expect(rows[0]).toContain('It does not delay the first eligible notification');
+ expect(rows[1]).toContain('Stops ordinary reminders for the same alert occurrence');
+ expect(rows[1]).toContain('Off does not mean “send on every poll” or “no rate limit”');
+ expect(rows[1]).toContain('does not turn off monitoring or initial delivery');
+ expect(text(guide)).toContain('not eligible for an ordinary reminder before 10:30');
+ expect(text(guide)).toContain('not a promise of a message at 10:30');
+ expect(text(guide)).toContain('cooldown off is not a way to silence them');
+ });
+
+ it('binds off and the independent hourly limit to the actual saved and reloaded payload', () => {
+ const snapshot = createDefaultAlertsConfigurationSnapshot();
+ snapshot.scheduleCooldown = { enabled: false, minutes: 30, maxAlerts: 3 };
+ const saved = buildAlertsConfigurationPayload({
+ snapshot,
+ rawOverridesConfig: {},
+ alertsActivationState: 'active',
+ alertsActivationConfig: { enabled: true },
+ });
+ expect(saved.alertConfig?.schedule?.cooldown).toBe(0);
+ expect(saved.alertConfig?.schedule?.maxAlertsHour).toBe(3);
+ const loaded = readAlertsConfigurationSnapshot(saved.alertConfig!);
+ expect(loaded.scheduleCooldown.enabled).toBe(false);
+ expect(loaded.scheduleCooldown.maxAlerts).toBe(3);
+ const guide = text(reminderHelp());
+ expect(guide).toContain(ALERT_CONFIG_COOLDOWN_MAX_ALERTS_LABEL);
+ expect(guide).toContain('over a rolling hour, not across the whole installation');
+ expect(guide).toContain('retains this separate limit; it does not remove it');
+ expect(guide).toContain('hourly limit is not a count of provider requests');
+ });
+
+ it('keeps recovery opt-in separate from occurrence, destination and human receipt evidence', () => {
+ const guide = text(reminderHelp());
+ expect(guide).toContain(`${ALERT_CONFIG_RECOVERY_TITLE} on`);
+ expect(guide).toContain('recorded successful firing delivery for that same alert occurrence');
+ expect(guide).toContain('does not send an all-clear to every configured destination');
+ expect(guide).toContain('Stops recovery messages, not detection of recovery');
+ expect(guide).toContain('Recovery follows the configured grouping window too');
+ expect(guide).toContain('provider acceptance, not that a person read the message');
+ expect(guide).toContain('enabling the toggle does not bypass those controls');
+ expect(guide).toContain(
+ 'A missing reminder or recovery message is not a workload health check',
+ );
+ expect(guide).toContain('Do not lower thresholds, create load or stop a workload');
+ });
+
+ it('ships exact mirrors and navigable diagnosis, holds and retained-failure links', () => {
+ for (const name of ['CONFIGURATION', 'TROUBLESHOOTING']) {
+ expect(read(`frontend-modern/public/docs/${name}.md`)).toBe(read(`docs/${name}.md`));
+ }
+ // Only render the linked sections: unrelated large recipes are not part of
+ // these link assertions, and repeated whole-guide parsing obscures failures.
+ const diagnosis = section('TROUBLESHOOTING', '#### Test succeeds but real alerts are missing');
+ const reminder = reminderHelp();
+ const backlink = diagnosis.querySelector(
+ 'a[href="/docs/CONFIGURATION#alert-reminders-and-recovery-notifications"]',
+ );
+ expect(backlink?.textContent).toBe('alert reminders and recovery notifications');
+ expect(backlink?.hasAttribute('data-doc-link')).toBe(true);
+ expect(reminder.querySelector('#alert-reminders-and-recovery-notifications')).not.toBeNull();
+ expect(text(diagnosis)).toContain(
+ 'Their absence does not establish whether the workload recovered',
+ );
+ for (const [href, target, heading] of [
+ [
+ '#acknowledge-and-snooze-existing-alerts',
+ 'CONFIGURATION',
+ '### Acknowledge and snooze existing alerts',
+ ],
+ [
+ '#quiet-hours-and-notification-holds',
+ 'CONFIGURATION',
+ '### Quiet hours and notification holds',
+ ],
+ [
+ '/docs/TROUBLESHOOTING#test-succeeds-but-real-alerts-are-missing',
+ 'TROUBLESHOOTING',
+ '#### Test succeeds but real alerts are missing',
+ ],
+ [
+ '/docs/TROUBLESHOOTING#recover-retained-delivery-failures',
+ 'TROUBLESHOOTING',
+ '#### Recover retained delivery failures',
+ ],
+ ]) {
+ expect(reminder.querySelector(`a[href="${href}"]`)).not.toBeNull();
+ expect(section(target, heading).querySelector(href.slice(href.indexOf('#')))).not.toBeNull();
+ }
+ });
+});