Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Make webhook retry guidance match normal queued delivery
pulse-triage[bot]8ebd62d141ab
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.mdindex f9c49e806f..74f3afa64a 100644--- a/docs/TROUBLESHOOTING.md+++ b/docs/TROUBLESHOOTING.md@@ -614,9 +614,15 @@ alert was generated, routed or delivered to the intended recipient.#### Recover retained delivery failuresPulse shows a delivery warning for failed or dead-lettered notifications in its-persistent queue, not for every recoverable retry. **Recent delivery activity**-includes safely redacted provider errors; completed attempts remain for 7 days-and dead-letter attempts for 30 days. Start with the failure class and timestamp:+persistent queue, not for every recoverable retry. Authentication,+configuration and rejected failures stop automatic retries as soon as they are+classified, even with attempts left. A terminal failure can therefore appear+without exhausting the retry budget; waiting alone will not resend it. See+[webhook retry behaviour](WEBHOOKS.md#-delivery-contract) for HTTP exceptions,+transient failures and the normal queued attempt limit.++**Recent delivery activity** includes safely redacted provider errors;+completed attempts remain for 7 days and dead-letter attempts for 30 days. Start with the failure class and timestamp:| Failure | Check before retrying || --- | --- |diff --git a/docs/WEBHOOKS.md b/docs/WEBHOOKS.mdindex 126cfe1eaf..7b26879430 100644--- a/docs/WEBHOOKS.md+++ b/docs/WEBHOOKS.md@@ -100,7 +100,35 @@ severity and other delivery policies still apply.**Resource tag routing.** Email and each alert webhook can be limited to resources with selected tags in **Alerts → Notifications**. An empty filter receives every alert. With multiple tags, choose **Match all tags** or **Match any tag**. Matching ignores case. Proxmox tags are matched as shown; Docker container and service labels are exposed as `key:value` tags (or `key` when the label value is empty). Recovery notifications follow the destinations that received the firing alert, even if a resource's tags change before recovery.-**Retries.** Failed deliveries retry with exponential backoff. The persistent notification queue makes up to 3 delivery attempts per notification; webhooks configured with transport-level retry add up to 3 more HTTP retries per attempt (1s doubling to a 30s cap, honoring `Retry-After` on HTTP 429). A receiver may therefore see the same logical event more than once.+**Retries and retained failures.**++Normal queued firing and recovery webhooks have a budget of **up to three+queue delivery attempts, including the initial attempt**. This is a ceiling,+not a promise to retry every failure or a guarantee of delivery. The normal+queued webhook sender has no extra transport retry loop; do not assume three+extra HTTP retries or a provider's `Retry-After` wait on this path.++| Observed failure | Automatic queue behaviour |+| --- | --- |+| Authentication, configuration or rejected request (most HTTP 4xx, including 400, 401 and 403) | Stops as soon as this failure is classified, even with attempts left; retains the delivery as a terminal failure. |+| HTTP 408, 421, 423, 425 or 429; HTTP 5xx; connectivity or unknown failure | Can retry with backoff while the saved attempt budget remains; exhaustion retains a terminal failure. |+| TLS failure | Can retry within the saved budget, but retrying does not repair certificate trust, expiry or hostname errors. Do not disable verification. |++A terminal failure means **no further automatic retry**, not that the delivery+was successful or its history was deleted. In **Alerts → Notifications**, use+**Recent delivery activity** to check the destination, timestamp, failure class+and HTTP status. A pending or held delivery is not a terminal failure; quiet+hours and other delivery policies can postpone it. There is no fixed delivery+deadline promised by the attempt count.++Before choosing **Retry retained deliveries**, follow+[retained-failure recovery](TROUBLESHOOTING.md#recover-retained-delivery-failures).+It acts on all retained terminal failures, not just one webhook, and keeps their+original destination settings. A successful **Test** uses current settings and+does not validate or resend those saved deliveries. Do not repeat tests or+batch retries to diagnose rate limiting. A receiver can see a duplicate if an+earlier request was accepted but Pulse did not receive its response; preserve+[receiver deduplication](#receiver-correlation-and-deduplication).**Correlation header.** Alert webhooks carry `X-Pulse-Event-ID` in the form`<alertID>:<event>` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). Retries retain it,diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.mdindex f9c49e806f..74f3afa64a 100644--- a/frontend-modern/public/docs/TROUBLESHOOTING.md+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md@@ -614,9 +614,15 @@ alert was generated, routed or delivered to the intended recipient.#### Recover retained delivery failuresPulse shows a delivery warning for failed or dead-lettered notifications in its-persistent queue, not for every recoverable retry. **Recent delivery activity**-includes safely redacted provider errors; completed attempts remain for 7 days-and dead-letter attempts for 30 days. Start with the failure class and timestamp:+persistent queue, not for every recoverable retry. Authentication,+configuration and rejected failures stop automatic retries as soon as they are+classified, even with attempts left. A terminal failure can therefore appear+without exhausting the retry budget; waiting alone will not resend it. See+[webhook retry behaviour](WEBHOOKS.md#-delivery-contract) for HTTP exceptions,+transient failures and the normal queued attempt limit.++**Recent delivery activity** includes safely redacted provider errors;+completed attempts remain for 7 days and dead-letter attempts for 30 days. Start with the failure class and timestamp:| Failure | Check before retrying || --- | --- |diff --git a/frontend-modern/public/docs/WEBHOOKS.md b/frontend-modern/public/docs/WEBHOOKS.mdindex 126cfe1eaf..7b26879430 100644--- a/frontend-modern/public/docs/WEBHOOKS.md+++ b/frontend-modern/public/docs/WEBHOOKS.md@@ -100,7 +100,35 @@ severity and other delivery policies still apply.**Resource tag routing.** Email and each alert webhook can be limited to resources with selected tags in **Alerts → Notifications**. An empty filter receives every alert. With multiple tags, choose **Match all tags** or **Match any tag**. Matching ignores case. Proxmox tags are matched as shown; Docker container and service labels are exposed as `key:value` tags (or `key` when the label value is empty). Recovery notifications follow the destinations that received the firing alert, even if a resource's tags change before recovery.-**Retries.** Failed deliveries retry with exponential backoff. The persistent notification queue makes up to 3 delivery attempts per notification; webhooks configured with transport-level retry add up to 3 more HTTP retries per attempt (1s doubling to a 30s cap, honoring `Retry-After` on HTTP 429). A receiver may therefore see the same logical event more than once.+**Retries and retained failures.**++Normal queued firing and recovery webhooks have a budget of **up to three+queue delivery attempts, including the initial attempt**. This is a ceiling,+not a promise to retry every failure or a guarantee of delivery. The normal+queued webhook sender has no extra transport retry loop; do not assume three+extra HTTP retries or a provider's `Retry-After` wait on this path.++| Observed failure | Automatic queue behaviour |+| --- | --- |+| Authentication, configuration or rejected request (most HTTP 4xx, including 400, 401 and 403) | Stops as soon as this failure is classified, even with attempts left; retains the delivery as a terminal failure. |+| HTTP 408, 421, 423, 425 or 429; HTTP 5xx; connectivity or unknown failure | Can retry with backoff while the saved attempt budget remains; exhaustion retains a terminal failure. |+| TLS failure | Can retry within the saved budget, but retrying does not repair certificate trust, expiry or hostname errors. Do not disable verification. |++A terminal failure means **no further automatic retry**, not that the delivery+was successful or its history was deleted. In **Alerts → Notifications**, use+**Recent delivery activity** to check the destination, timestamp, failure class+and HTTP status. A pending or held delivery is not a terminal failure; quiet+hours and other delivery policies can postpone it. There is no fixed delivery+deadline promised by the attempt count.++Before choosing **Retry retained deliveries**, follow+[retained-failure recovery](TROUBLESHOOTING.md#recover-retained-delivery-failures).+It acts on all retained terminal failures, not just one webhook, and keeps their+original destination settings. A successful **Test** uses current settings and+does not validate or resend those saved deliveries. Do not repeat tests or+batch retries to diagnose rate limiting. A receiver can see a duplicate if an+earlier request was accepted but Pulse did not receive its response; preserve+[receiver deduplication](#receiver-correlation-and-deduplication).**Correlation header.** Alert webhooks carry `X-Pulse-Event-ID` in the form`<alertID>:<event>` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). Retries retain it,diff --git a/frontend-modern/src/utils/__tests__/webhookRetryHelp.test.ts b/frontend-modern/src/utils/__tests__/webhookRetryHelp.test.tsnew file mode 100644index 0000000000..35cdeb111d--- /dev/null+++ b/frontend-modern/src/utils/__tests__/webhookRetryHelp.test.ts@@ -0,0 +1,141 @@+import { readFileSync } from 'node:fs';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { describe, expect, it } from 'vitest';++import { renderDocMarkdown } from '@/features/docs/docMarkdown';++const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');+const read = (relative: string): string => readFileSync(path.join(repoRoot, relative), 'utf8');+const article = (name: string): HTMLElement => {+ const element = document.createElement('article');+ element.innerHTML = renderDocMarkdown(read(`docs/${name}.md`), name);+ return element;+};+const text = (element: Element): string => element.textContent!.replace(/\s+/g, ' ').trim();+const retryText = (): string => {+ const start = read('docs/WEBHOOKS.md').indexOf('**Retries');+ const end = read('docs/WEBHOOKS.md').indexOf('**Correlation header.**');+ const element = document.createElement('article');+ element.innerHTML = renderDocMarkdown(read('docs/WEBHOOKS.md').slice(start, end), 'WEBHOOKS');+ return text(element);+};+const recoveryText = (): string => {+ const guide = read('docs/TROUBLESHOOTING.md');+ const element = document.createElement('article');+ element.innerHTML = renderDocMarkdown(+ guide.split('#### Recover retained delivery failures')[1].split('#### Emails not sending')[0],+ 'TROUBLESHOOTING',+ );+ return text(element);+};++describe('webhook retry help', () => {+ it('ships the exact tested retry and recovery guidance', () => {+ for (const name of ['WEBHOOKS', 'TROUBLESHOOTING']) {+ expect(read(`frontend-modern/public/docs/${name}.md`)).toBe(read(`docs/${name}.md`));+ }+ });++ it('counts the initial normal queued attempt without promising extra transport retries', () => {+ expect(retryText()).toContain('Normal queued firing and recovery webhooks');+ expect(retryText()).toContain(+ 'up to three queue delivery attempts, including the initial attempt',+ );+ expect(retryText()).toContain(+ 'not a promise to retry every failure or a guarantee of delivery',+ );+ expect(retryText()).toContain('queued webhook sender has no extra transport retry loop');+ expect(retryText()).toContain(+ "do not assume three extra HTTP retries or a provider's Retry-After wait on this path",+ );+ });++ it('separates terminal request failures, temporary HTTP exceptions and TLS repair', () => {+ const table = [...article('WEBHOOKS').querySelectorAll('table')].find((element) =>+ text(element).startsWith('Observed failure Automatic queue behaviour'),+ );+ expect(table, 'retry policy must be a readable two-column table').toBeDefined();+ const rows = [...table!.querySelectorAll('tbody tr')].map((row) =>+ [...row.querySelectorAll('td')].map(text),+ );+ expect(rows).toEqual([+ [+ 'Authentication, configuration or rejected request (most HTTP 4xx, including 400, 401 and 403)',+ 'Stops as soon as this failure is classified, even with attempts left; retains the delivery as a terminal failure.',+ ],+ [+ 'HTTP 408, 421, 423, 425 or 429; HTTP 5xx; connectivity or unknown failure',+ 'Can retry with backoff while the saved attempt budget remains; exhaustion retains a terminal failure.',+ ],+ [+ 'TLS failure',+ 'Can retry within the saved budget, but retrying does not repair certificate trust, expiry or hostname errors. Do not disable verification.',+ ],+ ]);+ });++ it('distinguishes terminal failure from success, history removal and held delivery', () => {+ expect(retryText()).toContain('A terminal failure means no further automatic retry');+ expect(retryText()).toContain(+ 'not that the delivery was successful or its history was deleted',+ );+ expect(retryText()).toContain('destination, timestamp, failure class and HTTP status');+ expect(retryText()).toContain('A pending or held delivery is not a terminal failure');+ expect(retryText()).toContain(+ 'There is no fixed delivery deadline promised by the attempt count',+ );+ });++ it('does not let a settings Test or blind batch retry stand in for saved delivery acceptance', () => {+ expect(retryText()).toContain('all retained terminal failures, not just one webhook');+ expect(retryText()).toContain('keeps their original destination settings');+ expect(retryText()).toContain(+ 'A successful Test uses current settings and does not validate or resend those saved deliveries',+ );+ expect(retryText()).toContain('Do not repeat tests or batch retries to diagnose rate limiting');+ expect(retryText()).toContain(+ 'A receiver can see a duplicate if an earlier request was accepted but Pulse did not receive its response',+ );+ });++ it('explains an unused retry budget at the recovery entry point', () => {+ expect(recoveryText()).toContain(+ 'Authentication, configuration and rejected failures stop automatic retries as soon as they are classified, even with attempts left',+ );+ expect(recoveryText()).toContain(+ 'A terminal failure can therefore appear without exhausting the retry budget; waiting alone will not resend it',+ );+ // Preserve the existing operator safety constraints rather than suggesting+ // that changing a credential silently rewrites an old delivery.+ expect(recoveryText()).toContain('does not replace that saved configuration');+ expect(recoveryText()).toContain('all retained terminal failures');+ expect(recoveryText()).toContain('leave the failures retained');+ expect(recoveryText()).toContain('Do not delete notification_queue.db');+ });++ it('routes recovery and retry links to actual shipped sections', () => {+ const cases = [+ [+ 'WEBHOOKS',+ 'TROUBLESHOOTING',+ 'recover-retained-delivery-failures',+ 'retained-failure recovery',+ ],+ ['TROUBLESHOOTING', 'WEBHOOKS', '-delivery-contract', 'webhook retry behaviour'],+ ];+ for (const [from, to, fragment, label] of cases) {+ const link = article(from).querySelector(`a[href="/docs/${to}#${fragment}"]`);+ expect(link?.textContent).toBe(label);+ expect(link?.hasAttribute('data-doc-link')).toBe(true);+ expect(article(to).querySelector(`[id="${fragment}"]`)).not.toBeNull();+ }+ const link = article('WEBHOOKS').querySelector(+ 'a[href="#receiver-correlation-and-deduplication"]',+ );+ expect(link).not.toBeNull();+ expect(+ article('WEBHOOKS').querySelector('#receiver-correlation-and-deduplication'),+ ).not.toBeNull();+ });+});