Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Make webhook retry guidance match normal queued delivery

pulse-triage[bot]8ebd62d141ab
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md
index f9c49e806f..74f3afa64a 100644
--- a/docs/TROUBLESHOOTING.md
+++ b/docs/TROUBLESHOOTING.md
@@ -614,9 +614,15 @@ alert was generated, routed or delivered to the intended recipient.
#### Recover retained delivery failures
Pulse shows a delivery warning for failed or dead-lettered notifications in its
-persistent queue, not for every recoverable retry. **Recent delivery activity**
-includes safely redacted provider errors; completed attempts remain for 7 days
-and dead-letter attempts for 30 days. Start with the failure class and timestamp:
+persistent queue, not for every recoverable retry. Authentication,
+configuration and rejected failures stop automatic retries as soon as they are
+classified, even with attempts left. A terminal failure can therefore appear
+without exhausting the retry budget; waiting alone will not resend it. See
+[webhook retry behaviour](WEBHOOKS.md#-delivery-contract) for HTTP exceptions,
+transient failures and the normal queued attempt limit.
+
+**Recent delivery activity** includes safely redacted provider errors;
+completed attempts remain for 7 days and dead-letter attempts for 30 days. Start with the failure class and timestamp:
| Failure | Check before retrying |
| --- | --- |
diff --git a/docs/WEBHOOKS.md b/docs/WEBHOOKS.md
index 126cfe1eaf..7b26879430 100644
--- a/docs/WEBHOOKS.md
+++ b/docs/WEBHOOKS.md
@@ -100,7 +100,35 @@ severity and other delivery policies still apply.
**Resource tag routing.** Email and each alert webhook can be limited to resources with selected tags in **Alerts → Notifications**. An empty filter receives every alert. With multiple tags, choose **Match all tags** or **Match any tag**. Matching ignores case. Proxmox tags are matched as shown; Docker container and service labels are exposed as `key:value` tags (or `key` when the label value is empty). Recovery notifications follow the destinations that received the firing alert, even if a resource's tags change before recovery.
-**Retries.** Failed deliveries retry with exponential backoff. The persistent notification queue makes up to 3 delivery attempts per notification; webhooks configured with transport-level retry add up to 3 more HTTP retries per attempt (1s doubling to a 30s cap, honoring `Retry-After` on HTTP 429). A receiver may therefore see the same logical event more than once.
+**Retries and retained failures.**
+
+Normal queued firing and recovery webhooks have a budget of **up to three
+queue delivery attempts, including the initial attempt**. This is a ceiling,
+not a promise to retry every failure or a guarantee of delivery. The normal
+queued webhook sender has no extra transport retry loop; do not assume three
+extra HTTP retries or a provider's `Retry-After` wait on this path.
+
+| Observed failure | Automatic queue behaviour |
+| --- | --- |
+| Authentication, configuration or rejected request (most HTTP 4xx, including 400, 401 and 403) | Stops as soon as this failure is classified, even with attempts left; retains the delivery as a terminal failure. |
+| HTTP 408, 421, 423, 425 or 429; HTTP 5xx; connectivity or unknown failure | Can retry with backoff while the saved attempt budget remains; exhaustion retains a terminal failure. |
+| TLS failure | Can retry within the saved budget, but retrying does not repair certificate trust, expiry or hostname errors. Do not disable verification. |
+
+A terminal failure means **no further automatic retry**, not that the delivery
+was successful or its history was deleted. In **Alerts → Notifications**, use
+**Recent delivery activity** to check the destination, timestamp, failure class
+and HTTP status. A pending or held delivery is not a terminal failure; quiet
+hours and other delivery policies can postpone it. There is no fixed delivery
+deadline promised by the attempt count.
+
+Before choosing **Retry retained deliveries**, follow
+[retained-failure recovery](TROUBLESHOOTING.md#recover-retained-delivery-failures).
+It acts on all retained terminal failures, not just one webhook, and keeps their
+original destination settings. A successful **Test** uses current settings and
+does not validate or resend those saved deliveries. Do not repeat tests or
+batch retries to diagnose rate limiting. A receiver can see a duplicate if an
+earlier request was accepted but Pulse did not receive its response; preserve
+[receiver deduplication](#receiver-correlation-and-deduplication).
**Correlation header.** Alert webhooks carry `X-Pulse-Event-ID` in the form
`<alertID>:<event>` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). Retries retain it,
diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.md
index f9c49e806f..74f3afa64a 100644
--- a/frontend-modern/public/docs/TROUBLESHOOTING.md
+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md
@@ -614,9 +614,15 @@ alert was generated, routed or delivered to the intended recipient.
#### Recover retained delivery failures
Pulse shows a delivery warning for failed or dead-lettered notifications in its
-persistent queue, not for every recoverable retry. **Recent delivery activity**
-includes safely redacted provider errors; completed attempts remain for 7 days
-and dead-letter attempts for 30 days. Start with the failure class and timestamp:
+persistent queue, not for every recoverable retry. Authentication,
+configuration and rejected failures stop automatic retries as soon as they are
+classified, even with attempts left. A terminal failure can therefore appear
+without exhausting the retry budget; waiting alone will not resend it. See
+[webhook retry behaviour](WEBHOOKS.md#-delivery-contract) for HTTP exceptions,
+transient failures and the normal queued attempt limit.
+
+**Recent delivery activity** includes safely redacted provider errors;
+completed attempts remain for 7 days and dead-letter attempts for 30 days. Start with the failure class and timestamp:
| Failure | Check before retrying |
| --- | --- |
diff --git a/frontend-modern/public/docs/WEBHOOKS.md b/frontend-modern/public/docs/WEBHOOKS.md
index 126cfe1eaf..7b26879430 100644
--- a/frontend-modern/public/docs/WEBHOOKS.md
+++ b/frontend-modern/public/docs/WEBHOOKS.md
@@ -100,7 +100,35 @@ severity and other delivery policies still apply.
**Resource tag routing.** Email and each alert webhook can be limited to resources with selected tags in **Alerts → Notifications**. An empty filter receives every alert. With multiple tags, choose **Match all tags** or **Match any tag**. Matching ignores case. Proxmox tags are matched as shown; Docker container and service labels are exposed as `key:value` tags (or `key` when the label value is empty). Recovery notifications follow the destinations that received the firing alert, even if a resource's tags change before recovery.
-**Retries.** Failed deliveries retry with exponential backoff. The persistent notification queue makes up to 3 delivery attempts per notification; webhooks configured with transport-level retry add up to 3 more HTTP retries per attempt (1s doubling to a 30s cap, honoring `Retry-After` on HTTP 429). A receiver may therefore see the same logical event more than once.
+**Retries and retained failures.**
+
+Normal queued firing and recovery webhooks have a budget of **up to three
+queue delivery attempts, including the initial attempt**. This is a ceiling,
+not a promise to retry every failure or a guarantee of delivery. The normal
+queued webhook sender has no extra transport retry loop; do not assume three
+extra HTTP retries or a provider's `Retry-After` wait on this path.
+
+| Observed failure | Automatic queue behaviour |
+| --- | --- |
+| Authentication, configuration or rejected request (most HTTP 4xx, including 400, 401 and 403) | Stops as soon as this failure is classified, even with attempts left; retains the delivery as a terminal failure. |
+| HTTP 408, 421, 423, 425 or 429; HTTP 5xx; connectivity or unknown failure | Can retry with backoff while the saved attempt budget remains; exhaustion retains a terminal failure. |
+| TLS failure | Can retry within the saved budget, but retrying does not repair certificate trust, expiry or hostname errors. Do not disable verification. |
+
+A terminal failure means **no further automatic retry**, not that the delivery
+was successful or its history was deleted. In **Alerts → Notifications**, use
+**Recent delivery activity** to check the destination, timestamp, failure class
+and HTTP status. A pending or held delivery is not a terminal failure; quiet
+hours and other delivery policies can postpone it. There is no fixed delivery
+deadline promised by the attempt count.
+
+Before choosing **Retry retained deliveries**, follow
+[retained-failure recovery](TROUBLESHOOTING.md#recover-retained-delivery-failures).
+It acts on all retained terminal failures, not just one webhook, and keeps their
+original destination settings. A successful **Test** uses current settings and
+does not validate or resend those saved deliveries. Do not repeat tests or
+batch retries to diagnose rate limiting. A receiver can see a duplicate if an
+earlier request was accepted but Pulse did not receive its response; preserve
+[receiver deduplication](#receiver-correlation-and-deduplication).
**Correlation header.** Alert webhooks carry `X-Pulse-Event-ID` in the form
`<alertID>:<event>` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). Retries retain it,
diff --git a/frontend-modern/src/utils/__tests__/webhookRetryHelp.test.ts b/frontend-modern/src/utils/__tests__/webhookRetryHelp.test.ts
new file mode 100644
index 0000000000..35cdeb111d
--- /dev/null
+++ b/frontend-modern/src/utils/__tests__/webhookRetryHelp.test.ts
@@ -0,0 +1,141 @@
+import { readFileSync } from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { describe, expect, it } from 'vitest';
+
+import { renderDocMarkdown } from '@/features/docs/docMarkdown';
+
+const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');
+const read = (relative: string): string => readFileSync(path.join(repoRoot, relative), 'utf8');
+const article = (name: string): HTMLElement => {
+ const element = document.createElement('article');
+ element.innerHTML = renderDocMarkdown(read(`docs/${name}.md`), name);
+ return element;
+};
+const text = (element: Element): string => element.textContent!.replace(/\s+/g, ' ').trim();
+const retryText = (): string => {
+ const start = read('docs/WEBHOOKS.md').indexOf('**Retries');
+ const end = read('docs/WEBHOOKS.md').indexOf('**Correlation header.**');
+ const element = document.createElement('article');
+ element.innerHTML = renderDocMarkdown(read('docs/WEBHOOKS.md').slice(start, end), 'WEBHOOKS');
+ return text(element);
+};
+const recoveryText = (): string => {
+ const guide = read('docs/TROUBLESHOOTING.md');
+ const element = document.createElement('article');
+ element.innerHTML = renderDocMarkdown(
+ guide.split('#### Recover retained delivery failures')[1].split('#### Emails not sending')[0],
+ 'TROUBLESHOOTING',
+ );
+ return text(element);
+};
+
+describe('webhook retry help', () => {
+ it('ships the exact tested retry and recovery guidance', () => {
+ for (const name of ['WEBHOOKS', 'TROUBLESHOOTING']) {
+ expect(read(`frontend-modern/public/docs/${name}.md`)).toBe(read(`docs/${name}.md`));
+ }
+ });
+
+ it('counts the initial normal queued attempt without promising extra transport retries', () => {
+ expect(retryText()).toContain('Normal queued firing and recovery webhooks');
+ expect(retryText()).toContain(
+ 'up to three queue delivery attempts, including the initial attempt',
+ );
+ expect(retryText()).toContain(
+ 'not a promise to retry every failure or a guarantee of delivery',
+ );
+ expect(retryText()).toContain('queued webhook sender has no extra transport retry loop');
+ expect(retryText()).toContain(
+ "do not assume three extra HTTP retries or a provider's Retry-After wait on this path",
+ );
+ });
+
+ it('separates terminal request failures, temporary HTTP exceptions and TLS repair', () => {
+ const table = [...article('WEBHOOKS').querySelectorAll('table')].find((element) =>
+ text(element).startsWith('Observed failure Automatic queue behaviour'),
+ );
+ expect(table, 'retry policy must be a readable two-column table').toBeDefined();
+ const rows = [...table!.querySelectorAll('tbody tr')].map((row) =>
+ [...row.querySelectorAll('td')].map(text),
+ );
+ expect(rows).toEqual([
+ [
+ 'Authentication, configuration or rejected request (most HTTP 4xx, including 400, 401 and 403)',
+ 'Stops as soon as this failure is classified, even with attempts left; retains the delivery as a terminal failure.',
+ ],
+ [
+ 'HTTP 408, 421, 423, 425 or 429; HTTP 5xx; connectivity or unknown failure',
+ 'Can retry with backoff while the saved attempt budget remains; exhaustion retains a terminal failure.',
+ ],
+ [
+ 'TLS failure',
+ 'Can retry within the saved budget, but retrying does not repair certificate trust, expiry or hostname errors. Do not disable verification.',
+ ],
+ ]);
+ });
+
+ it('distinguishes terminal failure from success, history removal and held delivery', () => {
+ expect(retryText()).toContain('A terminal failure means no further automatic retry');
+ expect(retryText()).toContain(
+ 'not that the delivery was successful or its history was deleted',
+ );
+ expect(retryText()).toContain('destination, timestamp, failure class and HTTP status');
+ expect(retryText()).toContain('A pending or held delivery is not a terminal failure');
+ expect(retryText()).toContain(
+ 'There is no fixed delivery deadline promised by the attempt count',
+ );
+ });
+
+ it('does not let a settings Test or blind batch retry stand in for saved delivery acceptance', () => {
+ expect(retryText()).toContain('all retained terminal failures, not just one webhook');
+ expect(retryText()).toContain('keeps their original destination settings');
+ expect(retryText()).toContain(
+ 'A successful Test uses current settings and does not validate or resend those saved deliveries',
+ );
+ expect(retryText()).toContain('Do not repeat tests or batch retries to diagnose rate limiting');
+ expect(retryText()).toContain(
+ 'A receiver can see a duplicate if an earlier request was accepted but Pulse did not receive its response',
+ );
+ });
+
+ it('explains an unused retry budget at the recovery entry point', () => {
+ expect(recoveryText()).toContain(
+ 'Authentication, configuration and rejected failures stop automatic retries as soon as they are classified, even with attempts left',
+ );
+ expect(recoveryText()).toContain(
+ 'A terminal failure can therefore appear without exhausting the retry budget; waiting alone will not resend it',
+ );
+ // Preserve the existing operator safety constraints rather than suggesting
+ // that changing a credential silently rewrites an old delivery.
+ expect(recoveryText()).toContain('does not replace that saved configuration');
+ expect(recoveryText()).toContain('all retained terminal failures');
+ expect(recoveryText()).toContain('leave the failures retained');
+ expect(recoveryText()).toContain('Do not delete notification_queue.db');
+ });
+
+ it('routes recovery and retry links to actual shipped sections', () => {
+ const cases = [
+ [
+ 'WEBHOOKS',
+ 'TROUBLESHOOTING',
+ 'recover-retained-delivery-failures',
+ 'retained-failure recovery',
+ ],
+ ['TROUBLESHOOTING', 'WEBHOOKS', '-delivery-contract', 'webhook retry behaviour'],
+ ];
+ for (const [from, to, fragment, label] of cases) {
+ const link = article(from).querySelector(`a[href="/docs/${to}#${fragment}"]`);
+ expect(link?.textContent).toBe(label);
+ expect(link?.hasAttribute('data-doc-link')).toBe(true);
+ expect(article(to).querySelector(`[id="${fragment}"]`)).not.toBeNull();
+ }
+ const link = article('WEBHOOKS').querySelector(
+ 'a[href="#receiver-correlation-and-deduplication"]',
+ );
+ expect(link).not.toBeNull();
+ expect(
+ article('WEBHOOKS').querySelector('#receiver-correlation-and-deduplication'),
+ ).not.toBeNull();
+ });
+});