Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Keep webhook bridge guidance from discarding later alerts

pulse-triage[bot]94a14c78c7f8
diff --git a/docs/WEBHOOKS.md b/docs/WEBHOOKS.md
index 3acf6649b4..cb0f8424fd 100644
--- a/docs/WEBHOOKS.md
+++ b/docs/WEBHOOKS.md
@@ -65,11 +65,18 @@ For generic webhooks, use Go templates to format the JSON payload.
These fields and behaviors are stable; ticket-routing integrations can rely on them.
-**Events.** Every webhook fires on both `alert` and `resolved` events. `{{.Event}}` is `"alert"` or `"resolved"` — there is no separate "info" event class.
+**Events.** `{{.Event}}` is `"alert"` or `"resolved"` — there is no separate
+"info" event class. Recovery delivery depends on **Notify on resolve** and a
+successful firing-delivery receipt for that occurrence and destination. Do not
+assume that every configured webhook receives both events.
-**Severity.** `{{.Level}}` is `"warning"` or `"critical"`. Pulse has exactly these two alert levels.
+**Severity.** Metric alerts commonly use `"warning"` or `"critical"`;
+informational conditions can use `"info"`. Event and severity are separate:
+an informational firing still has event `"alert"`. Preserve and flag an
+unrecognised level rather than silently dropping the notification. Minimum
+severity and other delivery policies still apply.
-**Alert type.** `{{.Type}}` is the metric or condition that fired: `cpu`, `memory`, `disk`, `diskRead`, `diskWrite`, `networkIn`, `networkOut`, `connectivity`, and similar. The alert ID (`{{.ID}}`) is stable for the lifetime of an alert occurrence, so the `resolved` event carries the same ID as the `alert` event it closes.
+**Alert type.** `{{.Type}}` is the metric or condition that fired: `cpu`, `memory`, `disk`, `diskRead`, `diskWrite`, `networkIn`, `networkOut`, `connectivity`, and similar. The alert ID (`{{.ID}}`) correlates firing and recovery, but can be reused when the same condition fires again. It is not a unique incident or delivery ID.
**Message key.** `{{.MessageKey}}` is the stable, language-neutral condition key for rebuilding or translating a notification. Canonical alerts use `<kind>.<type>` (for example, `metric-threshold.disk`); older alert paths fall back to `{{.Type}}`. Combine it with `{{.Event}}` and `{{.ResourceType}}` when the receiving system needs separate wording for firing/recovery events or different resource classes. Unlike a numeric message index, the symbolic key does not change when another alert type is added.
@@ -79,7 +86,13 @@ These fields and behaviors are stable; ticket-routing integrations can rely on t
**Retries.** Failed deliveries retry with exponential backoff. The persistent notification queue makes up to 3 delivery attempts per notification; webhooks configured with transport-level retry add up to 3 more HTTP retries per attempt (1s doubling to a 30s cap, honoring `Retry-After` on HTTP 429). A receiver may therefore see the same logical event more than once.
-**Idempotency.** Every alert delivery carries an `X-Pulse-Event-ID` header of the form `<alertID>:<event>` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). It is identical across all retries of the same logical event — deduplicate on it.
+**Correlation header.** Alert webhooks carry `X-Pulse-Event-ID` in the form
+`<alertID>:<event>` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). Retries retain it,
+but later occurrences, severity changes and reminders can share it too. A group
+uses its primary alert's ID, not a unique ID for every member or batch. **Do not
+deduplicate permanently on this header or on alert ID and event alone:** doing
+so can silently discard a later incident or a changed group. See
+[receiver correlation and deduplication](#receiver-correlation-and-deduplication).
**Signed deliveries.** Set a `signingSecret` on the webhook config to enable HMAC signing. Signed requests carry:
@@ -126,14 +139,12 @@ def verify(secret: str, timestamp: str, body: bytes, signature: str) -> bool:
A time window rejects old captures, **not every replay**: the same signed
request can arrive more than once within that window, and Pulse signs each
retry with its current send time. After verification, deduplicate atomically
-before creating a ticket or taking another action. Use a durable key scoped to
-the configured Pulse sender and destination, built from the authenticated
-payload's alert ID and event (`{{.ID}}` and `{{.Event}}` in your template); keep
-it across retries and receiver restarts. An `alert` and its later `resolved`
-are different events, not duplicates. The `X-Pulse-Event-ID` header is a useful
+using the [receiver rules below](#receiver-correlation-and-deduplication),
+not the signature, send timestamp or alert ID alone. An `alert` and its later
+`resolved` are different events, not duplicates. The `X-Pulse-Event-ID` header is a useful
correlation hint, but is **not covered by this HMAC**: do not let a changed
header bypass deduplication of an otherwise identical signed event. The sample
-PSA payloads below include `alertId` and `event` for this purpose. A valid HMAC
+PSA payload below includes each member's occurrence identity for this purpose. A valid HMAC
establishes integrity, not that processing succeeded; retain your receiver's
normal durable processing and retry handling. Never log the secret or full
credential-bearing request while diagnosing a rejection.
@@ -153,6 +164,44 @@ Content-Type: application/json
}
```
+### Receiver correlation and deduplication
+
+Keep ticket correlation separate from suppressing repeated processing:
+
+- Scope durable receiver records to the **configured sender and destination**
+ and tenant, not just a resource name or alert ID. Read identity from the
+ authenticated payload's fields after verifying the request; retain it across
+ receiver restarts. The tenant field is context, not permission to act in an
+ arbitrary account.
+- For normal queued firing and recovery notifications, use each member of
+ `{{.Alerts}}`, not only the primary `{{.ID}}`. The [full PSA template below](#sample-psa-payloads)
+ includes each member's `alertId` and full-precision UTC `startedAt`. These
+ identify its observed occurrence; a later start with the same ID must be
+ processed as a new occurrence. Firing and recovery use the same occurrence
+ record, so an old delayed recovery must not close a newer incident.
+- Within that record, handle `event` **and severity**. A warning becoming
+ critical is an update, not a duplicate warning. Decide explicitly whether
+ reminders update the existing ticket; do not create another ticket for each
+ retry or discard a severity increase. Group membership can change without
+ changing the header: process every member, even when the primary is unchanged.
+- The primary `{{.StartTime}}` string has only whole-second precision. Member
+ `StartTime` values in `{{.Alerts}}` are Go times and can be formatted with
+ fractional seconds as below. Do not assume the shorter string is universally
+ unique. A test or legacy payload can lack member identity; keep incomplete or
+ ambiguous events for reconciliation rather than silently treating them as
+ duplicates. **Send test** alone does not validate this lifecycle.
+- Commit receiver state and the ticket action atomically, or use your ticket
+ system's durable idempotent operation. Do not mark an event processed before
+ its action succeeds. Exact signed-byte replays can be rejected separately,
+ but a queue retry may render a new body or send timestamp: hashing those alone
+ is not logical-event deduplication. There is no universal exactly-once key in
+ this legacy header.
+
+Validate in an authorised test environment with firing, retry, warning-to-critical,
+recovery, recurrence of the same condition and a changed multi-alert group.
+Check actual tickets and receiver restart behaviour before enabling automatic
+actions. Do not cause a production outage or notification storm for this test.
+
## 🛡️ Security
- **Private IPs**: By default, webhooks to private IPs are blocked. Allow them in **Settings → System → Network → Webhook Security**.
@@ -204,7 +253,14 @@ Suspended or pending-deletion organizations return `403`, and an unknown org ID
There are two integration models. The push model is usually the right fit when tickets or incidents should open and close automatically.
-**Push (recommended): one outbound webhook per organization.** Create a **Generic** webhook for each organization and point it at your external system's inbound endpoint (an ITSM/PSA inbound webhook, an email connector, or middleware that opens service tickets). Shape the JSON with a [custom template](#-custom-templates) so it matches the receiving system's expected schema: every template variable listed above is available. Pulse fires on both `alert` and `resolved` events (`{{.Event}}` is `"alert"` or `"resolved"`), so the receiving system can open a ticket on alert and auto-resolve it on recovery. Add authentication as a custom header (e.g. `Authorization: Bearer ...`).
+**Push (recommended): one outbound webhook per organization.** Create a
+**Generic** webhook for each organization and point it at your external system's
+inbound endpoint (an ITSM/PSA inbound webhook, an email connector, or middleware
+that opens service tickets). Shape the JSON with a [custom template](#-custom-templates)
+to match the receiving system's schema. The bridge can open or update a ticket
+on `alert` and resolve that occurrence on `resolved`, subject to the [delivery
+conditions above](#-delivery-contract). Add authentication as a custom header
+in the settings form; keep its value private.
Configure it from the UI (**Alerts → Notifications → Add Webhook**) per org, or programmatically with an org-bound admin token:
@@ -221,20 +277,25 @@ Content-Type: application/json
"service": "generic",
"enabled": true,
"headers": { "Authorization": "Bearer <psa-token>" },
- "template": "{\"summary\":\"{{.Level}}: {{.ResourceName}} {{.Message | jsonString}}\",\"event\":\"{{.Event}}\",\"alertId\":\"{{.ID}}\"}"
+ "template": "{\"summary\":\"{{.Level | jsonString}}: {{.ResourceName | jsonString}} {{.Message | jsonString}}\",\"event\":\"{{.Event | jsonString}}\",\"alertId\":\"{{.ID | jsonString}}\",\"startedAt\":\"{{.StartTime | jsonString}}\"}"
}
```
-The exact ticket fields differ by platform (ConnectWise, Autotask, Halo, and others each expect their own inbound shape), so map the template to your platform's contract. The `alertId` round-trips through `{{.ID}}`, which lets the receiving system correlate the later `resolved` event to the ticket it opened.
+The exact ticket fields differ by platform (ConnectWise, Autotask, Halo, and
+others each expect their own inbound shape), so map the template to your
+platform's contract. This short example carries only the primary alert and a
+whole-second start. For a bridge handling real grouped notifications, use the
+member-aware template below; do not use the short example as an exactly-once key.
### Sample PSA payloads
-A fuller template suited to ticket routing, including tenant identity and the
-stable severity/type fields from the [delivery contract](#-delivery-contract):
+A fuller template for normal queued firing and recovery notifications includes
+tenant context and every member's occurrence, severity and condition. The
+primary fields remain convenient summary context, **not the whole batch**:
```json
{
- "event": "{{.Event}}",
+ "event": "{{.Event | jsonString}}",
"alertId": "{{.ID | jsonString}}",
"messageKey": "{{.MessageKey | jsonString}}",
"severity": "{{.Level | jsonString}}",
@@ -248,11 +309,24 @@ stable severity/type fields from the [delivery contract](#-delivery-contract):
"value": {{.Value}},
"threshold": {{.Threshold}},
"startedAt": "{{.StartTime | jsonString}}",
- "duration": "{{.Duration | jsonString}}"
+ "duration": "{{.Duration | jsonString}}",
+ "alertCount": {{.AlertCount}},
+ "alerts": [{{$comma := ""}}{{range .Alerts}}{{if .}}{{$comma}}
+ {
+ "alertId": "{{.ID | jsonString}}",
+ "startedAt": "{{.StartTime.UTC.Format "2006-01-02T15:04:05.999999999Z07:00" | jsonString}}",
+ "severity": "{{.Level | jsonString}}",
+ "alertType": "{{.Type | jsonString}}",
+ "resourceId": "{{.ResourceID | jsonString}}",
+ "resource": "{{.ResourceName | jsonString}}",
+ "summary": "{{.Message | jsonString}}"
+ }{{$comma = ","}}{{end}}{{end}}
+ ]
}
```
-What the receiver sees for a **critical** alert:
+An abridged primary summary for a **critical** alert (the template also emits
+the `alerts` member array):
```json
{
@@ -272,9 +346,12 @@ What the receiver sees for a **critical** alert:
}
```
-A **warning** alert is identical except `"severity": "warning"` — warning and critical are the only two severities Pulse emits, so a two-priority PSA mapping covers the full range.
+A **warning** alert is identical except `"severity": "warning"`. Map
+informational levels too; a two-priority mapping does not cover every condition.
-The **resolved** event reuses the same `alertId`, letting the bridge close the ticket it opened:
+The **resolved** event retains the original member `alertId` and `startedAt`,
+letting the bridge close that occurrence's ticket rather than a later incident
+with the same ID. Its primary summary is:
```json
{
@@ -294,7 +371,12 @@ The **resolved** event reuses the same `alertId`, letting the bridge close the t
}
```
-For ConnectWise specifically, point the webhook at a ConnectWise inbound API callback (or middleware that calls the ConnectWise REST API) and map `severity` to ticket priority, `tenantName` to the company, and `alertId` to your correlation field. Combine with a [`signingSecret`](#-delivery-contract) and the `X-Pulse-Event-ID` dedup header for a production-grade bridge.
+For ConnectWise specifically, point the webhook at a ConnectWise inbound API
+callback (or middleware that calls the ConnectWise REST API). Map each member's
+severity to ticket priority and the tenant to your configured company. Combine
+[`signingSecret`](#-delivery-contract) with the [receiver correlation rules](#receiver-correlation-and-deduplication),
+not header-only deduplication. Verify your platform's processing and recovery
+behaviour before treating the bridge as production-ready.
**Pull (poll): org-scoped read API.** Issue a `monitoring:read` token bound to each organization and poll that org's alerts. Send `X-Pulse-Org-ID` (or rely on the org-bound token) so you get only that organization's data:
diff --git a/frontend-modern/browser-tests/docs-fragment-navigation.tsx b/frontend-modern/browser-tests/docs-fragment-navigation.tsx
index 373a4dd878..628e095444 100644
--- a/frontend-modern/browser-tests/docs-fragment-navigation.tsx
+++ b/frontend-modern/browser-tests/docs-fragment-navigation.tsx
@@ -25,6 +25,7 @@ const targets: Record<string, string> = {
truenas: '/docs/TRUENAS',
installation: '/docs/INSTALL',
configuration: '/docs/CONFIGURATION#-api-tokens',
+ webhooks: '/docs/WEBHOOKS',
};
window.history.replaceState({}, '', targets[scenario] ?? targets.plain);
diff --git a/frontend-modern/browser-tests/webhook-guidance.cjs b/frontend-modern/browser-tests/webhook-guidance.cjs
new file mode 100644
index 0000000000..39da41afa3
--- /dev/null
+++ b/frontend-modern/browser-tests/webhook-guidance.cjs
@@ -0,0 +1,95 @@
+// Inspect the real shipped Docs page and copyable webhook template. This is
+// offline rendering, not a ticket receiver, notification or external delivery.
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const path = require('node:path');
+const crypto = require('node:crypto');
+const { chromium, webkit } = require('playwright');
+
+(async () => {
+ const root = '/workspace/frontend-modern';
+ const artifacts = path.join(root, 'node_modules', 'webhook-guidance-browser');
+ fs.mkdirSync(artifacts, { recursive: true });
+ const { createServer } = await import(path.join(root, 'node_modules/vite/dist/node/index.js'));
+ const server = await createServer({ root, configFile: path.join(root, 'vite.config.ts'),
+ cacheDir: path.join(root, 'node_modules', '.vite-webhook-guidance'),
+ server: { host: '127.0.0.1', port: 5253, strictPort: true },
+ optimizeDeps: { noDiscovery: true, include: [] } });
+ const docs = fs.readFileSync(path.join(root, 'public/docs/WEBHOOKS.md'), 'utf8');
+ const expectedTemplate = docs.split('### Sample PSA payloads')[1].match(/```json\n([\s\S]*?)\n```/)[1];
+ let browser;
+ const results = [];
+ try {
+ await server.listen();
+ for (const [engine, width] of [['chromium', 1280], ['webkit', 390]]) {
+ browser = await { chromium, webkit }[engine].launch(engine === 'chromium'
+ ? { headless: true, channel: 'chromium', args: ['--no-sandbox'] } : { headless: true });
+ const page = await browser.newPage({ viewport: { width, height: 1000 },
+ ...(engine === 'webkit' ? { isMobile: true, hasTouch: true } : {}) });
+ const errors = [];
+ page.on('pageerror', (error) => errors.push(error.message));
+ page.on('console', (message) => { if (message.type() === 'error') errors.push(message.text()); });
+ await page.goto('http://127.0.0.1:5253/browser-tests/docs-fragment-navigation.html?scenario=webhooks',
+ { waitUntil: 'domcontentloaded', timeout: 60_000 });
+ const heading = page.getByRole('heading', { name: 'Receiver correlation and deduplication', exact: true });
+ await heading.waitFor();
+ if (engine === 'webkit') await page.evaluate(() => document.documentElement.classList.add('dark'));
+ const body = (await page.locator('article').innerText()).replace(/\s+/g, ' ');
+ for (const text of ['not a unique incident or delivery ID', 'Do not deduplicate permanently',
+ 'old delayed recovery must not close a newer incident', 'event and severity',
+ 'whole-second precision', 'incomplete or ambiguous events',
+ 'successful firing-delivery receipt', 'informational conditions can use "info"']) {
+ assert.ok(body.includes(text), text);
+ }
+ assert.ok(!body.includes('deduplicate on it'));
+ const follow = async (name, href, target) => {
+ const link = page.getByRole('link', { name, exact: true });
+ assert.equal(await link.getAttribute('href'), href);
+ await link.focus();
+ await page.keyboard.press('Enter');
+ await page.waitForURL(`**${href}`);
+ await page.waitForFunction((id) => document.activeElement?.id === id, target);
+ };
+ await follow('receiver correlation and deduplication', '#receiver-correlation-and-deduplication',
+ 'receiver-correlation-and-deduplication');
+ const screenshot = async (name, headingName) => {
+ await page.getByRole('heading', { name: headingName, exact: true })
+ .evaluate((element) => element.scrollIntoView({ block: 'start' }));
+ await page.evaluate(async () => {
+ await new Promise((resolve) => requestAnimationFrame(() => requestAnimationFrame(resolve)));
+ await Promise.all(document.getAnimations().map((animation) => animation.finished.catch(() => {})));
+ });
+ assert.ok(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth + 1));
+ await page.screenshot({ path: path.join(artifacts, `${engine}-${name}.png`) });
+ };
+ await screenshot('receiver', 'Receiver correlation and deduplication');
+ await follow('full PSA template below', '#sample-psa-payloads', 'sample-psa-payloads');
+ const template = page.locator('pre code').filter({ hasText: '"alertCount":' });
+ assert.equal(await template.count(), 1);
+ assert.equal((await template.innerText()).trim(), expectedTemplate.trim());
+ assert.ok(await template.evaluate((element) => {
+ const pre = element.closest('pre');
+ return getComputedStyle(pre).overflowX === 'auto' || pre.scrollWidth <= pre.clientWidth + 1;
+ }));
+ await screenshot('template', 'Sample PSA payloads');
+ await page.reload({ waitUntil: 'domcontentloaded' });
+ await page.getByRole('heading', { name: 'Sample PSA payloads', exact: true }).waitFor();
+ await page.waitForFunction(() => document.activeElement?.id === 'sample-psa-payloads');
+ assert.deepEqual(errors, []);
+ results.push({ engine, browserVersion: browser.version(), width,
+ scope: engine === 'webkit' ? 'phone-emulated, not native' : 'desktop',
+ warningsRendered: true, keyboardReceiverLink: true, keyboardTemplateLink: true,
+ templateTextExact: true, reloadFragmentFocus: true, noDocumentOverflow: true, errors });
+ await browser.close();
+ browser = undefined;
+ }
+ const files = ['public/docs/WEBHOOKS.md', 'browser-tests/docs-fragment-navigation.tsx',
+ 'browser-tests/webhook-guidance.cjs', 'src/features/docs/docMarkdown.ts', 'src/pages/Docs.tsx']
+ .map((file) => ({ file, sha256: crypto.createHash('sha256').update(fs.readFileSync(path.join(root, file))).digest('hex') }));
+ fs.writeFileSync(path.join(artifacts, 'result.json'), JSON.stringify({ playwright: require('playwright/package.json').version,
+ files, results, limits: 'No real receiver, ticket, provider, native device or installed notification acceptance.' }, null, 2));
+ } finally {
+ if (browser) await browser.close();
+ await server.close();
+ }
+})().catch((error) => { console.error(error); process.exitCode = 1; });
diff --git a/frontend-modern/public/docs/WEBHOOKS.md b/frontend-modern/public/docs/WEBHOOKS.md
index 3acf6649b4..cb0f8424fd 100644
--- a/frontend-modern/public/docs/WEBHOOKS.md
+++ b/frontend-modern/public/docs/WEBHOOKS.md
@@ -65,11 +65,18 @@ For generic webhooks, use Go templates to format the JSON payload.
These fields and behaviors are stable; ticket-routing integrations can rely on them.
-**Events.** Every webhook fires on both `alert` and `resolved` events. `{{.Event}}` is `"alert"` or `"resolved"` — there is no separate "info" event class.
+**Events.** `{{.Event}}` is `"alert"` or `"resolved"` — there is no separate
+"info" event class. Recovery delivery depends on **Notify on resolve** and a
+successful firing-delivery receipt for that occurrence and destination. Do not
+assume that every configured webhook receives both events.
-**Severity.** `{{.Level}}` is `"warning"` or `"critical"`. Pulse has exactly these two alert levels.
+**Severity.** Metric alerts commonly use `"warning"` or `"critical"`;
+informational conditions can use `"info"`. Event and severity are separate:
+an informational firing still has event `"alert"`. Preserve and flag an
+unrecognised level rather than silently dropping the notification. Minimum
+severity and other delivery policies still apply.
-**Alert type.** `{{.Type}}` is the metric or condition that fired: `cpu`, `memory`, `disk`, `diskRead`, `diskWrite`, `networkIn`, `networkOut`, `connectivity`, and similar. The alert ID (`{{.ID}}`) is stable for the lifetime of an alert occurrence, so the `resolved` event carries the same ID as the `alert` event it closes.
+**Alert type.** `{{.Type}}` is the metric or condition that fired: `cpu`, `memory`, `disk`, `diskRead`, `diskWrite`, `networkIn`, `networkOut`, `connectivity`, and similar. The alert ID (`{{.ID}}`) correlates firing and recovery, but can be reused when the same condition fires again. It is not a unique incident or delivery ID.
**Message key.** `{{.MessageKey}}` is the stable, language-neutral condition key for rebuilding or translating a notification. Canonical alerts use `<kind>.<type>` (for example, `metric-threshold.disk`); older alert paths fall back to `{{.Type}}`. Combine it with `{{.Event}}` and `{{.ResourceType}}` when the receiving system needs separate wording for firing/recovery events or different resource classes. Unlike a numeric message index, the symbolic key does not change when another alert type is added.
@@ -79,7 +86,13 @@ These fields and behaviors are stable; ticket-routing integrations can rely on t
**Retries.** Failed deliveries retry with exponential backoff. The persistent notification queue makes up to 3 delivery attempts per notification; webhooks configured with transport-level retry add up to 3 more HTTP retries per attempt (1s doubling to a 30s cap, honoring `Retry-After` on HTTP 429). A receiver may therefore see the same logical event more than once.
-**Idempotency.** Every alert delivery carries an `X-Pulse-Event-ID` header of the form `<alertID>:<event>` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). It is identical across all retries of the same logical event — deduplicate on it.
+**Correlation header.** Alert webhooks carry `X-Pulse-Event-ID` in the form
+`<alertID>:<event>` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). Retries retain it,
+but later occurrences, severity changes and reminders can share it too. A group
+uses its primary alert's ID, not a unique ID for every member or batch. **Do not
+deduplicate permanently on this header or on alert ID and event alone:** doing
+so can silently discard a later incident or a changed group. See
+[receiver correlation and deduplication](#receiver-correlation-and-deduplication).
**Signed deliveries.** Set a `signingSecret` on the webhook config to enable HMAC signing. Signed requests carry:
@@ -126,14 +139,12 @@ def verify(secret: str, timestamp: str, body: bytes, signature: str) -> bool:
A time window rejects old captures, **not every replay**: the same signed
request can arrive more than once within that window, and Pulse signs each
retry with its current send time. After verification, deduplicate atomically
-before creating a ticket or taking another action. Use a durable key scoped to
-the configured Pulse sender and destination, built from the authenticated
-payload's alert ID and event (`{{.ID}}` and `{{.Event}}` in your template); keep
-it across retries and receiver restarts. An `alert` and its later `resolved`
-are different events, not duplicates. The `X-Pulse-Event-ID` header is a useful
+using the [receiver rules below](#receiver-correlation-and-deduplication),
+not the signature, send timestamp or alert ID alone. An `alert` and its later
+`resolved` are different events, not duplicates. The `X-Pulse-Event-ID` header is a useful
correlation hint, but is **not covered by this HMAC**: do not let a changed
header bypass deduplication of an otherwise identical signed event. The sample
-PSA payloads below include `alertId` and `event` for this purpose. A valid HMAC
+PSA payload below includes each member's occurrence identity for this purpose. A valid HMAC
establishes integrity, not that processing succeeded; retain your receiver's
normal durable processing and retry handling. Never log the secret or full
credential-bearing request while diagnosing a rejection.
@@ -153,6 +164,44 @@ Content-Type: application/json
}
```
+### Receiver correlation and deduplication
+
+Keep ticket correlation separate from suppressing repeated processing:
+
+- Scope durable receiver records to the **configured sender and destination**
+ and tenant, not just a resource name or alert ID. Read identity from the
+ authenticated payload's fields after verifying the request; retain it across
+ receiver restarts. The tenant field is context, not permission to act in an
+ arbitrary account.
+- For normal queued firing and recovery notifications, use each member of
+ `{{.Alerts}}`, not only the primary `{{.ID}}`. The [full PSA template below](#sample-psa-payloads)
+ includes each member's `alertId` and full-precision UTC `startedAt`. These
+ identify its observed occurrence; a later start with the same ID must be
+ processed as a new occurrence. Firing and recovery use the same occurrence
+ record, so an old delayed recovery must not close a newer incident.
+- Within that record, handle `event` **and severity**. A warning becoming
+ critical is an update, not a duplicate warning. Decide explicitly whether
+ reminders update the existing ticket; do not create another ticket for each
+ retry or discard a severity increase. Group membership can change without
+ changing the header: process every member, even when the primary is unchanged.
+- The primary `{{.StartTime}}` string has only whole-second precision. Member
+ `StartTime` values in `{{.Alerts}}` are Go times and can be formatted with
+ fractional seconds as below. Do not assume the shorter string is universally
+ unique. A test or legacy payload can lack member identity; keep incomplete or
+ ambiguous events for reconciliation rather than silently treating them as
+ duplicates. **Send test** alone does not validate this lifecycle.
+- Commit receiver state and the ticket action atomically, or use your ticket
+ system's durable idempotent operation. Do not mark an event processed before
+ its action succeeds. Exact signed-byte replays can be rejected separately,
+ but a queue retry may render a new body or send timestamp: hashing those alone
+ is not logical-event deduplication. There is no universal exactly-once key in
+ this legacy header.
+
+Validate in an authorised test environment with firing, retry, warning-to-critical,
+recovery, recurrence of the same condition and a changed multi-alert group.
+Check actual tickets and receiver restart behaviour before enabling automatic
+actions. Do not cause a production outage or notification storm for this test.
+
## 🛡️ Security
- **Private IPs**: By default, webhooks to private IPs are blocked. Allow them in **Settings → System → Network → Webhook Security**.
@@ -204,7 +253,14 @@ Suspended or pending-deletion organizations return `403`, and an unknown org ID
There are two integration models. The push model is usually the right fit when tickets or incidents should open and close automatically.
-**Push (recommended): one outbound webhook per organization.** Create a **Generic** webhook for each organization and point it at your external system's inbound endpoint (an ITSM/PSA inbound webhook, an email connector, or middleware that opens service tickets). Shape the JSON with a [custom template](#-custom-templates) so it matches the receiving system's expected schema: every template variable listed above is available. Pulse fires on both `alert` and `resolved` events (`{{.Event}}` is `"alert"` or `"resolved"`), so the receiving system can open a ticket on alert and auto-resolve it on recovery. Add authentication as a custom header (e.g. `Authorization: Bearer ...`).
+**Push (recommended): one outbound webhook per organization.** Create a
+**Generic** webhook for each organization and point it at your external system's
+inbound endpoint (an ITSM/PSA inbound webhook, an email connector, or middleware
+that opens service tickets). Shape the JSON with a [custom template](#-custom-templates)
+to match the receiving system's schema. The bridge can open or update a ticket
+on `alert` and resolve that occurrence on `resolved`, subject to the [delivery
+conditions above](#-delivery-contract). Add authentication as a custom header
+in the settings form; keep its value private.
Configure it from the UI (**Alerts → Notifications → Add Webhook**) per org, or programmatically with an org-bound admin token:
@@ -221,20 +277,25 @@ Content-Type: application/json
"service": "generic",
"enabled": true,
"headers": { "Authorization": "Bearer <psa-token>" },
- "template": "{\"summary\":\"{{.Level}}: {{.ResourceName}} {{.Message | jsonString}}\",\"event\":\"{{.Event}}\",\"alertId\":\"{{.ID}}\"}"
+ "template": "{\"summary\":\"{{.Level | jsonString}}: {{.ResourceName | jsonString}} {{.Message | jsonString}}\",\"event\":\"{{.Event | jsonString}}\",\"alertId\":\"{{.ID | jsonString}}\",\"startedAt\":\"{{.StartTime | jsonString}}\"}"
}
```
-The exact ticket fields differ by platform (ConnectWise, Autotask, Halo, and others each expect their own inbound shape), so map the template to your platform's contract. The `alertId` round-trips through `{{.ID}}`, which lets the receiving system correlate the later `resolved` event to the ticket it opened.
+The exact ticket fields differ by platform (ConnectWise, Autotask, Halo, and
+others each expect their own inbound shape), so map the template to your
+platform's contract. This short example carries only the primary alert and a
+whole-second start. For a bridge handling real grouped notifications, use the
+member-aware template below; do not use the short example as an exactly-once key.
### Sample PSA payloads
-A fuller template suited to ticket routing, including tenant identity and the
-stable severity/type fields from the [delivery contract](#-delivery-contract):
+A fuller template for normal queued firing and recovery notifications includes
+tenant context and every member's occurrence, severity and condition. The
+primary fields remain convenient summary context, **not the whole batch**:
```json
{
- "event": "{{.Event}}",
+ "event": "{{.Event | jsonString}}",
"alertId": "{{.ID | jsonString}}",
"messageKey": "{{.MessageKey | jsonString}}",
"severity": "{{.Level | jsonString}}",
@@ -248,11 +309,24 @@ stable severity/type fields from the [delivery contract](#-delivery-contract):
"value": {{.Value}},
"threshold": {{.Threshold}},
"startedAt": "{{.StartTime | jsonString}}",
- "duration": "{{.Duration | jsonString}}"
+ "duration": "{{.Duration | jsonString}}",
+ "alertCount": {{.AlertCount}},
+ "alerts": [{{$comma := ""}}{{range .Alerts}}{{if .}}{{$comma}}
+ {
+ "alertId": "{{.ID | jsonString}}",
+ "startedAt": "{{.StartTime.UTC.Format "2006-01-02T15:04:05.999999999Z07:00" | jsonString}}",
+ "severity": "{{.Level | jsonString}}",
+ "alertType": "{{.Type | jsonString}}",
+ "resourceId": "{{.ResourceID | jsonString}}",
+ "resource": "{{.ResourceName | jsonString}}",
+ "summary": "{{.Message | jsonString}}"
+ }{{$comma = ","}}{{end}}{{end}}
+ ]
}
```
-What the receiver sees for a **critical** alert:
+An abridged primary summary for a **critical** alert (the template also emits
+the `alerts` member array):
```json
{
@@ -272,9 +346,12 @@ What the receiver sees for a **critical** alert:
}
```
-A **warning** alert is identical except `"severity": "warning"` — warning and critical are the only two severities Pulse emits, so a two-priority PSA mapping covers the full range.
+A **warning** alert is identical except `"severity": "warning"`. Map
+informational levels too; a two-priority mapping does not cover every condition.
-The **resolved** event reuses the same `alertId`, letting the bridge close the ticket it opened:
+The **resolved** event retains the original member `alertId` and `startedAt`,
+letting the bridge close that occurrence's ticket rather than a later incident
+with the same ID. Its primary summary is:
```json
{
@@ -294,7 +371,12 @@ The **resolved** event reuses the same `alertId`, letting the bridge close the t
}
```
-For ConnectWise specifically, point the webhook at a ConnectWise inbound API callback (or middleware that calls the ConnectWise REST API) and map `severity` to ticket priority, `tenantName` to the company, and `alertId` to your correlation field. Combine with a [`signingSecret`](#-delivery-contract) and the `X-Pulse-Event-ID` dedup header for a production-grade bridge.
+For ConnectWise specifically, point the webhook at a ConnectWise inbound API
+callback (or middleware that calls the ConnectWise REST API). Map each member's
+severity to ticket priority and the tenant to your configured company. Combine
+[`signingSecret`](#-delivery-contract) with the [receiver correlation rules](#receiver-correlation-and-deduplication),
+not header-only deduplication. Verify your platform's processing and recovery
+behaviour before treating the bridge as production-ready.
**Pull (poll): org-scoped read API.** Issue a `monitoring:read` token bound to each organization and poll that org's alerts. Send `X-Pulse-Org-ID` (or rely on the org-bound token) so you get only that organization's data:
diff --git a/internal/notifications/notifications.go b/internal/notifications/notifications.go
index 18d01ea67a..61277858b0 100644
--- a/internal/notifications/notifications.go
+++ b/internal/notifications/notifications.go
@@ -280,7 +280,7 @@ type webhookHTTPResult struct {
type webhookRequestOptions struct {
alertType string
- eventID string // Idempotency token sent as X-Pulse-Event-ID (stable across retries)
+ eventID string // Legacy X-Pulse-Event-ID correlation header, not a unique delivery ID
timeout time.Duration
userAgent string
responseLogging bool
@@ -3344,9 +3344,9 @@ func signWebhookPayload(secret, timestamp string, payload []byte) string {
return hex.EncodeToString(mac.Sum(nil))
}
-// webhookEventID builds the idempotency token for a delivery: stable for the
-// same alert occurrence and event so receivers can deduplicate retries at
-// either retry layer (transport or queue).
+// webhookEventID builds the legacy correlation header, stable across retries.
+// It omits occurrence start, severity and grouped membership, so it is not a
+// unique incident/delivery identity or a sufficient durable deduplication key.
func webhookEventID(alertID, event string) string {
if alertID == "" {
return ""
diff --git a/internal/notifications/webhook_guide_test.go b/internal/notifications/webhook_guide_test.go
new file mode 100644
index 0000000000..37a27878f8
--- /dev/null
+++ b/internal/notifications/webhook_guide_test.go
@@ -0,0 +1,193 @@
+package notifications
+
+import (
+ "crypto/hmac"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "io"
+ "net/http"
+ "os"
+ "regexp"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/alerts"
+ "github.com/stretchr/testify/require"
+)
+
+// Exercise the copyable templates through the real sender, not a second
+// renderer or a model receiver. The input override permits an exact-parent
+// documentation negative control without modifying production source.
+func webhookGuide(t *testing.T) string {
+ t.Helper()
+ path := os.Getenv("PULSE_WEBHOOK_GUIDE_TEST_INPUT")
+ if path == "" {
+ path = "../../docs/WEBHOOKS.md"
+ }
+ data, err := os.ReadFile(path)
+ require.NoError(t, err)
+ return string(data)
+}
+
+func webhookGuidePSATemplate(t *testing.T) string {
+ t.Helper()
+ sections := strings.SplitN(webhookGuide(t), "### Sample PSA payloads", 2)
+ require.Len(t, sections, 2)
+ block := regexp.MustCompile("(?s)```json\\n(.*?)\\n```").FindStringSubmatch(sections[1])
+ require.Len(t, block, 2)
+ return block[1]
+}
+
+type guideMember struct {
+ ID string `json:"alertId"`
+ StartedAt string `json:"startedAt"`
+ Severity string `json:"severity"`
+ ResourceID string `json:"resourceId"`
+ Resource string `json:"resource"`
+ Summary string `json:"summary"`
+}
+
+type guidePayload struct {
+ Event string `json:"event"`
+ TenantID string `json:"tenantId"`
+ StartedAt string `json:"startedAt"`
+ AlertCount int `json:"alertCount"`
+ Members []guideMember `json:"alerts"`
+}
+
+type guideRequest struct {
+ body, eventID, timestamp, signature string
+}
+
+func webhookGuideSender(t *testing.T, payloadTemplate string) (*NotificationManager, WebhookConfig, <-chan guideRequest) {
+ t.Helper()
+ manager := NewNotificationManager("https://pulse.example.test")
+ require.NoError(t, manager.UpdateAllowedPrivateCIDRs("127.0.0.1"))
+ manager.SetTenantIdentityResolver(func() (string, string) { return "synthetic-tenant", "Synthetic tenant" })
+ requests := make(chan guideRequest, 16)
+ server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ body, err := io.ReadAll(r.Body)
+ if err != nil {
+ w.WriteHeader(http.StatusBadRequest)
+ return
+ }
+ requests <- guideRequest{string(body), r.Header.Get("X-Pulse-Event-ID"),
+ r.Header.Get("X-Pulse-Timestamp"), r.Header.Get("X-Pulse-Signature")}
+ w.WriteHeader(http.StatusOK)
+ }))
+ t.Cleanup(server.Close)
+ webhook := WebhookConfig{ID: "synthetic-guide", Name: "Synthetic guide receiver", URL: server.URL,
+ Enabled: true, Service: "generic", SigningSecret: "synthetic-guide-test-secret", Template: payloadTemplate}
+ return manager, webhook, requests
+}
+
+func webhookGuideRead(t *testing.T, requests <-chan guideRequest) (guideRequest, guidePayload) {
+ t.Helper()
+ select {
+ case request := <-requests:
+ var payload guidePayload
+ require.NoError(t, json.Unmarshal([]byte(request.body), &payload), "copied template must render valid JSON")
+ mac := hmac.New(sha256.New, []byte("synthetic-guide-test-secret"))
+ _, err := mac.Write([]byte(request.timestamp + "." + request.body))
+ require.NoError(t, err)
+ require.Equal(t, "v1="+hex.EncodeToString(mac.Sum(nil)), request.signature)
+ return request, payload
+ case <-time.After(5 * time.Second):
+ t.Fatal("no local sender request received")
+ return guideRequest{}, guidePayload{}
+ }
+}
+
+func TestWebhookGuidePSALifecycle(t *testing.T) {
+ manager, webhook, requests := webhookGuideSender(t, webhookGuidePSATemplate(t))
+ start := time.Date(2026, 10, 4, 7, 0, 0, 123456789, time.UTC)
+ alert := &alerts.Alert{ID: "host::metric-threshold:cpu", StartTime: start,
+ Level: alerts.AlertLevelWarning, Type: "cpu", ResourceID: "synthetic-host",
+ ResourceName: "quoted \"host\"\nwith é", Message: "synthetic \\ message\n\"quoted\"", Value: 95, Threshold: 90}
+
+ require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{alert}))
+ firstRequest, first := webhookGuideRead(t, requests)
+ require.Len(t, first.Members, 1)
+ require.Equal(t, "alert", first.Event)
+ require.Equal(t, "synthetic-tenant", first.TenantID)
+ require.Equal(t, 1, first.AlertCount)
+ require.Equal(t, guideMember{alert.ID, start.Format(time.RFC3339Nano), "warning",
+ alert.ResourceID, alert.ResourceName, alert.Message}, first.Members[0])
+
+ // A synthetic retry retains the member identity even when rendering anew.
+ require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{alert}))
+ retryRequest, retry := webhookGuideRead(t, requests)
+ require.Equal(t, first.Members, retry.Members)
+ require.Equal(t, firstRequest.eventID, retryRequest.eventID)
+
+ critical := alert.Clone()
+ critical.Level = alerts.AlertLevelCritical
+ require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{critical}))
+ criticalRequest, escalated := webhookGuideRead(t, requests)
+ require.Equal(t, firstRequest.eventID, criticalRequest.eventID, "header cannot identify a severity change")
+ require.Equal(t, first.Members[0].StartedAt, escalated.Members[0].StartedAt)
+ require.Equal(t, "critical", escalated.Members[0].Severity)
+
+ require.NoError(t, manager.sendResolvedWebhook(webhook, []*alerts.Alert{critical}, start.Add(time.Minute)))
+ _, resolved := webhookGuideRead(t, requests)
+ require.Equal(t, "resolved", resolved.Event)
+ require.Equal(t, escalated.Members, resolved.Members, "recovery must correlate to the original occurrence")
+
+ // A second occurrence in the SAME SECOND defeats both header-only and
+ // whole-second primary-start deduplication. Member precision preserves it.
+ recurrent := critical.Clone()
+ recurrent.StartTime = start.Add(100 * time.Nanosecond)
+ require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{recurrent}))
+ recurrentRequest, recurrence := webhookGuideRead(t, requests)
+ require.Equal(t, firstRequest.eventID, recurrentRequest.eventID)
+ require.Equal(t, first.StartedAt, recurrence.StartedAt)
+ require.NotEqual(t, first.Members[0].StartedAt, recurrence.Members[0].StartedAt)
+ require.Equal(t, recurrent.StartTime.Format(time.RFC3339Nano), recurrence.Members[0].StartedAt)
+}
+
+func TestWebhookGuidePSAGroupMembershipAndInfo(t *testing.T) {
+ manager, webhook, requests := webhookGuideSender(t, webhookGuidePSATemplate(t))
+ start := time.Date(2026, 10, 4, 9, 0, 0, 987654321, time.FixedZone("synthetic", 2*60*60))
+ primary := &alerts.Alert{ID: "primary", StartTime: start, Level: alerts.AlertLevelWarning}
+ info := &alerts.Alert{ID: "info-member", StartTime: start.Add(time.Second), Level: alerts.AlertLevelInfo}
+ critical := &alerts.Alert{ID: "critical-member", StartTime: start.Add(2 * time.Second), Level: alerts.AlertLevelCritical}
+ for _, members := range [][]*alerts.Alert{{primary, info}, {primary, critical}, {primary, critical, info}} {
+ require.NoError(t, manager.sendGroupedWebhook(webhook, members))
+ request, payload := webhookGuideRead(t, requests)
+ require.Equal(t, "primary:alert", request.eventID, "changed groups can have the same correlation header")
+ require.Equal(t, len(members), payload.AlertCount)
+ require.Len(t, payload.Members, len(members))
+ for index, alert := range members {
+ require.Equal(t, alert.ID, payload.Members[index].ID)
+ require.Equal(t, string(alert.Level), payload.Members[index].Severity)
+ require.Equal(t, alert.StartTime.UTC().Format(time.RFC3339Nano), payload.Members[index].StartedAt)
+ }
+ }
+ require.NoError(t, manager.sendResolvedWebhook(webhook, []*alerts.Alert{primary, critical, info}, start.Add(time.Minute)))
+ _, resolved := webhookGuideRead(t, requests)
+ require.Equal(t, "resolved", resolved.Event)
+ require.Len(t, resolved.Members, 3)
+ require.Equal(t, "info", resolved.Members[2].Severity)
+}
+
+func TestWebhookGuideShortTemplateEscapesStrings(t *testing.T) {
+ section := strings.SplitN(webhookGuide(t), "### Sample PSA payloads", 2)[0]
+ block := regexp.MustCompile("(?s)```http\\nPOST /api/notifications/webhooks\\nX-Pulse-Org-ID:.*?\\n\\n(.*?)\\n```").FindStringSubmatch(section)
+ require.Len(t, block, 2)
+ var config struct {
+ Template string `json:"template"`
+ }
+ require.NoError(t, json.Unmarshal([]byte(block[1]), &config))
+ manager, webhook, requests := webhookGuideSender(t, config.Template)
+ alert := &alerts.Alert{ID: "quoted\"id", ResourceName: "host\"\\\nname", Message: "message\"\\\n",
+ StartTime: time.Date(2026, 10, 4, 7, 0, 0, 123, time.UTC), Level: alerts.AlertLevelWarning}
+ require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{alert}))
+ request, _ := webhookGuideRead(t, requests)
+ var payload map[string]interface{}
+ require.NoError(t, json.Unmarshal([]byte(request.body), &payload))
+ require.Equal(t, alert.ID, payload["alertId"])
+ require.Equal(t, "warning: "+alert.ResourceName+" "+alert.Message, payload["summary"])
+ require.Equal(t, alert.StartTime.Format(time.RFC3339), payload["startedAt"])
+}
diff --git a/scripts/tests/test_webhook_verification_docs.py b/scripts/tests/test_webhook_verification_docs.py
index 6092c377b5..db57a6f58b 100644
--- a/scripts/tests/test_webhook_verification_docs.py
+++ b/scripts/tests/test_webhook_verification_docs.py
@@ -146,6 +146,22 @@ class WebhookVerificationDocsTest(unittest.TestCase):
"not from the request", "Never log the secret"):
self.assertIn(phrase, text)
+ def test_receiver_guidance_preserves_recurrences_and_all_group_members(self):
+ # The Go sender tests render the copied PSA templates over actual
+ # loopback HTTP. Keep the safety explanation beside those examples.
+ text = " ".join(DOC.read_text().split())
+ for phrase in ("not a unique incident or delivery ID", "later occurrences",
+ "Do not deduplicate permanently", "every member",
+ "old delayed recovery must not close a newer incident",
+ "event and severity", "whole-second precision",
+ "incomplete or ambiguous events", "atomically",
+ "Do not mark an event processed before its action succeeds"):
+ self.assertIn(phrase, text.replace("**", ""))
+ self.assertNotIn("deduplicate on it", text)
+ self.assertNotIn("two-priority PSA mapping covers the full range", text)
+ self.assertIn('"info"', text)
+ self.assertIn("successful firing-delivery receipt", text)
+
if __name__ == "__main__":
unittest.main()