Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep webhook bridge guidance from discarding later alerts
pulse-triage[bot]94a14c78c7f8
diff --git a/docs/WEBHOOKS.md b/docs/WEBHOOKS.mdindex 3acf6649b4..cb0f8424fd 100644--- a/docs/WEBHOOKS.md+++ b/docs/WEBHOOKS.md@@ -65,11 +65,18 @@ For generic webhooks, use Go templates to format the JSON payload.These fields and behaviors are stable; ticket-routing integrations can rely on them.-**Events.** Every webhook fires on both `alert` and `resolved` events. `{{.Event}}` is `"alert"` or `"resolved"` — there is no separate "info" event class.+**Events.** `{{.Event}}` is `"alert"` or `"resolved"` — there is no separate+"info" event class. Recovery delivery depends on **Notify on resolve** and a+successful firing-delivery receipt for that occurrence and destination. Do not+assume that every configured webhook receives both events.-**Severity.** `{{.Level}}` is `"warning"` or `"critical"`. Pulse has exactly these two alert levels.+**Severity.** Metric alerts commonly use `"warning"` or `"critical"`;+informational conditions can use `"info"`. Event and severity are separate:+an informational firing still has event `"alert"`. Preserve and flag an+unrecognised level rather than silently dropping the notification. Minimum+severity and other delivery policies still apply.-**Alert type.** `{{.Type}}` is the metric or condition that fired: `cpu`, `memory`, `disk`, `diskRead`, `diskWrite`, `networkIn`, `networkOut`, `connectivity`, and similar. The alert ID (`{{.ID}}`) is stable for the lifetime of an alert occurrence, so the `resolved` event carries the same ID as the `alert` event it closes.+**Alert type.** `{{.Type}}` is the metric or condition that fired: `cpu`, `memory`, `disk`, `diskRead`, `diskWrite`, `networkIn`, `networkOut`, `connectivity`, and similar. The alert ID (`{{.ID}}`) correlates firing and recovery, but can be reused when the same condition fires again. It is not a unique incident or delivery ID.**Message key.** `{{.MessageKey}}` is the stable, language-neutral condition key for rebuilding or translating a notification. Canonical alerts use `<kind>.<type>` (for example, `metric-threshold.disk`); older alert paths fall back to `{{.Type}}`. Combine it with `{{.Event}}` and `{{.ResourceType}}` when the receiving system needs separate wording for firing/recovery events or different resource classes. Unlike a numeric message index, the symbolic key does not change when another alert type is added.@@ -79,7 +86,13 @@ These fields and behaviors are stable; ticket-routing integrations can rely on t**Retries.** Failed deliveries retry with exponential backoff. The persistent notification queue makes up to 3 delivery attempts per notification; webhooks configured with transport-level retry add up to 3 more HTTP retries per attempt (1s doubling to a 30s cap, honoring `Retry-After` on HTTP 429). A receiver may therefore see the same logical event more than once.-**Idempotency.** Every alert delivery carries an `X-Pulse-Event-ID` header of the form `<alertID>:<event>` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). It is identical across all retries of the same logical event — deduplicate on it.+**Correlation header.** Alert webhooks carry `X-Pulse-Event-ID` in the form+`<alertID>:<event>` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). Retries retain it,+but later occurrences, severity changes and reminders can share it too. A group+uses its primary alert's ID, not a unique ID for every member or batch. **Do not+deduplicate permanently on this header or on alert ID and event alone:** doing+so can silently discard a later incident or a changed group. See+[receiver correlation and deduplication](#receiver-correlation-and-deduplication).**Signed deliveries.** Set a `signingSecret` on the webhook config to enable HMAC signing. Signed requests carry:@@ -126,14 +139,12 @@ def verify(secret: str, timestamp: str, body: bytes, signature: str) -> bool:A time window rejects old captures, **not every replay**: the same signedrequest can arrive more than once within that window, and Pulse signs eachretry with its current send time. After verification, deduplicate atomically-before creating a ticket or taking another action. Use a durable key scoped to-the configured Pulse sender and destination, built from the authenticated-payload's alert ID and event (`{{.ID}}` and `{{.Event}}` in your template); keep-it across retries and receiver restarts. An `alert` and its later `resolved`-are different events, not duplicates. The `X-Pulse-Event-ID` header is a useful+using the [receiver rules below](#receiver-correlation-and-deduplication),+not the signature, send timestamp or alert ID alone. An `alert` and its later+`resolved` are different events, not duplicates. The `X-Pulse-Event-ID` header is a usefulcorrelation hint, but is **not covered by this HMAC**: do not let a changedheader bypass deduplication of an otherwise identical signed event. The sample-PSA payloads below include `alertId` and `event` for this purpose. A valid HMAC+PSA payload below includes each member's occurrence identity for this purpose. A valid HMACestablishes integrity, not that processing succeeded; retain your receiver'snormal durable processing and retry handling. Never log the secret or fullcredential-bearing request while diagnosing a rejection.@@ -153,6 +164,44 @@ Content-Type: application/json}```+### Receiver correlation and deduplication++Keep ticket correlation separate from suppressing repeated processing:++- Scope durable receiver records to the **configured sender and destination**+ and tenant, not just a resource name or alert ID. Read identity from the+ authenticated payload's fields after verifying the request; retain it across+ receiver restarts. The tenant field is context, not permission to act in an+ arbitrary account.+- For normal queued firing and recovery notifications, use each member of+ `{{.Alerts}}`, not only the primary `{{.ID}}`. The [full PSA template below](#sample-psa-payloads)+ includes each member's `alertId` and full-precision UTC `startedAt`. These+ identify its observed occurrence; a later start with the same ID must be+ processed as a new occurrence. Firing and recovery use the same occurrence+ record, so an old delayed recovery must not close a newer incident.+- Within that record, handle `event` **and severity**. A warning becoming+ critical is an update, not a duplicate warning. Decide explicitly whether+ reminders update the existing ticket; do not create another ticket for each+ retry or discard a severity increase. Group membership can change without+ changing the header: process every member, even when the primary is unchanged.+- The primary `{{.StartTime}}` string has only whole-second precision. Member+ `StartTime` values in `{{.Alerts}}` are Go times and can be formatted with+ fractional seconds as below. Do not assume the shorter string is universally+ unique. A test or legacy payload can lack member identity; keep incomplete or+ ambiguous events for reconciliation rather than silently treating them as+ duplicates. **Send test** alone does not validate this lifecycle.+- Commit receiver state and the ticket action atomically, or use your ticket+ system's durable idempotent operation. Do not mark an event processed before+ its action succeeds. Exact signed-byte replays can be rejected separately,+ but a queue retry may render a new body or send timestamp: hashing those alone+ is not logical-event deduplication. There is no universal exactly-once key in+ this legacy header.++Validate in an authorised test environment with firing, retry, warning-to-critical,+recovery, recurrence of the same condition and a changed multi-alert group.+Check actual tickets and receiver restart behaviour before enabling automatic+actions. Do not cause a production outage or notification storm for this test.+## 🛡️ Security- **Private IPs**: By default, webhooks to private IPs are blocked. Allow them in **Settings → System → Network → Webhook Security**.@@ -204,7 +253,14 @@ Suspended or pending-deletion organizations return `403`, and an unknown org IDThere are two integration models. The push model is usually the right fit when tickets or incidents should open and close automatically.-**Push (recommended): one outbound webhook per organization.** Create a **Generic** webhook for each organization and point it at your external system's inbound endpoint (an ITSM/PSA inbound webhook, an email connector, or middleware that opens service tickets). Shape the JSON with a [custom template](#-custom-templates) so it matches the receiving system's expected schema: every template variable listed above is available. Pulse fires on both `alert` and `resolved` events (`{{.Event}}` is `"alert"` or `"resolved"`), so the receiving system can open a ticket on alert and auto-resolve it on recovery. Add authentication as a custom header (e.g. `Authorization: Bearer ...`).+**Push (recommended): one outbound webhook per organization.** Create a+**Generic** webhook for each organization and point it at your external system's+inbound endpoint (an ITSM/PSA inbound webhook, an email connector, or middleware+that opens service tickets). Shape the JSON with a [custom template](#-custom-templates)+to match the receiving system's schema. The bridge can open or update a ticket+on `alert` and resolve that occurrence on `resolved`, subject to the [delivery+conditions above](#-delivery-contract). Add authentication as a custom header+in the settings form; keep its value private.Configure it from the UI (**Alerts → Notifications → Add Webhook**) per org, or programmatically with an org-bound admin token:@@ -221,20 +277,25 @@ Content-Type: application/json"service": "generic","enabled": true,"headers": { "Authorization": "Bearer <psa-token>" },- "template": "{\"summary\":\"{{.Level}}: {{.ResourceName}} {{.Message | jsonString}}\",\"event\":\"{{.Event}}\",\"alertId\":\"{{.ID}}\"}"+ "template": "{\"summary\":\"{{.Level | jsonString}}: {{.ResourceName | jsonString}} {{.Message | jsonString}}\",\"event\":\"{{.Event | jsonString}}\",\"alertId\":\"{{.ID | jsonString}}\",\"startedAt\":\"{{.StartTime | jsonString}}\"}"}```-The exact ticket fields differ by platform (ConnectWise, Autotask, Halo, and others each expect their own inbound shape), so map the template to your platform's contract. The `alertId` round-trips through `{{.ID}}`, which lets the receiving system correlate the later `resolved` event to the ticket it opened.+The exact ticket fields differ by platform (ConnectWise, Autotask, Halo, and+others each expect their own inbound shape), so map the template to your+platform's contract. This short example carries only the primary alert and a+whole-second start. For a bridge handling real grouped notifications, use the+member-aware template below; do not use the short example as an exactly-once key.### Sample PSA payloads-A fuller template suited to ticket routing, including tenant identity and the-stable severity/type fields from the [delivery contract](#-delivery-contract):+A fuller template for normal queued firing and recovery notifications includes+tenant context and every member's occurrence, severity and condition. The+primary fields remain convenient summary context, **not the whole batch**:```json{- "event": "{{.Event}}",+ "event": "{{.Event | jsonString}}","alertId": "{{.ID | jsonString}}","messageKey": "{{.MessageKey | jsonString}}","severity": "{{.Level | jsonString}}",@@ -248,11 +309,24 @@ stable severity/type fields from the [delivery contract](#-delivery-contract):"value": {{.Value}},"threshold": {{.Threshold}},"startedAt": "{{.StartTime | jsonString}}",- "duration": "{{.Duration | jsonString}}"+ "duration": "{{.Duration | jsonString}}",+ "alertCount": {{.AlertCount}},+ "alerts": [{{$comma := ""}}{{range .Alerts}}{{if .}}{{$comma}}+ {+ "alertId": "{{.ID | jsonString}}",+ "startedAt": "{{.StartTime.UTC.Format "2006-01-02T15:04:05.999999999Z07:00" | jsonString}}",+ "severity": "{{.Level | jsonString}}",+ "alertType": "{{.Type | jsonString}}",+ "resourceId": "{{.ResourceID | jsonString}}",+ "resource": "{{.ResourceName | jsonString}}",+ "summary": "{{.Message | jsonString}}"+ }{{$comma = ","}}{{end}}{{end}}+ ]}```-What the receiver sees for a **critical** alert:+An abridged primary summary for a **critical** alert (the template also emits+the `alerts` member array):```json{@@ -272,9 +346,12 @@ What the receiver sees for a **critical** alert:}```-A **warning** alert is identical except `"severity": "warning"` — warning and critical are the only two severities Pulse emits, so a two-priority PSA mapping covers the full range.+A **warning** alert is identical except `"severity": "warning"`. Map+informational levels too; a two-priority mapping does not cover every condition.-The **resolved** event reuses the same `alertId`, letting the bridge close the ticket it opened:+The **resolved** event retains the original member `alertId` and `startedAt`,+letting the bridge close that occurrence's ticket rather than a later incident+with the same ID. Its primary summary is:```json{@@ -294,7 +371,12 @@ The **resolved** event reuses the same `alertId`, letting the bridge close the t}```-For ConnectWise specifically, point the webhook at a ConnectWise inbound API callback (or middleware that calls the ConnectWise REST API) and map `severity` to ticket priority, `tenantName` to the company, and `alertId` to your correlation field. Combine with a [`signingSecret`](#-delivery-contract) and the `X-Pulse-Event-ID` dedup header for a production-grade bridge.+For ConnectWise specifically, point the webhook at a ConnectWise inbound API+callback (or middleware that calls the ConnectWise REST API). Map each member's+severity to ticket priority and the tenant to your configured company. Combine+[`signingSecret`](#-delivery-contract) with the [receiver correlation rules](#receiver-correlation-and-deduplication),+not header-only deduplication. Verify your platform's processing and recovery+behaviour before treating the bridge as production-ready.**Pull (poll): org-scoped read API.** Issue a `monitoring:read` token bound to each organization and poll that org's alerts. Send `X-Pulse-Org-ID` (or rely on the org-bound token) so you get only that organization's data:diff --git a/frontend-modern/browser-tests/docs-fragment-navigation.tsx b/frontend-modern/browser-tests/docs-fragment-navigation.tsxindex 373a4dd878..628e095444 100644--- a/frontend-modern/browser-tests/docs-fragment-navigation.tsx+++ b/frontend-modern/browser-tests/docs-fragment-navigation.tsx@@ -25,6 +25,7 @@ const targets: Record<string, string> = {truenas: '/docs/TRUENAS',installation: '/docs/INSTALL',configuration: '/docs/CONFIGURATION#-api-tokens',+ webhooks: '/docs/WEBHOOKS',};window.history.replaceState({}, '', targets[scenario] ?? targets.plain);diff --git a/frontend-modern/browser-tests/webhook-guidance.cjs b/frontend-modern/browser-tests/webhook-guidance.cjsnew file mode 100644index 0000000000..39da41afa3--- /dev/null+++ b/frontend-modern/browser-tests/webhook-guidance.cjs@@ -0,0 +1,95 @@+// Inspect the real shipped Docs page and copyable webhook template. This is+// offline rendering, not a ticket receiver, notification or external delivery.+const assert = require('node:assert/strict');+const fs = require('node:fs');+const path = require('node:path');+const crypto = require('node:crypto');+const { chromium, webkit } = require('playwright');++(async () => {+ const root = '/workspace/frontend-modern';+ const artifacts = path.join(root, 'node_modules', 'webhook-guidance-browser');+ fs.mkdirSync(artifacts, { recursive: true });+ const { createServer } = await import(path.join(root, 'node_modules/vite/dist/node/index.js'));+ const server = await createServer({ root, configFile: path.join(root, 'vite.config.ts'),+ cacheDir: path.join(root, 'node_modules', '.vite-webhook-guidance'),+ server: { host: '127.0.0.1', port: 5253, strictPort: true },+ optimizeDeps: { noDiscovery: true, include: [] } });+ const docs = fs.readFileSync(path.join(root, 'public/docs/WEBHOOKS.md'), 'utf8');+ const expectedTemplate = docs.split('### Sample PSA payloads')[1].match(/```json\n([\s\S]*?)\n```/)[1];+ let browser;+ const results = [];+ try {+ await server.listen();+ for (const [engine, width] of [['chromium', 1280], ['webkit', 390]]) {+ browser = await { chromium, webkit }[engine].launch(engine === 'chromium'+ ? { headless: true, channel: 'chromium', args: ['--no-sandbox'] } : { headless: true });+ const page = await browser.newPage({ viewport: { width, height: 1000 },+ ...(engine === 'webkit' ? { isMobile: true, hasTouch: true } : {}) });+ const errors = [];+ page.on('pageerror', (error) => errors.push(error.message));+ page.on('console', (message) => { if (message.type() === 'error') errors.push(message.text()); });+ await page.goto('http://127.0.0.1:5253/browser-tests/docs-fragment-navigation.html?scenario=webhooks',+ { waitUntil: 'domcontentloaded', timeout: 60_000 });+ const heading = page.getByRole('heading', { name: 'Receiver correlation and deduplication', exact: true });+ await heading.waitFor();+ if (engine === 'webkit') await page.evaluate(() => document.documentElement.classList.add('dark'));+ const body = (await page.locator('article').innerText()).replace(/\s+/g, ' ');+ for (const text of ['not a unique incident or delivery ID', 'Do not deduplicate permanently',+ 'old delayed recovery must not close a newer incident', 'event and severity',+ 'whole-second precision', 'incomplete or ambiguous events',+ 'successful firing-delivery receipt', 'informational conditions can use "info"']) {+ assert.ok(body.includes(text), text);+ }+ assert.ok(!body.includes('deduplicate on it'));+ const follow = async (name, href, target) => {+ const link = page.getByRole('link', { name, exact: true });+ assert.equal(await link.getAttribute('href'), href);+ await link.focus();+ await page.keyboard.press('Enter');+ await page.waitForURL(`**${href}`);+ await page.waitForFunction((id) => document.activeElement?.id === id, target);+ };+ await follow('receiver correlation and deduplication', '#receiver-correlation-and-deduplication',+ 'receiver-correlation-and-deduplication');+ const screenshot = async (name, headingName) => {+ await page.getByRole('heading', { name: headingName, exact: true })+ .evaluate((element) => element.scrollIntoView({ block: 'start' }));+ await page.evaluate(async () => {+ await new Promise((resolve) => requestAnimationFrame(() => requestAnimationFrame(resolve)));+ await Promise.all(document.getAnimations().map((animation) => animation.finished.catch(() => {})));+ });+ assert.ok(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth + 1));+ await page.screenshot({ path: path.join(artifacts, `${engine}-${name}.png`) });+ };+ await screenshot('receiver', 'Receiver correlation and deduplication');+ await follow('full PSA template below', '#sample-psa-payloads', 'sample-psa-payloads');+ const template = page.locator('pre code').filter({ hasText: '"alertCount":' });+ assert.equal(await template.count(), 1);+ assert.equal((await template.innerText()).trim(), expectedTemplate.trim());+ assert.ok(await template.evaluate((element) => {+ const pre = element.closest('pre');+ return getComputedStyle(pre).overflowX === 'auto' || pre.scrollWidth <= pre.clientWidth + 1;+ }));+ await screenshot('template', 'Sample PSA payloads');+ await page.reload({ waitUntil: 'domcontentloaded' });+ await page.getByRole('heading', { name: 'Sample PSA payloads', exact: true }).waitFor();+ await page.waitForFunction(() => document.activeElement?.id === 'sample-psa-payloads');+ assert.deepEqual(errors, []);+ results.push({ engine, browserVersion: browser.version(), width,+ scope: engine === 'webkit' ? 'phone-emulated, not native' : 'desktop',+ warningsRendered: true, keyboardReceiverLink: true, keyboardTemplateLink: true,+ templateTextExact: true, reloadFragmentFocus: true, noDocumentOverflow: true, errors });+ await browser.close();+ browser = undefined;+ }+ const files = ['public/docs/WEBHOOKS.md', 'browser-tests/docs-fragment-navigation.tsx',+ 'browser-tests/webhook-guidance.cjs', 'src/features/docs/docMarkdown.ts', 'src/pages/Docs.tsx']+ .map((file) => ({ file, sha256: crypto.createHash('sha256').update(fs.readFileSync(path.join(root, file))).digest('hex') }));+ fs.writeFileSync(path.join(artifacts, 'result.json'), JSON.stringify({ playwright: require('playwright/package.json').version,+ files, results, limits: 'No real receiver, ticket, provider, native device or installed notification acceptance.' }, null, 2));+ } finally {+ if (browser) await browser.close();+ await server.close();+ }+})().catch((error) => { console.error(error); process.exitCode = 1; });diff --git a/frontend-modern/public/docs/WEBHOOKS.md b/frontend-modern/public/docs/WEBHOOKS.mdindex 3acf6649b4..cb0f8424fd 100644--- a/frontend-modern/public/docs/WEBHOOKS.md+++ b/frontend-modern/public/docs/WEBHOOKS.md@@ -65,11 +65,18 @@ For generic webhooks, use Go templates to format the JSON payload.These fields and behaviors are stable; ticket-routing integrations can rely on them.-**Events.** Every webhook fires on both `alert` and `resolved` events. `{{.Event}}` is `"alert"` or `"resolved"` — there is no separate "info" event class.+**Events.** `{{.Event}}` is `"alert"` or `"resolved"` — there is no separate+"info" event class. Recovery delivery depends on **Notify on resolve** and a+successful firing-delivery receipt for that occurrence and destination. Do not+assume that every configured webhook receives both events.-**Severity.** `{{.Level}}` is `"warning"` or `"critical"`. Pulse has exactly these two alert levels.+**Severity.** Metric alerts commonly use `"warning"` or `"critical"`;+informational conditions can use `"info"`. Event and severity are separate:+an informational firing still has event `"alert"`. Preserve and flag an+unrecognised level rather than silently dropping the notification. Minimum+severity and other delivery policies still apply.-**Alert type.** `{{.Type}}` is the metric or condition that fired: `cpu`, `memory`, `disk`, `diskRead`, `diskWrite`, `networkIn`, `networkOut`, `connectivity`, and similar. The alert ID (`{{.ID}}`) is stable for the lifetime of an alert occurrence, so the `resolved` event carries the same ID as the `alert` event it closes.+**Alert type.** `{{.Type}}` is the metric or condition that fired: `cpu`, `memory`, `disk`, `diskRead`, `diskWrite`, `networkIn`, `networkOut`, `connectivity`, and similar. The alert ID (`{{.ID}}`) correlates firing and recovery, but can be reused when the same condition fires again. It is not a unique incident or delivery ID.**Message key.** `{{.MessageKey}}` is the stable, language-neutral condition key for rebuilding or translating a notification. Canonical alerts use `<kind>.<type>` (for example, `metric-threshold.disk`); older alert paths fall back to `{{.Type}}`. Combine it with `{{.Event}}` and `{{.ResourceType}}` when the receiving system needs separate wording for firing/recovery events or different resource classes. Unlike a numeric message index, the symbolic key does not change when another alert type is added.@@ -79,7 +86,13 @@ These fields and behaviors are stable; ticket-routing integrations can rely on t**Retries.** Failed deliveries retry with exponential backoff. The persistent notification queue makes up to 3 delivery attempts per notification; webhooks configured with transport-level retry add up to 3 more HTTP retries per attempt (1s doubling to a 30s cap, honoring `Retry-After` on HTTP 429). A receiver may therefore see the same logical event more than once.-**Idempotency.** Every alert delivery carries an `X-Pulse-Event-ID` header of the form `<alertID>:<event>` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). It is identical across all retries of the same logical event — deduplicate on it.+**Correlation header.** Alert webhooks carry `X-Pulse-Event-ID` in the form+`<alertID>:<event>` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). Retries retain it,+but later occurrences, severity changes and reminders can share it too. A group+uses its primary alert's ID, not a unique ID for every member or batch. **Do not+deduplicate permanently on this header or on alert ID and event alone:** doing+so can silently discard a later incident or a changed group. See+[receiver correlation and deduplication](#receiver-correlation-and-deduplication).**Signed deliveries.** Set a `signingSecret` on the webhook config to enable HMAC signing. Signed requests carry:@@ -126,14 +139,12 @@ def verify(secret: str, timestamp: str, body: bytes, signature: str) -> bool:A time window rejects old captures, **not every replay**: the same signedrequest can arrive more than once within that window, and Pulse signs eachretry with its current send time. After verification, deduplicate atomically-before creating a ticket or taking another action. Use a durable key scoped to-the configured Pulse sender and destination, built from the authenticated-payload's alert ID and event (`{{.ID}}` and `{{.Event}}` in your template); keep-it across retries and receiver restarts. An `alert` and its later `resolved`-are different events, not duplicates. The `X-Pulse-Event-ID` header is a useful+using the [receiver rules below](#receiver-correlation-and-deduplication),+not the signature, send timestamp or alert ID alone. An `alert` and its later+`resolved` are different events, not duplicates. The `X-Pulse-Event-ID` header is a usefulcorrelation hint, but is **not covered by this HMAC**: do not let a changedheader bypass deduplication of an otherwise identical signed event. The sample-PSA payloads below include `alertId` and `event` for this purpose. A valid HMAC+PSA payload below includes each member's occurrence identity for this purpose. A valid HMACestablishes integrity, not that processing succeeded; retain your receiver'snormal durable processing and retry handling. Never log the secret or fullcredential-bearing request while diagnosing a rejection.@@ -153,6 +164,44 @@ Content-Type: application/json}```+### Receiver correlation and deduplication++Keep ticket correlation separate from suppressing repeated processing:++- Scope durable receiver records to the **configured sender and destination**+ and tenant, not just a resource name or alert ID. Read identity from the+ authenticated payload's fields after verifying the request; retain it across+ receiver restarts. The tenant field is context, not permission to act in an+ arbitrary account.+- For normal queued firing and recovery notifications, use each member of+ `{{.Alerts}}`, not only the primary `{{.ID}}`. The [full PSA template below](#sample-psa-payloads)+ includes each member's `alertId` and full-precision UTC `startedAt`. These+ identify its observed occurrence; a later start with the same ID must be+ processed as a new occurrence. Firing and recovery use the same occurrence+ record, so an old delayed recovery must not close a newer incident.+- Within that record, handle `event` **and severity**. A warning becoming+ critical is an update, not a duplicate warning. Decide explicitly whether+ reminders update the existing ticket; do not create another ticket for each+ retry or discard a severity increase. Group membership can change without+ changing the header: process every member, even when the primary is unchanged.+- The primary `{{.StartTime}}` string has only whole-second precision. Member+ `StartTime` values in `{{.Alerts}}` are Go times and can be formatted with+ fractional seconds as below. Do not assume the shorter string is universally+ unique. A test or legacy payload can lack member identity; keep incomplete or+ ambiguous events for reconciliation rather than silently treating them as+ duplicates. **Send test** alone does not validate this lifecycle.+- Commit receiver state and the ticket action atomically, or use your ticket+ system's durable idempotent operation. Do not mark an event processed before+ its action succeeds. Exact signed-byte replays can be rejected separately,+ but a queue retry may render a new body or send timestamp: hashing those alone+ is not logical-event deduplication. There is no universal exactly-once key in+ this legacy header.++Validate in an authorised test environment with firing, retry, warning-to-critical,+recovery, recurrence of the same condition and a changed multi-alert group.+Check actual tickets and receiver restart behaviour before enabling automatic+actions. Do not cause a production outage or notification storm for this test.+## 🛡️ Security- **Private IPs**: By default, webhooks to private IPs are blocked. Allow them in **Settings → System → Network → Webhook Security**.@@ -204,7 +253,14 @@ Suspended or pending-deletion organizations return `403`, and an unknown org IDThere are two integration models. The push model is usually the right fit when tickets or incidents should open and close automatically.-**Push (recommended): one outbound webhook per organization.** Create a **Generic** webhook for each organization and point it at your external system's inbound endpoint (an ITSM/PSA inbound webhook, an email connector, or middleware that opens service tickets). Shape the JSON with a [custom template](#-custom-templates) so it matches the receiving system's expected schema: every template variable listed above is available. Pulse fires on both `alert` and `resolved` events (`{{.Event}}` is `"alert"` or `"resolved"`), so the receiving system can open a ticket on alert and auto-resolve it on recovery. Add authentication as a custom header (e.g. `Authorization: Bearer ...`).+**Push (recommended): one outbound webhook per organization.** Create a+**Generic** webhook for each organization and point it at your external system's+inbound endpoint (an ITSM/PSA inbound webhook, an email connector, or middleware+that opens service tickets). Shape the JSON with a [custom template](#-custom-templates)+to match the receiving system's schema. The bridge can open or update a ticket+on `alert` and resolve that occurrence on `resolved`, subject to the [delivery+conditions above](#-delivery-contract). Add authentication as a custom header+in the settings form; keep its value private.Configure it from the UI (**Alerts → Notifications → Add Webhook**) per org, or programmatically with an org-bound admin token:@@ -221,20 +277,25 @@ Content-Type: application/json"service": "generic","enabled": true,"headers": { "Authorization": "Bearer <psa-token>" },- "template": "{\"summary\":\"{{.Level}}: {{.ResourceName}} {{.Message | jsonString}}\",\"event\":\"{{.Event}}\",\"alertId\":\"{{.ID}}\"}"+ "template": "{\"summary\":\"{{.Level | jsonString}}: {{.ResourceName | jsonString}} {{.Message | jsonString}}\",\"event\":\"{{.Event | jsonString}}\",\"alertId\":\"{{.ID | jsonString}}\",\"startedAt\":\"{{.StartTime | jsonString}}\"}"}```-The exact ticket fields differ by platform (ConnectWise, Autotask, Halo, and others each expect their own inbound shape), so map the template to your platform's contract. The `alertId` round-trips through `{{.ID}}`, which lets the receiving system correlate the later `resolved` event to the ticket it opened.+The exact ticket fields differ by platform (ConnectWise, Autotask, Halo, and+others each expect their own inbound shape), so map the template to your+platform's contract. This short example carries only the primary alert and a+whole-second start. For a bridge handling real grouped notifications, use the+member-aware template below; do not use the short example as an exactly-once key.### Sample PSA payloads-A fuller template suited to ticket routing, including tenant identity and the-stable severity/type fields from the [delivery contract](#-delivery-contract):+A fuller template for normal queued firing and recovery notifications includes+tenant context and every member's occurrence, severity and condition. The+primary fields remain convenient summary context, **not the whole batch**:```json{- "event": "{{.Event}}",+ "event": "{{.Event | jsonString}}","alertId": "{{.ID | jsonString}}","messageKey": "{{.MessageKey | jsonString}}","severity": "{{.Level | jsonString}}",@@ -248,11 +309,24 @@ stable severity/type fields from the [delivery contract](#-delivery-contract):"value": {{.Value}},"threshold": {{.Threshold}},"startedAt": "{{.StartTime | jsonString}}",- "duration": "{{.Duration | jsonString}}"+ "duration": "{{.Duration | jsonString}}",+ "alertCount": {{.AlertCount}},+ "alerts": [{{$comma := ""}}{{range .Alerts}}{{if .}}{{$comma}}+ {+ "alertId": "{{.ID | jsonString}}",+ "startedAt": "{{.StartTime.UTC.Format "2006-01-02T15:04:05.999999999Z07:00" | jsonString}}",+ "severity": "{{.Level | jsonString}}",+ "alertType": "{{.Type | jsonString}}",+ "resourceId": "{{.ResourceID | jsonString}}",+ "resource": "{{.ResourceName | jsonString}}",+ "summary": "{{.Message | jsonString}}"+ }{{$comma = ","}}{{end}}{{end}}+ ]}```-What the receiver sees for a **critical** alert:+An abridged primary summary for a **critical** alert (the template also emits+the `alerts` member array):```json{@@ -272,9 +346,12 @@ What the receiver sees for a **critical** alert:}```-A **warning** alert is identical except `"severity": "warning"` — warning and critical are the only two severities Pulse emits, so a two-priority PSA mapping covers the full range.+A **warning** alert is identical except `"severity": "warning"`. Map+informational levels too; a two-priority mapping does not cover every condition.-The **resolved** event reuses the same `alertId`, letting the bridge close the ticket it opened:+The **resolved** event retains the original member `alertId` and `startedAt`,+letting the bridge close that occurrence's ticket rather than a later incident+with the same ID. Its primary summary is:```json{@@ -294,7 +371,12 @@ The **resolved** event reuses the same `alertId`, letting the bridge close the t}```-For ConnectWise specifically, point the webhook at a ConnectWise inbound API callback (or middleware that calls the ConnectWise REST API) and map `severity` to ticket priority, `tenantName` to the company, and `alertId` to your correlation field. Combine with a [`signingSecret`](#-delivery-contract) and the `X-Pulse-Event-ID` dedup header for a production-grade bridge.+For ConnectWise specifically, point the webhook at a ConnectWise inbound API+callback (or middleware that calls the ConnectWise REST API). Map each member's+severity to ticket priority and the tenant to your configured company. Combine+[`signingSecret`](#-delivery-contract) with the [receiver correlation rules](#receiver-correlation-and-deduplication),+not header-only deduplication. Verify your platform's processing and recovery+behaviour before treating the bridge as production-ready.**Pull (poll): org-scoped read API.** Issue a `monitoring:read` token bound to each organization and poll that org's alerts. Send `X-Pulse-Org-ID` (or rely on the org-bound token) so you get only that organization's data:diff --git a/internal/notifications/notifications.go b/internal/notifications/notifications.goindex 18d01ea67a..61277858b0 100644--- a/internal/notifications/notifications.go+++ b/internal/notifications/notifications.go@@ -280,7 +280,7 @@ type webhookHTTPResult struct {type webhookRequestOptions struct {alertType string- eventID string // Idempotency token sent as X-Pulse-Event-ID (stable across retries)+ eventID string // Legacy X-Pulse-Event-ID correlation header, not a unique delivery IDtimeout time.DurationuserAgent stringresponseLogging bool@@ -3344,9 +3344,9 @@ func signWebhookPayload(secret, timestamp string, payload []byte) string {return hex.EncodeToString(mac.Sum(nil))}-// webhookEventID builds the idempotency token for a delivery: stable for the-// same alert occurrence and event so receivers can deduplicate retries at-// either retry layer (transport or queue).+// webhookEventID builds the legacy correlation header, stable across retries.+// It omits occurrence start, severity and grouped membership, so it is not a+// unique incident/delivery identity or a sufficient durable deduplication key.func webhookEventID(alertID, event string) string {if alertID == "" {return ""diff --git a/internal/notifications/webhook_guide_test.go b/internal/notifications/webhook_guide_test.gonew file mode 100644index 0000000000..37a27878f8--- /dev/null+++ b/internal/notifications/webhook_guide_test.go@@ -0,0 +1,193 @@+package notifications++import (+ "crypto/hmac"+ "crypto/sha256"+ "encoding/hex"+ "encoding/json"+ "io"+ "net/http"+ "os"+ "regexp"+ "strings"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/alerts"+ "github.com/stretchr/testify/require"+)++// Exercise the copyable templates through the real sender, not a second+// renderer or a model receiver. The input override permits an exact-parent+// documentation negative control without modifying production source.+func webhookGuide(t *testing.T) string {+ t.Helper()+ path := os.Getenv("PULSE_WEBHOOK_GUIDE_TEST_INPUT")+ if path == "" {+ path = "../../docs/WEBHOOKS.md"+ }+ data, err := os.ReadFile(path)+ require.NoError(t, err)+ return string(data)+}++func webhookGuidePSATemplate(t *testing.T) string {+ t.Helper()+ sections := strings.SplitN(webhookGuide(t), "### Sample PSA payloads", 2)+ require.Len(t, sections, 2)+ block := regexp.MustCompile("(?s)```json\\n(.*?)\\n```").FindStringSubmatch(sections[1])+ require.Len(t, block, 2)+ return block[1]+}++type guideMember struct {+ ID string `json:"alertId"`+ StartedAt string `json:"startedAt"`+ Severity string `json:"severity"`+ ResourceID string `json:"resourceId"`+ Resource string `json:"resource"`+ Summary string `json:"summary"`+}++type guidePayload struct {+ Event string `json:"event"`+ TenantID string `json:"tenantId"`+ StartedAt string `json:"startedAt"`+ AlertCount int `json:"alertCount"`+ Members []guideMember `json:"alerts"`+}++type guideRequest struct {+ body, eventID, timestamp, signature string+}++func webhookGuideSender(t *testing.T, payloadTemplate string) (*NotificationManager, WebhookConfig, <-chan guideRequest) {+ t.Helper()+ manager := NewNotificationManager("https://pulse.example.test")+ require.NoError(t, manager.UpdateAllowedPrivateCIDRs("127.0.0.1"))+ manager.SetTenantIdentityResolver(func() (string, string) { return "synthetic-tenant", "Synthetic tenant" })+ requests := make(chan guideRequest, 16)+ server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ body, err := io.ReadAll(r.Body)+ if err != nil {+ w.WriteHeader(http.StatusBadRequest)+ return+ }+ requests <- guideRequest{string(body), r.Header.Get("X-Pulse-Event-ID"),+ r.Header.Get("X-Pulse-Timestamp"), r.Header.Get("X-Pulse-Signature")}+ w.WriteHeader(http.StatusOK)+ }))+ t.Cleanup(server.Close)+ webhook := WebhookConfig{ID: "synthetic-guide", Name: "Synthetic guide receiver", URL: server.URL,+ Enabled: true, Service: "generic", SigningSecret: "synthetic-guide-test-secret", Template: payloadTemplate}+ return manager, webhook, requests+}++func webhookGuideRead(t *testing.T, requests <-chan guideRequest) (guideRequest, guidePayload) {+ t.Helper()+ select {+ case request := <-requests:+ var payload guidePayload+ require.NoError(t, json.Unmarshal([]byte(request.body), &payload), "copied template must render valid JSON")+ mac := hmac.New(sha256.New, []byte("synthetic-guide-test-secret"))+ _, err := mac.Write([]byte(request.timestamp + "." + request.body))+ require.NoError(t, err)+ require.Equal(t, "v1="+hex.EncodeToString(mac.Sum(nil)), request.signature)+ return request, payload+ case <-time.After(5 * time.Second):+ t.Fatal("no local sender request received")+ return guideRequest{}, guidePayload{}+ }+}++func TestWebhookGuidePSALifecycle(t *testing.T) {+ manager, webhook, requests := webhookGuideSender(t, webhookGuidePSATemplate(t))+ start := time.Date(2026, 10, 4, 7, 0, 0, 123456789, time.UTC)+ alert := &alerts.Alert{ID: "host::metric-threshold:cpu", StartTime: start,+ Level: alerts.AlertLevelWarning, Type: "cpu", ResourceID: "synthetic-host",+ ResourceName: "quoted \"host\"\nwith é", Message: "synthetic \\ message\n\"quoted\"", Value: 95, Threshold: 90}++ require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{alert}))+ firstRequest, first := webhookGuideRead(t, requests)+ require.Len(t, first.Members, 1)+ require.Equal(t, "alert", first.Event)+ require.Equal(t, "synthetic-tenant", first.TenantID)+ require.Equal(t, 1, first.AlertCount)+ require.Equal(t, guideMember{alert.ID, start.Format(time.RFC3339Nano), "warning",+ alert.ResourceID, alert.ResourceName, alert.Message}, first.Members[0])++ // A synthetic retry retains the member identity even when rendering anew.+ require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{alert}))+ retryRequest, retry := webhookGuideRead(t, requests)+ require.Equal(t, first.Members, retry.Members)+ require.Equal(t, firstRequest.eventID, retryRequest.eventID)++ critical := alert.Clone()+ critical.Level = alerts.AlertLevelCritical+ require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{critical}))+ criticalRequest, escalated := webhookGuideRead(t, requests)+ require.Equal(t, firstRequest.eventID, criticalRequest.eventID, "header cannot identify a severity change")+ require.Equal(t, first.Members[0].StartedAt, escalated.Members[0].StartedAt)+ require.Equal(t, "critical", escalated.Members[0].Severity)++ require.NoError(t, manager.sendResolvedWebhook(webhook, []*alerts.Alert{critical}, start.Add(time.Minute)))+ _, resolved := webhookGuideRead(t, requests)+ require.Equal(t, "resolved", resolved.Event)+ require.Equal(t, escalated.Members, resolved.Members, "recovery must correlate to the original occurrence")++ // A second occurrence in the SAME SECOND defeats both header-only and+ // whole-second primary-start deduplication. Member precision preserves it.+ recurrent := critical.Clone()+ recurrent.StartTime = start.Add(100 * time.Nanosecond)+ require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{recurrent}))+ recurrentRequest, recurrence := webhookGuideRead(t, requests)+ require.Equal(t, firstRequest.eventID, recurrentRequest.eventID)+ require.Equal(t, first.StartedAt, recurrence.StartedAt)+ require.NotEqual(t, first.Members[0].StartedAt, recurrence.Members[0].StartedAt)+ require.Equal(t, recurrent.StartTime.Format(time.RFC3339Nano), recurrence.Members[0].StartedAt)+}++func TestWebhookGuidePSAGroupMembershipAndInfo(t *testing.T) {+ manager, webhook, requests := webhookGuideSender(t, webhookGuidePSATemplate(t))+ start := time.Date(2026, 10, 4, 9, 0, 0, 987654321, time.FixedZone("synthetic", 2*60*60))+ primary := &alerts.Alert{ID: "primary", StartTime: start, Level: alerts.AlertLevelWarning}+ info := &alerts.Alert{ID: "info-member", StartTime: start.Add(time.Second), Level: alerts.AlertLevelInfo}+ critical := &alerts.Alert{ID: "critical-member", StartTime: start.Add(2 * time.Second), Level: alerts.AlertLevelCritical}+ for _, members := range [][]*alerts.Alert{{primary, info}, {primary, critical}, {primary, critical, info}} {+ require.NoError(t, manager.sendGroupedWebhook(webhook, members))+ request, payload := webhookGuideRead(t, requests)+ require.Equal(t, "primary:alert", request.eventID, "changed groups can have the same correlation header")+ require.Equal(t, len(members), payload.AlertCount)+ require.Len(t, payload.Members, len(members))+ for index, alert := range members {+ require.Equal(t, alert.ID, payload.Members[index].ID)+ require.Equal(t, string(alert.Level), payload.Members[index].Severity)+ require.Equal(t, alert.StartTime.UTC().Format(time.RFC3339Nano), payload.Members[index].StartedAt)+ }+ }+ require.NoError(t, manager.sendResolvedWebhook(webhook, []*alerts.Alert{primary, critical, info}, start.Add(time.Minute)))+ _, resolved := webhookGuideRead(t, requests)+ require.Equal(t, "resolved", resolved.Event)+ require.Len(t, resolved.Members, 3)+ require.Equal(t, "info", resolved.Members[2].Severity)+}++func TestWebhookGuideShortTemplateEscapesStrings(t *testing.T) {+ section := strings.SplitN(webhookGuide(t), "### Sample PSA payloads", 2)[0]+ block := regexp.MustCompile("(?s)```http\\nPOST /api/notifications/webhooks\\nX-Pulse-Org-ID:.*?\\n\\n(.*?)\\n```").FindStringSubmatch(section)+ require.Len(t, block, 2)+ var config struct {+ Template string `json:"template"`+ }+ require.NoError(t, json.Unmarshal([]byte(block[1]), &config))+ manager, webhook, requests := webhookGuideSender(t, config.Template)+ alert := &alerts.Alert{ID: "quoted\"id", ResourceName: "host\"\\\nname", Message: "message\"\\\n",+ StartTime: time.Date(2026, 10, 4, 7, 0, 0, 123, time.UTC), Level: alerts.AlertLevelWarning}+ require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{alert}))+ request, _ := webhookGuideRead(t, requests)+ var payload map[string]interface{}+ require.NoError(t, json.Unmarshal([]byte(request.body), &payload))+ require.Equal(t, alert.ID, payload["alertId"])+ require.Equal(t, "warning: "+alert.ResourceName+" "+alert.Message, payload["summary"])+ require.Equal(t, alert.StartTime.Format(time.RFC3339), payload["startedAt"])+}diff --git a/scripts/tests/test_webhook_verification_docs.py b/scripts/tests/test_webhook_verification_docs.pyindex 6092c377b5..db57a6f58b 100644--- a/scripts/tests/test_webhook_verification_docs.py+++ b/scripts/tests/test_webhook_verification_docs.py@@ -146,6 +146,22 @@ class WebhookVerificationDocsTest(unittest.TestCase):"not from the request", "Never log the secret"):self.assertIn(phrase, text)+ def test_receiver_guidance_preserves_recurrences_and_all_group_members(self):+ # The Go sender tests render the copied PSA templates over actual+ # loopback HTTP. Keep the safety explanation beside those examples.+ text = " ".join(DOC.read_text().split())+ for phrase in ("not a unique incident or delivery ID", "later occurrences",+ "Do not deduplicate permanently", "every member",+ "old delayed recovery must not close a newer incident",+ "event and severity", "whole-second precision",+ "incomplete or ambiguous events", "atomically",+ "Do not mark an event processed before its action succeeds"):+ self.assertIn(phrase, text.replace("**", ""))+ self.assertNotIn("deduplicate on it", text)+ self.assertNotIn("two-priority PSA mapping covers the full range", text)+ self.assertIn('"info"', text)+ self.assertIn("successful firing-delivery receipt", text)+if __name__ == "__main__":unittest.main()