Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Make audit webhook guidance match delivery and trust boundaries
pulse-triage[bot]31db5415036d
diff --git a/.github/workflows/public-docs.yml b/.github/workflows/public-docs.ymlindex 6fd4583567..54be3619a4 100644--- a/.github/workflows/public-docs.yml+++ b/.github/workflows/public-docs.yml@@ -15,6 +15,7 @@ on:- "scripts/tests/test_production_rollout_docs.py"- "scripts/tests/test_pmg_docs.py"- "scripts/tests/test_agent_profile_docs.py"+ - "scripts/tests/test_audit_recovery_docs.py"- "scripts/tests/test_pve_backup_troubleshooting_docs.py"- "scripts/tests/test_vm_disk_diagnostics.py"- "scripts/tests/test_memory_troubleshooting_docs.py"@@ -38,6 +39,7 @@ on:- "scripts/tests/test_production_rollout_docs.py"- "scripts/tests/test_pmg_docs.py"- "scripts/tests/test_agent_profile_docs.py"+ - "scripts/tests/test_audit_recovery_docs.py"- "scripts/tests/test_pve_backup_troubleshooting_docs.py"- "scripts/tests/test_vm_disk_diagnostics.py"- "scripts/tests/test_memory_troubleshooting_docs.py"@@ -84,6 +86,7 @@ jobs:test_production_rollout_docs.py \test_pmg_docs.py \test_agent_profile_docs.py \+ test_audit_recovery_docs.py \test_pve_backup_troubleshooting_docs.py \test_vm_disk_diagnostics.py \test_memory_troubleshooting_docs.py \diff --git a/docs/WEBHOOKS.md b/docs/WEBHOOKS.mdindex 32aa1ec509..27184ed561 100644--- a/docs/WEBHOOKS.md+++ b/docs/WEBHOOKS.md@@ -302,14 +302,89 @@ actions. Do not cause a production outage or notification storm for this test.## π§Ύ Audit Webhooks (Pro/legacy Pro+/Cloud)-Pro, legacy Pro+, and Cloud support dedicated audit webhooks for security event compliance. Unlike alert notifications, these webhooks deliver the raw, signed JSON payload of every security-relevant action (login, config change, group mapping).+Audit webhooks forward recorded security events, such as logins and settings+changes, to an authorised receiver. They are separate from **Alerts β+Notifications**: alert templates, signing secrets, retry policies and delivery+activity do not configure or describe this sender.++Delivery requires the `audit_logging` capability and a supporting runtime. An+active licence on the public Community runtime does not enable these hooks;+if the panel says **Pulse Pro runtime required**, follow **Download Pulse Pro**+for that deployment. Do not buy another licence or reset data to clear the gate.### Setup-1. Go to **Settings β Security β Audit Webhooks**.-2. Add your endpoint URL (e.g., `https://siem.corp.local/ingest/pulse`).+1. Sign in as an administrator and open **Settings β Security β Audit Webhooks**+ in the intended organisation. Configuration is organisation-scoped.+2. Add a receiver you control, using HTTPS with valid certificate trust, for+ example `https://audit.example.com/ingest/pulse`. This is a placeholder, not+ a destination to send real events to. Audit delivery blocks loopback,+ private/reserved IPs, internal hostnames and public names resolving to those+ addresses. The alert-webhook private-network setting does not relax this+ audit restriction. Do not bypass it by changing DNS, TLS verification or+ network isolation; a private-only SIEM needs a separately secured, authorised+ ingress design, not the `.local` URL previously shown here.+3. Saving the URL is configuration acceptance, not a delivery test. Check the+ receiver against an already-recorded event during normal operation. Do not+ create failed logins, change roles or trigger an alert just to test delivery.++The API configuration route is `/api/admin/webhooks/audit`; it requires+administrator access and the same licence capability, with `settings:read`+for GET and `settings:write` for updates. Its write body is a **complete URL+list**, not an append operation. Prefer the signed-in UI; do not copy bearer+tokens or session cookies into commands, URLs or reports.++### Delivery and missing events++Audit forwarding is **best effort**, not guaranteed delivery of every event.+Its in-memory queue holds at most 1,000 events and drops new events when full.+A restart does not replay that queue from stored history. Each destination gets+up to four attempts (the initial request plus three retries), with waits of+1, 5 and 30 seconds and a 30-second request timeout. These bounds are not a+total delivery deadline. Only an HTTP 2xx response counts as send success; it+does not prove the receiver verified or durably stored the event.++Three workers can deliver concurrently, so receivers must tolerate duplicates+and out-of-order arrival. After the attempts are exhausted there is **no audit+dead-letter queue or automatic history replay**. Alert **Recent delivery+activity**, **Dismiss** and **Retry** do not recover audit deliveries. Removing+an endpoint changes future routing; it is not a guaranteed cancellation of a+request already in flight.++For a missing event, reconcile the intended organisation's retained audit+history, a bounded local sender error and the receiver's own record. Absence at+the receiver is not proof that the action was never recorded. A history query+error is not an empty history. Use [private, filtered audit reads and exports](AUDIT_LOGGING.md#viewing-audit-events)+when authorised, not a broad export or a production replay. Preserve keys and+history; do not restart or clear data to make the discrepancy disappear.### Security-Audit webhooks are dispatched asynchronously. The payload includes a `signature` field which can be verified using the per-instance HMAC key stored (encrypted) at `.audit-signing.key` in the Pulse data directory. There is no `PULSE_AUDIT_SIGNING_KEY` override.+The POST envelope contains `event` (`audit.` plus the event type), `timestamp`+and `data` (the recorded event). Its event ID is `data.id`, also sent as+`X-Pulse-Event-ID`; keep receiver deduplication scoped to the originating+instance/organisation. Event IDs and headers alone do not authenticate a sender.++When signing was available at capture, the signature is **`data.signature`**,+not a top-level payload field. It authenticates the signer's canonical event+fields, **not the raw POST body**; the alert webhook `X-Pulse-Signature`+timestamp/body verifier above is not an audit verifier. Events captured without+signing can omit this field. Retain unsigned, failed or unknown verification+outcomes rather than treating them as authenticated events or as proof of+tampering. A valid signature does not establish complete history, freshness or+freedom from replay. Use the [stored-event verification guidance](AUDIT_LOGGING.md#tamper-detection)+and the signature format supported by the originating version before a receiver+acts on an event.++The default encrypted key is at **`audit/.audit-signing.key`** relative to the+instance or organisation data directory, alongside the audit store, and needs+its matching `.encryption.key`. A runtime can use a different audit directory+or a managed signing key; follow [its actual storage configuration](AUDIT_LOGGING.md#storage),+not a guessed root path. Do not copy an encrypted key into a raw-body HMAC+recipe, disclose keys or reset them to fix delivery. There is no+`PULSE_AUDIT_SIGNING_KEY` override.++Audit events can contain users, IPs, paths and private details; endpoint URLs+and sender errors can contain ingest secrets. Keep raw payloads, full URLs,+headers and logs private. Share only a locally reviewed, redacted excerpt.## π’ Provider-hosted MSP webhooksdiff --git a/frontend-modern/public/docs/WEBHOOKS.md b/frontend-modern/public/docs/WEBHOOKS.mdindex 32aa1ec509..27184ed561 100644--- a/frontend-modern/public/docs/WEBHOOKS.md+++ b/frontend-modern/public/docs/WEBHOOKS.md@@ -302,14 +302,89 @@ actions. Do not cause a production outage or notification storm for this test.## π§Ύ Audit Webhooks (Pro/legacy Pro+/Cloud)-Pro, legacy Pro+, and Cloud support dedicated audit webhooks for security event compliance. Unlike alert notifications, these webhooks deliver the raw, signed JSON payload of every security-relevant action (login, config change, group mapping).+Audit webhooks forward recorded security events, such as logins and settings+changes, to an authorised receiver. They are separate from **Alerts β+Notifications**: alert templates, signing secrets, retry policies and delivery+activity do not configure or describe this sender.++Delivery requires the `audit_logging` capability and a supporting runtime. An+active licence on the public Community runtime does not enable these hooks;+if the panel says **Pulse Pro runtime required**, follow **Download Pulse Pro**+for that deployment. Do not buy another licence or reset data to clear the gate.### Setup-1. Go to **Settings β Security β Audit Webhooks**.-2. Add your endpoint URL (e.g., `https://siem.corp.local/ingest/pulse`).+1. Sign in as an administrator and open **Settings β Security β Audit Webhooks**+ in the intended organisation. Configuration is organisation-scoped.+2. Add a receiver you control, using HTTPS with valid certificate trust, for+ example `https://audit.example.com/ingest/pulse`. This is a placeholder, not+ a destination to send real events to. Audit delivery blocks loopback,+ private/reserved IPs, internal hostnames and public names resolving to those+ addresses. The alert-webhook private-network setting does not relax this+ audit restriction. Do not bypass it by changing DNS, TLS verification or+ network isolation; a private-only SIEM needs a separately secured, authorised+ ingress design, not the `.local` URL previously shown here.+3. Saving the URL is configuration acceptance, not a delivery test. Check the+ receiver against an already-recorded event during normal operation. Do not+ create failed logins, change roles or trigger an alert just to test delivery.++The API configuration route is `/api/admin/webhooks/audit`; it requires+administrator access and the same licence capability, with `settings:read`+for GET and `settings:write` for updates. Its write body is a **complete URL+list**, not an append operation. Prefer the signed-in UI; do not copy bearer+tokens or session cookies into commands, URLs or reports.++### Delivery and missing events++Audit forwarding is **best effort**, not guaranteed delivery of every event.+Its in-memory queue holds at most 1,000 events and drops new events when full.+A restart does not replay that queue from stored history. Each destination gets+up to four attempts (the initial request plus three retries), with waits of+1, 5 and 30 seconds and a 30-second request timeout. These bounds are not a+total delivery deadline. Only an HTTP 2xx response counts as send success; it+does not prove the receiver verified or durably stored the event.++Three workers can deliver concurrently, so receivers must tolerate duplicates+and out-of-order arrival. After the attempts are exhausted there is **no audit+dead-letter queue or automatic history replay**. Alert **Recent delivery+activity**, **Dismiss** and **Retry** do not recover audit deliveries. Removing+an endpoint changes future routing; it is not a guaranteed cancellation of a+request already in flight.++For a missing event, reconcile the intended organisation's retained audit+history, a bounded local sender error and the receiver's own record. Absence at+the receiver is not proof that the action was never recorded. A history query+error is not an empty history. Use [private, filtered audit reads and exports](AUDIT_LOGGING.md#viewing-audit-events)+when authorised, not a broad export or a production replay. Preserve keys and+history; do not restart or clear data to make the discrepancy disappear.### Security-Audit webhooks are dispatched asynchronously. The payload includes a `signature` field which can be verified using the per-instance HMAC key stored (encrypted) at `.audit-signing.key` in the Pulse data directory. There is no `PULSE_AUDIT_SIGNING_KEY` override.+The POST envelope contains `event` (`audit.` plus the event type), `timestamp`+and `data` (the recorded event). Its event ID is `data.id`, also sent as+`X-Pulse-Event-ID`; keep receiver deduplication scoped to the originating+instance/organisation. Event IDs and headers alone do not authenticate a sender.++When signing was available at capture, the signature is **`data.signature`**,+not a top-level payload field. It authenticates the signer's canonical event+fields, **not the raw POST body**; the alert webhook `X-Pulse-Signature`+timestamp/body verifier above is not an audit verifier. Events captured without+signing can omit this field. Retain unsigned, failed or unknown verification+outcomes rather than treating them as authenticated events or as proof of+tampering. A valid signature does not establish complete history, freshness or+freedom from replay. Use the [stored-event verification guidance](AUDIT_LOGGING.md#tamper-detection)+and the signature format supported by the originating version before a receiver+acts on an event.++The default encrypted key is at **`audit/.audit-signing.key`** relative to the+instance or organisation data directory, alongside the audit store, and needs+its matching `.encryption.key`. A runtime can use a different audit directory+or a managed signing key; follow [its actual storage configuration](AUDIT_LOGGING.md#storage),+not a guessed root path. Do not copy an encrypted key into a raw-body HMAC+recipe, disclose keys or reset them to fix delivery. There is no+`PULSE_AUDIT_SIGNING_KEY` override.++Audit events can contain users, IPs, paths and private details; endpoint URLs+and sender errors can contain ingest secrets. Keep raw payloads, full URLs,+headers and logs private. Share only a locally reviewed, redacted excerpt.## π’ Provider-hosted MSP webhooksdiff --git a/scripts/tests/test_audit_recovery_docs.py b/scripts/tests/test_audit_recovery_docs.pyindex 8240521efa..d78798b598 100644--- a/scripts/tests/test_audit_recovery_docs.py+++ b/scripts/tests/test_audit_recovery_docs.py@@ -11,7 +11,7 @@ import unittestROOT = Path(__file__).resolve().parents[2]-GUIDES = ("AUDIT_LOGGING", "TROUBLESHOOTING", "CONFIGURATION", "DEPLOYMENT_MODELS")+GUIDES = ("AUDIT_LOGGING", "TROUBLESHOOTING", "CONFIGURATION", "DEPLOYMENT_MODELS", "WEBHOOKS")def read_guide(name: str) -> str:@@ -82,5 +82,117 @@ class AuditRecoveryDocsTest(unittest.TestCase):self.assertIn("runtime may supply a different audit directory or managed signing key", guide)+class AuditWebhookDocsTest(unittest.TestCase):+ """Static guidance/source binding, not delivery or signature execution.++ Override text for literal-parent/adverse controls without altering source+ or reducing a substantive failure to a shipped-mirror mismatch.+ """++ text = None++ def setUp(self):+ text = self.text if self.text is not None else (ROOT / "docs/WEBHOOKS.md").read_text()+ self.guide = " ".join(text.split("## π§Ύ Audit Webhooks", 1)[1].split(+ "## π’ Provider-hosted MSP webhooks", 1)[0].replace("**", "").split())+ self.sender = (ROOT / "pkg/audit/webhook.go").read_text()++ def test_audit_access_is_not_alert_configuration_or_licence_only(self):+ for phrase in ("separate from", "retry policies and delivery activity do not configure",+ "`audit_logging` capability and a supporting runtime",+ "public Community runtime does not enable", "Pulse Pro runtime required",+ "Download Pulse Pro", "Do not buy another licence or reset data",+ "Configuration is organisation-scoped", "administrator access",+ "`settings:read` for GET and `settings:write` for updates",+ "complete URL list", "not an append operation"):+ self.assertIn(phrase, self.guide)+ route = (ROOT / "internal/api/router_routes_licensing.go").read_text().split(+ "// Audit Webhook routes", 1)[1].split("type auditAdminEndpointAdapter", 1)[0]+ for boundary in ("ensureAdminSession", "featureAuditLoggingValue", "ScopeSettingsRead",+ "ScopeSettingsWrite", "auth.ResourceAuditLogs"):+ self.assertIn(boundary, route)+ handler = (ROOT / "internal/api/activity_audit_handlers.go").read_text().split(+ "func (h *AuditHandlers) HandleUpdateWebhooks", 1)[1].split(+ "// validateWebhookURL", 1)[0]+ self.assertIn('URLs []string `json:"urls"`', handler)+ self.assertIn("getLoggerForOrg(orgID)", handler)+ self.assertIn("logger.UpdateWebhookURLs(validatedURLs)", handler)++ def test_audit_endpoint_restrictions_are_not_an_alert_setting(self):+ for phrase in ("using HTTPS with valid certificate trust", "placeholder",+ "private/reserved IPs", "public names resolving to those addresses",+ "does not relax this audit restriction", "Do not bypass it",+ "separately secured, authorised ingress", "not a delivery test",+ "Do not create failed logins"):+ self.assertIn(phrase, self.guide)+ self.assertNotIn("https://siem.corp.local", self.guide)+ validator = self.sender.split("func validateWebhookURL(", 1)[1]+ for boundary in ('".local"', "isPrivateOrReservedIP(addr.IP)", "isPrivateOrReservedIP(ip)"):+ self.assertIn(boundary, validator)+ self.assertNotIn("WebhookAllowedCIDRs", self.sender)++ def test_queue_attempts_and_success_are_not_complete_delivery(self):+ constants = dict(re.findall(r"webhook(QueueSize|MaxRetries|WorkerCount)\s*=\s*(\d+)", self.sender))+ self.assertIn(f"at most {int(constants['QueueSize']):,} events", self.guide)+ self.assertEqual(int(constants["MaxRetries"]), 3)+ self.assertEqual(int(constants["WorkerCount"]), 3)+ self.assertIn("webhookTimeout = 30 * time.Second", self.sender)+ self.assertIn("[]time.Duration{1 * time.Second, 5 * time.Second, 30 * time.Second}", self.sender)+ for phrase in ("best effort", "drops new events when full", "in-memory queue",+ "restart does not replay", "up to four attempts", "1, 5 and 30 seconds",+ "30-second request timeout", "not a total delivery deadline",+ "Only an HTTP 2xx", "does not prove the receiver verified or durably stored"):+ self.assertIn(phrase, self.guide)+ self.assertNotIn("payload of every security-relevant action", self.guide)+ self.assertIn("make(chan Event, webhookQueueSize)", self.sender)+ self.assertIn("attempt <= webhookMaxRetries", self.sender)+ self.assertIn("resp.StatusCode < 200 || resp.StatusCode >= 300", self.sender)++ def test_receiver_and_missing_event_guidance_preserve_uncertainty(self):+ for phrase in ("duplicates and out-of-order arrival", "no audit dead-letter queue",+ "automatic history replay", "do not recover audit deliveries",+ "not a guaranteed cancellation", "bounded local sender error",+ "not proof that the action was never recorded",+ "history query error is not an empty history", "not a broad export",+ "Preserve keys and history", "do not restart or clear data"):+ self.assertIn(phrase, self.guide)+ for anchor in ("#viewing-audit-events", "#tamper-detection", "#storage"):+ self.assertIn(f"AUDIT_LOGGING.md{anchor}", self.guide)+ self.assertIn("urls := make([]string, len(w.urls))", self.sender)+ self.assertIn("copy(urls, w.urls)", self.sender)++ def test_signature_is_on_the_event_not_the_raw_post_body(self):+ for phrase in ("`data.id`", "`X-Pulse-Event-ID`", "instance/organisation",+ "headers alone do not authenticate", "`data.signature`",+ "not a top-level payload field", "not the raw POST body",+ "not an audit verifier", "without signing can omit this field",+ "unsigned, failed or unknown", "freedom from replay"):+ self.assertIn(phrase, self.guide)+ payload = self.sender.split("type WebhookPayload struct {", 1)[1].split("\n}", 1)[0]+ for field in ('`json:"event"`', '`json:"timestamp"`', 'Data Event `json:"data"`'):+ self.assertIn(field, payload)+ event = (ROOT / "pkg/audit/audit.go").read_text()+ self.assertIn('Signature string `json:"signature,omitempty"`', event)+ self.assertIn('req.Header.Set("X-Pulse-Event-ID", event.ID)', self.sender)+ signer = (ROOT / "pkg/audit/signer.go").read_text()+ self.assertIn("s.mac(s.canonicalV2Form(event))", signer)++ def test_key_paths_and_private_evidence_do_not_offer_key_disclosure_or_reset(self):+ for phrase in ("`audit/.audit-signing.key`", "instance or organisation data directory",+ "matching `.encryption.key`", "different audit directory",+ "managed signing key", "not a guessed root path",+ "Do not copy an encrypted key", "disclose keys or reset them",+ "Keep raw payloads, full URLs, headers and logs private",+ "locally reviewed, redacted excerpt"):+ self.assertIn(phrase, self.guide)+ store = (ROOT / "pkg/audit/sqlite_logger.go").read_text()+ self.assertIn('auditDir = filepath.Join(cfg.DataDir, "audit")', store)+ self.assertIn("NewSigner(auditDir, cfg.CryptoMgr)", store)+ self.assertIn("NewSignerWithKey(cfg.SigningKey)", store)+ signer = (ROOT / "pkg/audit/signer.go").read_text()+ self.assertIn('keyPath := filepath.Join(dataDir, ".audit-signing.key")', signer)+ self.assertIn("cryptoMgr.Decrypt(data)", signer)++if __name__ == "__main__":unittest.main()diff --git a/scripts/tests/test_public_docs_workflow.py b/scripts/tests/test_public_docs_workflow.pyindex 5124ce8ee3..f9d4b0379b 100644--- a/scripts/tests/test_public_docs_workflow.py+++ b/scripts/tests/test_public_docs_workflow.py@@ -26,6 +26,7 @@ TESTS = ("test_production_rollout_docs.py","test_pmg_docs.py","test_agent_profile_docs.py",+ "test_audit_recovery_docs.py","test_pve_backup_troubleshooting_docs.py","test_vm_disk_diagnostics.py","test_memory_troubleshooting_docs.py",