Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Explain withheld webhook replies without replaying deliveries
pulse-triage[bot]01a7bc896a96
diff --git a/docs/ISSUE_TRIAGE.md b/docs/ISSUE_TRIAGE.mdindex bf3f80135d..cae83c2586 100644--- a/docs/ISSUE_TRIAGE.md+++ b/docs/ISSUE_TRIAGE.md@@ -134,6 +134,15 @@ payloads private, even when an error or screenshot contains them. A missingnotification alone does not prove that alert evaluation failed; retain bothsymptoms when the thread supplies evidence for each.+Treat webhook `response body withheld` or `details withheld` messages as+intentional privacy boundaries, not missing evidence to request from a reporter.+Use the supplied HTTP status, failure class and time; a response-read failure+does not prove the receiver rejected the request. A Test's bounded response byte+count is not the sent payload size or proof of receipt. If a provider-specific+distinction matters, use an existing, locally reviewed error code or redacted+explanation, never a raw response, Debug capture or replay. Historical errors+and other exports are not retroactively scrubbed; the full entry remains private.+## Required dispositionBefore removing `needs-decomposition` or declaring a mixed report triaged:diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.mdindex 75d796c6f5..dd2e563462 100644--- a/docs/TROUBLESHOOTING.md+++ b/docs/TROUBLESHOOTING.md@@ -922,10 +922,13 @@ without exhausting the retry budget; waiting alone will not resend it. Seetransient failures and the normal queued attempt limit.**Recent delivery activity** masks recognised credentials in URLs, **not all-private information**. Provider error text can still include email addresses,-private destinations or credentials echoed outside a URL; alert identifiers and-resource names can also identify your infrastructure. Keep full entries, copied-responses and screenshots private. A `REDACTED` marker does not make the rest+private information**. Older records and other error sources can still include+email addresses, private destinations or credentials echoed outside a URL;+alert identifiers and resource names can also identify your infrastructure.+Withheld webhook response text is intentional, not a reason to collect it again;+see [withheld-response guidance](#webhook-response-body-is-withheld).+Keep full entries, copied responses and screenshots private.+A `REDACTED` marker does not make the restsafe to share. For a public report, extract only the relevant timestamp,delivery method, failure class, HTTP status or SMTP error code and a manuallyredacted error. See [notification log precautions](#inspect-notification-logs).@@ -1016,6 +1019,39 @@ or enable debug logging just to collect it. If needed, inspect[bounded notification logs](#inspect-notification-logs) and share only theconsequential, manually redacted error.+#### Webhook response body is withheld++`response body withheld`, `details withheld` and **Response body withheld**+are deliberate privacy messages, not the receiver's explanation of a failure.+Alert-webhook receivers can echo tokens, alert content or private provider data+in their replies. Pulse discards that response text instead of copying it into+delivery errors, logs or a Test result. Do not enable Debug, capture raw traffic,+weaken a safeguard or send another Test or queue retry to recover the text.++Use the original timestamp, destination type, failure class and HTTP status+from **Recent delivery activity**. An HTTP rejection still matters when its+body is withheld; withholding alone does not mean that delivery failed.+For example, HTTP 429 is still rate limiting, not missing diagnostic data.+If the error says **failed to read response body**, reading the reply failed:+the receiver may already have accepted the request, so retrying can duplicate+it. Neither HTTP 2xx nor a completed Test proves receipt at the intended+destination or the success of queued alert delivery.++A Test summary such as `Response body withheld (128 bytes read)` records only+how much Pulse read, not the response's contents or total size. **read incomplete**+means the read failed; **read limit reached** means it reached the **1 MiB** cap,+not proof that the response exceeded that cap. A complete empty reply may have+no summary. These are not byte counts of the alert Pulse sent.++If provider-specific detail is consequential, review the receiver's existing+record locally using its normal authorised access. Share only the relevant+error code or manually redacted explanation, not response bodies, headers,+payloads or credentials. Unknown receipt remains unknown; do not replay the+request to create evidence. Older logs and retained delivery errors are **not+retroactively scrubbed**, and this safeguard does not sanitise every other error+or export. Keep those private and follow the+[log and export precautions](#inspect-notification-logs).+#### Telegram Test works but real alerts say "message text is empty"With the built-in Telegram template (no custom template), a blank or incorrectdiff --git a/docs/WEBHOOKS.md b/docs/WEBHOOKS.mdindex 1a2e26830a..eca89a612a 100644--- a/docs/WEBHOOKS.md+++ b/docs/WEBHOOKS.md@@ -135,9 +135,17 @@ and HTTP status. A pending or held delivery is not a terminal failure; quiethours and other delivery policies can postpone it. There is no fixed deliverydeadline promised by the attempt count.-Delivery activity masks recognised URL credentials, not arbitrary provider-text or private infrastructure details. Keep full errors and screenshots-private; follow the [delivery evidence precautions](TROUBLESHOOTING.md#recover-retained-delivery-failures)+Alert-webhook response text is deliberately withheld from delivery errors,+logs and Test summaries because receivers can echo credentials or alert content.+Withholding is not a delivery verdict: retain the HTTP status and failure class.+For read failures, bounded byte counts and safe use of existing receiver records,+follow the [withheld-response guidance](TROUBLESHOOTING.md#webhook-response-body-is-withheld).+Do not enable Debug or replay a request to obtain the hidden text.++Delivery activity masks recognised URL credentials, not all private information.+Older records and other error sources can retain provider text or private+infrastructure details; they are not retroactively scrubbed. Keep full errors+and screenshots private; follow the [delivery evidence precautions](TROUBLESHOOTING.md#recover-retained-delivery-failures)before sharing a manually redacted excerpt. A `REDACTED` marker is not proofthat the remaining text is safe to post.diff --git a/frontend-modern/public/docs/ISSUE_TRIAGE.md b/frontend-modern/public/docs/ISSUE_TRIAGE.mdindex bf3f80135d..cae83c2586 100644--- a/frontend-modern/public/docs/ISSUE_TRIAGE.md+++ b/frontend-modern/public/docs/ISSUE_TRIAGE.md@@ -134,6 +134,15 @@ payloads private, even when an error or screenshot contains them. A missingnotification alone does not prove that alert evaluation failed; retain bothsymptoms when the thread supplies evidence for each.+Treat webhook `response body withheld` or `details withheld` messages as+intentional privacy boundaries, not missing evidence to request from a reporter.+Use the supplied HTTP status, failure class and time; a response-read failure+does not prove the receiver rejected the request. A Test's bounded response byte+count is not the sent payload size or proof of receipt. If a provider-specific+distinction matters, use an existing, locally reviewed error code or redacted+explanation, never a raw response, Debug capture or replay. Historical errors+and other exports are not retroactively scrubbed; the full entry remains private.+## Required dispositionBefore removing `needs-decomposition` or declaring a mixed report triaged:diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.mdindex 75d796c6f5..dd2e563462 100644--- a/frontend-modern/public/docs/TROUBLESHOOTING.md+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md@@ -922,10 +922,13 @@ without exhausting the retry budget; waiting alone will not resend it. Seetransient failures and the normal queued attempt limit.**Recent delivery activity** masks recognised credentials in URLs, **not all-private information**. Provider error text can still include email addresses,-private destinations or credentials echoed outside a URL; alert identifiers and-resource names can also identify your infrastructure. Keep full entries, copied-responses and screenshots private. A `REDACTED` marker does not make the rest+private information**. Older records and other error sources can still include+email addresses, private destinations or credentials echoed outside a URL;+alert identifiers and resource names can also identify your infrastructure.+Withheld webhook response text is intentional, not a reason to collect it again;+see [withheld-response guidance](#webhook-response-body-is-withheld).+Keep full entries, copied responses and screenshots private.+A `REDACTED` marker does not make the restsafe to share. For a public report, extract only the relevant timestamp,delivery method, failure class, HTTP status or SMTP error code and a manuallyredacted error. See [notification log precautions](#inspect-notification-logs).@@ -1016,6 +1019,39 @@ or enable debug logging just to collect it. If needed, inspect[bounded notification logs](#inspect-notification-logs) and share only theconsequential, manually redacted error.+#### Webhook response body is withheld++`response body withheld`, `details withheld` and **Response body withheld**+are deliberate privacy messages, not the receiver's explanation of a failure.+Alert-webhook receivers can echo tokens, alert content or private provider data+in their replies. Pulse discards that response text instead of copying it into+delivery errors, logs or a Test result. Do not enable Debug, capture raw traffic,+weaken a safeguard or send another Test or queue retry to recover the text.++Use the original timestamp, destination type, failure class and HTTP status+from **Recent delivery activity**. An HTTP rejection still matters when its+body is withheld; withholding alone does not mean that delivery failed.+For example, HTTP 429 is still rate limiting, not missing diagnostic data.+If the error says **failed to read response body**, reading the reply failed:+the receiver may already have accepted the request, so retrying can duplicate+it. Neither HTTP 2xx nor a completed Test proves receipt at the intended+destination or the success of queued alert delivery.++A Test summary such as `Response body withheld (128 bytes read)` records only+how much Pulse read, not the response's contents or total size. **read incomplete**+means the read failed; **read limit reached** means it reached the **1 MiB** cap,+not proof that the response exceeded that cap. A complete empty reply may have+no summary. These are not byte counts of the alert Pulse sent.++If provider-specific detail is consequential, review the receiver's existing+record locally using its normal authorised access. Share only the relevant+error code or manually redacted explanation, not response bodies, headers,+payloads or credentials. Unknown receipt remains unknown; do not replay the+request to create evidence. Older logs and retained delivery errors are **not+retroactively scrubbed**, and this safeguard does not sanitise every other error+or export. Keep those private and follow the+[log and export precautions](#inspect-notification-logs).+#### Telegram Test works but real alerts say "message text is empty"With the built-in Telegram template (no custom template), a blank or incorrectdiff --git a/frontend-modern/public/docs/WEBHOOKS.md b/frontend-modern/public/docs/WEBHOOKS.mdindex 1a2e26830a..eca89a612a 100644--- a/frontend-modern/public/docs/WEBHOOKS.md+++ b/frontend-modern/public/docs/WEBHOOKS.md@@ -135,9 +135,17 @@ and HTTP status. A pending or held delivery is not a terminal failure; quiethours and other delivery policies can postpone it. There is no fixed deliverydeadline promised by the attempt count.-Delivery activity masks recognised URL credentials, not arbitrary provider-text or private infrastructure details. Keep full errors and screenshots-private; follow the [delivery evidence precautions](TROUBLESHOOTING.md#recover-retained-delivery-failures)+Alert-webhook response text is deliberately withheld from delivery errors,+logs and Test summaries because receivers can echo credentials or alert content.+Withholding is not a delivery verdict: retain the HTTP status and failure class.+For read failures, bounded byte counts and safe use of existing receiver records,+follow the [withheld-response guidance](TROUBLESHOOTING.md#webhook-response-body-is-withheld).+Do not enable Debug or replay a request to obtain the hidden text.++Delivery activity masks recognised URL credentials, not all private information.+Older records and other error sources can retain provider text or private+infrastructure details; they are not retroactively scrubbed. Keep full errors+and screenshots private; follow the [delivery evidence precautions](TROUBLESHOOTING.md#recover-retained-delivery-failures)before sharing a manually redacted excerpt. A `REDACTED` marker is not proofthat the remaining text is safe to post.diff --git a/scripts/tests/test_notification_troubleshooting_docs.py b/scripts/tests/test_notification_troubleshooting_docs.pyindex c4da43d685..edddb95d73 100644--- a/scripts/tests/test_notification_troubleshooting_docs.py+++ b/scripts/tests/test_notification_troubleshooting_docs.py@@ -188,6 +188,53 @@ class NotificationTroubleshootingDocsTest(unittest.TestCase):guide = (ROOT / "docs/WEBHOOKS.md").read_text()self.assertIn("TROUBLESHOOTING.md#telegram-test-works-but-real-alerts-say-message-text-is-empty", guide)+ def test_withheld_webhook_reply_preserves_verdict_and_unknown_receipt(self):+ help_text = " ".join(section("Webhook response body is withheld", 4).split())+ sender = (ROOT / "internal/notifications/webhook_response.go").read_text()+ # Bind the help to the actual receiver-text boundary, not a generic+ # claim that every error or export has been sanitised.+ for marker in ("response body withheld", "details withheld", "Response body withheld",+ "read incomplete", "read limit reached"):+ self.assertIn(marker, sender)+ self.assertIn(marker, help_text)+ for distinction in (+ "deliberate privacy messages", "not the receiver's explanation",+ "withholding alone does not mean that delivery failed", "HTTP 429",+ "receiver may already have accepted the request", "retrying can duplicate",+ "Neither HTTP 2xx nor a completed Test proves receipt", "queued alert delivery",+ "not the response's contents or total size", "1 MiB",+ "not proof that the response exceeded that cap", "complete empty reply",+ "not byte counts of the alert Pulse sent", "Unknown receipt remains unknown",+ ):+ self.assertIn(distinction, help_text)+ self.assertNotRegex(help_text, r"```|https?://|\bcurl\b|--token")++ def test_withheld_reply_help_keeps_historical_evidence_private_without_replay(self):+ help_text = " ".join(section("Webhook response body is withheld", 4).split())+ for boundary in (+ "Do not enable Debug, capture raw traffic", "weaken a safeguard",+ "send another Test or queue retry", "receiver's existing record locally",+ "normal authorised access", "manually redacted explanation",+ "not response bodies, headers, payloads or credentials", "not replay",+ "not retroactively scrubbed", "does not sanitise every other error or export",+ "#inspect-notification-logs",+ ):+ self.assertIn(boundary, help_text)+ link = "TROUBLESHOOTING.md#webhook-response-body-is-withheld"+ guide = (ROOT / "docs/WEBHOOKS.md").read_text()+ self.assertIn(link, guide)+ self.assertIn("#webhook-response-body-is-withheld", section("Recover retained delivery failures", 4))+ triage = " ".join((ROOT / "docs/ISSUE_TRIAGE.md").read_text().split())+ for boundary in (+ "not missing evidence to request from a reporter", "supplied HTTP status",+ "does not prove the receiver rejected the request", "not the sent payload size",+ "never a raw response, Debug capture or replay", "not retroactively scrubbed",+ ):+ self.assertIn(boundary, triage)+ for name in ("WEBHOOKS.md", "ISSUE_TRIAGE.md"):+ self.assertEqual((ROOT / "docs" / name).read_bytes(),+ (ROOT / "frontend-modern/public/docs" / name).read_bytes())+def test_pause_is_cancellation_not_catchup_or_remote_containment(self):limits = " ".join(section("Pause and cancellation limits", 4).split())for boundary in (