Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Explain safe recovery of retained notification failures
pulse-triage[bot]aae30e2baac2
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.mdindex 213b11b17d..a263607230 100644--- a/docs/TROUBLESHOOTING.md+++ b/docs/TROUBLESHOOTING.md@@ -441,10 +441,22 @@ and dead-letter attempts for 30 days. Start with the failure class and timestamp| Server error / unknown | Destination service status and a relevant, bounded local error excerpt. |Save the corrected destination settings and send one test; check receipt at the-intended destination. **Retry retained deliveries** gives terminal failures a-fresh retry budget, but a destination that accepted an earlier attempt may-receive a duplicate. Review the confirmation's delivery count and provider-limits before retrying. A successful test does not itself retry retained items.+intended destination. **Retained deliveries keep the destination settings saved+when they were queued.** Editing a URL, recipient, credential, header or template+does not replace that saved configuration. A test uses the edited settings;+retrying an old delivery can still use the old endpoint or credential and fail+again. If the old destination must no longer receive data, disable it rather+than relying on a URL edit to redirect queued work.++**Retry retained deliveries** gives all retained terminal failures a fresh retry+budget, not just the destination you tested. Use it only when sending those+original deliveries is still intended and their original settings remain+appropriate, for example after a temporary provider outage. A destination that+accepted an earlier attempt may receive a duplicate. Review the confirmation's+delivery count and provider limits before retrying. A successful test does not+itself retry retained items or prove that their saved settings now work. If+uncertain, leave the failures retained and check the next normally occurring+alert instead; do not use a batch retry to test a settings edit.Use **Dismiss retained failures** only when those deliveries should not be sent.Dismissal clears the warning without retrying them; delivery history remains.@@ -474,6 +486,28 @@ or enable debug logging just to collect it. If needed, inspect[bounded notification logs](#inspect-notification-logs) and share only theconsequential, manually redacted error.+#### Telegram Test works but real alerts say "message text is empty"++With the built-in Telegram template (no custom template), a blank or incorrect+saved `Content-Type` can make Telegram ignore the JSON body. The HTTP 400 error+does not by itself prove that Pulse omitted the `text` field. Test can supply+the correct header even when real delivery uses the saved one.++In **Alerts → Notifications**, edit that Telegram destination's **Custom+headers**: set one static `Content-Type` header to `application/json` and remove+any conflicting duplicate, including differently capitalised names. Save,+leaving the bot URL, `chat_id`, template and grouping settings unchanged. Check+whether the next normally occurring alert reaches the intended chat and compare+its timestamp with **Recent delivery activity**; do not induce an alert or retry+the old rejected batch to test this edit. Retained jobs keep their original+settings as explained [above](#recover-retained-delivery-failures).++For a custom template, check its JSON and required Telegram fields locally+instead of assuming this built-in-template workaround applies. If reporting a+continuing failure, share only the version, redacted delivery error, timestamp+and whether a custom template is used. Keep the bot URL/token, chat ID and full+notification configuration private.+### TrueNAS#### "TrueNAS service unavailable"diff --git a/docs/WEBHOOKS.md b/docs/WEBHOOKS.mdindex cb0f8424fd..146334b3ab 100644--- a/docs/WEBHOOKS.md+++ b/docs/WEBHOOKS.md@@ -8,6 +8,13 @@ Pulse includes built-in templates for popular services and a generic JSON templa2. Click **Add Webhook**.3. Click the current service label (Generic by default) to open the service picker, choose the destination type, and paste the URL.+A successful **Test** is not proof of queued alert delivery. For missing alerts+or rejected requests, follow the [notification troubleshooting guide](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing)+before retrying retained failures. In particular, a destination edit does not+replace the settings saved in an old queued delivery; a retry can still use its+original URL and credentials. For built-in Telegram's "message text is empty"+error despite a successful Test, see the [static-header check](TROUBLESHOOTING.md#telegram-test-works-but-real-alerts-say-message-text-is-empty).+## 📝 Service URLs| Service | URL Format |diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.mdindex 213b11b17d..a263607230 100644--- a/frontend-modern/public/docs/TROUBLESHOOTING.md+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md@@ -441,10 +441,22 @@ and dead-letter attempts for 30 days. Start with the failure class and timestamp| Server error / unknown | Destination service status and a relevant, bounded local error excerpt. |Save the corrected destination settings and send one test; check receipt at the-intended destination. **Retry retained deliveries** gives terminal failures a-fresh retry budget, but a destination that accepted an earlier attempt may-receive a duplicate. Review the confirmation's delivery count and provider-limits before retrying. A successful test does not itself retry retained items.+intended destination. **Retained deliveries keep the destination settings saved+when they were queued.** Editing a URL, recipient, credential, header or template+does not replace that saved configuration. A test uses the edited settings;+retrying an old delivery can still use the old endpoint or credential and fail+again. If the old destination must no longer receive data, disable it rather+than relying on a URL edit to redirect queued work.++**Retry retained deliveries** gives all retained terminal failures a fresh retry+budget, not just the destination you tested. Use it only when sending those+original deliveries is still intended and their original settings remain+appropriate, for example after a temporary provider outage. A destination that+accepted an earlier attempt may receive a duplicate. Review the confirmation's+delivery count and provider limits before retrying. A successful test does not+itself retry retained items or prove that their saved settings now work. If+uncertain, leave the failures retained and check the next normally occurring+alert instead; do not use a batch retry to test a settings edit.Use **Dismiss retained failures** only when those deliveries should not be sent.Dismissal clears the warning without retrying them; delivery history remains.@@ -474,6 +486,28 @@ or enable debug logging just to collect it. If needed, inspect[bounded notification logs](#inspect-notification-logs) and share only theconsequential, manually redacted error.+#### Telegram Test works but real alerts say "message text is empty"++With the built-in Telegram template (no custom template), a blank or incorrect+saved `Content-Type` can make Telegram ignore the JSON body. The HTTP 400 error+does not by itself prove that Pulse omitted the `text` field. Test can supply+the correct header even when real delivery uses the saved one.++In **Alerts → Notifications**, edit that Telegram destination's **Custom+headers**: set one static `Content-Type` header to `application/json` and remove+any conflicting duplicate, including differently capitalised names. Save,+leaving the bot URL, `chat_id`, template and grouping settings unchanged. Check+whether the next normally occurring alert reaches the intended chat and compare+its timestamp with **Recent delivery activity**; do not induce an alert or retry+the old rejected batch to test this edit. Retained jobs keep their original+settings as explained [above](#recover-retained-delivery-failures).++For a custom template, check its JSON and required Telegram fields locally+instead of assuming this built-in-template workaround applies. If reporting a+continuing failure, share only the version, redacted delivery error, timestamp+and whether a custom template is used. Keep the bot URL/token, chat ID and full+notification configuration private.+### TrueNAS#### "TrueNAS service unavailable"diff --git a/frontend-modern/public/docs/WEBHOOKS.md b/frontend-modern/public/docs/WEBHOOKS.mdindex cb0f8424fd..146334b3ab 100644--- a/frontend-modern/public/docs/WEBHOOKS.md+++ b/frontend-modern/public/docs/WEBHOOKS.md@@ -8,6 +8,13 @@ Pulse includes built-in templates for popular services and a generic JSON templa2. Click **Add Webhook**.3. Click the current service label (Generic by default) to open the service picker, choose the destination type, and paste the URL.+A successful **Test** is not proof of queued alert delivery. For missing alerts+or rejected requests, follow the [notification troubleshooting guide](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing)+before retrying retained failures. In particular, a destination edit does not+replace the settings saved in an old queued delivery; a retry can still use its+original URL and credentials. For built-in Telegram's "message text is empty"+error despite a successful Test, see the [static-header check](TROUBLESHOOTING.md#telegram-test-works-but-real-alerts-say-message-text-is-empty).+## 📝 Service URLs| Service | URL Format |diff --git a/internal/notifications/queue_destination_snapshot_test.go b/internal/notifications/queue_destination_snapshot_test.gonew file mode 100644index 0000000000..599b4b19f5--- /dev/null+++ b/internal/notifications/queue_destination_snapshot_test.go@@ -0,0 +1,183 @@+package notifications++import (+ "bytes"+ "encoding/json"+ "io"+ "net/http"+ "sync"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/alerts"+)++// The recovery guide must not imply that a successful Test after editing a+// destination repairs retained work. Exercise enqueue, actual HTTP rejection,+// edit/Test, persistent restart, operator retry and a new normal alert together.+// This records existing snapshot semantics, not a new replay policy.+func TestQueuedWebhookDestinationEditDoesNotRewriteRetainedDelivery(t *testing.T) {+ for _, policy := range []string{"enabled", "destination-disabled", "removed", "globally-paused"} {+ t.Run(policy, func(t *testing.T) {+ type request struct {+ path, credential, generation string+ }+ var mu sync.Mutex+ var received []request+ server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ body, err := io.ReadAll(r.Body)+ if err != nil {+ t.Error(err)+ }+ var payload struct {+ Generation string `json:"generation"`+ }+ if err := json.Unmarshal(body, &payload); err != nil {+ t.Errorf("invalid request body: %v", err)+ }+ mu.Lock()+ received = append(received, request{r.URL.Path, r.Header.Get("Authorization"), payload.Generation})+ mu.Unlock()+ if r.URL.Path == "/original" {+ w.WriteHeader(http.StatusBadRequest)+ return+ }+ w.WriteHeader(http.StatusOK)+ }))+ defer server.Close()+ original := WebhookConfig{ID: "ops", Name: "ops", Enabled: true, Service: "generic", Method: "POST",+ URL: server.URL + "/original", Headers: map[string]string{"Authorization": "Bearer synthetic-original"},+ Template: `{"generation":"original","text":"{{.Message | jsonString}}"}`}+ edited := copyWebhookConfig(original)+ edited.URL = server.URL + "/edited"+ edited.Headers["Authorization"] = "Bearer synthetic-edited"+ edited.Template = `{"generation":"edited","text":"{{.Message | jsonString}}"}`+ dir := t.TempDir()+ open := func(config WebhookConfig) *NotificationManager {+ m := NewNotificationManagerWithDeferredQueue("", dir)+ if err := m.UpdateAllowedPrivateCIDRs("127.0.0.1/32"); err != nil {+ t.Fatal(err)+ }+ m.SetGroupingConfig(false, 0, false, false)+ m.AddWebhook(config)+ t.Cleanup(m.Stop)+ return m+ }+ pending := func(m *NotificationManager) *QueuedNotification {+ t.Helper()+ rows, err := m.queue.GetPending(10)+ if err != nil || len(rows) != 1 {+ t.Fatalf("pending = %d, %v; want one newly queued delivery", len(rows), err)+ }+ return rows[0]+ }+ wait := func(m *NotificationManager, id string, want NotificationQueueStatus, wantAudits int) {+ t.Helper()+ deadline := time.Now().Add(10 * time.Second)+ for {+ var status string+ var audits int+ if err := m.queue.db.QueryRow("SELECT status FROM notification_queue WHERE id = ?", id).Scan(&status); err != nil {+ t.Fatal(err)+ }+ if err := m.queue.db.QueryRow("SELECT count(*) FROM notification_audit WHERE notification_id = ?", id).Scan(&audits); err != nil {+ t.Fatal(err)+ }+ if status == string(want) && audits == wantAudits {+ return+ }+ if time.Now().After(deadline) {+ t.Fatalf("status=%s audits=%d; want %s/%d", status, audits, want, wantAudits)+ }+ time.Sleep(time.Millisecond)+ }+ }+ m := open(original)+ alert := &alerts.Alert{ID: "before-edit", ResourceName: "test-node", Type: "cpu",+ Level: alerts.AlertLevelWarning, Message: "synthetic alert", StartTime: time.Now().Add(-time.Minute)}+ m.SendAlert(alert)+ old := pending(m)+ originalConfig := append([]byte(nil), old.Config...)+ m.StartQueueProcessing()+ wait(m, old.ID, QueueStatusDLQ, 1)+ if err := m.UpdateWebhook(original.ID, edited); err != nil {+ t.Fatal(err)+ }+ if err := m.SendTestWebhook(m.GetWebhooks()[0]); err != nil {+ t.Fatalf("Test with edited destination: %v", err)+ }+ wait(m, old.ID, QueueStatusDLQ, 1) // Test neither retries nor removes the failure.+ m.Stop()+ m = open(edited)+ var retainedConfig []byte+ if err := m.queue.db.QueryRow("SELECT config FROM notification_queue WHERE id = ?", old.ID).Scan(&retainedConfig); err != nil {+ t.Fatal(err)+ }+ if !bytes.Equal(retainedConfig, originalConfig) {+ t.Fatal("editing/restarting rewrote retained destination settings")+ }+ fresh := alert.Clone()+ fresh.ID = "after-edit"+ m.SendAlert(fresh)+ newJob := pending(m)+ var newConfig WebhookConfig+ if err := json.Unmarshal(newJob.Config, &newConfig); err != nil {+ t.Fatal(err)+ }+ if newConfig.URL != edited.URL || newConfig.Headers["Authorization"] != edited.Headers["Authorization"] || newConfig.Template != edited.Template {+ t.Fatal("new alert did not capture the edited destination")+ }+ switch policy {+ case "destination-disabled":+ edited.Enabled = false+ if err := m.UpdateWebhook(edited.ID, edited); err != nil {+ t.Fatal(err)+ }+ case "removed":+ if err := m.DeleteWebhook(edited.ID); err != nil {+ t.Fatal(err)+ }+ case "globally-paused":+ m.SetEnabled(false)+ }+ if count, err := m.queue.RetryTerminalFailures(); err != nil || count != 1 {+ t.Fatalf("retry = %d, %v; want one retained failure", count, err)+ }+ m.StartQueueProcessing()+ if policy == "enabled" {+ wait(m, old.ID, QueueStatusDLQ, 2)+ wait(m, newJob.ID, QueueStatusSent, 1)+ } else {+ wait(m, old.ID, QueueStatusCancelled, 1)+ wait(m, newJob.ID, QueueStatusCancelled, 0)+ }+ m.Stop() // Drain before inspecting the complete receiver/audit counts.+ mu.Lock()+ defer mu.Unlock()+ counts := map[request]int{}+ for _, r := range received {+ counts[r]+++ }+ oldRequests, newRequests := 1, 1 // Initial rejection and successful Test.+ if policy == "enabled" {+ oldRequests, newRequests = 2, 2 // Old retry still fails; new normal alert succeeds.+ }+ if len(counts) != 2 || counts[request{"/original", "Bearer synthetic-original", "original"}] != oldRequests ||+ counts[request{"/edited", "Bearer synthetic-edited", "edited"}] != newRequests {+ t.Fatalf("receiver counts = %v; want original=%d edited=%d", counts, oldRequests, newRequests)+ }+ var failed, succeeded int+ audit := open(edited)+ if err := audit.queue.db.QueryRow("SELECT count(*) FROM notification_audit WHERE notification_id = ? AND success = 0", old.ID).Scan(&failed); err != nil {+ t.Fatal(err)+ }+ if err := audit.queue.db.QueryRow("SELECT count(*) FROM notification_audit WHERE notification_id = ? AND success = 1", old.ID).Scan(&succeeded); err != nil {+ t.Fatal(err)+ }+ if failed != oldRequests || succeeded != 0 {+ t.Fatalf("old audit failures=%d successes=%d; want %d/0", failed, succeeded, oldRequests)+ }+ t.Logf("%s: Test/new work uses edited URL/header/template; retained retry keeps original settings and failure history; current disable/removal/pause remains effective", policy)+ })+ }+}diff --git a/scripts/tests/test_notification_troubleshooting_docs.py b/scripts/tests/test_notification_troubleshooting_docs.pyindex 1531b86116..d08a6e4d92 100644--- a/scripts/tests/test_notification_troubleshooting_docs.py+++ b/scripts/tests/test_notification_troubleshooting_docs.py@@ -150,6 +150,32 @@ class NotificationTroubleshootingDocsTest(unittest.TestCase):self.assertIn("#recover-retained-delivery-failures", section(heading, 4))self.assertIn("#inspect-notification-logs", section(heading, 4))+ def test_recovery_distinguishes_new_settings_from_retained_work(self):+ recovery = " ".join(section("Recover retained delivery failures", 4).split())+ for boundary in ("settings saved when they were queued", "does not replace that saved configuration",+ "all retained terminal failures", "not just the destination you tested",+ "old endpoint or credential", "disable it", "original settings remain appropriate",+ "leave the failures retained", "do not use a batch retry to test a settings edit"):+ self.assertIn(boundary, recovery)+ # The connected queue regression exercises these distinctions with real+ # retained HTTP failures/restart/retry, not a source-text mock.+ guide = (ROOT / "docs/WEBHOOKS.md").read_text()+ self.assertIn("TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing", guide)+ self.assertIn("original URL and credentials", guide)++ def test_telegram_workaround_is_static_passive_and_private(self):+ heading = 'Telegram Test works but real alerts say "message text is empty"'+ telegram = " ".join(section(heading, 4).split())+ for boundary in ("no custom template", "ignore the JSON body", "does not by itself prove",+ "one static `Content-Type` header to `application/json`", "conflicting duplicate",+ "leaving the bot URL, `chat_id`, template and grouping settings unchanged",+ "next normally occurring alert", "do not induce an alert or retry",+ "#recover-retained-delivery-failures", "custom template", "private"):+ self.assertIn(boundary, telegram)+ self.assertNotRegex(telegram, r"https://api\.telegram\.org|curl|--token|Debug")+ guide = (ROOT / "docs/WEBHOOKS.md").read_text()+ self.assertIn("TROUBLESHOOTING.md#telegram-test-works-but-real-alerts-say-message-text-is-empty", guide)+if __name__ == "__main__":unittest.main()