Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Explain safe recovery of retained notification failures

pulse-triage[bot]aae30e2baac2
diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md
index 213b11b17d..a263607230 100644
--- a/docs/TROUBLESHOOTING.md
+++ b/docs/TROUBLESHOOTING.md
@@ -441,10 +441,22 @@ and dead-letter attempts for 30 days. Start with the failure class and timestamp
| Server error / unknown | Destination service status and a relevant, bounded local error excerpt. |
Save the corrected destination settings and send one test; check receipt at the
-intended destination. **Retry retained deliveries** gives terminal failures a
-fresh retry budget, but a destination that accepted an earlier attempt may
-receive a duplicate. Review the confirmation's delivery count and provider
-limits before retrying. A successful test does not itself retry retained items.
+intended destination. **Retained deliveries keep the destination settings saved
+when they were queued.** Editing a URL, recipient, credential, header or template
+does not replace that saved configuration. A test uses the edited settings;
+retrying an old delivery can still use the old endpoint or credential and fail
+again. If the old destination must no longer receive data, disable it rather
+than relying on a URL edit to redirect queued work.
+
+**Retry retained deliveries** gives all retained terminal failures a fresh retry
+budget, not just the destination you tested. Use it only when sending those
+original deliveries is still intended and their original settings remain
+appropriate, for example after a temporary provider outage. A destination that
+accepted an earlier attempt may receive a duplicate. Review the confirmation's
+delivery count and provider limits before retrying. A successful test does not
+itself retry retained items or prove that their saved settings now work. If
+uncertain, leave the failures retained and check the next normally occurring
+alert instead; do not use a batch retry to test a settings edit.
Use **Dismiss retained failures** only when those deliveries should not be sent.
Dismissal clears the warning without retrying them; delivery history remains.
@@ -474,6 +486,28 @@ or enable debug logging just to collect it. If needed, inspect
[bounded notification logs](#inspect-notification-logs) and share only the
consequential, manually redacted error.
+#### Telegram Test works but real alerts say "message text is empty"
+
+With the built-in Telegram template (no custom template), a blank or incorrect
+saved `Content-Type` can make Telegram ignore the JSON body. The HTTP 400 error
+does not by itself prove that Pulse omitted the `text` field. Test can supply
+the correct header even when real delivery uses the saved one.
+
+In **Alerts → Notifications**, edit that Telegram destination's **Custom
+headers**: set one static `Content-Type` header to `application/json` and remove
+any conflicting duplicate, including differently capitalised names. Save,
+leaving the bot URL, `chat_id`, template and grouping settings unchanged. Check
+whether the next normally occurring alert reaches the intended chat and compare
+its timestamp with **Recent delivery activity**; do not induce an alert or retry
+the old rejected batch to test this edit. Retained jobs keep their original
+settings as explained [above](#recover-retained-delivery-failures).
+
+For a custom template, check its JSON and required Telegram fields locally
+instead of assuming this built-in-template workaround applies. If reporting a
+continuing failure, share only the version, redacted delivery error, timestamp
+and whether a custom template is used. Keep the bot URL/token, chat ID and full
+notification configuration private.
+
### TrueNAS
#### "TrueNAS service unavailable"
diff --git a/docs/WEBHOOKS.md b/docs/WEBHOOKS.md
index cb0f8424fd..146334b3ab 100644
--- a/docs/WEBHOOKS.md
+++ b/docs/WEBHOOKS.md
@@ -8,6 +8,13 @@ Pulse includes built-in templates for popular services and a generic JSON templa
2. Click **Add Webhook**.
3. Click the current service label (Generic by default) to open the service picker, choose the destination type, and paste the URL.
+A successful **Test** is not proof of queued alert delivery. For missing alerts
+or rejected requests, follow the [notification troubleshooting guide](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing)
+before retrying retained failures. In particular, a destination edit does not
+replace the settings saved in an old queued delivery; a retry can still use its
+original URL and credentials. For built-in Telegram's "message text is empty"
+error despite a successful Test, see the [static-header check](TROUBLESHOOTING.md#telegram-test-works-but-real-alerts-say-message-text-is-empty).
+
## 📝 Service URLs
| Service | URL Format |
diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.md
index 213b11b17d..a263607230 100644
--- a/frontend-modern/public/docs/TROUBLESHOOTING.md
+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md
@@ -441,10 +441,22 @@ and dead-letter attempts for 30 days. Start with the failure class and timestamp
| Server error / unknown | Destination service status and a relevant, bounded local error excerpt. |
Save the corrected destination settings and send one test; check receipt at the
-intended destination. **Retry retained deliveries** gives terminal failures a
-fresh retry budget, but a destination that accepted an earlier attempt may
-receive a duplicate. Review the confirmation's delivery count and provider
-limits before retrying. A successful test does not itself retry retained items.
+intended destination. **Retained deliveries keep the destination settings saved
+when they were queued.** Editing a URL, recipient, credential, header or template
+does not replace that saved configuration. A test uses the edited settings;
+retrying an old delivery can still use the old endpoint or credential and fail
+again. If the old destination must no longer receive data, disable it rather
+than relying on a URL edit to redirect queued work.
+
+**Retry retained deliveries** gives all retained terminal failures a fresh retry
+budget, not just the destination you tested. Use it only when sending those
+original deliveries is still intended and their original settings remain
+appropriate, for example after a temporary provider outage. A destination that
+accepted an earlier attempt may receive a duplicate. Review the confirmation's
+delivery count and provider limits before retrying. A successful test does not
+itself retry retained items or prove that their saved settings now work. If
+uncertain, leave the failures retained and check the next normally occurring
+alert instead; do not use a batch retry to test a settings edit.
Use **Dismiss retained failures** only when those deliveries should not be sent.
Dismissal clears the warning without retrying them; delivery history remains.
@@ -474,6 +486,28 @@ or enable debug logging just to collect it. If needed, inspect
[bounded notification logs](#inspect-notification-logs) and share only the
consequential, manually redacted error.
+#### Telegram Test works but real alerts say "message text is empty"
+
+With the built-in Telegram template (no custom template), a blank or incorrect
+saved `Content-Type` can make Telegram ignore the JSON body. The HTTP 400 error
+does not by itself prove that Pulse omitted the `text` field. Test can supply
+the correct header even when real delivery uses the saved one.
+
+In **Alerts → Notifications**, edit that Telegram destination's **Custom
+headers**: set one static `Content-Type` header to `application/json` and remove
+any conflicting duplicate, including differently capitalised names. Save,
+leaving the bot URL, `chat_id`, template and grouping settings unchanged. Check
+whether the next normally occurring alert reaches the intended chat and compare
+its timestamp with **Recent delivery activity**; do not induce an alert or retry
+the old rejected batch to test this edit. Retained jobs keep their original
+settings as explained [above](#recover-retained-delivery-failures).
+
+For a custom template, check its JSON and required Telegram fields locally
+instead of assuming this built-in-template workaround applies. If reporting a
+continuing failure, share only the version, redacted delivery error, timestamp
+and whether a custom template is used. Keep the bot URL/token, chat ID and full
+notification configuration private.
+
### TrueNAS
#### "TrueNAS service unavailable"
diff --git a/frontend-modern/public/docs/WEBHOOKS.md b/frontend-modern/public/docs/WEBHOOKS.md
index cb0f8424fd..146334b3ab 100644
--- a/frontend-modern/public/docs/WEBHOOKS.md
+++ b/frontend-modern/public/docs/WEBHOOKS.md
@@ -8,6 +8,13 @@ Pulse includes built-in templates for popular services and a generic JSON templa
2. Click **Add Webhook**.
3. Click the current service label (Generic by default) to open the service picker, choose the destination type, and paste the URL.
+A successful **Test** is not proof of queued alert delivery. For missing alerts
+or rejected requests, follow the [notification troubleshooting guide](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing)
+before retrying retained failures. In particular, a destination edit does not
+replace the settings saved in an old queued delivery; a retry can still use its
+original URL and credentials. For built-in Telegram's "message text is empty"
+error despite a successful Test, see the [static-header check](TROUBLESHOOTING.md#telegram-test-works-but-real-alerts-say-message-text-is-empty).
+
## 📝 Service URLs
| Service | URL Format |
diff --git a/internal/notifications/queue_destination_snapshot_test.go b/internal/notifications/queue_destination_snapshot_test.go
new file mode 100644
index 0000000000..599b4b19f5
--- /dev/null
+++ b/internal/notifications/queue_destination_snapshot_test.go
@@ -0,0 +1,183 @@
+package notifications
+
+import (
+ "bytes"
+ "encoding/json"
+ "io"
+ "net/http"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/alerts"
+)
+
+// The recovery guide must not imply that a successful Test after editing a
+// destination repairs retained work. Exercise enqueue, actual HTTP rejection,
+// edit/Test, persistent restart, operator retry and a new normal alert together.
+// This records existing snapshot semantics, not a new replay policy.
+func TestQueuedWebhookDestinationEditDoesNotRewriteRetainedDelivery(t *testing.T) {
+ for _, policy := range []string{"enabled", "destination-disabled", "removed", "globally-paused"} {
+ t.Run(policy, func(t *testing.T) {
+ type request struct {
+ path, credential, generation string
+ }
+ var mu sync.Mutex
+ var received []request
+ server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ body, err := io.ReadAll(r.Body)
+ if err != nil {
+ t.Error(err)
+ }
+ var payload struct {
+ Generation string `json:"generation"`
+ }
+ if err := json.Unmarshal(body, &payload); err != nil {
+ t.Errorf("invalid request body: %v", err)
+ }
+ mu.Lock()
+ received = append(received, request{r.URL.Path, r.Header.Get("Authorization"), payload.Generation})
+ mu.Unlock()
+ if r.URL.Path == "/original" {
+ w.WriteHeader(http.StatusBadRequest)
+ return
+ }
+ w.WriteHeader(http.StatusOK)
+ }))
+ defer server.Close()
+ original := WebhookConfig{ID: "ops", Name: "ops", Enabled: true, Service: "generic", Method: "POST",
+ URL: server.URL + "/original", Headers: map[string]string{"Authorization": "Bearer synthetic-original"},
+ Template: `{"generation":"original","text":"{{.Message | jsonString}}"}`}
+ edited := copyWebhookConfig(original)
+ edited.URL = server.URL + "/edited"
+ edited.Headers["Authorization"] = "Bearer synthetic-edited"
+ edited.Template = `{"generation":"edited","text":"{{.Message | jsonString}}"}`
+ dir := t.TempDir()
+ open := func(config WebhookConfig) *NotificationManager {
+ m := NewNotificationManagerWithDeferredQueue("", dir)
+ if err := m.UpdateAllowedPrivateCIDRs("127.0.0.1/32"); err != nil {
+ t.Fatal(err)
+ }
+ m.SetGroupingConfig(false, 0, false, false)
+ m.AddWebhook(config)
+ t.Cleanup(m.Stop)
+ return m
+ }
+ pending := func(m *NotificationManager) *QueuedNotification {
+ t.Helper()
+ rows, err := m.queue.GetPending(10)
+ if err != nil || len(rows) != 1 {
+ t.Fatalf("pending = %d, %v; want one newly queued delivery", len(rows), err)
+ }
+ return rows[0]
+ }
+ wait := func(m *NotificationManager, id string, want NotificationQueueStatus, wantAudits int) {
+ t.Helper()
+ deadline := time.Now().Add(10 * time.Second)
+ for {
+ var status string
+ var audits int
+ if err := m.queue.db.QueryRow("SELECT status FROM notification_queue WHERE id = ?", id).Scan(&status); err != nil {
+ t.Fatal(err)
+ }
+ if err := m.queue.db.QueryRow("SELECT count(*) FROM notification_audit WHERE notification_id = ?", id).Scan(&audits); err != nil {
+ t.Fatal(err)
+ }
+ if status == string(want) && audits == wantAudits {
+ return
+ }
+ if time.Now().After(deadline) {
+ t.Fatalf("status=%s audits=%d; want %s/%d", status, audits, want, wantAudits)
+ }
+ time.Sleep(time.Millisecond)
+ }
+ }
+ m := open(original)
+ alert := &alerts.Alert{ID: "before-edit", ResourceName: "test-node", Type: "cpu",
+ Level: alerts.AlertLevelWarning, Message: "synthetic alert", StartTime: time.Now().Add(-time.Minute)}
+ m.SendAlert(alert)
+ old := pending(m)
+ originalConfig := append([]byte(nil), old.Config...)
+ m.StartQueueProcessing()
+ wait(m, old.ID, QueueStatusDLQ, 1)
+ if err := m.UpdateWebhook(original.ID, edited); err != nil {
+ t.Fatal(err)
+ }
+ if err := m.SendTestWebhook(m.GetWebhooks()[0]); err != nil {
+ t.Fatalf("Test with edited destination: %v", err)
+ }
+ wait(m, old.ID, QueueStatusDLQ, 1) // Test neither retries nor removes the failure.
+ m.Stop()
+ m = open(edited)
+ var retainedConfig []byte
+ if err := m.queue.db.QueryRow("SELECT config FROM notification_queue WHERE id = ?", old.ID).Scan(&retainedConfig); err != nil {
+ t.Fatal(err)
+ }
+ if !bytes.Equal(retainedConfig, originalConfig) {
+ t.Fatal("editing/restarting rewrote retained destination settings")
+ }
+ fresh := alert.Clone()
+ fresh.ID = "after-edit"
+ m.SendAlert(fresh)
+ newJob := pending(m)
+ var newConfig WebhookConfig
+ if err := json.Unmarshal(newJob.Config, &newConfig); err != nil {
+ t.Fatal(err)
+ }
+ if newConfig.URL != edited.URL || newConfig.Headers["Authorization"] != edited.Headers["Authorization"] || newConfig.Template != edited.Template {
+ t.Fatal("new alert did not capture the edited destination")
+ }
+ switch policy {
+ case "destination-disabled":
+ edited.Enabled = false
+ if err := m.UpdateWebhook(edited.ID, edited); err != nil {
+ t.Fatal(err)
+ }
+ case "removed":
+ if err := m.DeleteWebhook(edited.ID); err != nil {
+ t.Fatal(err)
+ }
+ case "globally-paused":
+ m.SetEnabled(false)
+ }
+ if count, err := m.queue.RetryTerminalFailures(); err != nil || count != 1 {
+ t.Fatalf("retry = %d, %v; want one retained failure", count, err)
+ }
+ m.StartQueueProcessing()
+ if policy == "enabled" {
+ wait(m, old.ID, QueueStatusDLQ, 2)
+ wait(m, newJob.ID, QueueStatusSent, 1)
+ } else {
+ wait(m, old.ID, QueueStatusCancelled, 1)
+ wait(m, newJob.ID, QueueStatusCancelled, 0)
+ }
+ m.Stop() // Drain before inspecting the complete receiver/audit counts.
+ mu.Lock()
+ defer mu.Unlock()
+ counts := map[request]int{}
+ for _, r := range received {
+ counts[r]++
+ }
+ oldRequests, newRequests := 1, 1 // Initial rejection and successful Test.
+ if policy == "enabled" {
+ oldRequests, newRequests = 2, 2 // Old retry still fails; new normal alert succeeds.
+ }
+ if len(counts) != 2 || counts[request{"/original", "Bearer synthetic-original", "original"}] != oldRequests ||
+ counts[request{"/edited", "Bearer synthetic-edited", "edited"}] != newRequests {
+ t.Fatalf("receiver counts = %v; want original=%d edited=%d", counts, oldRequests, newRequests)
+ }
+ var failed, succeeded int
+ audit := open(edited)
+ if err := audit.queue.db.QueryRow("SELECT count(*) FROM notification_audit WHERE notification_id = ? AND success = 0", old.ID).Scan(&failed); err != nil {
+ t.Fatal(err)
+ }
+ if err := audit.queue.db.QueryRow("SELECT count(*) FROM notification_audit WHERE notification_id = ? AND success = 1", old.ID).Scan(&succeeded); err != nil {
+ t.Fatal(err)
+ }
+ if failed != oldRequests || succeeded != 0 {
+ t.Fatalf("old audit failures=%d successes=%d; want %d/0", failed, succeeded, oldRequests)
+ }
+ t.Logf("%s: Test/new work uses edited URL/header/template; retained retry keeps original settings and failure history; current disable/removal/pause remains effective", policy)
+ })
+ }
+}
diff --git a/scripts/tests/test_notification_troubleshooting_docs.py b/scripts/tests/test_notification_troubleshooting_docs.py
index 1531b86116..d08a6e4d92 100644
--- a/scripts/tests/test_notification_troubleshooting_docs.py
+++ b/scripts/tests/test_notification_troubleshooting_docs.py
@@ -150,6 +150,32 @@ class NotificationTroubleshootingDocsTest(unittest.TestCase):
self.assertIn("#recover-retained-delivery-failures", section(heading, 4))
self.assertIn("#inspect-notification-logs", section(heading, 4))
+ def test_recovery_distinguishes_new_settings_from_retained_work(self):
+ recovery = " ".join(section("Recover retained delivery failures", 4).split())
+ for boundary in ("settings saved when they were queued", "does not replace that saved configuration",
+ "all retained terminal failures", "not just the destination you tested",
+ "old endpoint or credential", "disable it", "original settings remain appropriate",
+ "leave the failures retained", "do not use a batch retry to test a settings edit"):
+ self.assertIn(boundary, recovery)
+ # The connected queue regression exercises these distinctions with real
+ # retained HTTP failures/restart/retry, not a source-text mock.
+ guide = (ROOT / "docs/WEBHOOKS.md").read_text()
+ self.assertIn("TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing", guide)
+ self.assertIn("original URL and credentials", guide)
+
+ def test_telegram_workaround_is_static_passive_and_private(self):
+ heading = 'Telegram Test works but real alerts say "message text is empty"'
+ telegram = " ".join(section(heading, 4).split())
+ for boundary in ("no custom template", "ignore the JSON body", "does not by itself prove",
+ "one static `Content-Type` header to `application/json`", "conflicting duplicate",
+ "leaving the bot URL, `chat_id`, template and grouping settings unchanged",
+ "next normally occurring alert", "do not induce an alert or retry",
+ "#recover-retained-delivery-failures", "custom template", "private"):
+ self.assertIn(boundary, telegram)
+ self.assertNotRegex(telegram, r"https://api\.telegram\.org|curl|--token|Debug")
+ guide = (ROOT / "docs/WEBHOOKS.md").read_text()
+ self.assertIn("TROUBLESHOOTING.md#telegram-test-works-but-real-alerts-say-message-text-is-empty", guide)
+
if __name__ == "__main__":
unittest.main()