Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Preserve explicit webhook edits beside masked credentials

pulse-triage[bot]9ea8a760d4a0
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index c66b718d8b..84b04af654 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -3592,6 +3592,14 @@ Agent` secondary handoff against the live setup wizard instead of relying
## Current State
+Notification destination edits in the shared `internal/api/` subtree preserve
+only submitted masked values, honour neighbouring explicit map edits/removals
+and reject unrecoverable masks before publication. Restored custom-field
+credentials remain masked in the response. This is not an agent token rotation,
+identity change, installer permission or enrollment operation; those boundaries
+remain unchanged. The notification API's encrypted edit/reload and delivery
+controls cover this shared configuration-write boundary.
+
### Disk History supplementation does not change agent lifecycle
The disk History handler can replace a shallow API-backed thermal series with
diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md
index 36d4d374b6..10aa3367c4 100644
--- a/docs/release-control/v6/internal/subsystems/api-contracts.md
+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md
@@ -4782,6 +4782,26 @@ auto-register mutation boundary.
## Current State
+### Masked webhook updates preserve edits without disclosing saved fields
+
+The existing webhook PUT replacement-map contract resolves masks per submitted
+key, not per entire map. A masked header may match a saved name under HTTP's
+case-insensitive spelling; custom-field names remain exact after existing
+service alias normalisation. Explicit edits/additions/blanks/removals beside
+unchanged credentials are honoured. Omitted maps retain replacement semantics.
+Unknown, already-masked or ambiguous preserved values return HTTP 400 before
+persistence or manager mutation. Saved custom fields restored from a mask stay
+masked in the update response; explicitly submitted values still round-trip.
+No endpoint, permission, payload shape, routing, signing, SSRF or queue retry
+policy changes. The existing config-write mutex and save-before-publish rollback
+remain authoritative.
+
+`internal/api/alerting/notifications_test.go` verifies exact request/response
+values and no-write failure cases, then traverses masked reads/PUT, encrypted
+reload, grouped and individual firing/recovery, persistent receipts and HTTP
+payloads for built-in Telegram and custom Generic destinations. Existing signing,
+Pushover aliases, routing and persistence failure controls remain required.
+
### Credential-safe container diagnostics response (1 October 2026)
`POST /api/diagnostics/docker/prepare-token` retains its fields and durable,
diff --git a/docs/release-control/v6/internal/subsystems/notifications.md b/docs/release-control/v6/internal/subsystems/notifications.md
index b119154243..b314e8a17c 100644
--- a/docs/release-control/v6/internal/subsystems/notifications.md
+++ b/docs/release-control/v6/internal/subsystems/notifications.md
@@ -242,6 +242,31 @@ stable opaque routing identities and must not expose credentials.
## Current State
+### Masked destination edits preserve each submitted value independently
+
+Webhook editing restores each submitted `***REDACTED***` header/custom-field
+value from that destination, rather than replacing the entire submitted map.
+Explicit changes, additions, empty values and removals beside another masked
+credential survive live publication and encrypted reload. Header names match
+case-insensitively; custom fields retain exact, canonical service-specific names.
+An unknown, already-masked or ambiguous saved value fails with HTTP 400 before
+any save or live mutation. Omitted maps retain their existing replacement
+semantics. A response must not disclose a stored custom-field value restored
+from a mask; it echoes the canonical submitted value instead.
+
+`TestWebhookEditPreservesOnlyMaskedValues` and
+`TestWebhookEditRejectsUnrecoverableMasks` pin editing, deletion, blank values,
+case matching, no-mutation rejection and confidentiality.
+`TestWebhookMaskedEditPersistsAndDelivers` connects masked form reads/edits,
+encrypted save/reopen, ordinary grouped firing/recovery and disabling an active
+group to individual deliveries through the persistent sender. Built-in Telegram
+and custom Generic bodies retain all members, exact headers/fields and ordinary
+per-occurrence recovery receipts. This is loopback source proof, not native
+Telegram acceptance or a reproduction of the reporter's actual stored settings.
+Already admitted queue jobs keep their configuration snapshots and current
+pause/disable/removal checks; editing does not rewrite or replay them. The
+built-in template-header correction remains separate and unchanged.
+
### Apprise diagnostic confidentiality
Apprise CLI targets may carry credentials in arbitrary provider schemes. CLI
diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md
index 580373bbff..e5d1b24b26 100644
--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md
+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md
@@ -2954,6 +2954,14 @@ vdev layout is reported` in
## Current State
+Webhook destination edits restore submitted masks per key while preserving
+explicit map edits/removals. They retain encrypted persistence, serialized
+save-before-publish and owned rollback on live publication failure; invalid
+masks fail before either write. Restored secrets do not reach the response.
+The connected notification edit/reopen test verifies durable new settings;
+already admitted queue work remains a snapshot, not an implied retry or rewrite.
+No storage schema, retention, database internals or recovery mechanism changes.
+
### Stored disk I/O cannot suppress native thermal History
The existing disk History response can supplement only its absent/shallow
diff --git a/internal/api/alerting/notifications.go b/internal/api/alerting/notifications.go
index 3cde888f05..a6845c85c5 100644
--- a/internal/api/alerting/notifications.go
+++ b/internal/api/alerting/notifications.go
@@ -473,7 +473,39 @@ func (h *NotificationHandlers) CreateWebhook(w http.ResponseWriter, r *http.Requ
}
}
-// UpdateWebhook updates an existing webhook
+// restoreMaskedWebhookValues resolves only submitted placeholders. The editor
+// sends a replacement map: explicit edits and omitted keys must not be undone
+// just because another value is still masked. Header names are case-insensitive;
+// custom-field names are not. Never store a placeholder without a saved value.
+func restoreMaskedWebhookValues(incoming, existing map[string]string, headers bool) (map[string]string, error) {
+ if incoming == nil {
+ return nil, nil
+ }
+ resolved := make(map[string]string, len(incoming))
+ for key, value := range incoming {
+ if value == "***REDACTED***" {
+ stored, found := existing[key]
+ if !found && headers {
+ for oldKey, oldValue := range existing {
+ if strings.EqualFold(oldKey, key) {
+ if found && stored != oldValue {
+ return nil, fmt.Errorf("ambiguous saved value for masked header %q", key)
+ }
+ stored, found = oldValue, true
+ }
+ }
+ }
+ if !found || stored == "***REDACTED***" {
+ return nil, fmt.Errorf("no saved value for masked field %q", key)
+ }
+ value = stored
+ }
+ resolved[key] = value
+ }
+ return resolved, nil
+}
+
+// UpdateWebhook updates an existing webhook.
func (h *NotificationHandlers) UpdateWebhook(w http.ResponseWriter, r *http.Request) {
h.configMu.Lock()
defer h.configMu.Unlock()
@@ -510,12 +542,14 @@ func (h *NotificationHandlers) UpdateWebhook(w http.ResponseWriter, r *http.Requ
}
_ = json.Unmarshal(bodyBytes, &routingPresence)
webhook = notifications.NormalizeWebhookConfig(webhook)
+ // Keep the canonical submitted fields for the response. Resolving a mask
+ // must not expose an unchanged stored credential to the caller.
+ responseCustomFields := webhook.CustomFields
monitor := h.getMonitor(r.Context())
manager := monitor.GetNotificationManager()
- // Preserve original headers/customFields if the incoming values are redacted
- // This happens when the frontend sends back masked values from GetWebhooks
+ // Restore each masked value returned by GetWebhooks, not its entire map.
existingWebhooks := manager.GetWebhooks()
found := false
for _, existing := range existingWebhooks {
@@ -530,31 +564,15 @@ func (h *NotificationHandlers) UpdateWebhook(w http.ResponseWriter, r *http.Requ
if routingPresence.MinimumSeverity == nil {
webhook.MinimumSeverity = existing.MinimumSeverity
}
- // Preserve headers if incoming contains redacted values
- if len(webhook.Headers) > 0 && len(existing.Headers) > 0 {
- hasRedacted := false
- for _, v := range webhook.Headers {
- if v == "***REDACTED***" {
- hasRedacted = true
- break
- }
- }
- if hasRedacted {
- webhook.Headers = existing.Headers
- }
+ webhook.Headers, err = restoreMaskedWebhookValues(webhook.Headers, existing.Headers, true)
+ if err != nil {
+ http.Error(w, err.Error(), http.StatusBadRequest)
+ return
}
- // Preserve customFields if incoming contains redacted values
- if len(webhook.CustomFields) > 0 && len(existing.CustomFields) > 0 {
- hasRedacted := false
- for _, v := range webhook.CustomFields {
- if v == "***REDACTED***" {
- hasRedacted = true
- break
- }
- }
- if hasRedacted {
- webhook.CustomFields = existing.CustomFields
- }
+ webhook.CustomFields, err = restoreMaskedWebhookValues(webhook.CustomFields, existing.CustomFields, false)
+ if err != nil {
+ http.Error(w, err.Error(), http.StatusBadRequest)
+ return
}
// Preserve the signing secret if the incoming value is redacted
if webhook.SigningSecret == "***REDACTED***" {
@@ -606,7 +624,7 @@ func (h *NotificationHandlers) UpdateWebhook(w http.ResponseWriter, r *http.Requ
responseData = make(map[string]interface{})
}
responseData["id"] = webhookID
- responseData["customFields"] = webhook.CustomFields
+ responseData["customFields"] = responseCustomFields
responseData["tagFilter"] = webhook.TagFilter
responseData["tagFilterMode"] = webhook.TagMode
responseData["minimumSeverity"] = webhook.MinimumSeverity
diff --git a/internal/api/alerting/notifications_test.go b/internal/api/alerting/notifications_test.go
index 4207845704..33809b54ae 100644
--- a/internal/api/alerting/notifications_test.go
+++ b/internal/api/alerting/notifications_test.go
@@ -8,11 +8,16 @@ import (
"fmt"
"net/http"
"net/http/httptest"
+ "os"
+ "path/filepath"
+ "reflect"
"strings"
"sync"
"testing"
"time"
+ "github.com/rcourtman/pulse-go-rewrite/internal/alerts"
+ "github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/notifications"
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
@@ -1384,3 +1389,307 @@ func TestNotificationHandlersWebhookSigningSecretLifecycle(t *testing.T) {
mockManager.AssertExpectations(t)
})
}
+
+// A masked value means keep that value, not ignore the rest of the edit.
+// All values here are synthetic; neither the response nor errors may reveal
+// a preserved credential the editor did not submit.
+func TestWebhookEditPreservesOnlyMaskedValues(t *testing.T) {
+ const masked = "***REDACTED***"
+ stored := notifications.WebhookConfig{ID: "edit", URL: "https://example.test/hook", Enabled: true,
+ Headers: map[string]string{"Authorization": "synthetic-auth", "Content-Type": "text/plain", "X-Remove": "old"},
+ CustomFields: map[string]string{"token": "synthetic-token", "channel": "old", "remove": "old"}}
+ cases := []struct {
+ name string
+ headers map[string]string
+ fields map[string]string
+ wantHeaders map[string]string
+ wantFields map[string]string
+ invalid bool
+ }{
+ {"edit and remove beside masks", map[string]string{"Authorization": masked, "Content-Type": "application/json", "X-New": "new"},
+ map[string]string{"token": masked, "channel": "new", "added": "new"},
+ map[string]string{"Authorization": "synthetic-auth", "Content-Type": "application/json", "X-New": "new"},
+ map[string]string{"token": "synthetic-token", "channel": "new", "added": "new"}, false},
+ {"clear beside masks", map[string]string{"Authorization": masked, "Content-Type": ""},
+ map[string]string{"token": masked, "channel": ""},
+ map[string]string{"Authorization": "synthetic-auth", "Content-Type": ""},
+ map[string]string{"token": "synthetic-token", "channel": ""}, false},
+ {"header case is insensitive", map[string]string{"authorization": masked}, map[string]string{"token": masked},
+ map[string]string{"authorization": "synthetic-auth"}, map[string]string{"token": "synthetic-token"}, false},
+ {"explicit empty maps", map[string]string{}, map[string]string{}, map[string]string{}, map[string]string{}, false},
+ {"omitted maps keep replacement semantics", nil, nil, nil, nil, false},
+ {"unknown masked header", map[string]string{"X-Unknown": masked}, nil, nil, nil, true},
+ {"unknown masked field", nil, map[string]string{"unknown": masked}, nil, nil, true},
+ {"custom field case is significant", nil, map[string]string{"Token": masked}, nil, nil, true},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ manager := new(MockNotificationManager)
+ persistence := new(MockNotificationConfigPersistence)
+ monitor := new(MockNotificationMonitor)
+ monitor.On("GetNotificationManager").Return(manager)
+ monitor.On("GetConfigPersistence").Return(persistence)
+ manager.On("GetWebhooks").Return([]notifications.WebhookConfig{stored}).Once()
+ if !tc.invalid {
+ manager.On("ValidateWebhookURL", stored.URL).Return(nil).Once()
+ matches := func(got notifications.WebhookConfig) bool {
+ return reflect.DeepEqual(got.Headers, tc.wantHeaders) && reflect.DeepEqual(got.CustomFields, tc.wantFields)
+ }
+ persistence.On("SaveWebhooks", mock.MatchedBy(func(got []notifications.WebhookConfig) bool {
+ return len(got) == 1 && matches(got[0])
+ })).Return(nil).Once()
+ manager.On("UpdateWebhook", stored.ID, mock.MatchedBy(matches)).Return(nil).Once()
+ }
+ incoming := stored
+ incoming.Headers, incoming.CustomFields = tc.headers, tc.fields
+ body, err := json.Marshal(incoming)
+ assert.NoError(t, err)
+ rec := httptest.NewRecorder()
+ NewNotificationHandlers(nil, monitor).UpdateWebhook(rec, httptest.NewRequest(http.MethodPut,
+ "/api/notifications/webhooks/"+stored.ID, bytes.NewReader(body)))
+ wantStatus := http.StatusOK
+ if tc.invalid {
+ wantStatus = http.StatusBadRequest
+ manager.AssertNotCalled(t, "UpdateWebhook", mock.Anything, mock.Anything)
+ persistence.AssertNotCalled(t, "SaveWebhooks", mock.Anything)
+ }
+ assert.Equal(t, wantStatus, rec.Code, rec.Body.String())
+ assert.NotContains(t, rec.Body.String(), "synthetic-auth")
+ assert.NotContains(t, rec.Body.String(), "synthetic-token")
+ if !tc.invalid {
+ var response notifications.WebhookConfig
+ assert.NoError(t, json.Unmarshal(rec.Body.Bytes(), &response))
+ assert.Equal(t, tc.fields, response.CustomFields)
+ }
+ assert.Equal(t, "old", stored.CustomFields["channel"], "edit mutated the previous map")
+ assert.Equal(t, "text/plain", stored.Headers["Content-Type"], "edit mutated the previous map")
+ manager.AssertExpectations(t)
+ persistence.AssertExpectations(t)
+ })
+ }
+}
+
+func TestWebhookEditRejectsUnrecoverableMasks(t *testing.T) {
+ for _, stored := range []map[string]string{
+ {"Authorization": "***REDACTED***"},
+ {"AUTHORIZATION": "synthetic-first", "authorization": "synthetic-second"},
+ } {
+ if _, err := restoreMaskedWebhookValues(map[string]string{"Authorization": "***REDACTED***"}, stored, true); err == nil {
+ t.Fatal("unrecoverable or ambiguous saved mask must not become a credential")
+ }
+ }
+}
+
+type webhookEditMonitor struct {
+ manager *notifications.NotificationManager
+ persistence *config.ConfigPersistence
+}
+
+func (m *webhookEditMonitor) GetNotificationManager() NotificationManager { return m.manager }
+func (m *webhookEditMonitor) GetConfigPersistence() NotificationConfigPersistence {
+ return m.persistence
+}
+
+// This complements the #2540 single/static-header control: a real masked form
+// edit with another saved credential, grouped firing/recovery, disable-grouping
+// flush and the persistent autonomous sender. No native Telegram is contacted.
+func TestWebhookMaskedEditPersistsAndDelivers(t *testing.T) {
+ for _, service := range []string{"telegram", "generic"} {
+ for _, mode := range []string{"grouped", "disable-pending-group"} {
+ t.Run(service+"/"+mode, func(t *testing.T) {
+ type delivery struct {
+ header http.Header
+ text, channel, chat, event string
+ }
+ var mu sync.Mutex
+ var received []delivery
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ var payload struct {
+ Text string `json:"text"`
+ Channel string `json:"channel"`
+ Chat string `json:"chat_id"`
+ Event string `json:"event"`
+ }
+ if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
+ t.Error(err)
+ w.WriteHeader(http.StatusBadRequest)
+ return
+ }
+ if r.Header.Get("Content-Type") != "application/json" || payload.Text == "" {
+ w.WriteHeader(http.StatusBadRequest)
+ return
+ }
+ mu.Lock()
+ received = append(received, delivery{r.Header.Clone(), payload.Text, payload.Channel, payload.Chat, payload.Event})
+ mu.Unlock()
+ w.WriteHeader(http.StatusOK)
+ }))
+ defer server.Close()
+ dir := t.TempDir()
+ persistence := config.NewConfigPersistence(dir)
+ if !persistence.IsEncryptionEnabled() {
+ t.Fatal("connected control requires encrypted save/load")
+ }
+ hook := notifications.WebhookConfig{ID: "edit", Name: "edit", Enabled: true, Method: "POST", Service: service,
+ URL: server.URL + "/hook?chat_id=-1001234",
+ Headers: map[string]string{"Authorization": "synthetic-auth", "Content-Type": "text/plain", "X-Route": "old", "X-Remove": "old"},
+ CustomFields: map[string]string{"token": "synthetic-token", "channel": "old", "remove": "old"}}
+ if service == "generic" {
+ hook.Template = `{"text":"{{.Message | jsonString}}","channel":"{{.CustomFields.channel | jsonString}}","event":"{{.Event}}"}`
+ }
+ if err := persistence.SaveWebhooks([]notifications.WebhookConfig{hook}); err != nil {
+ t.Fatal(err)
+ }
+ open := func() *notifications.NotificationManager {
+ t.Helper()
+ loaded, err := config.NewConfigPersistence(dir).LoadWebhooks()
+ if err != nil || len(loaded) != 1 {
+ t.Fatalf("saved destination unavailable: %v", err)
+ }
+ m := notifications.NewNotificationManagerWithDeferredQueue("", dir)
+ if err := m.UpdateAllowedPrivateCIDRs("127.0.0.1/32,::1/128"); err != nil {
+ t.Fatal(err)
+ }
+ m.AddWebhook(loaded[0])
+ m.SetNotifyOnResolve(true)
+ m.SetGroupingConfig(true, 1, true, false)
+ t.Cleanup(m.Stop)
+ return m
+ }
+ m := open()
+ monitor := &webhookEditMonitor{m, persistence}
+ handler := NewNotificationHandlers(nil, monitor)
+ listed := httptest.NewRecorder()
+ handler.GetWebhooks(listed, httptest.NewRequest(http.MethodGet, "/api/notifications/webhooks", nil))
+ var editable []notifications.WebhookConfig
+ if err := json.Unmarshal(listed.Body.Bytes(), &editable); err != nil || len(editable) != 1 {
+ t.Fatal("cannot read masked destination")
+ }
+ edit := editable[0]
+ if edit.Headers["Authorization"] != "***REDACTED***" || edit.CustomFields["token"] != "***REDACTED***" {
+ t.Fatal("list disclosed credentials")
+ }
+ edit.Headers["Content-Type"], edit.Headers["X-Route"] = "application/json", "new"
+ delete(edit.Headers, "X-Remove")
+ edit.CustomFields["channel"] = "new"
+ delete(edit.CustomFields, "remove")
+ body, err := json.Marshal(edit)
+ if err != nil {
+ t.Fatal(err)
+ }
+ updated := httptest.NewRecorder()
+ handler.UpdateWebhook(updated, httptest.NewRequest(http.MethodPut, "/api/notifications/webhooks/edit", bytes.NewReader(body)))
+ if updated.Code != http.StatusOK || strings.Contains(updated.Body.String(), "synthetic-token") {
+ t.Fatal("masked update failed or exposed a credential")
+ }
+ m.Stop()
+ m = open()
+ monitor.manager = m
+ saved := m.GetWebhooks()[0]
+ if saved.Service != service || saved.Template != hook.Template || saved.URL != hook.URL ||
+ saved.Headers["Authorization"] != "synthetic-auth" || saved.Headers["Content-Type"] != "application/json" ||
+ saved.Headers["X-Route"] != "new" || len(saved.Headers) != 3 ||
+ saved.CustomFields["token"] != "synthetic-token" || saved.CustomFields["channel"] != "new" || len(saved.CustomFields) != 2 {
+ t.Fatal("masked edit did not survive encrypted reload with explicit edits/removals intact")
+ }
+ encrypted, err := os.ReadFile(filepath.Join(dir, "webhooks.enc"))
+ if err != nil || bytes.Contains(encrypted, []byte("synthetic-token")) || bytes.Contains(encrypted, []byte("synthetic-auth")) {
+ t.Fatal("saved credentials were not encrypted")
+ }
+ wait := func(done func() bool) {
+ t.Helper()
+ deadline := time.Now().Add(15 * time.Second)
+ for !done() {
+ if time.Now().After(deadline) {
+ t.Fatal("connected delivery did not complete")
+ }
+ time.Sleep(5 * time.Millisecond)
+ }
+ }
+ batch := []*alerts.Alert{
+ {ID: "edit-alpha", ResourceName: "edit-alpha", Node: "node", Type: "cpu", Message: "edit-alpha CPU above threshold", Level: alerts.AlertLevelWarning, StartTime: time.Now().Add(-time.Minute)},
+ {ID: "edit-bravo", ResourceName: "edit-bravo", Node: "node", Type: "cpu", Message: "edit-bravo CPU above threshold", Level: alerts.AlertLevelWarning, StartTime: time.Now().Add(-time.Minute)},
+ }
+ for _, a := range batch {
+ m.SendAlert(a)
+ }
+ wantFiring := 1
+ if mode == "disable-pending-group" {
+ m.SetGroupingConfig(false, 600, true, false)
+ wantFiring = 2
+ }
+ wait(func() bool { stats, err := m.GetQueueStats(); return err == nil && stats["pending"] == wantFiring })
+ pending, err := m.GetQueue().GetPending(10)
+ if err != nil || len(pending) != wantFiring {
+ t.Fatal("ordinary grouping/disable policy produced the wrong number of jobs")
+ }
+ for _, row := range pending {
+ var queued notifications.WebhookConfig
+ if err := json.Unmarshal(row.Config, &queued); err != nil || queued.Headers["X-Route"] != "new" || queued.CustomFields["channel"] != "new" {
+ t.Fatal("new admission did not snapshot the edited destination")
+ }
+ if len(row.Alerts) != 3-wantFiring {
+ t.Fatal("grouped log did not match actual group membership")
+ }
+ }
+ m.StartQueueProcessing()
+ wait(func() bool { stats, err := m.GetQueueStats(); return err == nil && stats["sent"] == wantFiring })
+ // Reopen after firing so recovery relies on durable receipts.
+ m.Stop()
+ m = open()
+ if mode == "disable-pending-group" {
+ m.SetGroupingConfig(false, 600, true, false)
+ }
+ for _, a := range batch {
+ m.SendResolvedAlert(&alerts.ResolvedAlert{Alert: a, ResolvedTime: time.Now()})
+ }
+ m.StartQueueProcessing()
+ wantTotal := wantFiring * 2
+ wait(func() bool { stats, err := m.GetQueueStats(); return err == nil && stats["sent"] == wantTotal })
+ wait(func() bool {
+ entries, err := m.GetDeliveryLog(time.Now().Add(-time.Hour), 20)
+ return err == nil && len(entries) == wantTotal
+ })
+ entries, err := m.GetDeliveryLog(time.Now().Add(-time.Hour), 20)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, e := range entries {
+ if !e.Success || e.Outcome != notifications.DeliveryOutcomeSent || e.Attempts != 1 || e.AlertCount != 3-wantFiring {
+ t.Fatal("delivery audit lost exact attempt or group membership")
+ }
+ }
+ m.Stop()
+ mu.Lock()
+ got := append([]delivery(nil), received...)
+ mu.Unlock()
+ if len(got) != wantTotal {
+ t.Fatalf("received %d payloads, want %d", len(got), wantTotal)
+ }
+ for phase := 0; phase < 2; phase++ {
+ combined := ""
+ for _, d := range got[phase*wantFiring : (phase+1)*wantFiring] {
+ if d.header.Get("Authorization") != "synthetic-auth" || d.header.Get("X-Route") != "new" || d.header.Get("X-Remove") != "" {
+ t.Fatal("delivery lost an explicit edit or masked credential")
+ }
+ if service == "telegram" && d.chat != "-1001234" {
+ t.Fatal("saved Telegram service/chat did not reach delivery")
+ }
+ if service == "generic" && d.channel != "new" {
+ t.Fatal("custom field edit did not reach the custom payload")
+ }
+ if phase == 1 && ((service == "telegram" && !strings.Contains(strings.ToLower(d.text), "resolved")) || (service == "generic" && d.event != "resolved")) {
+ t.Fatal("recovery rendered as a firing")
+ }
+ combined += d.text
+ }
+ for _, a := range batch {
+ if !strings.Contains(combined, a.ResourceName) {
+ t.Fatal("grouped/individual payload lost an alert")
+ }
+ }
+ }
+ })
+ }
+ }
+}