Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Preserve explicit webhook edits beside masked credentials
pulse-triage[bot]9ea8a760d4a0
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex c66b718d8b..84b04af654 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -3592,6 +3592,14 @@ Agent` secondary handoff against the live setup wizard instead of relying## Current State+Notification destination edits in the shared `internal/api/` subtree preserve+only submitted masked values, honour neighbouring explicit map edits/removals+and reject unrecoverable masks before publication. Restored custom-field+credentials remain masked in the response. This is not an agent token rotation,+identity change, installer permission or enrollment operation; those boundaries+remain unchanged. The notification API's encrypted edit/reload and delivery+controls cover this shared configuration-write boundary.+### Disk History supplementation does not change agent lifecycleThe disk History handler can replace a shallow API-backed thermal series withdiff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.mdindex 36d4d374b6..10aa3367c4 100644--- a/docs/release-control/v6/internal/subsystems/api-contracts.md+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md@@ -4782,6 +4782,26 @@ auto-register mutation boundary.## Current State+### Masked webhook updates preserve edits without disclosing saved fields++The existing webhook PUT replacement-map contract resolves masks per submitted+key, not per entire map. A masked header may match a saved name under HTTP's+case-insensitive spelling; custom-field names remain exact after existing+service alias normalisation. Explicit edits/additions/blanks/removals beside+unchanged credentials are honoured. Omitted maps retain replacement semantics.+Unknown, already-masked or ambiguous preserved values return HTTP 400 before+persistence or manager mutation. Saved custom fields restored from a mask stay+masked in the update response; explicitly submitted values still round-trip.+No endpoint, permission, payload shape, routing, signing, SSRF or queue retry+policy changes. The existing config-write mutex and save-before-publish rollback+remain authoritative.++`internal/api/alerting/notifications_test.go` verifies exact request/response+values and no-write failure cases, then traverses masked reads/PUT, encrypted+reload, grouped and individual firing/recovery, persistent receipts and HTTP+payloads for built-in Telegram and custom Generic destinations. Existing signing,+Pushover aliases, routing and persistence failure controls remain required.+### Credential-safe container diagnostics response (1 October 2026)`POST /api/diagnostics/docker/prepare-token` retains its fields and durable,diff --git a/docs/release-control/v6/internal/subsystems/notifications.md b/docs/release-control/v6/internal/subsystems/notifications.mdindex b119154243..b314e8a17c 100644--- a/docs/release-control/v6/internal/subsystems/notifications.md+++ b/docs/release-control/v6/internal/subsystems/notifications.md@@ -242,6 +242,31 @@ stable opaque routing identities and must not expose credentials.## Current State+### Masked destination edits preserve each submitted value independently++Webhook editing restores each submitted `***REDACTED***` header/custom-field+value from that destination, rather than replacing the entire submitted map.+Explicit changes, additions, empty values and removals beside another masked+credential survive live publication and encrypted reload. Header names match+case-insensitively; custom fields retain exact, canonical service-specific names.+An unknown, already-masked or ambiguous saved value fails with HTTP 400 before+any save or live mutation. Omitted maps retain their existing replacement+semantics. A response must not disclose a stored custom-field value restored+from a mask; it echoes the canonical submitted value instead.++`TestWebhookEditPreservesOnlyMaskedValues` and+`TestWebhookEditRejectsUnrecoverableMasks` pin editing, deletion, blank values,+case matching, no-mutation rejection and confidentiality.+`TestWebhookMaskedEditPersistsAndDelivers` connects masked form reads/edits,+encrypted save/reopen, ordinary grouped firing/recovery and disabling an active+group to individual deliveries through the persistent sender. Built-in Telegram+and custom Generic bodies retain all members, exact headers/fields and ordinary+per-occurrence recovery receipts. This is loopback source proof, not native+Telegram acceptance or a reproduction of the reporter's actual stored settings.+Already admitted queue jobs keep their configuration snapshots and current+pause/disable/removal checks; editing does not rewrite or replay them. The+built-in template-header correction remains separate and unchanged.+### Apprise diagnostic confidentialityApprise CLI targets may carry credentials in arbitrary provider schemes. CLIdiff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex 580373bbff..e5d1b24b26 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -2954,6 +2954,14 @@ vdev layout is reported` in## Current State+Webhook destination edits restore submitted masks per key while preserving+explicit map edits/removals. They retain encrypted persistence, serialized+save-before-publish and owned rollback on live publication failure; invalid+masks fail before either write. Restored secrets do not reach the response.+The connected notification edit/reopen test verifies durable new settings;+already admitted queue work remains a snapshot, not an implied retry or rewrite.+No storage schema, retention, database internals or recovery mechanism changes.+### Stored disk I/O cannot suppress native thermal HistoryThe existing disk History response can supplement only its absent/shallowdiff --git a/internal/api/alerting/notifications.go b/internal/api/alerting/notifications.goindex 3cde888f05..a6845c85c5 100644--- a/internal/api/alerting/notifications.go+++ b/internal/api/alerting/notifications.go@@ -473,7 +473,39 @@ func (h *NotificationHandlers) CreateWebhook(w http.ResponseWriter, r *http.Requ}}-// UpdateWebhook updates an existing webhook+// restoreMaskedWebhookValues resolves only submitted placeholders. The editor+// sends a replacement map: explicit edits and omitted keys must not be undone+// just because another value is still masked. Header names are case-insensitive;+// custom-field names are not. Never store a placeholder without a saved value.+func restoreMaskedWebhookValues(incoming, existing map[string]string, headers bool) (map[string]string, error) {+ if incoming == nil {+ return nil, nil+ }+ resolved := make(map[string]string, len(incoming))+ for key, value := range incoming {+ if value == "***REDACTED***" {+ stored, found := existing[key]+ if !found && headers {+ for oldKey, oldValue := range existing {+ if strings.EqualFold(oldKey, key) {+ if found && stored != oldValue {+ return nil, fmt.Errorf("ambiguous saved value for masked header %q", key)+ }+ stored, found = oldValue, true+ }+ }+ }+ if !found || stored == "***REDACTED***" {+ return nil, fmt.Errorf("no saved value for masked field %q", key)+ }+ value = stored+ }+ resolved[key] = value+ }+ return resolved, nil+}++// UpdateWebhook updates an existing webhook.func (h *NotificationHandlers) UpdateWebhook(w http.ResponseWriter, r *http.Request) {h.configMu.Lock()defer h.configMu.Unlock()@@ -510,12 +542,14 @@ func (h *NotificationHandlers) UpdateWebhook(w http.ResponseWriter, r *http.Requ}_ = json.Unmarshal(bodyBytes, &routingPresence)webhook = notifications.NormalizeWebhookConfig(webhook)+ // Keep the canonical submitted fields for the response. Resolving a mask+ // must not expose an unchanged stored credential to the caller.+ responseCustomFields := webhook.CustomFieldsmonitor := h.getMonitor(r.Context())manager := monitor.GetNotificationManager()- // Preserve original headers/customFields if the incoming values are redacted- // This happens when the frontend sends back masked values from GetWebhooks+ // Restore each masked value returned by GetWebhooks, not its entire map.existingWebhooks := manager.GetWebhooks()found := falsefor _, existing := range existingWebhooks {@@ -530,31 +564,15 @@ func (h *NotificationHandlers) UpdateWebhook(w http.ResponseWriter, r *http.Requif routingPresence.MinimumSeverity == nil {webhook.MinimumSeverity = existing.MinimumSeverity}- // Preserve headers if incoming contains redacted values- if len(webhook.Headers) > 0 && len(existing.Headers) > 0 {- hasRedacted := false- for _, v := range webhook.Headers {- if v == "***REDACTED***" {- hasRedacted = true- break- }- }- if hasRedacted {- webhook.Headers = existing.Headers- }+ webhook.Headers, err = restoreMaskedWebhookValues(webhook.Headers, existing.Headers, true)+ if err != nil {+ http.Error(w, err.Error(), http.StatusBadRequest)+ return}- // Preserve customFields if incoming contains redacted values- if len(webhook.CustomFields) > 0 && len(existing.CustomFields) > 0 {- hasRedacted := false- for _, v := range webhook.CustomFields {- if v == "***REDACTED***" {- hasRedacted = true- break- }- }- if hasRedacted {- webhook.CustomFields = existing.CustomFields- }+ webhook.CustomFields, err = restoreMaskedWebhookValues(webhook.CustomFields, existing.CustomFields, false)+ if err != nil {+ http.Error(w, err.Error(), http.StatusBadRequest)+ return}// Preserve the signing secret if the incoming value is redactedif webhook.SigningSecret == "***REDACTED***" {@@ -606,7 +624,7 @@ func (h *NotificationHandlers) UpdateWebhook(w http.ResponseWriter, r *http.RequresponseData = make(map[string]interface{})}responseData["id"] = webhookID- responseData["customFields"] = webhook.CustomFields+ responseData["customFields"] = responseCustomFieldsresponseData["tagFilter"] = webhook.TagFilterresponseData["tagFilterMode"] = webhook.TagModeresponseData["minimumSeverity"] = webhook.MinimumSeveritydiff --git a/internal/api/alerting/notifications_test.go b/internal/api/alerting/notifications_test.goindex 4207845704..33809b54ae 100644--- a/internal/api/alerting/notifications_test.go+++ b/internal/api/alerting/notifications_test.go@@ -8,11 +8,16 @@ import ("fmt""net/http""net/http/httptest"+ "os"+ "path/filepath"+ "reflect""strings""sync""testing""time"+ "github.com/rcourtman/pulse-go-rewrite/internal/alerts"+ "github.com/rcourtman/pulse-go-rewrite/internal/config""github.com/rcourtman/pulse-go-rewrite/internal/notifications""github.com/rs/zerolog""github.com/rs/zerolog/log"@@ -1384,3 +1389,307 @@ func TestNotificationHandlersWebhookSigningSecretLifecycle(t *testing.T) {mockManager.AssertExpectations(t)})}++// A masked value means keep that value, not ignore the rest of the edit.+// All values here are synthetic; neither the response nor errors may reveal+// a preserved credential the editor did not submit.+func TestWebhookEditPreservesOnlyMaskedValues(t *testing.T) {+ const masked = "***REDACTED***"+ stored := notifications.WebhookConfig{ID: "edit", URL: "https://example.test/hook", Enabled: true,+ Headers: map[string]string{"Authorization": "synthetic-auth", "Content-Type": "text/plain", "X-Remove": "old"},+ CustomFields: map[string]string{"token": "synthetic-token", "channel": "old", "remove": "old"}}+ cases := []struct {+ name string+ headers map[string]string+ fields map[string]string+ wantHeaders map[string]string+ wantFields map[string]string+ invalid bool+ }{+ {"edit and remove beside masks", map[string]string{"Authorization": masked, "Content-Type": "application/json", "X-New": "new"},+ map[string]string{"token": masked, "channel": "new", "added": "new"},+ map[string]string{"Authorization": "synthetic-auth", "Content-Type": "application/json", "X-New": "new"},+ map[string]string{"token": "synthetic-token", "channel": "new", "added": "new"}, false},+ {"clear beside masks", map[string]string{"Authorization": masked, "Content-Type": ""},+ map[string]string{"token": masked, "channel": ""},+ map[string]string{"Authorization": "synthetic-auth", "Content-Type": ""},+ map[string]string{"token": "synthetic-token", "channel": ""}, false},+ {"header case is insensitive", map[string]string{"authorization": masked}, map[string]string{"token": masked},+ map[string]string{"authorization": "synthetic-auth"}, map[string]string{"token": "synthetic-token"}, false},+ {"explicit empty maps", map[string]string{}, map[string]string{}, map[string]string{}, map[string]string{}, false},+ {"omitted maps keep replacement semantics", nil, nil, nil, nil, false},+ {"unknown masked header", map[string]string{"X-Unknown": masked}, nil, nil, nil, true},+ {"unknown masked field", nil, map[string]string{"unknown": masked}, nil, nil, true},+ {"custom field case is significant", nil, map[string]string{"Token": masked}, nil, nil, true},+ }+ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ manager := new(MockNotificationManager)+ persistence := new(MockNotificationConfigPersistence)+ monitor := new(MockNotificationMonitor)+ monitor.On("GetNotificationManager").Return(manager)+ monitor.On("GetConfigPersistence").Return(persistence)+ manager.On("GetWebhooks").Return([]notifications.WebhookConfig{stored}).Once()+ if !tc.invalid {+ manager.On("ValidateWebhookURL", stored.URL).Return(nil).Once()+ matches := func(got notifications.WebhookConfig) bool {+ return reflect.DeepEqual(got.Headers, tc.wantHeaders) && reflect.DeepEqual(got.CustomFields, tc.wantFields)+ }+ persistence.On("SaveWebhooks", mock.MatchedBy(func(got []notifications.WebhookConfig) bool {+ return len(got) == 1 && matches(got[0])+ })).Return(nil).Once()+ manager.On("UpdateWebhook", stored.ID, mock.MatchedBy(matches)).Return(nil).Once()+ }+ incoming := stored+ incoming.Headers, incoming.CustomFields = tc.headers, tc.fields+ body, err := json.Marshal(incoming)+ assert.NoError(t, err)+ rec := httptest.NewRecorder()+ NewNotificationHandlers(nil, monitor).UpdateWebhook(rec, httptest.NewRequest(http.MethodPut,+ "/api/notifications/webhooks/"+stored.ID, bytes.NewReader(body)))+ wantStatus := http.StatusOK+ if tc.invalid {+ wantStatus = http.StatusBadRequest+ manager.AssertNotCalled(t, "UpdateWebhook", mock.Anything, mock.Anything)+ persistence.AssertNotCalled(t, "SaveWebhooks", mock.Anything)+ }+ assert.Equal(t, wantStatus, rec.Code, rec.Body.String())+ assert.NotContains(t, rec.Body.String(), "synthetic-auth")+ assert.NotContains(t, rec.Body.String(), "synthetic-token")+ if !tc.invalid {+ var response notifications.WebhookConfig+ assert.NoError(t, json.Unmarshal(rec.Body.Bytes(), &response))+ assert.Equal(t, tc.fields, response.CustomFields)+ }+ assert.Equal(t, "old", stored.CustomFields["channel"], "edit mutated the previous map")+ assert.Equal(t, "text/plain", stored.Headers["Content-Type"], "edit mutated the previous map")+ manager.AssertExpectations(t)+ persistence.AssertExpectations(t)+ })+ }+}++func TestWebhookEditRejectsUnrecoverableMasks(t *testing.T) {+ for _, stored := range []map[string]string{+ {"Authorization": "***REDACTED***"},+ {"AUTHORIZATION": "synthetic-first", "authorization": "synthetic-second"},+ } {+ if _, err := restoreMaskedWebhookValues(map[string]string{"Authorization": "***REDACTED***"}, stored, true); err == nil {+ t.Fatal("unrecoverable or ambiguous saved mask must not become a credential")+ }+ }+}++type webhookEditMonitor struct {+ manager *notifications.NotificationManager+ persistence *config.ConfigPersistence+}++func (m *webhookEditMonitor) GetNotificationManager() NotificationManager { return m.manager }+func (m *webhookEditMonitor) GetConfigPersistence() NotificationConfigPersistence {+ return m.persistence+}++// This complements the #2540 single/static-header control: a real masked form+// edit with another saved credential, grouped firing/recovery, disable-grouping+// flush and the persistent autonomous sender. No native Telegram is contacted.+func TestWebhookMaskedEditPersistsAndDelivers(t *testing.T) {+ for _, service := range []string{"telegram", "generic"} {+ for _, mode := range []string{"grouped", "disable-pending-group"} {+ t.Run(service+"/"+mode, func(t *testing.T) {+ type delivery struct {+ header http.Header+ text, channel, chat, event string+ }+ var mu sync.Mutex+ var received []delivery+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ var payload struct {+ Text string `json:"text"`+ Channel string `json:"channel"`+ Chat string `json:"chat_id"`+ Event string `json:"event"`+ }+ if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {+ t.Error(err)+ w.WriteHeader(http.StatusBadRequest)+ return+ }+ if r.Header.Get("Content-Type") != "application/json" || payload.Text == "" {+ w.WriteHeader(http.StatusBadRequest)+ return+ }+ mu.Lock()+ received = append(received, delivery{r.Header.Clone(), payload.Text, payload.Channel, payload.Chat, payload.Event})+ mu.Unlock()+ w.WriteHeader(http.StatusOK)+ }))+ defer server.Close()+ dir := t.TempDir()+ persistence := config.NewConfigPersistence(dir)+ if !persistence.IsEncryptionEnabled() {+ t.Fatal("connected control requires encrypted save/load")+ }+ hook := notifications.WebhookConfig{ID: "edit", Name: "edit", Enabled: true, Method: "POST", Service: service,+ URL: server.URL + "/hook?chat_id=-1001234",+ Headers: map[string]string{"Authorization": "synthetic-auth", "Content-Type": "text/plain", "X-Route": "old", "X-Remove": "old"},+ CustomFields: map[string]string{"token": "synthetic-token", "channel": "old", "remove": "old"}}+ if service == "generic" {+ hook.Template = `{"text":"{{.Message | jsonString}}","channel":"{{.CustomFields.channel | jsonString}}","event":"{{.Event}}"}`+ }+ if err := persistence.SaveWebhooks([]notifications.WebhookConfig{hook}); err != nil {+ t.Fatal(err)+ }+ open := func() *notifications.NotificationManager {+ t.Helper()+ loaded, err := config.NewConfigPersistence(dir).LoadWebhooks()+ if err != nil || len(loaded) != 1 {+ t.Fatalf("saved destination unavailable: %v", err)+ }+ m := notifications.NewNotificationManagerWithDeferredQueue("", dir)+ if err := m.UpdateAllowedPrivateCIDRs("127.0.0.1/32,::1/128"); err != nil {+ t.Fatal(err)+ }+ m.AddWebhook(loaded[0])+ m.SetNotifyOnResolve(true)+ m.SetGroupingConfig(true, 1, true, false)+ t.Cleanup(m.Stop)+ return m+ }+ m := open()+ monitor := &webhookEditMonitor{m, persistence}+ handler := NewNotificationHandlers(nil, monitor)+ listed := httptest.NewRecorder()+ handler.GetWebhooks(listed, httptest.NewRequest(http.MethodGet, "/api/notifications/webhooks", nil))+ var editable []notifications.WebhookConfig+ if err := json.Unmarshal(listed.Body.Bytes(), &editable); err != nil || len(editable) != 1 {+ t.Fatal("cannot read masked destination")+ }+ edit := editable[0]+ if edit.Headers["Authorization"] != "***REDACTED***" || edit.CustomFields["token"] != "***REDACTED***" {+ t.Fatal("list disclosed credentials")+ }+ edit.Headers["Content-Type"], edit.Headers["X-Route"] = "application/json", "new"+ delete(edit.Headers, "X-Remove")+ edit.CustomFields["channel"] = "new"+ delete(edit.CustomFields, "remove")+ body, err := json.Marshal(edit)+ if err != nil {+ t.Fatal(err)+ }+ updated := httptest.NewRecorder()+ handler.UpdateWebhook(updated, httptest.NewRequest(http.MethodPut, "/api/notifications/webhooks/edit", bytes.NewReader(body)))+ if updated.Code != http.StatusOK || strings.Contains(updated.Body.String(), "synthetic-token") {+ t.Fatal("masked update failed or exposed a credential")+ }+ m.Stop()+ m = open()+ monitor.manager = m+ saved := m.GetWebhooks()[0]+ if saved.Service != service || saved.Template != hook.Template || saved.URL != hook.URL ||+ saved.Headers["Authorization"] != "synthetic-auth" || saved.Headers["Content-Type"] != "application/json" ||+ saved.Headers["X-Route"] != "new" || len(saved.Headers) != 3 ||+ saved.CustomFields["token"] != "synthetic-token" || saved.CustomFields["channel"] != "new" || len(saved.CustomFields) != 2 {+ t.Fatal("masked edit did not survive encrypted reload with explicit edits/removals intact")+ }+ encrypted, err := os.ReadFile(filepath.Join(dir, "webhooks.enc"))+ if err != nil || bytes.Contains(encrypted, []byte("synthetic-token")) || bytes.Contains(encrypted, []byte("synthetic-auth")) {+ t.Fatal("saved credentials were not encrypted")+ }+ wait := func(done func() bool) {+ t.Helper()+ deadline := time.Now().Add(15 * time.Second)+ for !done() {+ if time.Now().After(deadline) {+ t.Fatal("connected delivery did not complete")+ }+ time.Sleep(5 * time.Millisecond)+ }+ }+ batch := []*alerts.Alert{+ {ID: "edit-alpha", ResourceName: "edit-alpha", Node: "node", Type: "cpu", Message: "edit-alpha CPU above threshold", Level: alerts.AlertLevelWarning, StartTime: time.Now().Add(-time.Minute)},+ {ID: "edit-bravo", ResourceName: "edit-bravo", Node: "node", Type: "cpu", Message: "edit-bravo CPU above threshold", Level: alerts.AlertLevelWarning, StartTime: time.Now().Add(-time.Minute)},+ }+ for _, a := range batch {+ m.SendAlert(a)+ }+ wantFiring := 1+ if mode == "disable-pending-group" {+ m.SetGroupingConfig(false, 600, true, false)+ wantFiring = 2+ }+ wait(func() bool { stats, err := m.GetQueueStats(); return err == nil && stats["pending"] == wantFiring })+ pending, err := m.GetQueue().GetPending(10)+ if err != nil || len(pending) != wantFiring {+ t.Fatal("ordinary grouping/disable policy produced the wrong number of jobs")+ }+ for _, row := range pending {+ var queued notifications.WebhookConfig+ if err := json.Unmarshal(row.Config, &queued); err != nil || queued.Headers["X-Route"] != "new" || queued.CustomFields["channel"] != "new" {+ t.Fatal("new admission did not snapshot the edited destination")+ }+ if len(row.Alerts) != 3-wantFiring {+ t.Fatal("grouped log did not match actual group membership")+ }+ }+ m.StartQueueProcessing()+ wait(func() bool { stats, err := m.GetQueueStats(); return err == nil && stats["sent"] == wantFiring })+ // Reopen after firing so recovery relies on durable receipts.+ m.Stop()+ m = open()+ if mode == "disable-pending-group" {+ m.SetGroupingConfig(false, 600, true, false)+ }+ for _, a := range batch {+ m.SendResolvedAlert(&alerts.ResolvedAlert{Alert: a, ResolvedTime: time.Now()})+ }+ m.StartQueueProcessing()+ wantTotal := wantFiring * 2+ wait(func() bool { stats, err := m.GetQueueStats(); return err == nil && stats["sent"] == wantTotal })+ wait(func() bool {+ entries, err := m.GetDeliveryLog(time.Now().Add(-time.Hour), 20)+ return err == nil && len(entries) == wantTotal+ })+ entries, err := m.GetDeliveryLog(time.Now().Add(-time.Hour), 20)+ if err != nil {+ t.Fatal(err)+ }+ for _, e := range entries {+ if !e.Success || e.Outcome != notifications.DeliveryOutcomeSent || e.Attempts != 1 || e.AlertCount != 3-wantFiring {+ t.Fatal("delivery audit lost exact attempt or group membership")+ }+ }+ m.Stop()+ mu.Lock()+ got := append([]delivery(nil), received...)+ mu.Unlock()+ if len(got) != wantTotal {+ t.Fatalf("received %d payloads, want %d", len(got), wantTotal)+ }+ for phase := 0; phase < 2; phase++ {+ combined := ""+ for _, d := range got[phase*wantFiring : (phase+1)*wantFiring] {+ if d.header.Get("Authorization") != "synthetic-auth" || d.header.Get("X-Route") != "new" || d.header.Get("X-Remove") != "" {+ t.Fatal("delivery lost an explicit edit or masked credential")+ }+ if service == "telegram" && d.chat != "-1001234" {+ t.Fatal("saved Telegram service/chat did not reach delivery")+ }+ if service == "generic" && d.channel != "new" {+ t.Fatal("custom field edit did not reach the custom payload")+ }+ if phase == 1 && ((service == "telegram" && !strings.Contains(strings.ToLower(d.text), "resolved")) || (service == "generic" && d.event != "resolved")) {+ t.Fatal("recovery rendered as a firing")+ }+ combined += d.text+ }+ for _, a := range batch {+ if !strings.Contains(combined, a.ResourceName) {+ t.Fatal("grouped/individual payload lost an alert")+ }+ }+ }+ })+ }+ }+}