Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Checkpoint buffered firing groups across graceful restarts
pulse-triage[bot]f6c32ed5ad38
diff --git a/docs/release-control/v6/internal/subsystems/notifications.md b/docs/release-control/v6/internal/subsystems/notifications.mdindex dbf5478dc1..ceb70d9afe 100644--- a/docs/release-control/v6/internal/subsystems/notifications.md+++ b/docs/release-control/v6/internal/subsystems/notifications.md@@ -299,9 +299,19 @@ do not extend the original timer or establish delivery.instant equality across time-zone/monotonic representations, cloned payloads,concurrent callbacks, occurrence-qualified cancellation, all-destination andwebhook-only disable/zero-window flushes, and notification disable/re-enable.-Stopping the manager cancels its volatile firing timer and snapshots, rather-than letting an abandoned timer send directly after its queue is closed.-Committed queue work retains its existing restart semantics.+Stopping the manager cancels its firing timer and checkpoints its accepted+snapshots as ordinary delayed queue work after workers quiesce. It preserves+batch membership, current destination snapshots and the first admission's+original deadline (rounded up to queue precision), or a later quiet-hours+replay deadline. It does not flush, restart the window, invent a delivery+receipt, or fall back to an early direct send if checkpoint storage fails.+A restarted manager treats this committed batch as immutable; new admissions+start their own window rather than merging into the old process's batch.+Queue replay still applies current quiet hours and destination/enable policy.+`firing_group_checkpoint_test.go` verifies exact occurrence/link snapshots,+original deadlines after config edits, cancellation, pause/channel selection,+tenant-directory isolation, concurrent timer/stop ownership and storage-error+silence. Queue schema, drivers and event types are unchanged.`TestPendingFiringGroupStopCancelsTimer` checks actual timer/HTTP silence after stop.The connected `TestPendingFiringGroupHTTPAcrossRestart` uses actual timer expiry,persistent queue reopen, the autonomous sender and two loopback receivers: one@@ -350,9 +360,11 @@ lifecycle regression proof, not installed SMTP receipt or reporter confirmation.### Manager shutdown is a restart boundary, not a delivery verdict-`NotificationManager.Stop` first refuses new alerts and discards volatile-firing and recovery groups in one critical section, as Stop has always-discarded them. Work that reads the queue outside the manager lock holds a+`NotificationManager.Stop` first refuses new alerts and stops the firing timer+in one critical section, retaining its buffered snapshots for checkpointing.+Queue-less recovery buffers keep their existing volatile semantics; persistent+recovery grouping is already durable. Work that reads the queue outside the+manager lock holds alease taken in the critical section that reads it. An admitted alert, recoveryor timer group is written to the open queue. Cancellation, operator pause andchannel disable still run while Stop drains. Direct delivery started by@@ -365,8 +377,13 @@ quiet-hours hold, partition or claim: its job stays `pending` with no attempt,audit or receipt, and the next start delivers it. Work a worker began beforethat observation finishes while delivery is still permitted, recording itsstatus, audit and durable firing receipt. Only after every worker and lease-has ended does Stop withdraw delivery, detach the queue and close storage. A-recovery after restart therefore still reaches each destination that received+has ended does Stop withdraw delivery, detach the queue and close storage.+Between quiescing workers and closing admission leases, Stop checkpoints the+remaining firing group under the manager lock: resolution/pause either removes+its buffered snapshot first or cancels its committed row afterwards, never+falling between those states. Grouping edits during draining leave this+checkpoint intact. Storage failures are logged, not called durable or routed+to direct delivery. A recovery after restart still reaches each destination that receivedthe firing. Shutdown must never cancel queued work as "delivery disabled";that verdict belongs to the pause and destination controls. Resolutioncancellation and at-least-once replay of interrupted sends are unchanged.@@ -1654,3 +1671,20 @@ SMTP, rate-limit and routing preservation through edits and reloads with emailon or off. Backend configuration, transport, retry policy, TLS, queue recoveryand notification delivery are unchanged. Browser controls are synthetic, notnative SMTP acceptance or a containing release.++### Buffered warnings survive graceful process boundaries++`TestGroupedWarningGracefulProcessRestartAndReload` drives ordinary CheckHost+warning callbacks through real alert/notification managers, persists active+warning dispatch markers and performs actual SIGTERM/normal-exit process+restarts plus same-process manager reload. Zero-repeat and positive-cooldown+cases retain accepted pending batches; two loopback destinations receive the+two-warning group no earlier than its original window. Another real process+restart precedes occurrence/destination receipt-gated recoveries. No queue+clock edit or fabricated successful firing receipt is used.+`TestMonitorReloadKeepsBufferedWarning` separately covers the production+monitor constructor/stop/reconstruction with saved notification policy and+destinations: pending work remains delayed with no attempt or audit. These are+source/native process proofs, not installed-service, external provider, stable+release or reporter recovery. A crash before graceful checkpointing can still+lose volatile firing buffers; this repair makes no crash-durability promise.diff --git a/internal/monitoring/monitor_notification_startup_test.go b/internal/monitoring/monitor_notification_startup_test.goindex 4a5bdf4aab..780bcc21c9 100644--- a/internal/monitoring/monitor_notification_startup_test.go+++ b/internal/monitoring/monitor_notification_startup_test.go@@ -374,3 +374,45 @@ func TestNewRejectsUnreadableNotificationPolicyBeforeQueueCreation(t *testing.T)})}}++// The real monitor stop/reconstruction path must checkpoint a still-buffered+// warning before replacing the notifier, and bootstrap must leave its deadline+// alone. No provider or artificial queue timestamp is involved.+func TestMonitorReloadKeepsBufferedWarning(t *testing.T) {+ dir := t.TempDir()+ t.Setenv("PULSE_DATA_DIR", dir)+ persistence := config.NewConfigPersistence(dir)+ saved := alerts.AlertConfig{Enabled: true, ActivationState: alerts.ActivationActive,+ Schedule: alerts.ScheduleConfig{InitialNotify: "webhook", NotifyOnResolve: true,+ Grouping: alerts.GroupingConfig{Enabled: true, Window: 600, ByNode: true, ByGuest: true}}}+ if err := persistence.SaveAlertConfig(saved); err != nil {+ t.Fatal(err)+ }+ if err := persistence.SaveWebhooks([]notifications.WebhookConfig{{ID: "reload-ops", Name: "Reload ops", URL: "https://example.invalid/not-contacted", Service: "generic", Enabled: true}}); err != nil {+ t.Fatal(err)+ }+ m, err := New(&config.Config{DataPath: dir})+ if err != nil {+ t.Fatal(err)+ }+ m.GetNotificationManager().SendAlert(&alerts.Alert{ID: "reload-warning", ResourceID: "reload-host", ResourceName: "Reload host", Type: "cpu", Level: alerts.AlertLevelWarning, StartTime: time.Now().UTC()})+ m.Stop()+ m, err = New(&config.Config{DataPath: dir})+ if err != nil {+ t.Fatal(err)+ }+ defer m.Stop()+ q := m.GetNotificationManager().GetQueue()+ stats, err := q.GetQueueStats()+ if err != nil || stats["pending"] != 1 || stats["sent"] != 0 {+ t.Fatalf("reload lost or flushed accepted warning: %v, %v", stats, err)+ }+ ready, err := q.GetPending(10)+ if err != nil || len(ready) != 0 {+ t.Fatalf("reload shortened original grouping deadline: %+v, %v", ready, err)+ }+ entries, err := q.GetDeliveryLog(time.Now().Add(-time.Hour), 10)+ if err != nil || len(entries) != 0 {+ t.Fatalf("reload manufactured provider attempts: %+v, %v", entries, err)+ }+}diff --git a/internal/notifications/firing_group_checkpoint.go b/internal/notifications/firing_group_checkpoint.gonew file mode 100644index 0000000000..7972517fec--- /dev/null+++ b/internal/notifications/firing_group_checkpoint.go@@ -0,0 +1,63 @@+package notifications++import (+ "time"++ "github.com/rs/zerolog/log"+)++// checkpointPendingFiringGroupLocked transfers the accepted window into ordinary+// queued delivery while n.mu is held and queue workers are stopped. A restart is+// not a flush: retain the original membership, destination snapshots and deadline+// (or the later quiet-hours deadline). Later process admissions start a new+// window, just as work admitted after an ordinary flush does.+//+// Queue/policy cancellation remains available during the handoff. Never fall+// back to an immediate provider send on a checkpoint error: it would violate the+// window and quiet hours. Storage errors are reported rather than called durable.+func (n *NotificationManager) checkpointPendingFiringGroupLocked(queue *NotificationQueue) {+ defer func() {+ clear(n.pendingAlerts)+ n.pendingAlerts = n.pendingAlerts[:0]+ n.pendingAlertIndex = nil+ n.groupDeadline = time.Time{}+ }()+ if len(n.pendingAlerts) == 0 || !n.enabled {+ return+ }+ if queue == nil {+ log.Error().Int("alertCount", len(n.pendingAlerts)).Msg("cannot preserve buffered notifications at shutdown: queue unavailable")+ return+ }+ deadline := n.groupDeadline+ if deadline.IsZero() {+ // Legacy manually constructed managers may lack a timer deadline.+ deadline = time.Now().Add(n.groupWindow)+ }+ jobs := buildNotificationDeliveryJobsForTarget(copyEmailConfig(n.emailConfig), copyWebhookConfigs(n.webhooks),+ copyAppriseConfig(n.appriseConfig), n.pendingAlerts, eventAlert, time.Time{}, n.initialTarget)+ for _, job := range jobs {+ configJSON, err := configJSONForNotificationDeliveryJob(job)+ if err != nil {+ n.logNotificationJobError(job, err, "failed to preserve buffered notifications at shutdown")+ continue+ }+ for _, bucket := range notificationQueueBucketsForJob(job, time.Now()) {+ due := deadline+ if bucket.nextRetryAt != nil && bucket.nextRetryAt.After(due) {+ due = *bucket.nextRetryAt+ }+ // Queue timestamps have second precision; truncation must never+ // allow a send before either policy deadline.+ if rounded := due.Truncate(time.Second); rounded.Before(due) {+ due = rounded.Add(time.Second)+ }+ row := &QueuedNotification{Type: queueTypeForNotificationDeliveryJob(bucket.job),+ DestinationID: notificationDestinationIDForJob(bucket.job), Alerts: bucket.job.Alerts,+ Config: configJSON, MaxAttempts: 3, NextRetryAt: &due}+ if err := queue.Enqueue(row); err != nil {+ n.logNotificationJobError(bucket.job, err, "failed to preserve buffered notifications at shutdown; no early direct send")+ }+ }+ }+}diff --git a/internal/notifications/firing_group_checkpoint_test.go b/internal/notifications/firing_group_checkpoint_test.gonew file mode 100644index 0000000000..9ba2a14b9c--- /dev/null+++ b/internal/notifications/firing_group_checkpoint_test.go@@ -0,0 +1,215 @@+package notifications++import (+ "encoding/json"+ "sync"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/alerts"+ "github.com/stretchr/testify/require"+)++// The checkpoint becomes normal queue work, not a successful send. The grouping+// deadline belongs to first admission, regardless of later config edits/reopens.+func TestStopCheckpointsFiringWindowPolicy(t *testing.T) {+ for _, policy := range []string{"ordinary", "quiet-later", "cancel-old", "pause", "email-disabled", "webhook-only", "tenant-isolation"} {+ t.Run(policy, func(t *testing.T) {+ f := newShutdownAdmissionFixture(t)+ email := EmailConfig{Enabled: true, SMTPHost: "smtp.example.invalid", SMTPPort: 587, From: "[email protected]", To: []string{"[email protected]"}}+ apprise := AppriseConfig{Enabled: true, Targets: []string{"ntfy://example.invalid/pulse"}}+ m := f.open(func(m *NotificationManager) {+ m.SetEmailConfig(email)+ m.SetAppriseConfig(apprise)+ m.SetGroupingConfig(true, 600, true, true)+ })+ start := time.Now().UTC().Add(-time.Minute)+ old := pendingGroupSnapshot("reused", start, start, alerts.AlertLevelWarning, "old")+ current := pendingGroupSnapshot("reused", start.Add(time.Nanosecond), start, alerts.AlertLevelWarning, "current")+ current.OperationalRecord.ID += "-current"+ peer := pendingGroupSnapshot("peer", start, start, alerts.AlertLevelWarning, "peer")+ if policy == "quiet-later" {+ peer.Metadata[alerts.MetadataQuietHoursReplayAt] = time.Now().Add(2 * time.Hour).UTC().Format(time.RFC3339Nano)+ }+ if policy == "webhook-only" {+ m.SetInitialNotifyTarget("webhook")+ }+ for _, a := range []*alerts.Alert{old, current, peer} {+ m.SendAlert(a)+ }+ m.mu.RLock()+ deadline := m.groupDeadline+ m.mu.RUnlock()+ current.LastSeen = start.Add(time.Second)+ current.Message = "latest"+ m.SendAlert(current)+ // Editing a live window never resets an existing timer.+ m.SetGroupingWindow(3600)+ switch policy {+ case "cancel-old":+ require.True(t, m.CancelResolvedAlert(old))+ case "pause":+ m.SetEnabled(false)+ case "email-disabled":+ email.Enabled = false+ m.SetEmailConfig(email)+ }+ m.Stop()+ m = f.open(func(m *NotificationManager) { m.SetEmailConfig(email); m.SetAppriseConfig(apprise) })+ rows := quietReplayRows(t, m.GetQueue())+ if policy == "pause" {+ require.Empty(t, rows)+ return+ }+ want := 3+ switch policy {+ case "quiet-later":+ want = 6+ case "email-disabled":+ want = 2+ case "webhook-only":+ want = 1+ }+ require.Len(t, rows, want)+ for _, row := range rows {+ require.Equal(t, QueueStatusPending, row.Status)+ require.Zero(t, row.Attempts)+ require.Nil(t, row.LastAttempt)+ require.NotNil(t, row.NextRetryAt)+ require.False(t, row.NextRetryAt.Before(deadline), "shutdown rounded the original window down")+ if policy != "quiet-later" || row.Alerts[0].ID != peer.ID {+ require.Less(t, row.NextRetryAt.Sub(deadline), time.Second, "shutdown reset the window to edited configuration")+ }+ wantAlerts := 3+ switch policy {+ case "cancel-old":+ wantAlerts = 2+ case "quiet-later":+ wantAlerts = 2+ if row.Alerts[0].ID == peer.ID {+ wantAlerts = 1+ require.True(t, row.NextRetryAt.After(time.Now().Add(time.Hour)))+ }+ }+ require.Len(t, row.Alerts, wantAlerts)+ require.Len(t, row.Links, wantAlerts)+ for i, a := range row.Alerts {+ require.Equal(t, a.LatestTransition.ID, row.Links[i].TransitionID)+ if a.StartTime.Equal(current.StartTime) && a.ID == current.ID {+ require.Equal(t, "latest", a.Message)+ }+ if policy == "cancel-old" {+ require.False(t, a.ID == old.ID && a.StartTime.Equal(old.StartTime))+ }+ }+ if row.Type == "webhook" {+ var saved WebhookConfig+ require.NoError(t, json.Unmarshal(row.Config, &saved))+ require.Equal(t, f.hook, saved)+ }+ }+ due, err := m.GetQueue().GetPending(100)+ require.NoError(t, err)+ require.Empty(t, due, "delayed pending work must not be ready before its original deadline")+ if policy == "tenant-isolation" {+ other := NewNotificationManagerWithDeferredQueue("", t.TempDir())+ defer other.Stop()+ require.Empty(t, quietReplayRows(t, other.GetQueue()), "a second tenant directory must not inherit this buffer")+ }+ select {+ case <-f.spawned:+ t.Fatal("checkpoint dispatched direct delivery")+ default:+ }+ })+ }+}++// Concurrent resolutions and pause retain the existing lifecycle lease and the+// queue's cancellation semantics; no gap between memory and durable queue rows.+func TestStopCheckpointConcurrentCancellation(t *testing.T) {+ for _, policy := range []string{"resolve", "pause", "disable-channel", "disable-grouping"} {+ t.Run(policy, func(t *testing.T) {+ f := newShutdownAdmissionFixture(t)+ m := f.open(func(m *NotificationManager) { m.SetGroupingConfig(true, 600, false, false) })+ start := time.Now().UTC().Add(-time.Minute)+ a := quietReplayAlert("buffered", alerts.AlertLevelWarning, start)+ m.SendAlert(a)+ // An owned direct send holds the draining stage open. This avoids+ // treating an operation begun after shutdown closed as a cancellation.+ require.True(t, m.lifecycle.join())+ stopped := make(chan struct{})+ go func() { m.Stop(); close(stopped) }()+ awaitShutdownCondition(t, "draining lifecycle", func() bool { return !m.lifecycle.accepting() })+ switch policy {+ case "resolve":+ m.CancelResolvedAlert(a)+ case "pause":+ m.SetEnabled(false)+ case "disable-channel":+ hook := f.hook+ hook.Enabled = false+ require.NoError(t, m.UpdateWebhook(hook.ID, hook))+ case "disable-grouping":+ m.SetGroupingConfig(false, 0, false, false)+ }+ m.lifecycle.release()+ awaitShutdownSignal(t, "Stop after operator policy", stopped)+ m = f.open(nil)+ rows := quietReplayRows(t, m.GetQueue())+ for _, row := range rows {+ if policy == "resolve" || policy == "pause" {+ require.Equal(t, QueueStatusCancelled, row.Status)+ } else {+ require.Equal(t, QueueStatusPending, row.Status)+ require.NotNil(t, row.NextRetryAt)+ }+ }+ if policy == "disable-grouping" {+ require.Len(t, rows, 1, "a policy edit during drain lost the buffer")+ }+ select {+ case <-f.spawned:+ t.Fatal("operator policy during shutdown sent directly")+ default:+ }+ })+ }+}++func TestStopCheckpointStorageErrorNeverSendsEarly(t *testing.T) {+ f := newShutdownAdmissionFixture(t)+ m := f.open(func(m *NotificationManager) { m.SetGroupingConfig(true, 600, false, false) })+ _, err := m.GetQueue().db.Exec(`CREATE TRIGGER reject_checkpoint BEFORE INSERT ON notification_queue BEGIN SELECT RAISE(ABORT, 'fixture checkpoint rejected'); END`)+ require.NoError(t, err)+ m.SendAlert(quietReplayAlert("pending", alerts.AlertLevelWarning, time.Now().UTC()))+ m.Stop()+ select {+ case <-f.spawned:+ t.Fatal("checkpoint failure bypassed the grouping window with a direct send")+ default:+ }+}++// A timer already taking a lease and a concurrent Stop may race, but exactly one+// path owns the batch. No provider is activated in this persistent queue check.+func TestStopCheckpointRacesTimerOnce(t *testing.T) {+ f := newShutdownAdmissionFixture(t)+ for range 12 {+ m := f.open(func(m *NotificationManager) { m.SetGroupingConfig(true, 600, false, false) })+ a := quietReplayAlert(time.Now().UTC().Format(time.RFC3339Nano), alerts.AlertLevelWarning, time.Now().UTC())+ m.SendAlert(a)+ var wg sync.WaitGroup+ wg.Add(2)+ go func() { defer wg.Done(); m.sendGroupedAlerts() }()+ go func() { defer wg.Done(); m.Stop() }()+ wg.Wait()+ }+ rows := quietReplayRows(t, f.open(nil).GetQueue())+ require.Len(t, rows, 12)+ for _, row := range rows {+ require.Zero(t, row.Attempts)+ require.Len(t, row.Alerts, 1)+ require.Equal(t, QueueStatusPending, row.Status)+ }+}diff --git a/internal/notifications/firing_group_process_test.go b/internal/notifications/firing_group_process_test.gonew file mode 100644index 0000000000..8cbdbafc4e--- /dev/null+++ b/internal/notifications/firing_group_process_test.go@@ -0,0 +1,354 @@+package notifications_test++import (+ "bufio"+ "bytes"+ "encoding/json"+ "fmt"+ "io"+ "net/http"+ "net/http/httptest"+ "os"+ "os/exec"+ "os/signal"+ "strconv"+ "strings"+ "syscall"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/alerts"+ "github.com/rcourtman/pulse-go-rewrite/internal/models"+ "github.com/rcourtman/pulse-go-rewrite/internal/notifications"+ "github.com/stretchr/testify/require"+)++const checkpointProcessMarker = "GROUPED_PROCESS "++type checkpointProcessState struct {+ Phase string `json:"phase"`+ Stats map[string]int `json:"stats"`+ Log []notifications.DeliveryLogEntry `json:"entries"`+ Started time.Time `json:"started"`+ Active int `json:"active"`+}++// This registered helper runs real alert/notification managers in a separate+// process with persistent state, guest-local receivers and an actual SIGTERM.+// It is not the installed server executable or a system-service qualification.+func TestGroupedWarningProcessHelper(t *testing.T) {+ dir := os.Getenv("PULSE_GROUPED_TEST_DIR")+ if dir == "" {+ t.Skip("subprocess helper")+ }+ t.Setenv("PULSE_DATA_DIR", dir)+ url := os.Getenv("PULSE_GROUPED_TEST_RECEIVER")+ cooldown, err := strconv.Atoi(os.Getenv("PULSE_GROUPED_TEST_COOLDOWN"))+ require.NoError(t, err)+ var am *alerts.Manager+ var nm *notifications.NotificationManager+ fired := make(chan *alerts.Alert, 8)+ open := func() {+ nm = notifications.NewNotificationManagerWithDeferredQueue("", dir)+ require.NotNil(t, nm.GetQueue())+ require.NoError(t, nm.UpdateAllowedPrivateCIDRs("127.0.0.1/32"))+ nm.SetCooldown(cooldown)+ nm.SetGroupingConfig(true, 4, true, true)+ for _, id := range []string{"left", "right"} {+ nm.AddWebhook(notifications.WebhookConfig{ID: id, Name: "Fixture " + id, URL: url + "/" + id, Enabled: true, Service: "generic"})+ }+ am = alerts.NewManagerWithDataDir(dir, alerts.WithDurableAlertStore())+ cfg := am.GetConfig()+ cfg.Enabled = true+ cfg.ActivationState = alerts.ActivationActive+ cfg.FlappingEnabled = false+ cfg.Schedule.Cooldown = cooldown+ cfg.Schedule.MaxAlertsHour = 100+ cfg.TimeThresholds = map[string]int{"agent": 0, "storage": 0}+ cfg.SuppressionWindow = 0+ am.UpdateConfig(cfg)+ am.SetAlertCallback(func(a *alerts.Alert) { nm.SendAlert(a); fired <- a.Clone() })+ nm.StartQueueProcessing()+ }+ stop := func() {+ if am != nil {+ am.Stop()+ am = nil+ }+ if nm != nil {+ nm.Stop()+ nm = nil+ }+ }+ defer stop()+ open()+ started := time.Now().UTC()+ host := models.Host{ID: "process-host", Hostname: "Process host", Sensors: models.HostSensorSummary{Custom: []models.HostCustomSensorMetric{+ {ID: "probe-one", Name: "Probe one", Status: "warning", ObservedAt: started},+ {ID: "probe-two", Name: "Probe two", Status: "warning", ObservedAt: started},+ }}}+ if os.Getenv("PULSE_GROUPED_TEST_SEED") == "1" {+ am.CheckHost(host)+ for range 2 {+ select {+ case a := <-fired:+ require.Equal(t, alerts.AlertLevelWarning, a.Level)+ case <-time.After(5 * time.Second):+ t.Fatal("CheckHost did not dispatch both warnings")+ }+ }+ require.NoError(t, am.SaveActiveAlerts())+ } else {+ active := am.GetActiveAlerts()+ require.Len(t, active, 2)+ for _, a := range active {+ require.NotNil(t, a.LastNotified, "dispatch marker must survive restore")+ }+ am.CheckHost(host)+ }+ signals := make(chan os.Signal, 1)+ signal.Notify(signals, syscall.SIGTERM)+ defer signal.Stop(signals)+ commands := make(chan string)+ go func() {+ scanner := bufio.NewScanner(os.Stdin)+ for scanner.Scan() {+ commands <- scanner.Text()+ }+ close(commands)+ }()+ emit := func(phase string) {+ stats, err := nm.GetQueue().GetQueueStats()+ require.NoError(t, err)+ entries, err := nm.GetQueue().GetDeliveryLog(time.Now().Add(-time.Hour), 100)+ require.NoError(t, err)+ encoded, err := json.Marshal(checkpointProcessState{phase, stats, entries, started, len(am.GetActiveAlerts())})+ require.NoError(t, err)+ fmt.Println(checkpointProcessMarker + string(encoded))+ }+ emit("ready")+ for {+ select {+ case <-signals:+ stop()+ fmt.Println(checkpointProcessMarker + `{"phase":"stopped"}`)+ return+ case command, ok := <-commands:+ if !ok {+ return+ }+ switch command {+ case "reload":+ stop()+ open()+ am.CheckHost(host)+ emit("reloaded")+ case "status":+ emit("status")+ case "resolve":+ // Recovery dispatch uses the same public occurrence-cancellation+ // and receipt-gated entry points as the monitor callback.+ nm.SetGroupingConfig(false, 0, false, false)+ for _, a := range am.GetActiveAlerts() {+ nm.CancelResolvedAlert(a)+ nm.SendResolvedAlert(&alerts.ResolvedAlert{Alert: a, ResolvedTime: time.Now().UTC()})+ }+ emit("resolved")+ case "stop":+ stop()+ fmt.Println(checkpointProcessMarker + `{"phase":"stopped"}`)+ return+ default:+ t.Fatalf("unknown fixture command %q", command)+ }+ }+ }+}++func TestGroupedWarningGracefulProcessRestartAndReload(t *testing.T) {+ for _, tc := range []struct {+ mode string+ cooldown int+ }{{"restart", 0}, {"reload", 5}} {+ t.Run(tc.mode, func(t *testing.T) {+ type delivery struct {+ At time.Time+ Path string+ Event string `json:"event"`+ Alerts []*alerts.Alert `json:"alerts"`+ Count int `json:"count"`+ Grouped bool `json:"grouped"`+ }+ received := make(chan delivery, 16)+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ var d delivery+ if err := json.NewDecoder(r.Body).Decode(&d); err != nil {+ t.Error(err)+ w.WriteHeader(400)+ return+ }+ d.At, d.Path = time.Now().UTC(), r.URL.Path+ received <- d+ w.WriteHeader(200)+ }))+ defer server.Close()+ dir := t.TempDir()+ type child struct {+ cmd *exec.Cmd+ input io.WriteCloser+ states chan checkpointProcessState+ stderr *bytes.Buffer+ }+ start := func(seed bool) *child {+ cmd := exec.Command(os.Args[0], "-test.run=^TestGroupedWarningProcessHelper$", "-test.timeout=45s", "-test.v")+ cmd.Env = append(os.Environ(), "PULSE_GROUPED_TEST_DIR="+dir, "PULSE_GROUPED_TEST_RECEIVER="+server.URL, "PULSE_GROUPED_TEST_COOLDOWN="+strconv.Itoa(tc.cooldown), "PULSE_GROUPED_TEST_SEED="+map[bool]string{true: "1", false: "0"}[seed])+ stdout, err := cmd.StdoutPipe()+ require.NoError(t, err)+ input, err := cmd.StdinPipe()+ require.NoError(t, err)+ c := &child{cmd, input, make(chan checkpointProcessState, 32), &bytes.Buffer{}}+ cmd.Stderr = c.stderr+ require.NoError(t, cmd.Start())+ go func() {+ defer close(c.states)+ scanner := bufio.NewScanner(stdout)+ for scanner.Scan() {+ line := scanner.Text()+ if strings.HasPrefix(line, checkpointProcessMarker) {+ var state checkpointProcessState+ if json.Unmarshal([]byte(strings.TrimPrefix(line, checkpointProcessMarker)), &state) == nil {+ c.states <- state+ }+ }+ }+ }()+ return c+ }+ read := func(c *child, phase string) checkpointProcessState {+ t.Helper()+ select {+ case state, ok := <-c.states:+ require.True(t, ok, "child ended before %s", phase)+ require.Equal(t, phase, state.Phase)+ return state+ case <-time.After(15 * time.Second):+ t.Fatalf("no child phase %s", phase)+ return checkpointProcessState{}+ }+ }+ finish := func(c *child, sig bool) {+ t.Helper()+ if sig {+ require.NoError(t, c.cmd.Process.Signal(syscall.SIGTERM))+ } else {+ _, err := fmt.Fprintln(c.input, "stop")+ require.NoError(t, err)+ }+ read(c, "stopped")+ require.NoError(t, c.input.Close())+ err := c.cmd.Wait()+ if err != nil {+ t.Log(c.stderr.String())+ }+ require.NoError(t, err, "helper must exit normally after graceful stop")+ }+ c := start(true)+ // Cleanup only the child owned by this test, even on an assertion failure.+ t.Cleanup(func() {+ if c != nil && c.cmd.ProcessState == nil {+ _ = c.input.Close()+ _ = c.cmd.Process.Signal(syscall.SIGTERM)+ _ = c.cmd.Wait()+ }+ })+ seed := read(c, "ready")+ require.Equal(t, 2, seed.Active)+ require.Empty(t, seed.Log, "buffer admission is not a provider attempt")+ if tc.mode == "restart" {+ finish(c, true)+ c = start(false)+ read(c, "ready")+ } else {+ _, err := fmt.Fprintln(c.input, "reload")+ require.NoError(t, err)+ read(c, "reloaded")+ }+ _, err := fmt.Fprintln(c.input, "status")+ require.NoError(t, err)+ pending := read(c, "status")+ require.Equal(t, 2, pending.Stats["pending"], "restored dispatch markers must not strand accepted warnings")+ require.Empty(t, pending.Log)+ t.Logf("GROUPED_BROWSER_SNAPSHOT %s", mustCheckpointJSON(t, pending))+ seen := map[string]bool{}+ for range 2 {+ select {+ case d := <-received:+ require.False(t, d.At.Before(seed.Started.Add(4*time.Second)), "shutdown shortened grouping window")+ require.True(t, d.Grouped)+ require.Equal(t, 2, d.Count)+ require.Len(t, d.Alerts, 2)+ require.Equal(t, "alert", d.Event)+ require.False(t, seen[d.Path])+ seen[d.Path] = true+ case <-time.After(10 * time.Second):+ t.Fatal("wanted firing did not reach both receivers after restart/reload")+ }+ }+ // Wait for durable status/audit completion, not merely receiver arrival.+ waitSent := func(want int) checkpointProcessState {+ t.Helper()+ deadline := time.Now().Add(5 * time.Second)+ for {+ _, err := fmt.Fprintln(c.input, "status")+ require.NoError(t, err)+ state := read(c, "status")+ if state.Stats["sent"] == want {+ return state+ }+ require.True(t, time.Now().Before(deadline), "durable completion missing")+ time.Sleep(5 * time.Millisecond)+ }+ }+ sent := waitSent(2)+ t.Logf("GROUPED_BROWSER_SNAPSHOT %s", mustCheckpointJSON(t, sent))+ // Another real process boundary proves each occurrence/destination+ // firing receipt, not the in-memory delivery map, authorises recovery.+ finish(c, true)+ c = start(false)+ read(c, "ready")+ _, err = fmt.Fprintln(c.input, "resolve")+ require.NoError(t, err)+ read(c, "resolved")+ recoveries := map[string]int{}+ for range 4 {+ select {+ case d := <-received:+ require.Equal(t, "resolved", d.Event)+ require.Len(t, d.Alerts, 1)+ require.True(t, seen[d.Path])+ recoveries[d.Path]+++ case <-time.After(10 * time.Second):+ t.Fatal("receipt-gated recovery missing after process restart")+ }+ }+ require.Equal(t, map[string]int{"/left": 2, "/right": 2}, recoveries)+ resolved := waitSent(6)+ t.Logf("GROUPED_BROWSER_SNAPSHOT %s", mustCheckpointJSON(t, resolved))+ finish(c, false)+ c = nil+ select {+ case d := <-received:+ t.Fatalf("duplicate receiver message %+v", d)+ default:+ }+ t.Log("CheckHost warning markers, original grouped deadline, normal SIGTERM exit/reload, two scoped grouped HTTP firings and four receipt-gated recoveries verified without edited queue clocks or provider access")+ })+ }+}++func mustCheckpointJSON(t *testing.T, value any) string {+ t.Helper()+ b, err := json.Marshal(value)+ require.NoError(t, err)+ return string(b)+}diff --git a/internal/notifications/manager_lifecycle_test.go b/internal/notifications/manager_lifecycle_test.goindex 5e0c538e90..fe066d1f47 100644--- a/internal/notifications/manager_lifecycle_test.go+++ b/internal/notifications/manager_lifecycle_test.go@@ -267,7 +267,7 @@ func TestStopDrainKeepsPauseAndChannelDisable(t *testing.T) {}// A deferred queue that never started keeps admitted rows for the next start,-// and Stop still discards a firing group that was only buffered in memory.+// and Stop checkpoints the buffered group without starting provider delivery.func TestStopNeverStartedQueueAndBufferedGroup(t *testing.T) {f := newShutdownAdmissionFixture(t)m := f.open(nil)@@ -301,11 +301,14 @@ func TestStopNeverStartedQueueAndBufferedGroup(t *testing.T) {statuses[alert.ID] = row.Status}}- if len(rows) != 2 || statuses["queued-1"] != QueueStatusPending || statuses["queued-2"] != QueueStatusPending {+ if len(rows) != 3 || statuses["queued-1"] != QueueStatusPending || statuses["queued-2"] != QueueStatusPending || statuses["buffered"] != QueueStatusPending {t.Fatalf("never-started queue lost admitted rows across Stop: %+v", rows)}- if _, persisted := statuses["buffered"]; persisted {- t.Fatal("Stop persisted a buffered firing group; buffered groups are discarded at shutdown")+ for _, row := range rows {+ if row.Alerts[0].ID == "buffered" {+ require.NotNil(t, row.NextRetryAt)+ require.True(t, row.NextRetryAt.After(time.Now().Add(3500*time.Second)), "shutdown must not flush or reset the original deadline")+ }}}diff --git a/internal/notifications/notifications.go b/internal/notifications/notifications.goindex 6a7b5d713c..d130964d49 100644--- a/internal/notifications/notifications.go+++ b/internal/notifications/notifications.go@@ -322,6 +322,7 @@ type NotificationManager struct {pendingAlerts []*alerts.AlertpendingAlertIndex map[pendingFiringOccurrence]intgroupTimer *time.Timer+ groupDeadline time.Time // First admission fixes this window; shutdown must not restart or shorten itpendingResolved []*alerts.AlertresolvedGroupTimer *time.TimergroupByNode bool@@ -1143,10 +1144,10 @@ func (n *NotificationManager) SetGroupingConfig(enabled bool, seconds int, byNodvar appriseConfig AppriseConfigvar initialTarget notificationDeliveryTargetvar queue *NotificationQueue- if !enabled || seconds == 0 {+ if (!enabled || seconds == 0) && n.lifecycle.accepting() {resolved = n.takePendingResolvedLocked()- // Stop clears pending groups in the critical section that refuses- // admission, so a pending group here is always admitted.+ // Only a live manager may flush to delivery; Stop checkpoints any+ // remaining buffer without bypassing its deadline.if len(n.pendingAlerts) > 0 && n.lifecycle.admit() {pending = append(pending, n.pendingAlerts...)}@@ -1156,6 +1157,7 @@ func (n *NotificationManager) SetGroupingConfig(enabled bool, seconds int, byNodif n.groupTimer != nil {n.groupTimer.Stop()n.groupTimer = nil+ n.groupDeadline = time.Time{}}emailConfig = copyEmailConfig(n.emailConfig)webhooks = copyWebhookConfigs(n.webhooks)@@ -1272,6 +1274,7 @@ func (n *NotificationManager) SetEnabled(enabled bool) {if n.groupTimer != nil {n.groupTimer.Stop()n.groupTimer = nil+ n.groupDeadline = time.Time{}}if n.queue != nil && n.lifecycle.join() {queue = n.queue@@ -1415,7 +1418,8 @@ func (n *NotificationManager) sendAlert(alert *alerts.Alert, options alertSendOp// If this is the first alert in the group, start the timerif n.groupTimer == nil {- n.groupTimer = time.AfterFunc(n.groupWindow, func() {+ n.groupDeadline = time.Now().Add(n.groupWindow)+ n.groupTimer = time.AfterFunc(time.Until(n.groupDeadline), func() {n.sendGroupedAlerts()})log.Debug().@@ -1855,6 +1859,7 @@ func (n *NotificationManager) cancelAlert(alertID string, start *time.Time) boollog.Debug().Str("alertID", alertID).Msg("stopped grouping timer after alert cancellation")}n.groupTimer = nil+ n.groupDeadline = time.Time{}}}@@ -1898,8 +1903,8 @@ func (n *NotificationManager) sendGroupedAlerts() {n.mu.Unlock()return}- // Stop clears pending groups in the critical section that refuses- // admission, so a timer that finds a group here is always admitted.+ // A timer racing Stop must leave the buffer for the shutdown checkpoint,+ // not dispatch it directly or lose it after admission has closed.if !n.lifecycle.admit() {n.mu.Unlock()return@@ -1918,6 +1923,7 @@ func (n *NotificationManager) sendGroupedAlerts() {n.groupTimer.Stop()}n.groupTimer = nil+ n.groupDeadline = time.Time{}log.Info().Int("alertCount", len(alertsToSend)).@@ -4534,15 +4540,12 @@ func (n *NotificationManager) cleanupOldNotificationRecords() {// Stop gracefully stops the notification managerfunc (n *NotificationManager) Stop() {n.stopOnce.Do(func() {- // Refuse new alerts and discard volatile groups in one critical- // section, so a pending group is either already admitted with a lease- // or discarded here, as Stop has always discarded it.+ // Refuse new admissions and stop the firing timer, but keep its snapshots+ // until the queue is quiescent. Shutdown checkpoints the accepted batch+ // without sending it or waiting for its grouping deadline.n.mu.Lock()n.lifecycle.drain()n.takePendingResolvedLocked()- clear(n.pendingAlerts)- n.pendingAlerts = n.pendingAlerts[:0]- n.pendingAlertIndex = nilif n.groupTimer != nil {n.groupTimer.Stop()n.groupTimer = nil@@ -4559,6 +4562,14 @@ func (n *NotificationManager) Stop() {if queue != nil {queue.stopProcessing()}+ // With workers stopped, no checkpoint can race a provider attempt. Keep+ // n.mu across the buffer-to-queue handoff: a concurrent resolution/pause+ // either removes the snapshot first or cancels the committed queue row.+ // Neither can fall between removing the buffer and persisting the row.+ n.mu.Lock()+ n.checkpointPendingFiringGroupLocked(queue)+ n.mu.Unlock()+// Wait for admitted writes, cancellations, operator changes and direct// deliveries. No lease can be taken once this returns.n.lifecycle.close()