Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
fix(notifications): preserve queued work through restart
rcourtman5d11f80d7d67
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 489f7d34dd..7fe16d2ddd 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -10769,3 +10769,21 @@ settings remain live and unchanged. This is not a backup, credential recovery,agent/guest probe or installed recovery. The normal HTTP contract`TestContractSupportBundleCredentialBoundary` and `log_bundle_test.go` cover thesummary, scope denials, state preservation and configured-file/buffer fallbacks.++### Notification policy precedes persistent queue activation++`monitoring.New` reads notification system settings from the shared installation+directory before creating notification workers. Tenant-local settings cannot+grant private destinations or replace the installation public URL. The+constructor validates the private CIDR policy, restores the saved public URL+while preserving a deployment-enforced URL, and installs the policy before activating persisted work. Missing+settings keep the default private-destination denial. Unreadable settings or an+invalid CIDR policy fail construction before queue creation, so accepted rows+cannot acquire a terminal failure merely because bootstrap has not restored+policy yet. Router settings refresh remains responsible for later live changes.++`TestNewRestoresSharedWebhookPolicyBeforeQueueActivation` covers a real pending+HTTP delivery before any router exists for default and tenant paths, saved+public URL precedence and deployment lock, and a tenant grant that cannot override installation revocation.+`TestNewRejectsUnreadableNotificationPolicyBeforeQueueCreation` covers corrupt+settings and invalid CIDRs without creating queue storage.diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 4dda4b8827..4166077c47 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -9271,3 +9271,21 @@ alert engine emits them; every other projection uses `RecordChange` and stayssilent. Both paths write the same deterministic change row. Proof:`TestAlertTimelineAnnouncesOnlyLiveLifecycleChanges` in`internal/monitoring/monitor_alert_handling_test.go`.++### Notification policy precedes persistent queue activation++`monitoring.New` reads notification system settings from the shared installation+directory before creating notification workers. Tenant-local settings cannot+grant private destinations or replace the installation public URL. The+constructor validates the private CIDR policy, restores the saved public URL+while preserving a deployment-enforced URL, and installs the policy before activating persisted work. Missing+settings keep the default private-destination denial. Unreadable settings or an+invalid CIDR policy fail construction before queue creation, so accepted rows+cannot acquire a terminal failure merely because bootstrap has not restored+policy yet. Router settings refresh remains responsible for later live changes.++`TestNewRestoresSharedWebhookPolicyBeforeQueueActivation` covers a real pending+HTTP delivery before any router exists for default and tenant paths, saved+public URL precedence and deployment lock, and a tenant grant that cannot override installation revocation.+`TestNewRejectsUnreadableNotificationPolicyBeforeQueueCreation` covers corrupt+settings and invalid CIDRs without creating queue storage.diff --git a/docs/release-control/v6/internal/subsystems/notifications.md b/docs/release-control/v6/internal/subsystems/notifications.mdindex 464354aa40..dbf5478dc1 100644--- a/docs/release-control/v6/internal/subsystems/notifications.md+++ b/docs/release-control/v6/internal/subsystems/notifications.md@@ -348,6 +348,54 @@ notifier, replaces the router monitor, and exercises stats, Retry and Dismiss;replacement and unavailable-manager reads with the race detector. This islifecycle regression proof, not installed SMTP receipt or reporter confirmation.+### Manager shutdown is a restart boundary, not a delivery verdict++`NotificationManager.Stop` first refuses new alerts and discards volatile+firing and recovery groups in one critical section, as Stop has always+discarded them. Work that reads the queue outside the manager lock holds a+lease taken in the critical section that reads it. An admitted alert, recovery+or timer group is written to the open queue. Cancellation, operator pause and+channel disable still run while Stop drains. Direct delivery started by+admitted work is owned until it finishes and records its receipt. An attempt+made after Stop begins is refused, neither queued nor sent directly, and no+direct delivery starts once Stop has finished.++Stop then quiesces the queue. A worker that observes the stop begins no+quiet-hours hold, partition or claim: its job stays `pending` with no attempt,+audit or receipt, and the next start delivers it. Work a worker began before+that observation finishes while delivery is still permitted, recording its+status, audit and durable firing receipt. Only after every worker and lease+has ended does Stop withdraw delivery, detach the queue and close storage. A+recovery after restart therefore still reaches each destination that received+the firing. Shutdown must never cancel queued work as "delivery disabled";+that verdict belongs to the pause and destination controls. Resolution+cancellation and at-least-once replay of interrupted sends are unchanged.+Delivery attempts retain their existing provider timeouts and SMTP retry+budget. Stop waits for their durable outcomes instead of closing storage+under an active writer. There is no separate aggregate drain deadline. The+queue logs the processing stop after refusing new attempts and before waiting,+so process-signal proofs can release held provider responses at that boundary.++`TestManagerStopKeepsQueuedDeliveryForRestart` holds five real HTTP sends in+the worker pool while a sixth admitted job waits for a slot, stops the manager,+reopens the persistent queue, and checks the kept job, the in-flight outcomes,+their durable receipts and a recovery delivered after restart.+`manager_lifecycle_test.go` holds an admitted alert, recovery or timer group at+queue serialisation while two concurrent Stop calls run. It checks the write+after reopen, the refusal of a later attempt, and pause and channel disable+during the drain. It also checks a never-started queue, the discarded buffered+group, and direct delivery that Stop waits for and refuses afterwards. These+are loopback source proofs, not process-signal, reload or installed-provider+acceptance.++Monitor bootstrap also restores the shared installation webhook policy before+activating persistent work, including tenant queues. A later router refresh is+too late for a due private-webhook job. Missing settings retain default denial;+unreadable settings or invalid CIDRs fail construction before queue creation.+The monitor startup tests cover delivery with no router, installation revocation+against tenant-local grants, and invalid-policy admission. This boundary keeps+shutdown-preserved rows from becoming terminal failures during reconstruction.+Own notification delivery transport, provider configuration, queueing, andnotification-management API surfaces.diff --git a/internal/api/router_webhook_cidr_reload_test.go b/internal/api/router_webhook_cidr_reload_test.goindex 5b8ebf0a54..0b78eb9e7a 100644--- a/internal/api/router_webhook_cidr_reload_test.go+++ b/internal/api/router_webhook_cidr_reload_test.go@@ -10,11 +10,9 @@ import ("github.com/rcourtman/pulse-go-rewrite/internal/monitoring")-// TestReloadSystemSettings_AppliesWebhookCIDRsToNewMonitor verifies the fix-// for issue #1507: after a monitor reload (which recreates the notification-// manager), ReloadSystemSettings must re-apply the persisted webhook private-// CIDR allowlist to the new notification manager. Without this, webhooks to-// private IPs fail after any auto-registration-triggered monitor reload.+// A recreated monitor restores its bootstrap policy before any router exists.+// ReloadSystemSettings still owns later persisted changes, including revoking+// a previously allowed private destination (the live replacement path of #1507).func TestReloadSystemSettings_AppliesWebhookCIDRsToNewMonitor(t *testing.T) {tempDir := t.TempDir()@@ -59,41 +57,50 @@ func TestReloadSystemSettings_AppliesWebhookCIDRsToNewMonitor(t *testing.T) {t.Fatalf("initial monitor: expected CIDR to allow 192.168.1.50, got: %v", err)}- // Simulate a monitor reload: create a fresh monitor with a brand-new- // notification manager (empty CIDR allowlist).+ // Recreated managers must already have saved policy before queued work runs.monitor2, err := monitoring.New(cfg)if err != nil {t.Fatalf("monitoring.New (reload): %v", err)}t.Cleanup(func() { monitor2.Stop() })- // Before the fix: the reload path in server.go never called- // ReloadSystemSettings, so the new notification manager would have- // an empty allowlist and reject private IPs.nm2 := monitor2.GetNotificationManager()if nm2 == nil {t.Fatal("reloaded monitor has no notification manager")}- if err := nm2.ValidateWebhookURL("http://192.168.1.50/hook"); err == nil {- t.Fatal("fresh notification manager should reject private IP before settings are re-applied")+ if err := nm2.ValidateWebhookURL("http://192.168.1.50/hook"); err != nil {+ t.Fatalf("fresh manager must restore policy before router refresh: %v", err)}- // Now update the router's monitor reference (as SetMonitor does during reload).+ // Persist a real policy replacement after construction. Saving alone must+ // not mutate the live manager; refreshing must replace, not merge, grants.+ settings.WebhookAllowedPrivateCIDRs = "172.16.0.0/16"+ if err := persistence.SaveSystemSettings(*settings); err != nil {+ t.Fatalf("SaveSystemSettings replacement: %v", err)+ }+ if err := nm2.ValidateWebhookURL("http://192.168.1.50/hook"); err != nil {+ t.Fatalf("persisting replacement unexpectedly changed live policy: %v", err)+ }+ if err := nm2.ValidateWebhookURL("http://172.16.0.1/hook"); err == nil {+ t.Fatal("replacement grant reached the live manager before refresh")+ }router.monitor = monitor2-- // Call ReloadSystemSettings — this is the fix.router.ReloadSystemSettings()-- // The new notification manager should now have the persisted CIDR allowlist.- if err := nm2.ValidateWebhookURL("http://192.168.1.50/hook"); err != nil {- t.Fatalf("after ReloadSystemSettings: expected CIDR to allow 192.168.1.50, got: %v", err)+ if err := nm2.ValidateWebhookURL("http://172.16.0.1/hook"); err != nil {+ t.Fatalf("after refresh: replacement destination rejected: %v", err)}- if err := nm2.ValidateWebhookURL("http://10.5.5.5/hook"); err != nil {- t.Fatalf("after ReloadSystemSettings: expected CIDR to allow 10.5.5.5, got: %v", err)+ for _, revoked := range []string{"http://192.168.1.50/hook", "http://10.5.5.5/hook"} {+ if err := nm2.ValidateWebhookURL(revoked); err == nil {+ t.Fatalf("after refresh: old grant retained for %s", revoked)+ }}- // IPs outside the allowlist should still be rejected.+ settings.WebhookAllowedPrivateCIDRs = ""+ if err := persistence.SaveSystemSettings(*settings); err != nil {+ t.Fatalf("SaveSystemSettings revocation: %v", err)+ }+ router.ReloadSystemSettings()if err := nm2.ValidateWebhookURL("http://172.16.0.1/hook"); err == nil {- t.Fatal("after ReloadSystemSettings: expected 172.16.0.1 to be rejected (not in allowlist)")+ t.Fatal("clearing the installation policy did not revoke the final grant")}}diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.goindex c9d4b63a5f..6590db2f3c 100644--- a/internal/monitoring/monitor.go+++ b/internal/monitoring/monitor.go@@ -1839,6 +1839,31 @@ func New(cfg *config.Config) (*Monitor, error) {return nil, fmt.Errorf("load Docker report order journal: %w", err)}+ // Queued deliveries can run before a router exists. Restore the instance+ // policy before creating workers, including for tenant persistence paths.+ // An unreadable policy must not turn accepted work into a terminal failure.+ notificationPersistence := config.NewConfigPersistence(cfg.DataPath)+ notificationSystemPersistence := notificationPersistence+ if sharedDir := notificationPersistence.SharedInstallationDataDir(); sharedDir != notificationPersistence.GetConfigDir() {+ notificationSystemPersistence = config.NewConfigPersistence(sharedDir)+ }+ notificationSystemSettings, notificationSettingsErr := notificationSystemPersistence.LoadSystemSettings()+ if notificationSettingsErr != nil {+ return nil, fmt.Errorf("load notification system settings: %w", notificationSettingsErr)+ }+ notificationPrivateCIDRs := ""+ notificationPublicURL := cfg.PublicURL+ if notificationSystemSettings != nil {+ notificationPrivateCIDRs = notificationSystemSettings.WebhookAllowedPrivateCIDRs+ if !cfg.EnvOverrides["publicURL"] && notificationSystemSettings.PublicURL != "" {+ notificationPublicURL = notificationSystemSettings.PublicURL+ }+ }+ notificationPrivateNetworks, notificationPolicyErr := notifications.ParseAllowedPrivateCIDRs(notificationPrivateCIDRs)+ if notificationPolicyErr != nil {+ return nil, fmt.Errorf("load notification private CIDR policy: %w", notificationPolicyErr)+ }+// Initialize temperature collector with sensors SSH key// Will use root user for now - can be made configurable laterhomeDir := os.Getenv("HOME")@@ -2006,9 +2031,9 @@ func New(cfg *config.Config) (*Monitor, error) {mockModeAligned: builtInMockMode,mockModeSwitchesSeen: switchesSeen,incidentStore: incidentStore,- notificationMgr: notifications.NewNotificationManagerWithDeferredQueue(cfg.PublicURL, cfg.DataPath),+ notificationMgr: notifications.NewNotificationManagerWithDeferredQueue(notificationPublicURL, cfg.DataPath),deadMan: newDeadManRuntime(config.ResolveRuntimeDataDir(cfg.DataPath)),- configPersist: config.NewConfigPersistence(cfg.DataPath),+ configPersist: notificationPersistence,discoveryService: nil, // Will be initialized in Start()authFailures: make(map[string]int),lastAuthAttempt: make(map[string]time.Time),@@ -2159,8 +2184,9 @@ func New(cfg *config.Config) (*Monitor, error) {log.Warn().Err(err).Msg("failed to load webhook configuration")}// The queue worker starts with the manager but has no processor until all- // saved destinations are installed. Activate it only after the last load;- // a slow migration or config read must not cancel due persisted work.+ // saved destinations and shared delivery policy are installed. A router+ // may not exist yet, so its later settings refresh cannot own this boundary.+ m.notificationMgr.ApplyAllowedPrivateCIDRs(notificationPrivateCIDRs, notificationPrivateNetworks)m.notificationMgr.SetQuietHoursPolicyProvider(m.alertManager.QuietHoursNotificationPolicy)m.notificationMgr.StartQueueProcessing()diff --git a/internal/monitoring/monitor_notification_startup_test.go b/internal/monitoring/monitor_notification_startup_test.goindex 0dc557f8a4..4a5bdf4aab 100644--- a/internal/monitoring/monitor_notification_startup_test.go+++ b/internal/monitoring/monitor_notification_startup_test.go@@ -4,7 +4,10 @@ import ("encoding/json""net/http""net/http/httptest"+ "os"+ "path/filepath""reflect"+ "strings""sync""sync/atomic""testing"@@ -218,3 +221,156 @@ func TestMonitorResolvedSnapshotKeepsRefiringPBSDelivery(t *testing.T) {}assertOccurrenceHTTP(t, n, receipts, current)}++// No router or post-construction setter may be needed to resume accepted work.+// A tenant must use the installation policy, including revocation, rather than+// a policy file placed in its own persistence directory.+func TestNewRestoresSharedWebhookPolicyBeforeQueueActivation(t *testing.T) {+ for _, tc := range []struct {+ name string+ tenant, allowed bool+ configuredURL string+ urlLocked bool+ }{+ {name: "default", allowed: true},+ {name: "tenant", tenant: true, allowed: true},+ {name: "tenant-cannot-grant", tenant: true},+ {name: "saved-url-replaces-stale-config", allowed: true, configuredURL: "https://stale-config.example.test"},+ {name: "deployment-url-lock", allowed: true, configuredURL: "https://deployment.example.test", urlLocked: true},+ } {+ t.Run(tc.name, func(t *testing.T) {+ root := t.TempDir()+ t.Setenv("PULSE_DATA_DIR", root)+ received := make(chan string, 4)+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ var body struct {+ Instance string `json:"instance"`+ }+ if err := json.NewDecoder(r.Body).Decode(&body); err != nil {+ t.Error(err)+ }+ received <- body.Instance+ w.WriteHeader(http.StatusOK)+ }))+ defer server.Close()+ global := config.NewConfigPersistence(root)+ settings := config.DefaultSystemSettings()+ settings.PublicURL = "https://notification-bootstrap.example.test"+ if tc.allowed {+ settings.WebhookAllowedPrivateCIDRs = "127.0.0.1/32"+ }+ if err := global.SaveSystemSettings(*settings); err != nil {+ t.Fatal(err)+ }+ dir := root+ if tc.tenant {+ dir = filepath.Join(root, "orgs", "queue-policy")+ }+ persistence := config.NewConfigPersistence(dir)+ if tc.tenant {+ local := config.DefaultSystemSettings()+ local.PublicURL = "https://tenant-policy-must-not-win.example.test"+ if !tc.allowed {+ local.WebhookAllowedPrivateCIDRs = "127.0.0.1/32"+ }+ if err := persistence.SaveSystemSettings(*local); err != nil {+ t.Fatal(err)+ }+ }+ if err := persistence.SaveAlertConfig(alerts.AlertConfig{Enabled: true, ActivationState: alerts.ActivationActive}); err != nil {+ t.Fatal(err)+ }+ hook := notifications.WebhookConfig{ID: "startup-policy", Name: "startup-policy", Enabled: true, URL: server.URL, Service: "generic", Template: `{"instance":"{{.Instance | jsonString}}"}`}+ if err := persistence.SaveWebhooks([]notifications.WebhookConfig{hook}); err != nil {+ t.Fatal(err)+ }+ configJSON, err := json.Marshal(hook)+ if err != nil {+ t.Fatal(err)+ }+ seed, err := notifications.NewNotificationQueue(dir)+ if err != nil {+ t.Fatal(err)+ }+ if err := seed.Enqueue(¬ifications.QueuedNotification{ID: "accepted-before-restart", Type: "webhook", Config: configJSON, Alerts: []*alerts.Alert{{ID: "accepted-policy", Type: "cpu", Level: alerts.AlertLevelWarning, StartTime: time.Now().Add(-time.Minute)}}}); err != nil {+ t.Fatal(err)+ }+ if err := seed.Stop(); err != nil {+ t.Fatal(err)+ }+ m, err := New(&config.Config{DataPath: dir, ConfigPath: dir, PublicURL: tc.configuredURL,+ EnvOverrides: map[string]bool{"publicURL": tc.urlLocked}})+ if err != nil {+ t.Fatal(err)+ }+ defer m.Stop()+ q := m.GetNotificationManager().GetQueue()+ want := "dlq"+ if tc.allowed {+ want = "sent"+ }+ deadline := time.Now().Add(5 * time.Second)+ for {+ stats, err := q.GetQueueStats()+ if err != nil {+ t.Fatal(err)+ }+ if stats[want] == 1 {+ break+ }+ if time.Now().After(deadline) {+ t.Fatalf("BOOTSTRAP_POLICY: accepted row did not reach %s before router construction: %v", want, stats)+ }+ time.Sleep(10 * time.Millisecond)+ }+ if tc.allowed {+ select {+ case got := <-received:+ wantURL := settings.PublicURL+ if tc.urlLocked {+ wantURL = tc.configuredURL+ }+ if got != wantURL {+ t.Fatalf("bootstrap URL=%q, want effective installation URL %q", got, wantURL)+ }+ case <-time.After(time.Second):+ t.Fatal("accepted row has no actual provider receipt")+ }+ } else if len(received) != 0 {+ t.Fatal("tenant policy granted a destination denied by the installation")+ }+ })+ }+}++func TestNewRejectsUnreadableNotificationPolicyBeforeQueueCreation(t *testing.T) {+ for _, bad := range []string{"invalid-json", "invalid-cidr"} {+ t.Run(bad, func(t *testing.T) {+ dir := t.TempDir()+ t.Setenv("PULSE_DATA_DIR", dir)+ persistence := config.NewConfigPersistence(dir)+ if bad == "invalid-json" {+ if err := os.WriteFile(filepath.Join(dir, "system.json"), []byte(`{"broken":`), 0600); err != nil {+ t.Fatal(err)+ }+ } else {+ settings := config.DefaultSystemSettings()+ settings.WebhookAllowedPrivateCIDRs = "not-a-network"+ if err := persistence.SaveSystemSettings(*settings); err != nil {+ t.Fatal(err)+ }+ }+ m, err := New(&config.Config{DataPath: dir})+ if m != nil {+ m.Stop()+ t.Fatal("monitor admitted runtime before notification policy was readable")+ }+ if err == nil || !strings.Contains(err.Error(), "notification") {+ t.Fatalf("missing notification policy error: %v", err)+ }+ if _, err := os.Stat(filepath.Join(dir, "notification_queue.db")); !os.IsNotExist(err) {+ t.Fatalf("policy failure created queue storage: %v", err)+ }+ })+ }+}diff --git a/internal/notifications/manager_lifecycle_test.go b/internal/notifications/manager_lifecycle_test.gonew file mode 100644index 0000000000..5e0c538e90--- /dev/null+++ b/internal/notifications/manager_lifecycle_test.go@@ -0,0 +1,378 @@+package notifications++import (+ "net/http"+ "sync"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/alerts"+ "github.com/stretchr/testify/require"+)++// admissionBarrier holds the first serialisation of the alert that carries it.+// Enqueue serialises an alert's payload after the admission path has read the+// queue and before it writes the row, so the barrier pauses admitted work at+// the point where Stop could otherwise close storage underneath it.+type admissionBarrier struct {+ once sync.Once+ entered chan struct{}+ release chan struct{}+}++func newAdmissionBarrier() *admissionBarrier {+ return &admissionBarrier{entered: make(chan struct{}), release: make(chan struct{})}+}++func (b *admissionBarrier) MarshalJSON() ([]byte, error) {+ b.once.Do(func() {+ close(b.entered)+ <-b.release+ })+ return []byte(`"released"`), nil+}++// shutdownAdmissionFixture is a deferred manager on persistent storage whose+// queue is never activated, so only the work under test takes the queue lock+// while Stop runs. Direct delivery is captured, never run.+type shutdownAdmissionFixture struct {+ t *testing.T+ dir string+ hook WebhookConfig+ spawned chan func()+}++func newShutdownAdmissionFixture(t *testing.T) *shutdownAdmissionFixture {+ t.Helper()+ f := &shutdownAdmissionFixture{+ t: t,+ dir: t.TempDir(),+ hook: WebhookConfig{ID: "ops", URL: "http://127.0.0.1:9/never-contacted", Enabled: true, Service: "generic"},+ spawned: make(chan func(), 8),+ }+ origSpawn := spawnAsync+ spawnAsync = func(deliver func()) { f.spawned <- deliver }+ t.Cleanup(func() { spawnAsync = origSpawn })+ return f+}++func (f *shutdownAdmissionFixture) open(configure func(*NotificationManager)) *NotificationManager {+ f.t.Helper()+ m := NewNotificationManagerWithDeferredQueue("", f.dir)+ f.t.Cleanup(m.Stop)+ // No wall-clock batch may contend for the queue lock in this window.+ m.GetQueue().processorTicker.Stop()+ m.SetGroupingConfig(false, 0, false, false)+ m.AddWebhook(f.hook)+ if configure != nil {+ configure(m)+ }+ return m+}++// stopPassedAdmission reports that Stop has gone as far as it can while+// admitted work is held: it closed storage without waiting for that work, or+// it stopped the queue workers and is waiting with delivery still permitted.+func stopPassedAdmission(m *NotificationManager, q *NotificationQueue) bool {+ if q.db.Ping() != nil {+ return true+ }+ select {+ case <-q.stopChan:+ return m.IsEnabled()+ default:+ return false+ }+}++func awaitShutdownCondition(t *testing.T, what string, condition func() bool) {+ t.Helper()+ deadline := time.Now().Add(10 * time.Second)+ for !condition() {+ if time.Now().After(deadline) {+ t.Fatalf("timed out waiting for %s", what)+ }+ time.Sleep(time.Millisecond)+ }+}++func awaitShutdownSignal(t *testing.T, what string, signal <-chan struct{}) {+ t.Helper()+ select {+ case <-signal:+ case <-time.After(10 * time.Second):+ t.Fatalf("timed out waiting for %s", what)+ }+}++// An alert, recovery or timer group admitted before Stop must be written+// before Stop closes storage, and an attempt made once Stop has begun must be+// refused rather than wait on the queue or fall back to direct delivery. Two+// concurrent Stop calls share one shutdown.+func TestStopWaitsForAdmittedNotificationWrites(t *testing.T) {+ for _, path := range []string{"firing", "recovery", "group-timer"} {+ t.Run(path, func(t *testing.T) {+ f := newShutdownAdmissionFixture(t)+ m := f.open(func(m *NotificationManager) {+ if path == "group-timer" {+ // sendGroupedAlerts is the timer callback; a long window keeps+ // the real timer from also running it.+ m.SetGroupingConfig(true, 3600, false, false)+ }+ })+ q := m.GetQueue()+ barrier := newAdmissionBarrier()+ alert := quietReplayAlert("admitted-"+path, alerts.AlertLevelWarning, time.Now().UTC().Add(-time.Minute))+ alert.Metadata = map[string]interface{}{"barrier": barrier}++ admitted := make(chan struct{})+ switch path {+ case "firing":+ go func() {+ defer close(admitted)+ m.SendAlert(alert)+ }()+ case "recovery":+ m.recordSuccessfulDelivery(notificationDeliveryJob{Type: "webhook", Event: eventAlert,+ Alerts: []*alerts.Alert{alert}, WebhookConfig: &f.hook}, time.Now())+ go func() {+ defer close(admitted)+ m.SendResolvedAlert(&alerts.ResolvedAlert{Alert: alert, ResolvedTime: time.Now()})+ }()+ case "group-timer":+ m.SendAlert(alert)+ go func() {+ defer close(admitted)+ m.sendGroupedAlerts()+ }()+ }+ awaitShutdownSignal(t, "admitted work to reach queue serialisation", barrier.entered)++ stopped := make(chan struct{}, 2)+ for range 2 {+ go func() {+ m.Stop()+ stopped <- struct{}{}+ }()+ }+ awaitShutdownCondition(t, "Stop to pass the admitted work", func() bool { return stopPassedAdmission(m, q) })++ late := quietReplayAlert("late-"+path, alerts.AlertLevelWarning, time.Now().UTC())+ refused := make(chan struct{})+ go func() {+ defer close(refused)+ m.SendAlert(late)+ m.SendResolvedAlert(&alerts.ResolvedAlert{Alert: late, ResolvedTime: time.Now()})+ }()+ awaitShutdownSignal(t, "an attempt made during Stop to be refused", refused)++ close(barrier.release)+ awaitShutdownSignal(t, "admitted work to finish", admitted)+ for range 2 {+ select {+ case <-stopped:+ case <-time.After(10 * time.Second):+ t.Fatal("Stop did not finish after admitted work completed")+ }+ }+ select {+ case <-f.spawned:+ t.Errorf("SHUTDOWN_ADMISSION_DIRECT_SEND: %s admitted before Stop could not be written because Stop closed the queue under it, and went to direct delivery after shutdown", path)+ default:+ }++ rows := quietReplayRows(t, f.open(nil).GetQueue())+ if len(rows) != 1 || len(rows[0].Alerts) != 1 || rows[0].Alerts[0].ID != alert.ID ||+ rows[0].Status != QueueStatusPending || rows[0].Attempts != 0 {+ t.Fatalf("SHUTDOWN_ADMISSION_LOST: %s admitted before Stop is not pending after restart: %+v", path, rows)+ }+ })+ }+}++// Operator pause and channel disable keep their meaning while Stop drains:+// they cancel queued work, including an alert admitted just before Stop.+func TestStopDrainKeepsPauseAndChannelDisable(t *testing.T) {+ email := EmailConfig{Enabled: true, SMTPHost: "127.0.0.1", SMTPPort: 9,+ From: "[email protected]", To: []string{"[email protected]"}}+ for _, tc := range []struct {+ name string+ configure func(*NotificationManager)+ act func(*NotificationManager)+ visible func(*NotificationManager) bool+ reason string+ }{+ {+ name: "pause",+ act: func(m *NotificationManager) { m.SetEnabled(false) },+ visible: func(m *NotificationManager) bool { return !m.IsEnabled() },+ reason: "Notifications disabled",+ },+ {+ name: "email-disabled",+ configure: func(m *NotificationManager) {+ m.SetEmailConfig(email)+ _ = m.DeleteWebhook("ops")+ },+ act: func(m *NotificationManager) {+ disabled := email+ disabled.Enabled = false+ m.SetEmailConfig(disabled)+ },+ visible: func(m *NotificationManager) bool { return !m.GetEmailConfig().Enabled },+ reason: "Email notifications disabled",+ },+ } {+ t.Run(tc.name, func(t *testing.T) {+ f := newShutdownAdmissionFixture(t)+ m := f.open(tc.configure)+ q := m.GetQueue()+ barrier := newAdmissionBarrier()+ alert := quietReplayAlert("paused-"+tc.name, alerts.AlertLevelWarning, time.Now().UTC().Add(-time.Minute))+ alert.Metadata = map[string]interface{}{"barrier": barrier}+ admitted := make(chan struct{})+ go func() {+ defer close(admitted)+ m.SendAlert(alert)+ }()+ awaitShutdownSignal(t, "admitted work to reach queue serialisation", barrier.entered)++ stopped := make(chan struct{})+ go func() {+ m.Stop()+ close(stopped)+ }()+ awaitShutdownCondition(t, "Stop to pass the admitted work", func() bool { return stopPassedAdmission(m, q) })++ // The change takes its lease in the critical section that publishes+ // it, so once it is visible its cancellation is owed to the queue.+ acted := make(chan struct{})+ go func() {+ defer close(acted)+ tc.act(m)+ }()+ awaitShutdownCondition(t, tc.name+" to take effect", func() bool { return tc.visible(m) })+ close(barrier.release)+ awaitShutdownSignal(t, "admitted work to finish", admitted)+ awaitShutdownSignal(t, tc.name+" to finish", acted)+ awaitShutdownSignal(t, "Stop to finish", stopped)++ rows := quietReplayRows(t, f.open(nil).GetQueue())+ if len(rows) != 1 || rows[0].Status != QueueStatusCancelled ||+ rows[0].LastError == nil || *rows[0].LastError != tc.reason {+ t.Fatalf("SHUTDOWN_POLICY_LOST: %s during the drain did not cancel the alert admitted before Stop: %+v", tc.name, rows)+ }+ })+ }+}++// A deferred queue that never started keeps admitted rows for the next start,+// and Stop still discards a firing group that was only buffered in memory.+func TestStopNeverStartedQueueAndBufferedGroup(t *testing.T) {+ f := newShutdownAdmissionFixture(t)+ m := f.open(nil)+ start := time.Now().UTC().Add(-time.Minute)+ m.SendAlert(quietReplayAlert("queued-1", alerts.AlertLevelWarning, start))+ m.SendAlert(quietReplayAlert("queued-2", alerts.AlertLevelWarning, start))+ m.SetGroupingConfig(true, 3600, false, false)+ m.SendAlert(quietReplayAlert("buffered", alerts.AlertLevelWarning, start))++ stopped := make(chan struct{})+ go func() {+ m.Stop()+ close(stopped)+ }()+ awaitShutdownSignal(t, "Stop of a never-started queue", stopped)+ // A timer that runs after Stop finds nothing to admit.+ m.sendGroupedAlerts()++ select {+ case <-f.spawned:+ t.Fatal("stopped manager handed work to direct delivery")+ default:+ }+ statuses := map[string]NotificationQueueStatus{}+ rows := quietReplayRows(t, f.open(nil).GetQueue())+ for _, row := range rows {+ if row.Attempts != 0 {+ t.Errorf("never-started queue consumed an attempt: %+v", row)+ }+ for _, alert := range row.Alerts {+ statuses[alert.ID] = row.Status+ }+ }+ if len(rows) != 2 || statuses["queued-1"] != QueueStatusPending || statuses["queued-2"] != QueueStatusPending {+ t.Fatalf("never-started queue lost admitted rows across Stop: %+v", rows)+ }+ if _, persisted := statuses["buffered"]; persisted {+ t.Fatal("Stop persisted a buffered firing group; buffered groups are discarded at shutdown")+ }+}++// Direct delivery started before Stop is owned: Stop waits for it, so its+// firing receipt is durable and the recovery still reaches the destination+// after restart. Direct delivery cannot start once Stop has finished.+func TestStopOwnsDirectFallbackDelivery(t *testing.T) {+ received := make(chan struct{}, 4)+ release := make(chan struct{})+ server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ received <- struct{}{}+ <-release+ w.WriteHeader(http.StatusOK)+ }))+ t.Cleanup(server.Close)+ var releaseOnce sync.Once+ releaseDelivery := func() { releaseOnce.Do(func() { close(release) }) }++ dir := t.TempDir()+ hook := WebhookConfig{ID: "ops", URL: server.URL + "/direct", Enabled: true, Service: "generic"}+ open := func() *NotificationManager {+ m := NewNotificationManagerWithDeferredQueue("", dir)+ t.Cleanup(m.Stop)+ m.webhookClient = server.Client()+ require.NoError(t, m.UpdateAllowedPrivateCIDRs("127.0.0.1/32"))+ m.SetGroupingConfig(false, 0, false, false)+ m.AddWebhook(hook)+ return m+ }+ m := open()+ // Registered after the manager, so a held delivery is released before Stop.+ t.Cleanup(releaseDelivery)+ q := m.GetQueue()+ // A storage failure, not shutdown, sends this alert down the direct path.+ _, err := q.db.Exec(`CREATE TRIGGER reject_admission BEFORE INSERT ON notification_queue+ BEGIN SELECT RAISE(ABORT, 'fixture queue write failed'); END`)+ require.NoError(t, err)++ alert := quietReplayAlert("direct", alerts.AlertLevelWarning, time.Now().UTC().Add(-time.Minute))+ m.SendAlert(alert)+ select {+ case <-received:+ case <-time.After(10 * time.Second):+ t.Fatal("enqueue failure did not start direct delivery")+ }++ stopped := make(chan struct{})+ go func() {+ m.Stop()+ close(stopped)+ }()+ awaitShutdownCondition(t, "Stop to pass the direct delivery", func() bool { return stopPassedAdmission(m, q) })+ releaseDelivery()+ awaitShutdownSignal(t, "Stop to finish", stopped)++ spawned := 0+ origSpawn := spawnAsync+ spawnAsync = func(func()) { spawned++ }+ m.dispatchNotificationJobAsync(notificationDeliveryJob{Type: "webhook", Event: eventAlert,+ Alerts: []*alerts.Alert{alert}, WebhookConfig: &hook}, "direct delivery after Stop")+ spawnAsync = origSpawn+ if spawned != 0 {+ t.Error("FALLBACK_AFTER_SHUTDOWN: direct delivery started after Stop returned")+ }++ jobs := buildNotificationDeliveryJobs(EmailConfig{}, []WebhookConfig{hook}, AppriseConfig{}, []*alerts.Alert{alert}, eventResolved, time.Now())+ if len(open().filterResolvedJobsByDeliveryReceipt(jobs)) != 1 {+ t.Fatal("FALLBACK_RECEIPT_LOST: direct delivery accepted during shutdown left no durable firing receipt, so its recovery is suppressed after restart")+ }+}diff --git a/internal/notifications/notifications.go b/internal/notifications/notifications.goindex fbb433d2ee..6a7b5d713c 100644--- a/internal/notifications/notifications.go+++ b/internal/notifications/notifications.go@@ -221,11 +221,92 @@ type webhookRateLimit struct {sentCount int}+type notificationLifecycleState int++const (+ notificationLifecycleRunning notificationLifecycleState = iota+ notificationLifecycleDraining+ notificationLifecycleClosed+)++// notificationLifecycle owns work that holds the queue outside the manager+// lock. Each lease is taken in the critical section that reads the queue, so+// a lease holder always writes to an open queue. Stop drains the lifecycle,+// refusing new alerts while cancellations, operator pause and direct delivery+// already owed continue, then closes it once every lease has ended.+type notificationLifecycle struct {+ mu sync.Mutex+ idle sync.Cond+ state notificationLifecycleState+ leases int+}++// admit leases new alert or recovery work. Stop refuses it from the start.+func (l *notificationLifecycle) admit() bool {+ return l.acquire(notificationLifecycleRunning)+}++// join leases work that must still reach an open queue while Stop drains:+// cancellation, operator pause or channel disable, and direct delivery+// started by admitted work. It is refused only once the drain has finished.+func (l *notificationLifecycle) join() bool {+ return l.acquire(notificationLifecycleDraining)+}++func (l *notificationLifecycle) acquire(latest notificationLifecycleState) bool {+ l.mu.Lock()+ defer l.mu.Unlock()+ if l.state > latest {+ return false+ }+ l.leases+++ return true+}++func (l *notificationLifecycle) release() {+ l.mu.Lock()+ l.leases--+ if l.leases == 0 {+ l.idle.Broadcast()+ }+ l.mu.Unlock()+}++// accepting reports whether new work may still be buffered for later+// admission.+func (l *notificationLifecycle) accepting() bool {+ l.mu.Lock()+ defer l.mu.Unlock()+ return l.state == notificationLifecycleRunning+}++func (l *notificationLifecycle) drain() {+ l.mu.Lock()+ if l.state == notificationLifecycleRunning {+ l.state = notificationLifecycleDraining+ }+ l.mu.Unlock()+}++// close waits for every lease to end, then refuses all later work.+func (l *notificationLifecycle) close() {+ l.mu.Lock()+ defer l.mu.Unlock()+ if l.idle.L == nil {+ l.idle.L = &l.mu+ }+ for l.leases > 0 {+ l.idle.Wait()+ }+ l.state = notificationLifecycleClosed+}+// NotificationManager handles sending notificationstype NotificationManager struct {mu sync.RWMutexstopOnce sync.OncecleanupWG sync.WaitGroup+ lifecycle notificationLifecycle // Admission and shutdown ownership of queue and direct delivery workemailConfig EmailConfigemailManager *EnhancedEmailManager // Shared email manager for rate limitingwebhooks []WebhookConfig@@ -936,11 +1017,14 @@ func (n *NotificationManager) SetEmailConfig(config EmailConfig) {AuthRequired: config.Username != "" && config.Password != "",}n.emailManager = NewEnhancedEmailManager(providerConfig)- queue := n.queue- disabled := !config.Enabled+ var queue *NotificationQueue+ if !config.Enabled && n.queue != nil && n.lifecycle.join() {+ queue = n.queue+ }n.mu.Unlock()- if disabled && queue != nil {+ if queue != nil {+ defer n.lifecycle.release()if err := queue.CancelByTypes([]string{"email", "email" + queueTypeSuffixResolved}, "Email notifications disabled"); err != nil {log.Error().Err(err).Msg("failed to cancel queued email notifications after disabling email delivery")}@@ -951,11 +1035,14 @@ func (n *NotificationManager) SetEmailConfig(config EmailConfig) {func (n *NotificationManager) SetAppriseConfig(config AppriseConfig) {n.mu.Lock()n.appriseConfig = NormalizeAppriseConfig(config)- queue := n.queue- disabled := !n.appriseConfig.Enabled+ var queue *NotificationQueue+ if !n.appriseConfig.Enabled && n.queue != nil && n.lifecycle.join() {+ queue = n.queue+ }n.mu.Unlock()- if disabled && queue != nil {+ if queue != nil {+ defer n.lifecycle.release()if err := queue.CancelByTypes([]string{"apprise", "apprise" + queueTypeSuffixResolved}, "Apprise notifications disabled"); err != nil {log.Error().Err(err).Msg("failed to cancel queued Apprise notifications after disabling delivery")}@@ -1058,7 +1145,11 @@ func (n *NotificationManager) SetGroupingConfig(enabled bool, seconds int, byNodvar queue *NotificationQueueif !enabled || seconds == 0 {resolved = n.takePendingResolvedLocked()- pending = append(pending, n.pendingAlerts...)+ // Stop clears pending groups in the critical section that refuses+ // admission, so a pending group here is always admitted.+ if len(n.pendingAlerts) > 0 && n.lifecycle.admit() {+ pending = append(pending, n.pendingAlerts...)+ }clear(n.pendingAlerts)n.pendingAlerts = n.pendingAlerts[:0]n.pendingAlertIndex = nil@@ -1073,6 +1164,9 @@ func (n *NotificationManager) SetGroupingConfig(enabled bool, seconds int, byNodqueue = n.queue}n.mu.Unlock()+ if len(pending) > 0 {+ defer n.lifecycle.release()+ }for _, alert := range resolved {n.dispatchResolvedAlerts([]*alerts.Alert{alert})@@ -1179,7 +1273,9 @@ func (n *NotificationManager) SetEnabled(enabled bool) {n.groupTimer.Stop()n.groupTimer = nil}- queue = n.queue+ if n.queue != nil && n.lifecycle.join() {+ queue = n.queue+ }}n.mu.Unlock()@@ -1187,7 +1283,8 @@ func (n *NotificationManager) SetEnabled(enabled bool) {log.Info().Bool("enabled", enabled).Msg("updated notification manager enabled state")}- if !enabled && queue != nil {+ if queue != nil {+ defer n.lifecycle.release()types := []string{"email", "email" + queueTypeSuffixResolved,"webhook", "webhook" + queueTypeSuffixResolved,@@ -1259,6 +1356,14 @@ func (n *NotificationManager) sendAlert(alert *alerts.Alert, options alertSendOpn.mu.Unlock()return}+ // The lease covers the queue read below until its write completes, so Stop+ // cannot close storage under an admitted alert.+ if !n.lifecycle.admit() {+ log.Debug().Str("alertID", alert.ID).Msg("notification manager stopping, alert not admitted")+ n.mu.Unlock()+ return+ }+ defer n.lifecycle.release()// Check cooldown. A disabled cooldown means "do not send repeat// notifications for the same active alert occurrence". A higher severity is@@ -1623,7 +1728,7 @@ func (n *NotificationManager) SendResolvedAlert(resolved *alerts.ResolvedAlert)annotateResolvedMetadata(alertCopy, resolvedAt)n.mu.Lock()- if !n.enabled || !n.notifyOnResolve {+ if !n.enabled || !n.notifyOnResolve || !n.lifecycle.accepting() {n.mu.Unlock()return}@@ -1674,10 +1779,13 @@ func (n *NotificationManager) dispatchResolvedAlerts(alertList []*alerts.Alert)webhooks := copyWebhookConfigs(n.webhooks)appriseConfig := copyAppriseConfig(n.appriseConfig)queue := n.queue+ // The lease precedes the receipt read below, which already uses the queue.+ admitted := enabled && n.lifecycle.admit()n.mu.RUnlock()- if !enabled {+ if !admitted {return}+ defer n.lifecycle.release()jobs := buildNotificationDeliveryJobs(emailConfig, webhooks, appriseConfig, alertList, eventResolved, resolvedTimeFromAlerts(alertList))// Filter each destination independently at dispatch, not merely when the// alert enters the window. Never announce a recovery for an undelivered firing.@@ -1711,6 +1819,11 @@ func (n *NotificationManager) CancelResolvedAlert(alert *alerts.Alert) bool {func (n *NotificationManager) cancelAlert(alertID string, start *time.Time) bool {n.mu.Lock()queue := n.queue+ // Cancellation still runs while Stop drains; its lease keeps the queue+ // open until the cancellation is committed.+ if queue != nil && !n.lifecycle.join() {+ queue = nil+ }record, firingDelivered := n.lastNotified[alertID]firingDelivered = firingDelivered && (start == nil || record.alertStart.Equal(*start))@@ -1755,6 +1868,7 @@ func (n *NotificationManager) cancelAlert(alertID string, start *time.Time) bool// Cancel any queued notifications containing this alertcancelledPending := 0if queue != nil {+ defer n.lifecycle.release()var err errorif start == nil {cancelledPending, err = queue.CancelByAlertIdentifiers([]string{alertID})@@ -1784,6 +1898,13 @@ func (n *NotificationManager) sendGroupedAlerts() {n.mu.Unlock()return}+ // Stop clears pending groups in the critical section that refuses+ // admission, so a timer that finds a group here is always admitted.+ if !n.lifecycle.admit() {+ n.mu.Unlock()+ return+ }+ defer n.lifecycle.release()// Copy alerts to sendalertsToSend := make([]*alerts.Alert, len(n.pendingAlerts))@@ -2107,7 +2228,18 @@ func (n *NotificationManager) sendResolvedNotificationsDirect(emailConfig EmailC}func (n *NotificationManager) dispatchNotificationJobAsync(job notificationDeliveryJob, failureMessage string) {+ // Direct delivery stays owned until it finishes, so Stop waits for it and+ // nothing is sent, or recorded only in memory, after Stop returns.+ if !n.lifecycle.join() {+ log.Warn().+ Str("type", job.Type).+ Str("event", string(job.Event)).+ Int("alertCount", len(job.Alerts)).+ Msg("notification manager stopped; direct delivery refused")+ return+ }spawnAsync(func() {+ defer n.lifecycle.release()if err := n.deliverNotificationJob(job); err != nil {n.logNotificationJobError(job, err, failureMessage)return@@ -4402,8 +4534,11 @@ func (n *NotificationManager) cleanupOldNotificationRecords() {// Stop gracefully stops the notification managerfunc (n *NotificationManager) Stop() {n.stopOnce.Do(func() {+ // Refuse new alerts and discard volatile groups in one critical+ // section, so a pending group is either already admitted with a lease+ // or discarded here, as Stop has always discarded it.n.mu.Lock()- n.enabled = false+ n.lifecycle.drain()n.takePendingResolvedLocked()clear(n.pendingAlerts)n.pendingAlerts = n.pendingAlerts[:0]@@ -4413,6 +4548,23 @@ func (n *NotificationManager) Stop() {n.groupTimer = nil}queue := n.queue+ n.mu.Unlock()++ // Shutdown is a restart boundary, not a delivery verdict. Quiesce the+ // queue while this manager still permits delivery and records receipts+ // durably: a queued job reaching the processor after delivery is+ // withdrawn below would be cancelled as disabled, and a send completing+ // after the queue is detached would keep its firing receipt only in+ // memory. Rows no worker has begun stay pending for the next start.+ if queue != nil {+ queue.stopProcessing()+ }+ // Wait for admitted writes, cancellations, operator changes and direct+ // deliveries. No lease can be taken once this returns.+ n.lifecycle.close()++ n.mu.Lock()+ n.enabled = falsecleanupDone := n.cleanupDoneclient := n.webhookClientdiff --git a/internal/notifications/notifications_test.go b/internal/notifications/notifications_test.goindex 589b1e6199..ba11741978 100644--- a/internal/notifications/notifications_test.go+++ b/internal/notifications/notifications_test.go@@ -4846,8 +4846,12 @@ func TestGetWebhooksReturnsDeepCopies(t *testing.T) {}func TestSendGroupedAlertsQueueEnqueueFailureDoesNotDeadlock(t *testing.T) {+ // Hold the fallback rather than run it: the cooldown check below needs it+ // not to have delivered. Stop waits for direct delivery it owns, so the+ // held fallback completes against an in-memory SMTP server before Stop.+ var held []func()origSpawn := spawnAsync- spawnAsync = func(func()) {}+ spawnAsync = func(f func()) { held = append(held, f) }t.Cleanup(func() { spawnAsync = origSpawn })nm := NewNotificationManager("")@@ -4901,6 +4905,13 @@ func TestSendGroupedAlertsQueueEnqueueFailureDoesNotDeadlock(t *testing.T) {if notified {t.Fatal("did not expect a cooldown record when fallback delivery never ran")}+ if len(held) == 0 {+ t.Fatal("enqueue failure did not hand the group to direct delivery")+ }+ stubSMTPDialSuccess(t, nil)+ for _, deliver := range held {+ deliver()+ }}func TestNormalizeWebhookConfig_CanonicalizesPushoverCustomFields(t *testing.T) {diff --git a/internal/notifications/queue.go b/internal/notifications/queue.goindex 00bc9d1a14..2eef8fbc79 100644--- a/internal/notifications/queue.go+++ b/internal/notifications/queue.go@@ -179,6 +179,7 @@ type NotificationQueue struct {deliveryGateMu sync.MutexdeliveryGates map[string]*notificationDeliveryGatestopOnce sync.Once+ stopProcessingOnce sync.OncestopErr errordb *sql.DBdbPath string@@ -1791,6 +1792,12 @@ func (nq *NotificationQueue) processNotification(notif *QueuedNotification) {nq.notifyDeliveryHealthChanged()}}()+ // A worker that observes the stop begins no hold, partition or claim, so+ // its row stays pending for the next start. A worker already past this+ // check finishes normally; stopProcessing waits for it.+ if nq.stopping() {+ return+ }if quietHoursPolicy != nil {ready, err := nq.prepareQuietHoursDelivery(notif, quietHoursPolicy)if err != nil {@@ -2119,14 +2126,37 @@ func (nq *NotificationQueue) performCleanup() {}}-// Stop gracefully stops the queue processor-func (nq *NotificationQueue) Stop() error {- nq.stopOnce.Do(func() {+// stopping reports whether processing has begun to stop. Queues built+// without a stop channel are never stopping.+func (nq *NotificationQueue) stopping() bool {+ select {+ case <-nq.stopChan:+ return true+ default:+ return false+ }+}++// stopProcessing ends delivery work but keeps storage open. Workers that+// observe the stop leave their rows pending for the next start; a worker+// already past that check finishes its hold, partition or attempt and records+// the outcome before this returns. Stop calls it, and an owner that must+// withdraw delivery permission calls it first so no worker sees that change.+func (nq *NotificationQueue) stopProcessing() {+ nq.stopProcessingOnce.Do(func() {close(nq.stopChan)+ log.Info().Str("component", "notification_queue").Msg("Stopping notification queue processing")nq.wg.Wait()nq.processorTicker.Stop()nq.cleanupTicker.Stop()+ })+}++// Stop gracefully stops the queue processor+func (nq *NotificationQueue) Stop() error {+ nq.stopOnce.Do(func() {+ nq.stopProcessing()if err := nq.db.Close(); err != nil {nq.stopErr = fmt.Errorf("failed to close database: %w", err)diff --git a/internal/notifications/queue_shutdown_test.go b/internal/notifications/queue_shutdown_test.gonew file mode 100644index 0000000000..3fe793d229--- /dev/null+++ b/internal/notifications/queue_shutdown_test.go@@ -0,0 +1,187 @@+package notifications++import (+ "encoding/json"+ "fmt"+ "net/http"+ "sort"+ "sync"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/alerts"+ "github.com/stretchr/testify/require"+)++// Shutdown is a restart boundary, not an operator decision to stop delivery.+// Five real sends occupy every queue worker while a sixth admitted job waits+// for a slot, which is where shutdown finds an ordinary backlog. The waiting+// job must survive for the next start rather than be cancelled as "delivery+// disabled", and sends completing during shutdown must keep durable firing+// receipts so their recoveries are still delivered after restart.+func TestManagerStopKeepsQueuedDeliveryForRestart(t *testing.T) {+ type message struct {+ Event string `json:"event"`+ Alerts []*alerts.Alert `json:"alerts"`+ }+ received := make(chan message, 16)+ release := make(chan struct{})+ server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ defer r.Body.Close()+ var msg message+ if err := json.NewDecoder(r.Body).Decode(&msg); err != nil || len(msg.Alerts) != 1 {+ t.Errorf("decode delivery: %v (%d alerts)", err, len(msg.Alerts))+ w.WriteHeader(http.StatusBadRequest)+ return+ }+ select {+ case received <- msg:+ default:+ t.Error("unexpected excess deliveries")+ }+ <-release+ w.WriteHeader(http.StatusOK)+ }))+ t.Cleanup(server.Close)+ var releaseOnce sync.Once+ releaseSends := func() { releaseOnce.Do(func() { close(release) }) }++ dir := t.TempDir()+ hook := WebhookConfig{ID: "ops", URL: server.URL, Enabled: true, Service: "generic"}+ open := func() *NotificationManager {+ m := NewNotificationManagerWithDeferredQueue("", dir)+ t.Cleanup(m.Stop)+ m.webhookClient = server.Client()+ require.NoError(t, m.UpdateAllowedPrivateCIDRs("127.0.0.1/32"))+ m.SetGroupingConfig(false, 0, false, false)+ m.AddWebhook(hook)+ return m+ }+ m := open()+ // Registered after the manager, so held sends are released before its Stop.+ t.Cleanup(releaseSends)++ start := time.Now().UTC().Add(-time.Minute)+ admitted := map[string]*alerts.Alert{}+ for i := range 6 {+ alert := quietReplayAlert(fmt.Sprintf("shutdown-%d", i), alerts.AlertLevelWarning, start)+ admitted[alert.ID] = alert+ m.SendAlert(alert)+ }+ pending, err := m.GetQueue().GetPending(20)+ require.NoError(t, err)+ require.Len(t, pending, 6, "every job must be admitted before workers start")++ q := m.GetQueue()+ m.StartQueueProcessing()+ inFlight := map[string]bool{}+ for range 5 {+ select {+ case msg := <-received:+ inFlight[msg.Alerts[0].ID] = true+ case <-time.After(10 * time.Second):+ t.Fatal("five queued sends did not occupy the worker pool")+ }+ }++ stopped := make(chan struct{})+ go func() {+ m.Stop()+ close(stopped)+ }()+ // Either half of shutdown may come first; release the held sends only+ // once the manager has begun to stop.+ stopBegun := func() bool {+ select {+ case <-q.stopChan:+ return true+ default:+ return !m.IsEnabled()+ }+ }+ deadline := time.Now().Add(10 * time.Second)+ for !stopBegun() {+ if time.Now().After(deadline) {+ t.Fatal("manager shutdown did not begin")+ }+ time.Sleep(time.Millisecond)+ }+ releaseSends()+ select {+ case <-stopped:+ case <-time.After(10 * time.Second):+ t.Fatal("manager shutdown did not finish its in-flight sends")+ }+ select {+ case extra := <-received:+ t.Fatalf("shutdown started a new provider attempt: %+v", extra)+ default:+ }++ m = open()+ q = m.GetQueue()+ var waiting *QueuedNotification+ for _, row := range quietReplayRows(t, q) {+ require.Len(t, row.Alerts, 1)+ if inFlight[row.Alerts[0].ID] {+ require.Equal(t, QueueStatusSent, row.Status, "a send completed during shutdown keeps its outcome")+ require.Equal(t, 1, row.Attempts)+ continue+ }+ require.Nil(t, waiting, "exactly one job waited for a worker")+ waiting = row+ }+ require.NotNil(t, waiting)++ delivered := make([]string, 0, len(inFlight))+ for id := range inFlight {+ delivered = append(delivered, id)+ }+ sort.Strings(delivered)+ for _, id := range delivered {+ jobs := buildNotificationDeliveryJobs(EmailConfig{}, []WebhookConfig{hook}, AppriseConfig{}, []*alerts.Alert{admitted[id]}, eventResolved, time.Now())+ if len(m.filterResolvedJobsByDeliveryReceipt(jobs)) != 1 {+ t.Errorf("SHUTDOWN_RECEIPT_LOST: %s reached the receiver during shutdown, but its recovery is suppressed after restart", id)+ }+ }+ if waiting.Status != QueueStatusPending || waiting.Attempts != 0 {+ lastError := ""+ if waiting.LastError != nil {+ lastError = *waiting.LastError+ }+ t.Fatalf("SHUTDOWN_CANCELLED_UNCLAIMED: job for %s that waited for a worker became %s after %d attempt(s) (%q); it must stay pending for the next start",+ waiting.Alerts[0].ID, waiting.Status, waiting.Attempts, lastError)+ }++ m.StartQueueProcessing()+ select {+ case msg := <-received:+ require.Equal(t, waiting.Alerts[0].ID, msg.Alerts[0].ID)+ case <-time.After(10 * time.Second):+ t.Fatal("the job kept across shutdown was not delivered after restart")+ }+ deadline = time.Now().Add(10 * time.Second)+ for loadQuietReplay(t, q, waiting.ID).Status != QueueStatusSent {+ if time.Now().After(deadline) {+ t.Fatal("the job kept across shutdown was not recorded as sent")+ }+ time.Sleep(time.Millisecond)+ }++ // The user-visible end of a receipt: a recovery after restart still+ // reaches the destination that received the firing during shutdown.+ recovered := delivered[0]+ m.SendResolvedAlert(&alerts.ResolvedAlert{Alert: admitted[recovered], ResolvedTime: time.Now()})+ select {+ case msg := <-received:+ require.Equal(t, "resolved", msg.Event)+ require.Equal(t, recovered, msg.Alerts[0].ID)+ case <-time.After(10 * time.Second):+ t.Fatalf("recovery for %s was not delivered after restart", recovered)+ }+ select {+ case extra := <-received:+ t.Fatalf("duplicate provider message: %+v", extra)+ default:+ }+}