Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

fix(notifications): preserve queued work through restart

rcourtman5d11f80d7d67
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 489f7d34dd..7fe16d2ddd 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -10769,3 +10769,21 @@ settings remain live and unchanged. This is not a backup, credential recovery,
agent/guest probe or installed recovery. The normal HTTP contract
`TestContractSupportBundleCredentialBoundary` and `log_bundle_test.go` cover the
summary, scope denials, state preservation and configured-file/buffer fallbacks.
+
+### Notification policy precedes persistent queue activation
+
+`monitoring.New` reads notification system settings from the shared installation
+directory before creating notification workers. Tenant-local settings cannot
+grant private destinations or replace the installation public URL. The
+constructor validates the private CIDR policy, restores the saved public URL
+while preserving a deployment-enforced URL, and installs the policy before activating persisted work. Missing
+settings keep the default private-destination denial. Unreadable settings or an
+invalid CIDR policy fail construction before queue creation, so accepted rows
+cannot acquire a terminal failure merely because bootstrap has not restored
+policy yet. Router settings refresh remains responsible for later live changes.
+
+`TestNewRestoresSharedWebhookPolicyBeforeQueueActivation` covers a real pending
+HTTP delivery before any router exists for default and tenant paths, saved
+public URL precedence and deployment lock, and a tenant grant that cannot override installation revocation.
+`TestNewRejectsUnreadableNotificationPolicyBeforeQueueCreation` covers corrupt
+settings and invalid CIDRs without creating queue storage.
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 4dda4b8827..4166077c47 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -9271,3 +9271,21 @@ alert engine emits them; every other projection uses `RecordChange` and stays
silent. Both paths write the same deterministic change row. Proof:
`TestAlertTimelineAnnouncesOnlyLiveLifecycleChanges` in
`internal/monitoring/monitor_alert_handling_test.go`.
+
+### Notification policy precedes persistent queue activation
+
+`monitoring.New` reads notification system settings from the shared installation
+directory before creating notification workers. Tenant-local settings cannot
+grant private destinations or replace the installation public URL. The
+constructor validates the private CIDR policy, restores the saved public URL
+while preserving a deployment-enforced URL, and installs the policy before activating persisted work. Missing
+settings keep the default private-destination denial. Unreadable settings or an
+invalid CIDR policy fail construction before queue creation, so accepted rows
+cannot acquire a terminal failure merely because bootstrap has not restored
+policy yet. Router settings refresh remains responsible for later live changes.
+
+`TestNewRestoresSharedWebhookPolicyBeforeQueueActivation` covers a real pending
+HTTP delivery before any router exists for default and tenant paths, saved
+public URL precedence and deployment lock, and a tenant grant that cannot override installation revocation.
+`TestNewRejectsUnreadableNotificationPolicyBeforeQueueCreation` covers corrupt
+settings and invalid CIDRs without creating queue storage.
diff --git a/docs/release-control/v6/internal/subsystems/notifications.md b/docs/release-control/v6/internal/subsystems/notifications.md
index 464354aa40..dbf5478dc1 100644
--- a/docs/release-control/v6/internal/subsystems/notifications.md
+++ b/docs/release-control/v6/internal/subsystems/notifications.md
@@ -348,6 +348,54 @@ notifier, replaces the router monitor, and exercises stats, Retry and Dismiss;
replacement and unavailable-manager reads with the race detector. This is
lifecycle regression proof, not installed SMTP receipt or reporter confirmation.
+### Manager shutdown is a restart boundary, not a delivery verdict
+
+`NotificationManager.Stop` first refuses new alerts and discards volatile
+firing and recovery groups in one critical section, as Stop has always
+discarded them. Work that reads the queue outside the manager lock holds a
+lease taken in the critical section that reads it. An admitted alert, recovery
+or timer group is written to the open queue. Cancellation, operator pause and
+channel disable still run while Stop drains. Direct delivery started by
+admitted work is owned until it finishes and records its receipt. An attempt
+made after Stop begins is refused, neither queued nor sent directly, and no
+direct delivery starts once Stop has finished.
+
+Stop then quiesces the queue. A worker that observes the stop begins no
+quiet-hours hold, partition or claim: its job stays `pending` with no attempt,
+audit or receipt, and the next start delivers it. Work a worker began before
+that observation finishes while delivery is still permitted, recording its
+status, audit and durable firing receipt. Only after every worker and lease
+has ended does Stop withdraw delivery, detach the queue and close storage. A
+recovery after restart therefore still reaches each destination that received
+the firing. Shutdown must never cancel queued work as "delivery disabled";
+that verdict belongs to the pause and destination controls. Resolution
+cancellation and at-least-once replay of interrupted sends are unchanged.
+Delivery attempts retain their existing provider timeouts and SMTP retry
+budget. Stop waits for their durable outcomes instead of closing storage
+under an active writer. There is no separate aggregate drain deadline. The
+queue logs the processing stop after refusing new attempts and before waiting,
+so process-signal proofs can release held provider responses at that boundary.
+
+`TestManagerStopKeepsQueuedDeliveryForRestart` holds five real HTTP sends in
+the worker pool while a sixth admitted job waits for a slot, stops the manager,
+reopens the persistent queue, and checks the kept job, the in-flight outcomes,
+their durable receipts and a recovery delivered after restart.
+`manager_lifecycle_test.go` holds an admitted alert, recovery or timer group at
+queue serialisation while two concurrent Stop calls run. It checks the write
+after reopen, the refusal of a later attempt, and pause and channel disable
+during the drain. It also checks a never-started queue, the discarded buffered
+group, and direct delivery that Stop waits for and refuses afterwards. These
+are loopback source proofs, not process-signal, reload or installed-provider
+acceptance.
+
+Monitor bootstrap also restores the shared installation webhook policy before
+activating persistent work, including tenant queues. A later router refresh is
+too late for a due private-webhook job. Missing settings retain default denial;
+unreadable settings or invalid CIDRs fail construction before queue creation.
+The monitor startup tests cover delivery with no router, installation revocation
+against tenant-local grants, and invalid-policy admission. This boundary keeps
+shutdown-preserved rows from becoming terminal failures during reconstruction.
+
Own notification delivery transport, provider configuration, queueing, and
notification-management API surfaces.
diff --git a/internal/api/router_webhook_cidr_reload_test.go b/internal/api/router_webhook_cidr_reload_test.go
index 5b8ebf0a54..0b78eb9e7a 100644
--- a/internal/api/router_webhook_cidr_reload_test.go
+++ b/internal/api/router_webhook_cidr_reload_test.go
@@ -10,11 +10,9 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
)
-// TestReloadSystemSettings_AppliesWebhookCIDRsToNewMonitor verifies the fix
-// for issue #1507: after a monitor reload (which recreates the notification
-// manager), ReloadSystemSettings must re-apply the persisted webhook private
-// CIDR allowlist to the new notification manager. Without this, webhooks to
-// private IPs fail after any auto-registration-triggered monitor reload.
+// A recreated monitor restores its bootstrap policy before any router exists.
+// ReloadSystemSettings still owns later persisted changes, including revoking
+// a previously allowed private destination (the live replacement path of #1507).
func TestReloadSystemSettings_AppliesWebhookCIDRsToNewMonitor(t *testing.T) {
tempDir := t.TempDir()
@@ -59,41 +57,50 @@ func TestReloadSystemSettings_AppliesWebhookCIDRsToNewMonitor(t *testing.T) {
t.Fatalf("initial monitor: expected CIDR to allow 192.168.1.50, got: %v", err)
}
- // Simulate a monitor reload: create a fresh monitor with a brand-new
- // notification manager (empty CIDR allowlist).
+ // Recreated managers must already have saved policy before queued work runs.
monitor2, err := monitoring.New(cfg)
if err != nil {
t.Fatalf("monitoring.New (reload): %v", err)
}
t.Cleanup(func() { monitor2.Stop() })
- // Before the fix: the reload path in server.go never called
- // ReloadSystemSettings, so the new notification manager would have
- // an empty allowlist and reject private IPs.
nm2 := monitor2.GetNotificationManager()
if nm2 == nil {
t.Fatal("reloaded monitor has no notification manager")
}
- if err := nm2.ValidateWebhookURL("http://192.168.1.50/hook"); err == nil {
- t.Fatal("fresh notification manager should reject private IP before settings are re-applied")
+ if err := nm2.ValidateWebhookURL("http://192.168.1.50/hook"); err != nil {
+ t.Fatalf("fresh manager must restore policy before router refresh: %v", err)
}
- // Now update the router's monitor reference (as SetMonitor does during reload).
+ // Persist a real policy replacement after construction. Saving alone must
+ // not mutate the live manager; refreshing must replace, not merge, grants.
+ settings.WebhookAllowedPrivateCIDRs = "172.16.0.0/16"
+ if err := persistence.SaveSystemSettings(*settings); err != nil {
+ t.Fatalf("SaveSystemSettings replacement: %v", err)
+ }
+ if err := nm2.ValidateWebhookURL("http://192.168.1.50/hook"); err != nil {
+ t.Fatalf("persisting replacement unexpectedly changed live policy: %v", err)
+ }
+ if err := nm2.ValidateWebhookURL("http://172.16.0.1/hook"); err == nil {
+ t.Fatal("replacement grant reached the live manager before refresh")
+ }
router.monitor = monitor2
-
- // Call ReloadSystemSettings — this is the fix.
router.ReloadSystemSettings()
-
- // The new notification manager should now have the persisted CIDR allowlist.
- if err := nm2.ValidateWebhookURL("http://192.168.1.50/hook"); err != nil {
- t.Fatalf("after ReloadSystemSettings: expected CIDR to allow 192.168.1.50, got: %v", err)
+ if err := nm2.ValidateWebhookURL("http://172.16.0.1/hook"); err != nil {
+ t.Fatalf("after refresh: replacement destination rejected: %v", err)
}
- if err := nm2.ValidateWebhookURL("http://10.5.5.5/hook"); err != nil {
- t.Fatalf("after ReloadSystemSettings: expected CIDR to allow 10.5.5.5, got: %v", err)
+ for _, revoked := range []string{"http://192.168.1.50/hook", "http://10.5.5.5/hook"} {
+ if err := nm2.ValidateWebhookURL(revoked); err == nil {
+ t.Fatalf("after refresh: old grant retained for %s", revoked)
+ }
}
- // IPs outside the allowlist should still be rejected.
+ settings.WebhookAllowedPrivateCIDRs = ""
+ if err := persistence.SaveSystemSettings(*settings); err != nil {
+ t.Fatalf("SaveSystemSettings revocation: %v", err)
+ }
+ router.ReloadSystemSettings()
if err := nm2.ValidateWebhookURL("http://172.16.0.1/hook"); err == nil {
- t.Fatal("after ReloadSystemSettings: expected 172.16.0.1 to be rejected (not in allowlist)")
+ t.Fatal("clearing the installation policy did not revoke the final grant")
}
}
diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.go
index c9d4b63a5f..6590db2f3c 100644
--- a/internal/monitoring/monitor.go
+++ b/internal/monitoring/monitor.go
@@ -1839,6 +1839,31 @@ func New(cfg *config.Config) (*Monitor, error) {
return nil, fmt.Errorf("load Docker report order journal: %w", err)
}
+ // Queued deliveries can run before a router exists. Restore the instance
+ // policy before creating workers, including for tenant persistence paths.
+ // An unreadable policy must not turn accepted work into a terminal failure.
+ notificationPersistence := config.NewConfigPersistence(cfg.DataPath)
+ notificationSystemPersistence := notificationPersistence
+ if sharedDir := notificationPersistence.SharedInstallationDataDir(); sharedDir != notificationPersistence.GetConfigDir() {
+ notificationSystemPersistence = config.NewConfigPersistence(sharedDir)
+ }
+ notificationSystemSettings, notificationSettingsErr := notificationSystemPersistence.LoadSystemSettings()
+ if notificationSettingsErr != nil {
+ return nil, fmt.Errorf("load notification system settings: %w", notificationSettingsErr)
+ }
+ notificationPrivateCIDRs := ""
+ notificationPublicURL := cfg.PublicURL
+ if notificationSystemSettings != nil {
+ notificationPrivateCIDRs = notificationSystemSettings.WebhookAllowedPrivateCIDRs
+ if !cfg.EnvOverrides["publicURL"] && notificationSystemSettings.PublicURL != "" {
+ notificationPublicURL = notificationSystemSettings.PublicURL
+ }
+ }
+ notificationPrivateNetworks, notificationPolicyErr := notifications.ParseAllowedPrivateCIDRs(notificationPrivateCIDRs)
+ if notificationPolicyErr != nil {
+ return nil, fmt.Errorf("load notification private CIDR policy: %w", notificationPolicyErr)
+ }
+
// Initialize temperature collector with sensors SSH key
// Will use root user for now - can be made configurable later
homeDir := os.Getenv("HOME")
@@ -2006,9 +2031,9 @@ func New(cfg *config.Config) (*Monitor, error) {
mockModeAligned: builtInMockMode,
mockModeSwitchesSeen: switchesSeen,
incidentStore: incidentStore,
- notificationMgr: notifications.NewNotificationManagerWithDeferredQueue(cfg.PublicURL, cfg.DataPath),
+ notificationMgr: notifications.NewNotificationManagerWithDeferredQueue(notificationPublicURL, cfg.DataPath),
deadMan: newDeadManRuntime(config.ResolveRuntimeDataDir(cfg.DataPath)),
- configPersist: config.NewConfigPersistence(cfg.DataPath),
+ configPersist: notificationPersistence,
discoveryService: nil, // Will be initialized in Start()
authFailures: make(map[string]int),
lastAuthAttempt: make(map[string]time.Time),
@@ -2159,8 +2184,9 @@ func New(cfg *config.Config) (*Monitor, error) {
log.Warn().Err(err).Msg("failed to load webhook configuration")
}
// The queue worker starts with the manager but has no processor until all
- // saved destinations are installed. Activate it only after the last load;
- // a slow migration or config read must not cancel due persisted work.
+ // saved destinations and shared delivery policy are installed. A router
+ // may not exist yet, so its later settings refresh cannot own this boundary.
+ m.notificationMgr.ApplyAllowedPrivateCIDRs(notificationPrivateCIDRs, notificationPrivateNetworks)
m.notificationMgr.SetQuietHoursPolicyProvider(m.alertManager.QuietHoursNotificationPolicy)
m.notificationMgr.StartQueueProcessing()
diff --git a/internal/monitoring/monitor_notification_startup_test.go b/internal/monitoring/monitor_notification_startup_test.go
index 0dc557f8a4..4a5bdf4aab 100644
--- a/internal/monitoring/monitor_notification_startup_test.go
+++ b/internal/monitoring/monitor_notification_startup_test.go
@@ -4,7 +4,10 @@ import (
"encoding/json"
"net/http"
"net/http/httptest"
+ "os"
+ "path/filepath"
"reflect"
+ "strings"
"sync"
"sync/atomic"
"testing"
@@ -218,3 +221,156 @@ func TestMonitorResolvedSnapshotKeepsRefiringPBSDelivery(t *testing.T) {
}
assertOccurrenceHTTP(t, n, receipts, current)
}
+
+// No router or post-construction setter may be needed to resume accepted work.
+// A tenant must use the installation policy, including revocation, rather than
+// a policy file placed in its own persistence directory.
+func TestNewRestoresSharedWebhookPolicyBeforeQueueActivation(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ tenant, allowed bool
+ configuredURL string
+ urlLocked bool
+ }{
+ {name: "default", allowed: true},
+ {name: "tenant", tenant: true, allowed: true},
+ {name: "tenant-cannot-grant", tenant: true},
+ {name: "saved-url-replaces-stale-config", allowed: true, configuredURL: "https://stale-config.example.test"},
+ {name: "deployment-url-lock", allowed: true, configuredURL: "https://deployment.example.test", urlLocked: true},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ root := t.TempDir()
+ t.Setenv("PULSE_DATA_DIR", root)
+ received := make(chan string, 4)
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ var body struct {
+ Instance string `json:"instance"`
+ }
+ if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
+ t.Error(err)
+ }
+ received <- body.Instance
+ w.WriteHeader(http.StatusOK)
+ }))
+ defer server.Close()
+ global := config.NewConfigPersistence(root)
+ settings := config.DefaultSystemSettings()
+ settings.PublicURL = "https://notification-bootstrap.example.test"
+ if tc.allowed {
+ settings.WebhookAllowedPrivateCIDRs = "127.0.0.1/32"
+ }
+ if err := global.SaveSystemSettings(*settings); err != nil {
+ t.Fatal(err)
+ }
+ dir := root
+ if tc.tenant {
+ dir = filepath.Join(root, "orgs", "queue-policy")
+ }
+ persistence := config.NewConfigPersistence(dir)
+ if tc.tenant {
+ local := config.DefaultSystemSettings()
+ local.PublicURL = "https://tenant-policy-must-not-win.example.test"
+ if !tc.allowed {
+ local.WebhookAllowedPrivateCIDRs = "127.0.0.1/32"
+ }
+ if err := persistence.SaveSystemSettings(*local); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if err := persistence.SaveAlertConfig(alerts.AlertConfig{Enabled: true, ActivationState: alerts.ActivationActive}); err != nil {
+ t.Fatal(err)
+ }
+ hook := notifications.WebhookConfig{ID: "startup-policy", Name: "startup-policy", Enabled: true, URL: server.URL, Service: "generic", Template: `{"instance":"{{.Instance | jsonString}}"}`}
+ if err := persistence.SaveWebhooks([]notifications.WebhookConfig{hook}); err != nil {
+ t.Fatal(err)
+ }
+ configJSON, err := json.Marshal(hook)
+ if err != nil {
+ t.Fatal(err)
+ }
+ seed, err := notifications.NewNotificationQueue(dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := seed.Enqueue(&notifications.QueuedNotification{ID: "accepted-before-restart", Type: "webhook", Config: configJSON, Alerts: []*alerts.Alert{{ID: "accepted-policy", Type: "cpu", Level: alerts.AlertLevelWarning, StartTime: time.Now().Add(-time.Minute)}}}); err != nil {
+ t.Fatal(err)
+ }
+ if err := seed.Stop(); err != nil {
+ t.Fatal(err)
+ }
+ m, err := New(&config.Config{DataPath: dir, ConfigPath: dir, PublicURL: tc.configuredURL,
+ EnvOverrides: map[string]bool{"publicURL": tc.urlLocked}})
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer m.Stop()
+ q := m.GetNotificationManager().GetQueue()
+ want := "dlq"
+ if tc.allowed {
+ want = "sent"
+ }
+ deadline := time.Now().Add(5 * time.Second)
+ for {
+ stats, err := q.GetQueueStats()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if stats[want] == 1 {
+ break
+ }
+ if time.Now().After(deadline) {
+ t.Fatalf("BOOTSTRAP_POLICY: accepted row did not reach %s before router construction: %v", want, stats)
+ }
+ time.Sleep(10 * time.Millisecond)
+ }
+ if tc.allowed {
+ select {
+ case got := <-received:
+ wantURL := settings.PublicURL
+ if tc.urlLocked {
+ wantURL = tc.configuredURL
+ }
+ if got != wantURL {
+ t.Fatalf("bootstrap URL=%q, want effective installation URL %q", got, wantURL)
+ }
+ case <-time.After(time.Second):
+ t.Fatal("accepted row has no actual provider receipt")
+ }
+ } else if len(received) != 0 {
+ t.Fatal("tenant policy granted a destination denied by the installation")
+ }
+ })
+ }
+}
+
+func TestNewRejectsUnreadableNotificationPolicyBeforeQueueCreation(t *testing.T) {
+ for _, bad := range []string{"invalid-json", "invalid-cidr"} {
+ t.Run(bad, func(t *testing.T) {
+ dir := t.TempDir()
+ t.Setenv("PULSE_DATA_DIR", dir)
+ persistence := config.NewConfigPersistence(dir)
+ if bad == "invalid-json" {
+ if err := os.WriteFile(filepath.Join(dir, "system.json"), []byte(`{"broken":`), 0600); err != nil {
+ t.Fatal(err)
+ }
+ } else {
+ settings := config.DefaultSystemSettings()
+ settings.WebhookAllowedPrivateCIDRs = "not-a-network"
+ if err := persistence.SaveSystemSettings(*settings); err != nil {
+ t.Fatal(err)
+ }
+ }
+ m, err := New(&config.Config{DataPath: dir})
+ if m != nil {
+ m.Stop()
+ t.Fatal("monitor admitted runtime before notification policy was readable")
+ }
+ if err == nil || !strings.Contains(err.Error(), "notification") {
+ t.Fatalf("missing notification policy error: %v", err)
+ }
+ if _, err := os.Stat(filepath.Join(dir, "notification_queue.db")); !os.IsNotExist(err) {
+ t.Fatalf("policy failure created queue storage: %v", err)
+ }
+ })
+ }
+}
diff --git a/internal/notifications/manager_lifecycle_test.go b/internal/notifications/manager_lifecycle_test.go
new file mode 100644
index 0000000000..5e0c538e90
--- /dev/null
+++ b/internal/notifications/manager_lifecycle_test.go
@@ -0,0 +1,378 @@
+package notifications
+
+import (
+ "net/http"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/alerts"
+ "github.com/stretchr/testify/require"
+)
+
+// admissionBarrier holds the first serialisation of the alert that carries it.
+// Enqueue serialises an alert's payload after the admission path has read the
+// queue and before it writes the row, so the barrier pauses admitted work at
+// the point where Stop could otherwise close storage underneath it.
+type admissionBarrier struct {
+ once sync.Once
+ entered chan struct{}
+ release chan struct{}
+}
+
+func newAdmissionBarrier() *admissionBarrier {
+ return &admissionBarrier{entered: make(chan struct{}), release: make(chan struct{})}
+}
+
+func (b *admissionBarrier) MarshalJSON() ([]byte, error) {
+ b.once.Do(func() {
+ close(b.entered)
+ <-b.release
+ })
+ return []byte(`"released"`), nil
+}
+
+// shutdownAdmissionFixture is a deferred manager on persistent storage whose
+// queue is never activated, so only the work under test takes the queue lock
+// while Stop runs. Direct delivery is captured, never run.
+type shutdownAdmissionFixture struct {
+ t *testing.T
+ dir string
+ hook WebhookConfig
+ spawned chan func()
+}
+
+func newShutdownAdmissionFixture(t *testing.T) *shutdownAdmissionFixture {
+ t.Helper()
+ f := &shutdownAdmissionFixture{
+ t: t,
+ dir: t.TempDir(),
+ hook: WebhookConfig{ID: "ops", URL: "http://127.0.0.1:9/never-contacted", Enabled: true, Service: "generic"},
+ spawned: make(chan func(), 8),
+ }
+ origSpawn := spawnAsync
+ spawnAsync = func(deliver func()) { f.spawned <- deliver }
+ t.Cleanup(func() { spawnAsync = origSpawn })
+ return f
+}
+
+func (f *shutdownAdmissionFixture) open(configure func(*NotificationManager)) *NotificationManager {
+ f.t.Helper()
+ m := NewNotificationManagerWithDeferredQueue("", f.dir)
+ f.t.Cleanup(m.Stop)
+ // No wall-clock batch may contend for the queue lock in this window.
+ m.GetQueue().processorTicker.Stop()
+ m.SetGroupingConfig(false, 0, false, false)
+ m.AddWebhook(f.hook)
+ if configure != nil {
+ configure(m)
+ }
+ return m
+}
+
+// stopPassedAdmission reports that Stop has gone as far as it can while
+// admitted work is held: it closed storage without waiting for that work, or
+// it stopped the queue workers and is waiting with delivery still permitted.
+func stopPassedAdmission(m *NotificationManager, q *NotificationQueue) bool {
+ if q.db.Ping() != nil {
+ return true
+ }
+ select {
+ case <-q.stopChan:
+ return m.IsEnabled()
+ default:
+ return false
+ }
+}
+
+func awaitShutdownCondition(t *testing.T, what string, condition func() bool) {
+ t.Helper()
+ deadline := time.Now().Add(10 * time.Second)
+ for !condition() {
+ if time.Now().After(deadline) {
+ t.Fatalf("timed out waiting for %s", what)
+ }
+ time.Sleep(time.Millisecond)
+ }
+}
+
+func awaitShutdownSignal(t *testing.T, what string, signal <-chan struct{}) {
+ t.Helper()
+ select {
+ case <-signal:
+ case <-time.After(10 * time.Second):
+ t.Fatalf("timed out waiting for %s", what)
+ }
+}
+
+// An alert, recovery or timer group admitted before Stop must be written
+// before Stop closes storage, and an attempt made once Stop has begun must be
+// refused rather than wait on the queue or fall back to direct delivery. Two
+// concurrent Stop calls share one shutdown.
+func TestStopWaitsForAdmittedNotificationWrites(t *testing.T) {
+ for _, path := range []string{"firing", "recovery", "group-timer"} {
+ t.Run(path, func(t *testing.T) {
+ f := newShutdownAdmissionFixture(t)
+ m := f.open(func(m *NotificationManager) {
+ if path == "group-timer" {
+ // sendGroupedAlerts is the timer callback; a long window keeps
+ // the real timer from also running it.
+ m.SetGroupingConfig(true, 3600, false, false)
+ }
+ })
+ q := m.GetQueue()
+ barrier := newAdmissionBarrier()
+ alert := quietReplayAlert("admitted-"+path, alerts.AlertLevelWarning, time.Now().UTC().Add(-time.Minute))
+ alert.Metadata = map[string]interface{}{"barrier": barrier}
+
+ admitted := make(chan struct{})
+ switch path {
+ case "firing":
+ go func() {
+ defer close(admitted)
+ m.SendAlert(alert)
+ }()
+ case "recovery":
+ m.recordSuccessfulDelivery(notificationDeliveryJob{Type: "webhook", Event: eventAlert,
+ Alerts: []*alerts.Alert{alert}, WebhookConfig: &f.hook}, time.Now())
+ go func() {
+ defer close(admitted)
+ m.SendResolvedAlert(&alerts.ResolvedAlert{Alert: alert, ResolvedTime: time.Now()})
+ }()
+ case "group-timer":
+ m.SendAlert(alert)
+ go func() {
+ defer close(admitted)
+ m.sendGroupedAlerts()
+ }()
+ }
+ awaitShutdownSignal(t, "admitted work to reach queue serialisation", barrier.entered)
+
+ stopped := make(chan struct{}, 2)
+ for range 2 {
+ go func() {
+ m.Stop()
+ stopped <- struct{}{}
+ }()
+ }
+ awaitShutdownCondition(t, "Stop to pass the admitted work", func() bool { return stopPassedAdmission(m, q) })
+
+ late := quietReplayAlert("late-"+path, alerts.AlertLevelWarning, time.Now().UTC())
+ refused := make(chan struct{})
+ go func() {
+ defer close(refused)
+ m.SendAlert(late)
+ m.SendResolvedAlert(&alerts.ResolvedAlert{Alert: late, ResolvedTime: time.Now()})
+ }()
+ awaitShutdownSignal(t, "an attempt made during Stop to be refused", refused)
+
+ close(barrier.release)
+ awaitShutdownSignal(t, "admitted work to finish", admitted)
+ for range 2 {
+ select {
+ case <-stopped:
+ case <-time.After(10 * time.Second):
+ t.Fatal("Stop did not finish after admitted work completed")
+ }
+ }
+ select {
+ case <-f.spawned:
+ t.Errorf("SHUTDOWN_ADMISSION_DIRECT_SEND: %s admitted before Stop could not be written because Stop closed the queue under it, and went to direct delivery after shutdown", path)
+ default:
+ }
+
+ rows := quietReplayRows(t, f.open(nil).GetQueue())
+ if len(rows) != 1 || len(rows[0].Alerts) != 1 || rows[0].Alerts[0].ID != alert.ID ||
+ rows[0].Status != QueueStatusPending || rows[0].Attempts != 0 {
+ t.Fatalf("SHUTDOWN_ADMISSION_LOST: %s admitted before Stop is not pending after restart: %+v", path, rows)
+ }
+ })
+ }
+}
+
+// Operator pause and channel disable keep their meaning while Stop drains:
+// they cancel queued work, including an alert admitted just before Stop.
+func TestStopDrainKeepsPauseAndChannelDisable(t *testing.T) {
+ email := EmailConfig{Enabled: true, SMTPHost: "127.0.0.1", SMTPPort: 9,
+ From: "[email protected]", To: []string{"[email protected]"}}
+ for _, tc := range []struct {
+ name string
+ configure func(*NotificationManager)
+ act func(*NotificationManager)
+ visible func(*NotificationManager) bool
+ reason string
+ }{
+ {
+ name: "pause",
+ act: func(m *NotificationManager) { m.SetEnabled(false) },
+ visible: func(m *NotificationManager) bool { return !m.IsEnabled() },
+ reason: "Notifications disabled",
+ },
+ {
+ name: "email-disabled",
+ configure: func(m *NotificationManager) {
+ m.SetEmailConfig(email)
+ _ = m.DeleteWebhook("ops")
+ },
+ act: func(m *NotificationManager) {
+ disabled := email
+ disabled.Enabled = false
+ m.SetEmailConfig(disabled)
+ },
+ visible: func(m *NotificationManager) bool { return !m.GetEmailConfig().Enabled },
+ reason: "Email notifications disabled",
+ },
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ f := newShutdownAdmissionFixture(t)
+ m := f.open(tc.configure)
+ q := m.GetQueue()
+ barrier := newAdmissionBarrier()
+ alert := quietReplayAlert("paused-"+tc.name, alerts.AlertLevelWarning, time.Now().UTC().Add(-time.Minute))
+ alert.Metadata = map[string]interface{}{"barrier": barrier}
+ admitted := make(chan struct{})
+ go func() {
+ defer close(admitted)
+ m.SendAlert(alert)
+ }()
+ awaitShutdownSignal(t, "admitted work to reach queue serialisation", barrier.entered)
+
+ stopped := make(chan struct{})
+ go func() {
+ m.Stop()
+ close(stopped)
+ }()
+ awaitShutdownCondition(t, "Stop to pass the admitted work", func() bool { return stopPassedAdmission(m, q) })
+
+ // The change takes its lease in the critical section that publishes
+ // it, so once it is visible its cancellation is owed to the queue.
+ acted := make(chan struct{})
+ go func() {
+ defer close(acted)
+ tc.act(m)
+ }()
+ awaitShutdownCondition(t, tc.name+" to take effect", func() bool { return tc.visible(m) })
+ close(barrier.release)
+ awaitShutdownSignal(t, "admitted work to finish", admitted)
+ awaitShutdownSignal(t, tc.name+" to finish", acted)
+ awaitShutdownSignal(t, "Stop to finish", stopped)
+
+ rows := quietReplayRows(t, f.open(nil).GetQueue())
+ if len(rows) != 1 || rows[0].Status != QueueStatusCancelled ||
+ rows[0].LastError == nil || *rows[0].LastError != tc.reason {
+ t.Fatalf("SHUTDOWN_POLICY_LOST: %s during the drain did not cancel the alert admitted before Stop: %+v", tc.name, rows)
+ }
+ })
+ }
+}
+
+// A deferred queue that never started keeps admitted rows for the next start,
+// and Stop still discards a firing group that was only buffered in memory.
+func TestStopNeverStartedQueueAndBufferedGroup(t *testing.T) {
+ f := newShutdownAdmissionFixture(t)
+ m := f.open(nil)
+ start := time.Now().UTC().Add(-time.Minute)
+ m.SendAlert(quietReplayAlert("queued-1", alerts.AlertLevelWarning, start))
+ m.SendAlert(quietReplayAlert("queued-2", alerts.AlertLevelWarning, start))
+ m.SetGroupingConfig(true, 3600, false, false)
+ m.SendAlert(quietReplayAlert("buffered", alerts.AlertLevelWarning, start))
+
+ stopped := make(chan struct{})
+ go func() {
+ m.Stop()
+ close(stopped)
+ }()
+ awaitShutdownSignal(t, "Stop of a never-started queue", stopped)
+ // A timer that runs after Stop finds nothing to admit.
+ m.sendGroupedAlerts()
+
+ select {
+ case <-f.spawned:
+ t.Fatal("stopped manager handed work to direct delivery")
+ default:
+ }
+ statuses := map[string]NotificationQueueStatus{}
+ rows := quietReplayRows(t, f.open(nil).GetQueue())
+ for _, row := range rows {
+ if row.Attempts != 0 {
+ t.Errorf("never-started queue consumed an attempt: %+v", row)
+ }
+ for _, alert := range row.Alerts {
+ statuses[alert.ID] = row.Status
+ }
+ }
+ if len(rows) != 2 || statuses["queued-1"] != QueueStatusPending || statuses["queued-2"] != QueueStatusPending {
+ t.Fatalf("never-started queue lost admitted rows across Stop: %+v", rows)
+ }
+ if _, persisted := statuses["buffered"]; persisted {
+ t.Fatal("Stop persisted a buffered firing group; buffered groups are discarded at shutdown")
+ }
+}
+
+// Direct delivery started before Stop is owned: Stop waits for it, so its
+// firing receipt is durable and the recovery still reaches the destination
+// after restart. Direct delivery cannot start once Stop has finished.
+func TestStopOwnsDirectFallbackDelivery(t *testing.T) {
+ received := make(chan struct{}, 4)
+ release := make(chan struct{})
+ server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ received <- struct{}{}
+ <-release
+ w.WriteHeader(http.StatusOK)
+ }))
+ t.Cleanup(server.Close)
+ var releaseOnce sync.Once
+ releaseDelivery := func() { releaseOnce.Do(func() { close(release) }) }
+
+ dir := t.TempDir()
+ hook := WebhookConfig{ID: "ops", URL: server.URL + "/direct", Enabled: true, Service: "generic"}
+ open := func() *NotificationManager {
+ m := NewNotificationManagerWithDeferredQueue("", dir)
+ t.Cleanup(m.Stop)
+ m.webhookClient = server.Client()
+ require.NoError(t, m.UpdateAllowedPrivateCIDRs("127.0.0.1/32"))
+ m.SetGroupingConfig(false, 0, false, false)
+ m.AddWebhook(hook)
+ return m
+ }
+ m := open()
+ // Registered after the manager, so a held delivery is released before Stop.
+ t.Cleanup(releaseDelivery)
+ q := m.GetQueue()
+ // A storage failure, not shutdown, sends this alert down the direct path.
+ _, err := q.db.Exec(`CREATE TRIGGER reject_admission BEFORE INSERT ON notification_queue
+ BEGIN SELECT RAISE(ABORT, 'fixture queue write failed'); END`)
+ require.NoError(t, err)
+
+ alert := quietReplayAlert("direct", alerts.AlertLevelWarning, time.Now().UTC().Add(-time.Minute))
+ m.SendAlert(alert)
+ select {
+ case <-received:
+ case <-time.After(10 * time.Second):
+ t.Fatal("enqueue failure did not start direct delivery")
+ }
+
+ stopped := make(chan struct{})
+ go func() {
+ m.Stop()
+ close(stopped)
+ }()
+ awaitShutdownCondition(t, "Stop to pass the direct delivery", func() bool { return stopPassedAdmission(m, q) })
+ releaseDelivery()
+ awaitShutdownSignal(t, "Stop to finish", stopped)
+
+ spawned := 0
+ origSpawn := spawnAsync
+ spawnAsync = func(func()) { spawned++ }
+ m.dispatchNotificationJobAsync(notificationDeliveryJob{Type: "webhook", Event: eventAlert,
+ Alerts: []*alerts.Alert{alert}, WebhookConfig: &hook}, "direct delivery after Stop")
+ spawnAsync = origSpawn
+ if spawned != 0 {
+ t.Error("FALLBACK_AFTER_SHUTDOWN: direct delivery started after Stop returned")
+ }
+
+ jobs := buildNotificationDeliveryJobs(EmailConfig{}, []WebhookConfig{hook}, AppriseConfig{}, []*alerts.Alert{alert}, eventResolved, time.Now())
+ if len(open().filterResolvedJobsByDeliveryReceipt(jobs)) != 1 {
+ t.Fatal("FALLBACK_RECEIPT_LOST: direct delivery accepted during shutdown left no durable firing receipt, so its recovery is suppressed after restart")
+ }
+}
diff --git a/internal/notifications/notifications.go b/internal/notifications/notifications.go
index fbb433d2ee..6a7b5d713c 100644
--- a/internal/notifications/notifications.go
+++ b/internal/notifications/notifications.go
@@ -221,11 +221,92 @@ type webhookRateLimit struct {
sentCount int
}
+type notificationLifecycleState int
+
+const (
+ notificationLifecycleRunning notificationLifecycleState = iota
+ notificationLifecycleDraining
+ notificationLifecycleClosed
+)
+
+// notificationLifecycle owns work that holds the queue outside the manager
+// lock. Each lease is taken in the critical section that reads the queue, so
+// a lease holder always writes to an open queue. Stop drains the lifecycle,
+// refusing new alerts while cancellations, operator pause and direct delivery
+// already owed continue, then closes it once every lease has ended.
+type notificationLifecycle struct {
+ mu sync.Mutex
+ idle sync.Cond
+ state notificationLifecycleState
+ leases int
+}
+
+// admit leases new alert or recovery work. Stop refuses it from the start.
+func (l *notificationLifecycle) admit() bool {
+ return l.acquire(notificationLifecycleRunning)
+}
+
+// join leases work that must still reach an open queue while Stop drains:
+// cancellation, operator pause or channel disable, and direct delivery
+// started by admitted work. It is refused only once the drain has finished.
+func (l *notificationLifecycle) join() bool {
+ return l.acquire(notificationLifecycleDraining)
+}
+
+func (l *notificationLifecycle) acquire(latest notificationLifecycleState) bool {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ if l.state > latest {
+ return false
+ }
+ l.leases++
+ return true
+}
+
+func (l *notificationLifecycle) release() {
+ l.mu.Lock()
+ l.leases--
+ if l.leases == 0 {
+ l.idle.Broadcast()
+ }
+ l.mu.Unlock()
+}
+
+// accepting reports whether new work may still be buffered for later
+// admission.
+func (l *notificationLifecycle) accepting() bool {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ return l.state == notificationLifecycleRunning
+}
+
+func (l *notificationLifecycle) drain() {
+ l.mu.Lock()
+ if l.state == notificationLifecycleRunning {
+ l.state = notificationLifecycleDraining
+ }
+ l.mu.Unlock()
+}
+
+// close waits for every lease to end, then refuses all later work.
+func (l *notificationLifecycle) close() {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ if l.idle.L == nil {
+ l.idle.L = &l.mu
+ }
+ for l.leases > 0 {
+ l.idle.Wait()
+ }
+ l.state = notificationLifecycleClosed
+}
+
// NotificationManager handles sending notifications
type NotificationManager struct {
mu sync.RWMutex
stopOnce sync.Once
cleanupWG sync.WaitGroup
+ lifecycle notificationLifecycle // Admission and shutdown ownership of queue and direct delivery work
emailConfig EmailConfig
emailManager *EnhancedEmailManager // Shared email manager for rate limiting
webhooks []WebhookConfig
@@ -936,11 +1017,14 @@ func (n *NotificationManager) SetEmailConfig(config EmailConfig) {
AuthRequired: config.Username != "" && config.Password != "",
}
n.emailManager = NewEnhancedEmailManager(providerConfig)
- queue := n.queue
- disabled := !config.Enabled
+ var queue *NotificationQueue
+ if !config.Enabled && n.queue != nil && n.lifecycle.join() {
+ queue = n.queue
+ }
n.mu.Unlock()
- if disabled && queue != nil {
+ if queue != nil {
+ defer n.lifecycle.release()
if err := queue.CancelByTypes([]string{"email", "email" + queueTypeSuffixResolved}, "Email notifications disabled"); err != nil {
log.Error().Err(err).Msg("failed to cancel queued email notifications after disabling email delivery")
}
@@ -951,11 +1035,14 @@ func (n *NotificationManager) SetEmailConfig(config EmailConfig) {
func (n *NotificationManager) SetAppriseConfig(config AppriseConfig) {
n.mu.Lock()
n.appriseConfig = NormalizeAppriseConfig(config)
- queue := n.queue
- disabled := !n.appriseConfig.Enabled
+ var queue *NotificationQueue
+ if !n.appriseConfig.Enabled && n.queue != nil && n.lifecycle.join() {
+ queue = n.queue
+ }
n.mu.Unlock()
- if disabled && queue != nil {
+ if queue != nil {
+ defer n.lifecycle.release()
if err := queue.CancelByTypes([]string{"apprise", "apprise" + queueTypeSuffixResolved}, "Apprise notifications disabled"); err != nil {
log.Error().Err(err).Msg("failed to cancel queued Apprise notifications after disabling delivery")
}
@@ -1058,7 +1145,11 @@ func (n *NotificationManager) SetGroupingConfig(enabled bool, seconds int, byNod
var queue *NotificationQueue
if !enabled || seconds == 0 {
resolved = n.takePendingResolvedLocked()
- pending = append(pending, n.pendingAlerts...)
+ // Stop clears pending groups in the critical section that refuses
+ // admission, so a pending group here is always admitted.
+ if len(n.pendingAlerts) > 0 && n.lifecycle.admit() {
+ pending = append(pending, n.pendingAlerts...)
+ }
clear(n.pendingAlerts)
n.pendingAlerts = n.pendingAlerts[:0]
n.pendingAlertIndex = nil
@@ -1073,6 +1164,9 @@ func (n *NotificationManager) SetGroupingConfig(enabled bool, seconds int, byNod
queue = n.queue
}
n.mu.Unlock()
+ if len(pending) > 0 {
+ defer n.lifecycle.release()
+ }
for _, alert := range resolved {
n.dispatchResolvedAlerts([]*alerts.Alert{alert})
@@ -1179,7 +1273,9 @@ func (n *NotificationManager) SetEnabled(enabled bool) {
n.groupTimer.Stop()
n.groupTimer = nil
}
- queue = n.queue
+ if n.queue != nil && n.lifecycle.join() {
+ queue = n.queue
+ }
}
n.mu.Unlock()
@@ -1187,7 +1283,8 @@ func (n *NotificationManager) SetEnabled(enabled bool) {
log.Info().Bool("enabled", enabled).Msg("updated notification manager enabled state")
}
- if !enabled && queue != nil {
+ if queue != nil {
+ defer n.lifecycle.release()
types := []string{
"email", "email" + queueTypeSuffixResolved,
"webhook", "webhook" + queueTypeSuffixResolved,
@@ -1259,6 +1356,14 @@ func (n *NotificationManager) sendAlert(alert *alerts.Alert, options alertSendOp
n.mu.Unlock()
return
}
+ // The lease covers the queue read below until its write completes, so Stop
+ // cannot close storage under an admitted alert.
+ if !n.lifecycle.admit() {
+ log.Debug().Str("alertID", alert.ID).Msg("notification manager stopping, alert not admitted")
+ n.mu.Unlock()
+ return
+ }
+ defer n.lifecycle.release()
// Check cooldown. A disabled cooldown means "do not send repeat
// notifications for the same active alert occurrence". A higher severity is
@@ -1623,7 +1728,7 @@ func (n *NotificationManager) SendResolvedAlert(resolved *alerts.ResolvedAlert)
annotateResolvedMetadata(alertCopy, resolvedAt)
n.mu.Lock()
- if !n.enabled || !n.notifyOnResolve {
+ if !n.enabled || !n.notifyOnResolve || !n.lifecycle.accepting() {
n.mu.Unlock()
return
}
@@ -1674,10 +1779,13 @@ func (n *NotificationManager) dispatchResolvedAlerts(alertList []*alerts.Alert)
webhooks := copyWebhookConfigs(n.webhooks)
appriseConfig := copyAppriseConfig(n.appriseConfig)
queue := n.queue
+ // The lease precedes the receipt read below, which already uses the queue.
+ admitted := enabled && n.lifecycle.admit()
n.mu.RUnlock()
- if !enabled {
+ if !admitted {
return
}
+ defer n.lifecycle.release()
jobs := buildNotificationDeliveryJobs(emailConfig, webhooks, appriseConfig, alertList, eventResolved, resolvedTimeFromAlerts(alertList))
// Filter each destination independently at dispatch, not merely when the
// alert enters the window. Never announce a recovery for an undelivered firing.
@@ -1711,6 +1819,11 @@ func (n *NotificationManager) CancelResolvedAlert(alert *alerts.Alert) bool {
func (n *NotificationManager) cancelAlert(alertID string, start *time.Time) bool {
n.mu.Lock()
queue := n.queue
+ // Cancellation still runs while Stop drains; its lease keeps the queue
+ // open until the cancellation is committed.
+ if queue != nil && !n.lifecycle.join() {
+ queue = nil
+ }
record, firingDelivered := n.lastNotified[alertID]
firingDelivered = firingDelivered && (start == nil || record.alertStart.Equal(*start))
@@ -1755,6 +1868,7 @@ func (n *NotificationManager) cancelAlert(alertID string, start *time.Time) bool
// Cancel any queued notifications containing this alert
cancelledPending := 0
if queue != nil {
+ defer n.lifecycle.release()
var err error
if start == nil {
cancelledPending, err = queue.CancelByAlertIdentifiers([]string{alertID})
@@ -1784,6 +1898,13 @@ func (n *NotificationManager) sendGroupedAlerts() {
n.mu.Unlock()
return
}
+ // Stop clears pending groups in the critical section that refuses
+ // admission, so a timer that finds a group here is always admitted.
+ if !n.lifecycle.admit() {
+ n.mu.Unlock()
+ return
+ }
+ defer n.lifecycle.release()
// Copy alerts to send
alertsToSend := make([]*alerts.Alert, len(n.pendingAlerts))
@@ -2107,7 +2228,18 @@ func (n *NotificationManager) sendResolvedNotificationsDirect(emailConfig EmailC
}
func (n *NotificationManager) dispatchNotificationJobAsync(job notificationDeliveryJob, failureMessage string) {
+ // Direct delivery stays owned until it finishes, so Stop waits for it and
+ // nothing is sent, or recorded only in memory, after Stop returns.
+ if !n.lifecycle.join() {
+ log.Warn().
+ Str("type", job.Type).
+ Str("event", string(job.Event)).
+ Int("alertCount", len(job.Alerts)).
+ Msg("notification manager stopped; direct delivery refused")
+ return
+ }
spawnAsync(func() {
+ defer n.lifecycle.release()
if err := n.deliverNotificationJob(job); err != nil {
n.logNotificationJobError(job, err, failureMessage)
return
@@ -4402,8 +4534,11 @@ func (n *NotificationManager) cleanupOldNotificationRecords() {
// Stop gracefully stops the notification manager
func (n *NotificationManager) Stop() {
n.stopOnce.Do(func() {
+ // Refuse new alerts and discard volatile groups in one critical
+ // section, so a pending group is either already admitted with a lease
+ // or discarded here, as Stop has always discarded it.
n.mu.Lock()
- n.enabled = false
+ n.lifecycle.drain()
n.takePendingResolvedLocked()
clear(n.pendingAlerts)
n.pendingAlerts = n.pendingAlerts[:0]
@@ -4413,6 +4548,23 @@ func (n *NotificationManager) Stop() {
n.groupTimer = nil
}
queue := n.queue
+ n.mu.Unlock()
+
+ // Shutdown is a restart boundary, not a delivery verdict. Quiesce the
+ // queue while this manager still permits delivery and records receipts
+ // durably: a queued job reaching the processor after delivery is
+ // withdrawn below would be cancelled as disabled, and a send completing
+ // after the queue is detached would keep its firing receipt only in
+ // memory. Rows no worker has begun stay pending for the next start.
+ if queue != nil {
+ queue.stopProcessing()
+ }
+ // Wait for admitted writes, cancellations, operator changes and direct
+ // deliveries. No lease can be taken once this returns.
+ n.lifecycle.close()
+
+ n.mu.Lock()
+ n.enabled = false
cleanupDone := n.cleanupDone
client := n.webhookClient
diff --git a/internal/notifications/notifications_test.go b/internal/notifications/notifications_test.go
index 589b1e6199..ba11741978 100644
--- a/internal/notifications/notifications_test.go
+++ b/internal/notifications/notifications_test.go
@@ -4846,8 +4846,12 @@ func TestGetWebhooksReturnsDeepCopies(t *testing.T) {
}
func TestSendGroupedAlertsQueueEnqueueFailureDoesNotDeadlock(t *testing.T) {
+ // Hold the fallback rather than run it: the cooldown check below needs it
+ // not to have delivered. Stop waits for direct delivery it owns, so the
+ // held fallback completes against an in-memory SMTP server before Stop.
+ var held []func()
origSpawn := spawnAsync
- spawnAsync = func(func()) {}
+ spawnAsync = func(f func()) { held = append(held, f) }
t.Cleanup(func() { spawnAsync = origSpawn })
nm := NewNotificationManager("")
@@ -4901,6 +4905,13 @@ func TestSendGroupedAlertsQueueEnqueueFailureDoesNotDeadlock(t *testing.T) {
if notified {
t.Fatal("did not expect a cooldown record when fallback delivery never ran")
}
+ if len(held) == 0 {
+ t.Fatal("enqueue failure did not hand the group to direct delivery")
+ }
+ stubSMTPDialSuccess(t, nil)
+ for _, deliver := range held {
+ deliver()
+ }
}
func TestNormalizeWebhookConfig_CanonicalizesPushoverCustomFields(t *testing.T) {
diff --git a/internal/notifications/queue.go b/internal/notifications/queue.go
index 00bc9d1a14..2eef8fbc79 100644
--- a/internal/notifications/queue.go
+++ b/internal/notifications/queue.go
@@ -179,6 +179,7 @@ type NotificationQueue struct {
deliveryGateMu sync.Mutex
deliveryGates map[string]*notificationDeliveryGate
stopOnce sync.Once
+ stopProcessingOnce sync.Once
stopErr error
db *sql.DB
dbPath string
@@ -1791,6 +1792,12 @@ func (nq *NotificationQueue) processNotification(notif *QueuedNotification) {
nq.notifyDeliveryHealthChanged()
}
}()
+ // A worker that observes the stop begins no hold, partition or claim, so
+ // its row stays pending for the next start. A worker already past this
+ // check finishes normally; stopProcessing waits for it.
+ if nq.stopping() {
+ return
+ }
if quietHoursPolicy != nil {
ready, err := nq.prepareQuietHoursDelivery(notif, quietHoursPolicy)
if err != nil {
@@ -2119,14 +2126,37 @@ func (nq *NotificationQueue) performCleanup() {
}
}
-// Stop gracefully stops the queue processor
-func (nq *NotificationQueue) Stop() error {
- nq.stopOnce.Do(func() {
+// stopping reports whether processing has begun to stop. Queues built
+// without a stop channel are never stopping.
+func (nq *NotificationQueue) stopping() bool {
+ select {
+ case <-nq.stopChan:
+ return true
+ default:
+ return false
+ }
+}
+
+// stopProcessing ends delivery work but keeps storage open. Workers that
+// observe the stop leave their rows pending for the next start; a worker
+// already past that check finishes its hold, partition or attempt and records
+// the outcome before this returns. Stop calls it, and an owner that must
+// withdraw delivery permission calls it first so no worker sees that change.
+func (nq *NotificationQueue) stopProcessing() {
+ nq.stopProcessingOnce.Do(func() {
close(nq.stopChan)
+ log.Info().Str("component", "notification_queue").Msg("Stopping notification queue processing")
nq.wg.Wait()
nq.processorTicker.Stop()
nq.cleanupTicker.Stop()
+ })
+}
+
+// Stop gracefully stops the queue processor
+func (nq *NotificationQueue) Stop() error {
+ nq.stopOnce.Do(func() {
+ nq.stopProcessing()
if err := nq.db.Close(); err != nil {
nq.stopErr = fmt.Errorf("failed to close database: %w", err)
diff --git a/internal/notifications/queue_shutdown_test.go b/internal/notifications/queue_shutdown_test.go
new file mode 100644
index 0000000000..3fe793d229
--- /dev/null
+++ b/internal/notifications/queue_shutdown_test.go
@@ -0,0 +1,187 @@
+package notifications
+
+import (
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "sort"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/alerts"
+ "github.com/stretchr/testify/require"
+)
+
+// Shutdown is a restart boundary, not an operator decision to stop delivery.
+// Five real sends occupy every queue worker while a sixth admitted job waits
+// for a slot, which is where shutdown finds an ordinary backlog. The waiting
+// job must survive for the next start rather than be cancelled as "delivery
+// disabled", and sends completing during shutdown must keep durable firing
+// receipts so their recoveries are still delivered after restart.
+func TestManagerStopKeepsQueuedDeliveryForRestart(t *testing.T) {
+ type message struct {
+ Event string `json:"event"`
+ Alerts []*alerts.Alert `json:"alerts"`
+ }
+ received := make(chan message, 16)
+ release := make(chan struct{})
+ server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ defer r.Body.Close()
+ var msg message
+ if err := json.NewDecoder(r.Body).Decode(&msg); err != nil || len(msg.Alerts) != 1 {
+ t.Errorf("decode delivery: %v (%d alerts)", err, len(msg.Alerts))
+ w.WriteHeader(http.StatusBadRequest)
+ return
+ }
+ select {
+ case received <- msg:
+ default:
+ t.Error("unexpected excess deliveries")
+ }
+ <-release
+ w.WriteHeader(http.StatusOK)
+ }))
+ t.Cleanup(server.Close)
+ var releaseOnce sync.Once
+ releaseSends := func() { releaseOnce.Do(func() { close(release) }) }
+
+ dir := t.TempDir()
+ hook := WebhookConfig{ID: "ops", URL: server.URL, Enabled: true, Service: "generic"}
+ open := func() *NotificationManager {
+ m := NewNotificationManagerWithDeferredQueue("", dir)
+ t.Cleanup(m.Stop)
+ m.webhookClient = server.Client()
+ require.NoError(t, m.UpdateAllowedPrivateCIDRs("127.0.0.1/32"))
+ m.SetGroupingConfig(false, 0, false, false)
+ m.AddWebhook(hook)
+ return m
+ }
+ m := open()
+ // Registered after the manager, so held sends are released before its Stop.
+ t.Cleanup(releaseSends)
+
+ start := time.Now().UTC().Add(-time.Minute)
+ admitted := map[string]*alerts.Alert{}
+ for i := range 6 {
+ alert := quietReplayAlert(fmt.Sprintf("shutdown-%d", i), alerts.AlertLevelWarning, start)
+ admitted[alert.ID] = alert
+ m.SendAlert(alert)
+ }
+ pending, err := m.GetQueue().GetPending(20)
+ require.NoError(t, err)
+ require.Len(t, pending, 6, "every job must be admitted before workers start")
+
+ q := m.GetQueue()
+ m.StartQueueProcessing()
+ inFlight := map[string]bool{}
+ for range 5 {
+ select {
+ case msg := <-received:
+ inFlight[msg.Alerts[0].ID] = true
+ case <-time.After(10 * time.Second):
+ t.Fatal("five queued sends did not occupy the worker pool")
+ }
+ }
+
+ stopped := make(chan struct{})
+ go func() {
+ m.Stop()
+ close(stopped)
+ }()
+ // Either half of shutdown may come first; release the held sends only
+ // once the manager has begun to stop.
+ stopBegun := func() bool {
+ select {
+ case <-q.stopChan:
+ return true
+ default:
+ return !m.IsEnabled()
+ }
+ }
+ deadline := time.Now().Add(10 * time.Second)
+ for !stopBegun() {
+ if time.Now().After(deadline) {
+ t.Fatal("manager shutdown did not begin")
+ }
+ time.Sleep(time.Millisecond)
+ }
+ releaseSends()
+ select {
+ case <-stopped:
+ case <-time.After(10 * time.Second):
+ t.Fatal("manager shutdown did not finish its in-flight sends")
+ }
+ select {
+ case extra := <-received:
+ t.Fatalf("shutdown started a new provider attempt: %+v", extra)
+ default:
+ }
+
+ m = open()
+ q = m.GetQueue()
+ var waiting *QueuedNotification
+ for _, row := range quietReplayRows(t, q) {
+ require.Len(t, row.Alerts, 1)
+ if inFlight[row.Alerts[0].ID] {
+ require.Equal(t, QueueStatusSent, row.Status, "a send completed during shutdown keeps its outcome")
+ require.Equal(t, 1, row.Attempts)
+ continue
+ }
+ require.Nil(t, waiting, "exactly one job waited for a worker")
+ waiting = row
+ }
+ require.NotNil(t, waiting)
+
+ delivered := make([]string, 0, len(inFlight))
+ for id := range inFlight {
+ delivered = append(delivered, id)
+ }
+ sort.Strings(delivered)
+ for _, id := range delivered {
+ jobs := buildNotificationDeliveryJobs(EmailConfig{}, []WebhookConfig{hook}, AppriseConfig{}, []*alerts.Alert{admitted[id]}, eventResolved, time.Now())
+ if len(m.filterResolvedJobsByDeliveryReceipt(jobs)) != 1 {
+ t.Errorf("SHUTDOWN_RECEIPT_LOST: %s reached the receiver during shutdown, but its recovery is suppressed after restart", id)
+ }
+ }
+ if waiting.Status != QueueStatusPending || waiting.Attempts != 0 {
+ lastError := ""
+ if waiting.LastError != nil {
+ lastError = *waiting.LastError
+ }
+ t.Fatalf("SHUTDOWN_CANCELLED_UNCLAIMED: job for %s that waited for a worker became %s after %d attempt(s) (%q); it must stay pending for the next start",
+ waiting.Alerts[0].ID, waiting.Status, waiting.Attempts, lastError)
+ }
+
+ m.StartQueueProcessing()
+ select {
+ case msg := <-received:
+ require.Equal(t, waiting.Alerts[0].ID, msg.Alerts[0].ID)
+ case <-time.After(10 * time.Second):
+ t.Fatal("the job kept across shutdown was not delivered after restart")
+ }
+ deadline = time.Now().Add(10 * time.Second)
+ for loadQuietReplay(t, q, waiting.ID).Status != QueueStatusSent {
+ if time.Now().After(deadline) {
+ t.Fatal("the job kept across shutdown was not recorded as sent")
+ }
+ time.Sleep(time.Millisecond)
+ }
+
+ // The user-visible end of a receipt: a recovery after restart still
+ // reaches the destination that received the firing during shutdown.
+ recovered := delivered[0]
+ m.SendResolvedAlert(&alerts.ResolvedAlert{Alert: admitted[recovered], ResolvedTime: time.Now()})
+ select {
+ case msg := <-received:
+ require.Equal(t, "resolved", msg.Event)
+ require.Equal(t, recovered, msg.Alerts[0].ID)
+ case <-time.After(10 * time.Second):
+ t.Fatalf("recovery for %s was not delivered after restart", recovered)
+ }
+ select {
+ case extra := <-received:
+ t.Fatalf("duplicate provider message: %+v", extra)
+ default:
+ }
+}