Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Scope Patrol's expected-offline silence to availability
rcourtman2ef16ccea5ef
diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.mdindex 07523d6c53..a4ecc131ed 100644--- a/docs/release-control/v6/internal/subsystems/ai-runtime.md+++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md@@ -5867,7 +5867,8 @@ were removed with the unreachable handlers. The `IntentionallyOffline` branch iscounterpart — same auto-dismiss but with`operator_state_cause: intentionally_offline` and no`maintenance_end_at` field because the suppression has no scheduled-end. Maintenance windows take priority over intentionally-offline+end — and, like `expected_offline`, it silences only availability findings+(see "Patrol consumes canonical monitoring and lifecycle policy"). Maintenance windows take priority over intentionally-offlinewhen both are active, because the time-bounded suppression is themore honest one to surface to the operator. Deployments without aprovider keep the original new-finding behavior — suppression is@@ -8128,7 +8129,23 @@ The findings operator-state projection carries canonical `MonitoringMode` andlock state. Active maintenance, expected-offline, muted, and retired policyproduce distinct lifecycle causes in durable finding history. Suppression isreversible when the canonical cause lifts. Legacy providers that project only-`IntentionallyOffline` retain their established cause and behavior.+`IntentionallyOffline` (no `MonitoringMode`) retain their established+`intentionally_offline` cause and read as expected-offline.++Expected-offline is availability-specific in Patrol exactly as it is in the+alert runtime. `ResourceOperatorStateProjection.monitoringSuppressionCauseFor`+decides per finding: retired and muted silence every finding, while+expected-offline (and the legacy flag) silence only findings in the shared+down condition class (`findingConditionClass` in+`internal/ai/findings_alert_mirror.go`: stopped, exited, offline, unreachable,+PDM offline, Proxmox guest stopped). Capacity, performance, backup, security,+and disk-health findings on an expected-offline resource stay visible; SMART+"offline uncorrectable" reports are disk health, not the resource being down.+The wake path judges the re-detected finding with the same helper, so a+finding silenced under muted wakes when the resource moves to expected-offline+unless it is itself an availability finding.+`internal/ai/findings_expected_offline_test.go` pins the split with muted as+the negative control.Retired lifecycle also projects an effective remediation lock into Patrolinvestigation and the shared execution admission paths, so a resource cannotdiff --git a/docs/release-control/v6/internal/subsystems/patrol-intelligence.md b/docs/release-control/v6/internal/subsystems/patrol-intelligence.mdindex 6c1ae7b5bf..cd20b09b83 100644--- a/docs/release-control/v6/internal/subsystems/patrol-intelligence.md+++ b/docs/release-control/v6/internal/subsystems/patrol-intelligence.md@@ -2578,7 +2578,11 @@ both: maintenance window suppression takes priority and emits`maintenance_end_at` metadata; the `IntentionallyOffline` branch isthe indefinite fallback that emits `operator_state_cause:intentionally_offline` with no end-at field because the suppression-has no scheduled end. Both branches set `DismissedReason =+has no scheduled end. That branch, like canonical `expected_offline`,+silences only availability findings (the shared down condition class);+capacity, performance, backup, security, and disk-health findings on the+resource stay visible, matching the alert runtime. Only `muted`, retired+lifecycle, and active maintenance silence every finding. Both branches set `DismissedReason =expected_behavior` and write a `UserNote` naming the cause so theoperator can audit why future findings stayed quiet withoutexpanding each row. Default deployments without a provider behavediff --git a/internal/ai/findings.go b/internal/ai/findings.goindex 2189b43fea..6e351bdb21 100644--- a/internal/ai/findings.go+++ b/internal/ai/findings.go@@ -877,7 +877,14 @@ type ResourceOperatorStateProjection struct {Criticality string}-func (p ResourceOperatorStateProjection) monitoringSuppressionCause() string {+// monitoringSuppressionCauseFor names the monitoring or lifecycle policy that+// silences f, or "" when none does. Retired and muted silence every finding.+// Expected-offline is availability-specific, as it is in the alert runtime: it+// silences only findings in the shared down condition class and leaves health,+// capacity, backup, and security findings visible. A projection without a+// MonitoringMode that sets only the legacy IntentionallyOffline flag is the+// pre-enum spelling of expected-offline and carries the same scope.+func (p ResourceOperatorStateProjection) monitoringSuppressionCauseFor(f *Finding) string {if strings.EqualFold(strings.TrimSpace(p.LifecycleState), "retired") {return "resource_retired"}@@ -885,16 +892,19 @@ func (p ResourceOperatorStateProjection) monitoringSuppressionCause() string {case "muted":return "monitoring_muted"case "expected_offline":- return "expected_offline"- }- if p.IntentionallyOffline {- return "intentionally_offline"+ if findingConditionClass(f) == mirrorConditionDown {+ return "expected_offline"+ }+ case "":+ if p.IntentionallyOffline && findingConditionClass(f) == mirrorConditionDown {+ return "intentionally_offline"+ }}return ""}-func (p ResourceOperatorStateProjection) suppressesFindings() bool {- return p.MaintenanceWindow != nil || p.monitoringSuppressionCause() != ""+func (p ResourceOperatorStateProjection) suppressesFinding(f *Finding) bool {+ return p.MaintenanceWindow != nil || p.monitoringSuppressionCauseFor(f) != ""}// ResourceOperatorStateProvider is the narrow interface the findings@@ -1939,7 +1949,7 @@ func (s *FindingsStore) Add(f *Finding) bool {if !willFixReminderDue && existing.DismissedReason == "expected_behavior" && !existing.Suppressed {if cause := findOperatorStateDismissCause(existing); cause != "" {if operatorStateProviderPresent {- currentlySuppressing := operatorProjectionOK && operatorProjection.suppressesFindings()+ currentlySuppressing := operatorProjectionOK && operatorProjection.suppressesFinding(f)operatorStateLifted = !currentlySuppressing} else {// No provider currently wired but the finding was@@ -2124,9 +2134,10 @@ func (s *FindingsStore) Add(f *Finding) bool {// intent that suppresses findings, auto-dismiss the finding as// expected_behavior with a reason that names the cause. Maintenance// windows take priority because they're time-bounded ("expect this- // noise until the window closes"); IntentionallyOffline is the- // fallback branch ("expect this resource to stay quiet- // indefinitely"). The finding still lands in durable history so+ // noise until the window closes"); monitoring and lifecycle policy is+ // the indefinite fallback, scoped per finding by+ // monitoringSuppressionCauseFor (expected-offline silences only+ // availability findings). The finding still lands in durable history so// the operator can audit what tripped during the suppression;// severity escalation paths still wake it later if the underlying// condition stays after the operator clears the state.@@ -2149,13 +2160,16 @@ func (s *FindingsStore) Add(f *Finding) bool {"maintenance_end_at": window.EndAt.Format(time.RFC3339),})s.syncLoopStateLocked(f)- case projection.monitoringSuppressionCause() != "":- cause := projection.monitoringSuppressionCause()+ case projection.monitoringSuppressionCauseFor(f) != "":+ cause := projection.monitoringSuppressionCauseFor(f)f.DismissedReason = "expected_behavior"f.AcknowledgedAt = &now- if cause == "intentionally_offline" {- f.UserNote = "Resource is marked intentionally offline by the operator. Clear the flag on the resource detail surface to resume notifications."- } else {+ switch cause {+ case "intentionally_offline":+ f.UserNote = "Resource is marked intentionally offline by the operator, so Patrol keeps its availability findings quiet. Clear the flag on the resource detail surface to resume them."+ case "expected_offline":+ f.UserNote = "Resource is marked expected offline, so Patrol keeps its availability findings quiet. Set monitoring back to Normal on the resource detail surface to resume them."+ default:f.UserNote = "Resource monitoring policy suppresses Patrol attention. Return monitoring and lifecycle to active defaults on the resource detail surface to resume findings."}s.appendLifecycleLocked(f, "dismissed", "Auto-acknowledged: resource monitoring policy suppresses attention", f.LoopState, string(FindingLoopStateDismissed), map[string]string{diff --git a/internal/ai/findings_alert_mirror.go b/internal/ai/findings_alert_mirror.goindex 27e931744b..23571199a7 100644--- a/internal/ai/findings_alert_mirror.go+++ b/internal/ai/findings_alert_mirror.go@@ -111,6 +111,10 @@ func findingConditionClass(f *Finding) string {return mirrorConditionRestartLoopcase contains("unhealthy", "health check"):return mirrorConditionContainerHealth+ case key == "smart-failure", contains("smart", "uncorrectable"):+ // SMART reports count "offline uncorrectable" sectors; that is+ // disk health, not the resource being down.+ return ""case contains("stopped", "exited", "offline", "powered off", "is down", "unreachable", "not running"):return mirrorConditionDown}diff --git a/internal/ai/findings_alert_mirror_test.go b/internal/ai/findings_alert_mirror_test.goindex 0a3d743ed3..5a2b0e40ca 100644--- a/internal/ai/findings_alert_mirror_test.go+++ b/internal/ai/findings_alert_mirror_test.go@@ -59,6 +59,8 @@ func TestFindingConditionClass_CoversAlertOwnedConditions(t *testing.T) {{"snapshot age", &Finding{Category: FindingCategoryBackup, Title: `Snapshot "base" is 349 days old`}, mirrorConditionSnapshotAge},{"backup age", &Finding{Category: FindingCategoryBackup, Title: `No recent backup for VM 104`}, mirrorConditionBackupAge},{"pdm offline", &Finding{ID: PDMAlertFindingPrefix + ":dc/node/pve1", Category: FindingCategoryReliability, Title: `PDM: node "pve1" is offline`}, mirrorConditionDown},+ {"smart offline uncorrectable is disk health", &Finding{Category: FindingCategoryReliability, Title: `Disk sda reports 8 offline uncorrectable sectors`}, ""},+ {"smart-failure key is disk health", &Finding{Key: "smart-failure", Category: FindingCategoryReliability, Title: `Disk sdb is failing and may go offline`}, ""},{"mail gateway degraded is Patrol-owned", &Finding{Category: FindingCategoryReliability, Title: `Mail gateway "mail-gateway-us" is degraded`}, ""},{"security finding never mirrors", &Finding{Category: FindingCategorySecurity, Title: `Container runs as root and is offline`}, ""},}diff --git a/internal/ai/findings_expected_offline_test.go b/internal/ai/findings_expected_offline_test.gonew file mode 100644index 0000000000..6a5b503c39--- /dev/null+++ b/internal/ai/findings_expected_offline_test.go@@ -0,0 +1,151 @@+package ai++import (+ "testing"+ "time"+)++// Expected-offline is availability-specific in every runtime that reads the+// canonical monitoring mode: the alert runtime suppresses only offline and+// availability signals, and Patrol must silence only availability findings.+// Muted is the posture that silences everything.++func expectedOfflineTestStore(projection ResourceOperatorStateProjection) *FindingsStore {+ store := NewFindingsStore()+ store.SetResourceOperatorStateProvider(+ ResourceOperatorStateProviderFunc(func(_ string, _ time.Time) (ResourceOperatorStateProjection, bool) {+ return projection, true+ }),+ )+ return store+}++func operatorStateDismissCause(f *Finding) string {+ if f == nil {+ return ""+ }+ return findOperatorStateDismissCause(f)+}++func expectedOfflineTestFindings() (availability *Finding, others []*Finding) {+ availability = &Finding{+ ID: "f-down", ResourceID: "vm:standby", Severity: FindingSeverityWarning,+ Category: FindingCategoryReliability, Title: `VM "standby" is offline`,+ }+ others = []*Finding{+ {+ ID: "f-disk", ResourceID: "vm:standby", Severity: FindingSeverityWarning,+ Category: FindingCategoryCapacity, Title: `Disk usage on "standby" is at 95%`,+ },+ {+ ID: "f-memory", ResourceID: "vm:standby", Severity: FindingSeverityWarning,+ Category: FindingCategoryPerformance, Title: `Memory usage on "standby" is at 93%`,+ },+ {+ ID: "f-smart", ResourceID: "vm:standby", Severity: FindingSeverityCritical,+ Key: "smart-failure", Category: FindingCategoryReliability,+ Title: `Disk sda reports 8 offline uncorrectable sectors`,+ },+ }+ return availability, others+}++func TestFindingsStore_Add_ExpectedOfflineSuppressesOnlyAvailabilityFindings(t *testing.T) {+ for _, tc := range []struct {+ name string+ projection ResourceOperatorStateProjection+ wantCause string+ }{+ {"monitoring mode", ResourceOperatorStateProjection{MonitoringMode: "expected_offline", IntentionallyOffline: true}, "expected_offline"},+ {"legacy flag only", ResourceOperatorStateProjection{IntentionallyOffline: true}, "intentionally_offline"},+ } {+ t.Run(tc.name, func(t *testing.T) {+ store := expectedOfflineTestStore(tc.projection)+ availability, others := expectedOfflineTestFindings()++ store.Add(availability)+ down := store.Get(availability.ID)+ if down == nil || down.DismissedReason != "expected_behavior" || down.IsActive() {+ t.Fatalf("availability finding on an expected-offline resource must be auto-dismissed; got %+v", down)+ }+ if cause := operatorStateDismissCause(down); cause != tc.wantCause {+ t.Fatalf("availability finding operator_state_cause = %q, want %q", cause, tc.wantCause)+ }++ for _, f := range others {+ store.Add(f)+ got := store.Get(f.ID)+ if got == nil {+ t.Fatalf("%s: finding must be persisted", f.ID)+ }+ if got.DismissedReason != "" || got.AcknowledgedAt != nil || !got.IsActive() {+ t.Errorf("%s (%s) must stay visible on an expected-offline resource; got DismissedReason=%q AcknowledgedAt=%v", f.ID, f.Category, got.DismissedReason, got.AcknowledgedAt)+ }+ if cause := operatorStateDismissCause(got); cause != "" {+ t.Errorf("%s must carry no operator-state dismissal; got cause %q", f.ID, cause)+ }+ }+ })+ }+}++// Negative control: the same non-availability findings are silenced when the+// resource is muted, so the visibility above comes from expected-offline's+// scope and not from a provider the store ignored.+func TestFindingsStore_Add_MutedSuppressesEveryFindingCategory(t *testing.T) {+ store := expectedOfflineTestStore(ResourceOperatorStateProjection{MonitoringMode: "muted"})+ availability, others := expectedOfflineTestFindings()+ for _, f := range append([]*Finding{availability}, others...) {+ store.Add(f)+ got := store.Get(f.ID)+ if got == nil || got.DismissedReason != "expected_behavior" || got.IsActive() {+ t.Fatalf("%s must be auto-dismissed on a muted resource; got %+v", f.ID, got)+ }+ if cause := operatorStateDismissCause(got); cause != "monitoring_muted" {+ t.Errorf("%s operator_state_cause = %q, want monitoring_muted", f.ID, cause)+ }+ }+}++// Moving a resource from muted to expected-offline wakes the non-availability+// findings muted silenced, while the availability finding stays quiet.+func TestFindingsStore_Add_MutedToExpectedOfflineWakesOnlyNonAvailabilityFindings(t *testing.T) {+ projection := ResourceOperatorStateProjection{MonitoringMode: "muted"}+ store := NewFindingsStore()+ store.SetResourceOperatorStateProvider(+ ResourceOperatorStateProviderFunc(func(_ string, _ time.Time) (ResourceOperatorStateProjection, bool) {+ return projection, true+ }),+ )+ availability, others := expectedOfflineTestFindings()+ disk := others[0]+ store.Add(copyFindingForTest(availability))+ store.Add(copyFindingForTest(disk))++ projection = ResourceOperatorStateProjection{MonitoringMode: "expected_offline", IntentionallyOffline: true}+ store.Add(copyFindingForTest(availability))+ store.Add(copyFindingForTest(disk))++ if down := store.Get(availability.ID); down == nil || down.DismissedReason != "expected_behavior" {+ t.Fatalf("availability finding must stay dismissed under expected-offline; got %+v", down)+ }+ woken := store.Get(disk.ID)+ if woken == nil || woken.DismissedReason != "" || !woken.IsActive() {+ t.Fatalf("disk finding must wake once only expected-offline applies; got %+v", woken)+ }+ lifted := false+ for _, ev := range woken.Lifecycle {+ if ev.Type == "suppression_lifted" && ev.Metadata["previous_cause"] == "monitoring_muted" {+ lifted = true+ }+ }+ if !lifted {+ t.Error("expected a suppression_lifted lifecycle event naming monitoring_muted")+ }+}++func copyFindingForTest(f *Finding) *Finding {+ c := *f+ c.Lifecycle = nil+ return &c+}diff --git a/internal/ai/findings_test.go b/internal/ai/findings_test.goindex 0a1525d6bf..877b7aeea2 100644--- a/internal/ai/findings_test.go+++ b/internal/ai/findings_test.go@@ -1804,11 +1804,12 @@ func TestFindingsStore_Add_DoesNotAutoAcknowledgeOutsideMaintenanceWindow(t *tesfunc TestFindingsStore_Add_AutoAcknowledgesOnIntentionallyOfflineResource(t *testing.T) {// When the operator has marked a resource as intentionally offline,- // any new finding raised against it must be auto-dismissed as- // expected_behavior with operator_state_cause=intentionally_offline.- // The intent is an indefinite "shut up about this resource" — no- // time window — so the lifecycle event must not include a- // maintenance_end_at field.+ // a new availability finding raised against it must be auto-dismissed+ // as expected_behavior with operator_state_cause=intentionally_offline.+ // The intent is an indefinite "this resource is expected to be down" —+ // no time window — so the lifecycle event must not include a+ // maintenance_end_at field. Non-availability findings stay visible; see+ // findings_expected_offline_test.go.store := NewFindingsStore()provider := ResourceOperatorStateProviderFunc(func(canonicalID string, now time.Time) (ResourceOperatorStateProjection, bool) {if canonicalID != "vm:archived" {@@ -1824,6 +1825,7 @@ func TestFindingsStore_Add_AutoAcknowledgesOnIntentionallyOfflineResource(t *tesID: "f-archived",ResourceID: "vm:archived",Severity: FindingSeverityWarning,+ Category: FindingCategoryReliability,Title: "Resource is offline",})@@ -2005,6 +2007,7 @@ func TestFindingsStore_Add_WakesAfterIntentionallyOfflineCleared(t *testing.T) {ID: "f-offline-wake",ResourceID: "vm:archived",Severity: FindingSeverityWarning,+ Category: FindingCategoryReliability,Title: "Resource is offline",})if got := store.Get("f-offline-wake"); got == nil || got.DismissedReason != "expected_behavior" {@@ -2021,6 +2024,7 @@ func TestFindingsStore_Add_WakesAfterIntentionallyOfflineCleared(t *testing.T) {ID: "f-offline-wake",ResourceID: "vm:archived",Severity: FindingSeverityWarning,+ Category: FindingCategoryReliability,Title: "Resource is offline",})got := store.Get("f-offline-wake")@@ -2135,13 +2139,18 @@ func TestFindingsStore_OperatorStateSuppressedFindingsDoNotTriggerInvestigation(cases := []struct {name stringsignal ResourceOperatorStateProjection+ category FindingCategory+ title stringwantCause string}{{+ // Intentionally offline silences availability findings only.name: "intentionally_offline",signal: ResourceOperatorStateProjection{IntentionallyOffline: true,},+ category: FindingCategoryReliability,+ title: "VM 101 is offline",wantCause: "intentionally_offline",},{@@ -2152,6 +2161,8 @@ func TestFindingsStore_OperatorStateSuppressedFindingsDoNotTriggerInvestigation(EndAt: time.Now().Add(time.Hour),},},+ category: FindingCategoryPerformance,+ title: "CPU saturated",wantCause: "maintenance_window",},}@@ -2170,7 +2181,8 @@ func TestFindingsStore_OperatorStateSuppressedFindingsDoNotTriggerInvestigation(ID: "f-suppressed",ResourceID: "vm:101",Severity: FindingSeverityWarning,- Title: "CPU saturated",+ Category: tc.category,+ Title: tc.title,})got := store.Get("f-suppressed")