Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Scope Patrol's expected-offline silence to availability

rcourtman2ef16ccea5ef
diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.md
index 07523d6c53..a4ecc131ed 100644
--- a/docs/release-control/v6/internal/subsystems/ai-runtime.md
+++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md
@@ -5867,7 +5867,8 @@ were removed with the unreachable handlers. The `IntentionallyOffline` branch is
counterpart — same auto-dismiss but with
`operator_state_cause: intentionally_offline` and no
`maintenance_end_at` field because the suppression has no scheduled
-end. Maintenance windows take priority over intentionally-offline
+end — and, like `expected_offline`, it silences only availability findings
+(see "Patrol consumes canonical monitoring and lifecycle policy"). Maintenance windows take priority over intentionally-offline
when both are active, because the time-bounded suppression is the
more honest one to surface to the operator. Deployments without a
provider keep the original new-finding behavior — suppression is
@@ -8128,7 +8129,23 @@ The findings operator-state projection carries canonical `MonitoringMode` and
lock state. Active maintenance, expected-offline, muted, and retired policy
produce distinct lifecycle causes in durable finding history. Suppression is
reversible when the canonical cause lifts. Legacy providers that project only
-`IntentionallyOffline` retain their established cause and behavior.
+`IntentionallyOffline` (no `MonitoringMode`) retain their established
+`intentionally_offline` cause and read as expected-offline.
+
+Expected-offline is availability-specific in Patrol exactly as it is in the
+alert runtime. `ResourceOperatorStateProjection.monitoringSuppressionCauseFor`
+decides per finding: retired and muted silence every finding, while
+expected-offline (and the legacy flag) silence only findings in the shared
+down condition class (`findingConditionClass` in
+`internal/ai/findings_alert_mirror.go`: stopped, exited, offline, unreachable,
+PDM offline, Proxmox guest stopped). Capacity, performance, backup, security,
+and disk-health findings on an expected-offline resource stay visible; SMART
+"offline uncorrectable" reports are disk health, not the resource being down.
+The wake path judges the re-detected finding with the same helper, so a
+finding silenced under muted wakes when the resource moves to expected-offline
+unless it is itself an availability finding.
+`internal/ai/findings_expected_offline_test.go` pins the split with muted as
+the negative control.
Retired lifecycle also projects an effective remediation lock into Patrol
investigation and the shared execution admission paths, so a resource cannot
diff --git a/docs/release-control/v6/internal/subsystems/patrol-intelligence.md b/docs/release-control/v6/internal/subsystems/patrol-intelligence.md
index 6c1ae7b5bf..cd20b09b83 100644
--- a/docs/release-control/v6/internal/subsystems/patrol-intelligence.md
+++ b/docs/release-control/v6/internal/subsystems/patrol-intelligence.md
@@ -2578,7 +2578,11 @@ both: maintenance window suppression takes priority and emits
`maintenance_end_at` metadata; the `IntentionallyOffline` branch is
the indefinite fallback that emits `operator_state_cause:
intentionally_offline` with no end-at field because the suppression
-has no scheduled end. Both branches set `DismissedReason =
+has no scheduled end. That branch, like canonical `expected_offline`,
+silences only availability findings (the shared down condition class);
+capacity, performance, backup, security, and disk-health findings on the
+resource stay visible, matching the alert runtime. Only `muted`, retired
+lifecycle, and active maintenance silence every finding. Both branches set `DismissedReason =
expected_behavior` and write a `UserNote` naming the cause so the
operator can audit why future findings stayed quiet without
expanding each row. Default deployments without a provider behave
diff --git a/internal/ai/findings.go b/internal/ai/findings.go
index 2189b43fea..6e351bdb21 100644
--- a/internal/ai/findings.go
+++ b/internal/ai/findings.go
@@ -877,7 +877,14 @@ type ResourceOperatorStateProjection struct {
Criticality string
}
-func (p ResourceOperatorStateProjection) monitoringSuppressionCause() string {
+// monitoringSuppressionCauseFor names the monitoring or lifecycle policy that
+// silences f, or "" when none does. Retired and muted silence every finding.
+// Expected-offline is availability-specific, as it is in the alert runtime: it
+// silences only findings in the shared down condition class and leaves health,
+// capacity, backup, and security findings visible. A projection without a
+// MonitoringMode that sets only the legacy IntentionallyOffline flag is the
+// pre-enum spelling of expected-offline and carries the same scope.
+func (p ResourceOperatorStateProjection) monitoringSuppressionCauseFor(f *Finding) string {
if strings.EqualFold(strings.TrimSpace(p.LifecycleState), "retired") {
return "resource_retired"
}
@@ -885,16 +892,19 @@ func (p ResourceOperatorStateProjection) monitoringSuppressionCause() string {
case "muted":
return "monitoring_muted"
case "expected_offline":
- return "expected_offline"
- }
- if p.IntentionallyOffline {
- return "intentionally_offline"
+ if findingConditionClass(f) == mirrorConditionDown {
+ return "expected_offline"
+ }
+ case "":
+ if p.IntentionallyOffline && findingConditionClass(f) == mirrorConditionDown {
+ return "intentionally_offline"
+ }
}
return ""
}
-func (p ResourceOperatorStateProjection) suppressesFindings() bool {
- return p.MaintenanceWindow != nil || p.monitoringSuppressionCause() != ""
+func (p ResourceOperatorStateProjection) suppressesFinding(f *Finding) bool {
+ return p.MaintenanceWindow != nil || p.monitoringSuppressionCauseFor(f) != ""
}
// ResourceOperatorStateProvider is the narrow interface the findings
@@ -1939,7 +1949,7 @@ func (s *FindingsStore) Add(f *Finding) bool {
if !willFixReminderDue && existing.DismissedReason == "expected_behavior" && !existing.Suppressed {
if cause := findOperatorStateDismissCause(existing); cause != "" {
if operatorStateProviderPresent {
- currentlySuppressing := operatorProjectionOK && operatorProjection.suppressesFindings()
+ currentlySuppressing := operatorProjectionOK && operatorProjection.suppressesFinding(f)
operatorStateLifted = !currentlySuppressing
} else {
// No provider currently wired but the finding was
@@ -2124,9 +2134,10 @@ func (s *FindingsStore) Add(f *Finding) bool {
// intent that suppresses findings, auto-dismiss the finding as
// expected_behavior with a reason that names the cause. Maintenance
// windows take priority because they're time-bounded ("expect this
- // noise until the window closes"); IntentionallyOffline is the
- // fallback branch ("expect this resource to stay quiet
- // indefinitely"). The finding still lands in durable history so
+ // noise until the window closes"); monitoring and lifecycle policy is
+ // the indefinite fallback, scoped per finding by
+ // monitoringSuppressionCauseFor (expected-offline silences only
+ // availability findings). The finding still lands in durable history so
// the operator can audit what tripped during the suppression;
// severity escalation paths still wake it later if the underlying
// condition stays after the operator clears the state.
@@ -2149,13 +2160,16 @@ func (s *FindingsStore) Add(f *Finding) bool {
"maintenance_end_at": window.EndAt.Format(time.RFC3339),
})
s.syncLoopStateLocked(f)
- case projection.monitoringSuppressionCause() != "":
- cause := projection.monitoringSuppressionCause()
+ case projection.monitoringSuppressionCauseFor(f) != "":
+ cause := projection.monitoringSuppressionCauseFor(f)
f.DismissedReason = "expected_behavior"
f.AcknowledgedAt = &now
- if cause == "intentionally_offline" {
- f.UserNote = "Resource is marked intentionally offline by the operator. Clear the flag on the resource detail surface to resume notifications."
- } else {
+ switch cause {
+ case "intentionally_offline":
+ f.UserNote = "Resource is marked intentionally offline by the operator, so Patrol keeps its availability findings quiet. Clear the flag on the resource detail surface to resume them."
+ case "expected_offline":
+ f.UserNote = "Resource is marked expected offline, so Patrol keeps its availability findings quiet. Set monitoring back to Normal on the resource detail surface to resume them."
+ default:
f.UserNote = "Resource monitoring policy suppresses Patrol attention. Return monitoring and lifecycle to active defaults on the resource detail surface to resume findings."
}
s.appendLifecycleLocked(f, "dismissed", "Auto-acknowledged: resource monitoring policy suppresses attention", f.LoopState, string(FindingLoopStateDismissed), map[string]string{
diff --git a/internal/ai/findings_alert_mirror.go b/internal/ai/findings_alert_mirror.go
index 27e931744b..23571199a7 100644
--- a/internal/ai/findings_alert_mirror.go
+++ b/internal/ai/findings_alert_mirror.go
@@ -111,6 +111,10 @@ func findingConditionClass(f *Finding) string {
return mirrorConditionRestartLoop
case contains("unhealthy", "health check"):
return mirrorConditionContainerHealth
+ case key == "smart-failure", contains("smart", "uncorrectable"):
+ // SMART reports count "offline uncorrectable" sectors; that is
+ // disk health, not the resource being down.
+ return ""
case contains("stopped", "exited", "offline", "powered off", "is down", "unreachable", "not running"):
return mirrorConditionDown
}
diff --git a/internal/ai/findings_alert_mirror_test.go b/internal/ai/findings_alert_mirror_test.go
index 0a3d743ed3..5a2b0e40ca 100644
--- a/internal/ai/findings_alert_mirror_test.go
+++ b/internal/ai/findings_alert_mirror_test.go
@@ -59,6 +59,8 @@ func TestFindingConditionClass_CoversAlertOwnedConditions(t *testing.T) {
{"snapshot age", &Finding{Category: FindingCategoryBackup, Title: `Snapshot "base" is 349 days old`}, mirrorConditionSnapshotAge},
{"backup age", &Finding{Category: FindingCategoryBackup, Title: `No recent backup for VM 104`}, mirrorConditionBackupAge},
{"pdm offline", &Finding{ID: PDMAlertFindingPrefix + ":dc/node/pve1", Category: FindingCategoryReliability, Title: `PDM: node "pve1" is offline`}, mirrorConditionDown},
+ {"smart offline uncorrectable is disk health", &Finding{Category: FindingCategoryReliability, Title: `Disk sda reports 8 offline uncorrectable sectors`}, ""},
+ {"smart-failure key is disk health", &Finding{Key: "smart-failure", Category: FindingCategoryReliability, Title: `Disk sdb is failing and may go offline`}, ""},
{"mail gateway degraded is Patrol-owned", &Finding{Category: FindingCategoryReliability, Title: `Mail gateway "mail-gateway-us" is degraded`}, ""},
{"security finding never mirrors", &Finding{Category: FindingCategorySecurity, Title: `Container runs as root and is offline`}, ""},
}
diff --git a/internal/ai/findings_expected_offline_test.go b/internal/ai/findings_expected_offline_test.go
new file mode 100644
index 0000000000..6a5b503c39
--- /dev/null
+++ b/internal/ai/findings_expected_offline_test.go
@@ -0,0 +1,151 @@
+package ai
+
+import (
+ "testing"
+ "time"
+)
+
+// Expected-offline is availability-specific in every runtime that reads the
+// canonical monitoring mode: the alert runtime suppresses only offline and
+// availability signals, and Patrol must silence only availability findings.
+// Muted is the posture that silences everything.
+
+func expectedOfflineTestStore(projection ResourceOperatorStateProjection) *FindingsStore {
+ store := NewFindingsStore()
+ store.SetResourceOperatorStateProvider(
+ ResourceOperatorStateProviderFunc(func(_ string, _ time.Time) (ResourceOperatorStateProjection, bool) {
+ return projection, true
+ }),
+ )
+ return store
+}
+
+func operatorStateDismissCause(f *Finding) string {
+ if f == nil {
+ return ""
+ }
+ return findOperatorStateDismissCause(f)
+}
+
+func expectedOfflineTestFindings() (availability *Finding, others []*Finding) {
+ availability = &Finding{
+ ID: "f-down", ResourceID: "vm:standby", Severity: FindingSeverityWarning,
+ Category: FindingCategoryReliability, Title: `VM "standby" is offline`,
+ }
+ others = []*Finding{
+ {
+ ID: "f-disk", ResourceID: "vm:standby", Severity: FindingSeverityWarning,
+ Category: FindingCategoryCapacity, Title: `Disk usage on "standby" is at 95%`,
+ },
+ {
+ ID: "f-memory", ResourceID: "vm:standby", Severity: FindingSeverityWarning,
+ Category: FindingCategoryPerformance, Title: `Memory usage on "standby" is at 93%`,
+ },
+ {
+ ID: "f-smart", ResourceID: "vm:standby", Severity: FindingSeverityCritical,
+ Key: "smart-failure", Category: FindingCategoryReliability,
+ Title: `Disk sda reports 8 offline uncorrectable sectors`,
+ },
+ }
+ return availability, others
+}
+
+func TestFindingsStore_Add_ExpectedOfflineSuppressesOnlyAvailabilityFindings(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ projection ResourceOperatorStateProjection
+ wantCause string
+ }{
+ {"monitoring mode", ResourceOperatorStateProjection{MonitoringMode: "expected_offline", IntentionallyOffline: true}, "expected_offline"},
+ {"legacy flag only", ResourceOperatorStateProjection{IntentionallyOffline: true}, "intentionally_offline"},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ store := expectedOfflineTestStore(tc.projection)
+ availability, others := expectedOfflineTestFindings()
+
+ store.Add(availability)
+ down := store.Get(availability.ID)
+ if down == nil || down.DismissedReason != "expected_behavior" || down.IsActive() {
+ t.Fatalf("availability finding on an expected-offline resource must be auto-dismissed; got %+v", down)
+ }
+ if cause := operatorStateDismissCause(down); cause != tc.wantCause {
+ t.Fatalf("availability finding operator_state_cause = %q, want %q", cause, tc.wantCause)
+ }
+
+ for _, f := range others {
+ store.Add(f)
+ got := store.Get(f.ID)
+ if got == nil {
+ t.Fatalf("%s: finding must be persisted", f.ID)
+ }
+ if got.DismissedReason != "" || got.AcknowledgedAt != nil || !got.IsActive() {
+ t.Errorf("%s (%s) must stay visible on an expected-offline resource; got DismissedReason=%q AcknowledgedAt=%v", f.ID, f.Category, got.DismissedReason, got.AcknowledgedAt)
+ }
+ if cause := operatorStateDismissCause(got); cause != "" {
+ t.Errorf("%s must carry no operator-state dismissal; got cause %q", f.ID, cause)
+ }
+ }
+ })
+ }
+}
+
+// Negative control: the same non-availability findings are silenced when the
+// resource is muted, so the visibility above comes from expected-offline's
+// scope and not from a provider the store ignored.
+func TestFindingsStore_Add_MutedSuppressesEveryFindingCategory(t *testing.T) {
+ store := expectedOfflineTestStore(ResourceOperatorStateProjection{MonitoringMode: "muted"})
+ availability, others := expectedOfflineTestFindings()
+ for _, f := range append([]*Finding{availability}, others...) {
+ store.Add(f)
+ got := store.Get(f.ID)
+ if got == nil || got.DismissedReason != "expected_behavior" || got.IsActive() {
+ t.Fatalf("%s must be auto-dismissed on a muted resource; got %+v", f.ID, got)
+ }
+ if cause := operatorStateDismissCause(got); cause != "monitoring_muted" {
+ t.Errorf("%s operator_state_cause = %q, want monitoring_muted", f.ID, cause)
+ }
+ }
+}
+
+// Moving a resource from muted to expected-offline wakes the non-availability
+// findings muted silenced, while the availability finding stays quiet.
+func TestFindingsStore_Add_MutedToExpectedOfflineWakesOnlyNonAvailabilityFindings(t *testing.T) {
+ projection := ResourceOperatorStateProjection{MonitoringMode: "muted"}
+ store := NewFindingsStore()
+ store.SetResourceOperatorStateProvider(
+ ResourceOperatorStateProviderFunc(func(_ string, _ time.Time) (ResourceOperatorStateProjection, bool) {
+ return projection, true
+ }),
+ )
+ availability, others := expectedOfflineTestFindings()
+ disk := others[0]
+ store.Add(copyFindingForTest(availability))
+ store.Add(copyFindingForTest(disk))
+
+ projection = ResourceOperatorStateProjection{MonitoringMode: "expected_offline", IntentionallyOffline: true}
+ store.Add(copyFindingForTest(availability))
+ store.Add(copyFindingForTest(disk))
+
+ if down := store.Get(availability.ID); down == nil || down.DismissedReason != "expected_behavior" {
+ t.Fatalf("availability finding must stay dismissed under expected-offline; got %+v", down)
+ }
+ woken := store.Get(disk.ID)
+ if woken == nil || woken.DismissedReason != "" || !woken.IsActive() {
+ t.Fatalf("disk finding must wake once only expected-offline applies; got %+v", woken)
+ }
+ lifted := false
+ for _, ev := range woken.Lifecycle {
+ if ev.Type == "suppression_lifted" && ev.Metadata["previous_cause"] == "monitoring_muted" {
+ lifted = true
+ }
+ }
+ if !lifted {
+ t.Error("expected a suppression_lifted lifecycle event naming monitoring_muted")
+ }
+}
+
+func copyFindingForTest(f *Finding) *Finding {
+ c := *f
+ c.Lifecycle = nil
+ return &c
+}
diff --git a/internal/ai/findings_test.go b/internal/ai/findings_test.go
index 0a1525d6bf..877b7aeea2 100644
--- a/internal/ai/findings_test.go
+++ b/internal/ai/findings_test.go
@@ -1804,11 +1804,12 @@ func TestFindingsStore_Add_DoesNotAutoAcknowledgeOutsideMaintenanceWindow(t *tes
func TestFindingsStore_Add_AutoAcknowledgesOnIntentionallyOfflineResource(t *testing.T) {
// When the operator has marked a resource as intentionally offline,
- // any new finding raised against it must be auto-dismissed as
- // expected_behavior with operator_state_cause=intentionally_offline.
- // The intent is an indefinite "shut up about this resource" — no
- // time window — so the lifecycle event must not include a
- // maintenance_end_at field.
+ // a new availability finding raised against it must be auto-dismissed
+ // as expected_behavior with operator_state_cause=intentionally_offline.
+ // The intent is an indefinite "this resource is expected to be down" —
+ // no time window — so the lifecycle event must not include a
+ // maintenance_end_at field. Non-availability findings stay visible; see
+ // findings_expected_offline_test.go.
store := NewFindingsStore()
provider := ResourceOperatorStateProviderFunc(func(canonicalID string, now time.Time) (ResourceOperatorStateProjection, bool) {
if canonicalID != "vm:archived" {
@@ -1824,6 +1825,7 @@ func TestFindingsStore_Add_AutoAcknowledgesOnIntentionallyOfflineResource(t *tes
ID: "f-archived",
ResourceID: "vm:archived",
Severity: FindingSeverityWarning,
+ Category: FindingCategoryReliability,
Title: "Resource is offline",
})
@@ -2005,6 +2007,7 @@ func TestFindingsStore_Add_WakesAfterIntentionallyOfflineCleared(t *testing.T) {
ID: "f-offline-wake",
ResourceID: "vm:archived",
Severity: FindingSeverityWarning,
+ Category: FindingCategoryReliability,
Title: "Resource is offline",
})
if got := store.Get("f-offline-wake"); got == nil || got.DismissedReason != "expected_behavior" {
@@ -2021,6 +2024,7 @@ func TestFindingsStore_Add_WakesAfterIntentionallyOfflineCleared(t *testing.T) {
ID: "f-offline-wake",
ResourceID: "vm:archived",
Severity: FindingSeverityWarning,
+ Category: FindingCategoryReliability,
Title: "Resource is offline",
})
got := store.Get("f-offline-wake")
@@ -2135,13 +2139,18 @@ func TestFindingsStore_OperatorStateSuppressedFindingsDoNotTriggerInvestigation(
cases := []struct {
name string
signal ResourceOperatorStateProjection
+ category FindingCategory
+ title string
wantCause string
}{
{
+ // Intentionally offline silences availability findings only.
name: "intentionally_offline",
signal: ResourceOperatorStateProjection{
IntentionallyOffline: true,
},
+ category: FindingCategoryReliability,
+ title: "VM 101 is offline",
wantCause: "intentionally_offline",
},
{
@@ -2152,6 +2161,8 @@ func TestFindingsStore_OperatorStateSuppressedFindingsDoNotTriggerInvestigation(
EndAt: time.Now().Add(time.Hour),
},
},
+ category: FindingCategoryPerformance,
+ title: "CPU saturated",
wantCause: "maintenance_window",
},
}
@@ -2170,7 +2181,8 @@ func TestFindingsStore_OperatorStateSuppressedFindingsDoNotTriggerInvestigation(
ID: "f-suppressed",
ResourceID: "vm:101",
Severity: FindingSeverityWarning,
- Title: "CPU saturated",
+ Category: tc.category,
+ Title: tc.title,
})
got := store.Get("f-suppressed")