Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Preserve held threshold alerts’ breach dates in live state
pulse-triage[bot]5709c6073c2d
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 841ac9578c..e0fdf6ff73 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -141,6 +141,11 @@ the alerts subsystem. Agent report admission, host continuity andre-enrollment never read or write it, and it is not persisted with agent orstate records.+`MetricAlertStatus.lastBreachAt` is an optional server-evaluated breach date,+not an agent report timestamp. It stays with the volatile alert status and+neither changes agent admission nor adds persistent agent/state fields. The+model wire/clone controls cover known, offset, unknown and Go-zero dates.+### Source-record lookup preserves report admissionHost and Docker report admission still performs canonical host-view matchingdiff --git a/docs/release-control/v6/internal/subsystems/alerts.md b/docs/release-control/v6/internal/subsystems/alerts.mdindex 61fb8cae35..80d99f4d91 100644--- a/docs/release-control/v6/internal/subsystems/alerts.md+++ b/docs/release-control/v6/internal/subsystems/alerts.md@@ -102,6 +102,20 @@ day is not auto-resolved as unmonitored. Frontend surfaces format the statusthrough `features/alerts/metricAlertPresentation.ts` and never re-derive thephase or recovery timing.+The optional `metricStatus.lastBreachAt` dates the evaluated value retained in+`Alert.Value`, not the newer `observedAt` of a hold or recovery. A breaching+sample (including equality at the trigger) advances it; a held or recovering+sample uses the existing alert's `LastSeen`. Unknown/zero dates are omitted,+never replaced by the occurrence start, restart or poll time. It remains+volatile with the status, so restart rebuilds it from the saved breach and a+hold does not rewrite the recovery mirror. Resolution drops the whole status.++`TestMetricBreachTimeTracksTheEvaluatedBreach` checks holds, recovery progress+and reset, exact-trigger and renewed breaches. The existing live-status and+checkpoint controls also assert the breach date; restart controls cover both+durable active state and the JSON mirror, unchanged mirror bytes/mtime, and+legacy unknown-date recovery through a new breach.+### Alert card and open incident lead with the live reading — issue #2068The Alerts overview card leads an open threshold alert with the presentation'sdiff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.mdindex 05ea050f9e..2b60032623 100644--- a/docs/release-control/v6/internal/subsystems/api-contracts.md+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md@@ -127,6 +127,18 @@ while recovering, `recoveryStartedAt` and `recoveryElapsedSeconds`. Legacytrigger. The field is absent for non-threshold alerts and until the firstevaluation after a restart, and clients then fall back to `message`.+`metricStatus.lastBreachAt` optionally dates the alert's retained evaluated+breach value. Holds and recovery do not advance it to `observedAt`, and an+unknown/zero date is omitted. The websocket and `/api/state` projection still+omit legacy `lastSeen`; `/api/alerts/active` retains it. Old clients may ignore+the additive field; consumers without it may use a valid legacy `lastSeen`+but must not fabricate the date from `startTime` or the current observation.+After restart no live status exists until evaluation; resolved rows have none.++`TestMetricBreachTimeStateDeltasPreserveAndWithdraw` reconstructs the client+baseline through held, recovery, renewed-breach, unknown-date and no-status+keyed patches, including removal rather than retention of an obsolete date.+### Organisation deletion retains data after incomplete monitoring shutdownAuthenticated organisation-owner deletion waits for tenant-loop exit and sealeddiff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex c53a8c63d7..7ec82afccf 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -243,6 +243,13 @@ lists carry the reading being evaluated now for a threshold alert held belowits trigger, while `value` and `message` keep the last breach. The projectionadds no copy of its own: `GetActiveAlerts` already returns deep clones.+The optional status field `lastBreachAt` carries the held breach's original+`LastSeen` through that same projection, without publishing the poll-varying+`LastSeen` of other lifecycle alerts. `observedAt` still dates the current+comparison. `TestSyncAlertsToStateCarriesLiveMetricStatusOfHeldAlert` checks+the concrete frontend JSON as well as the in-memory held reading, with an+independent breach date and omitted legacy `lastSeen`.+### Fresh broadcast projection owns one resource/target captureCanonical live, fallback and standalone-host continuity broadcast reads preferdiff --git a/internal/alerts/alerts_test.go b/internal/alerts/alerts_test.goindex fbdea21e4b..18f1a96a94 100644--- a/internal/alerts/alerts_test.go+++ b/internal/alerts/alerts_test.go@@ -22472,7 +22472,7 @@ func TestCleanupRetentionRequiresObservationInactivity(t *testing.T) {func TestMetricStatusStaysOutOfDurableAlertSnapshots(t *testing.T) {m, clock := newMetricStatusTestManager(t)- fireNodeTemperatureAlert(t, m, clock, 85)+ fired := fireNodeTemperatureAlert(t, m, clock, 85)clock.advance(30 * time.Second)checkMetricStatusNode(m, 78)@@ -22480,6 +22480,9 @@ func TestMetricStatusStaysOutOfDurableAlertSnapshots(t *testing.T) {if len(live) != 1 || live[0].MetricStatus == nil || live[0].MetricStatus.Value != 78 {t.Fatalf("live alerts do not carry the current reading: %+v", live)}+ if got := metricBreachTime(t, live[0].MetricStatus); !got.Equal(fired.LastSeen) {+ t.Fatalf("live status lost the breach's date: %v, want %v", got, fired.LastSeen)+ }for _, alert := range m.snapshotActiveAlerts() {if alert.MetricStatus != nil {t.Fatalf("durable checkpoint carries volatile status for %s: %+v", alert.ID, alert.MetricStatus)diff --git a/internal/alerts/canonical_metric.go b/internal/alerts/canonical_metric.goindex 8d29198075..d1d5bf40ff 100644--- a/internal/alerts/canonical_metric.go+++ b/internal/alerts/canonical_metric.go@@ -282,6 +282,15 @@ func (m *Manager) evaluateCanonicalMetricAlert(spec alertspecs.ResourceAlertSpecalertMetadata[k] = v}}+ // Date the evaluated breach, not a newer hold/recovery observation.+ // A restored alert retains its breach's LastSeen; without that evidence+ // the date stays unknown rather than becoming the restart or poll time.+ var lastBreachAt time.Time+ if triggered {+ lastBreachAt = observedAt+ } else if exists && existingAlert != nil {+ lastBreachAt = existingAlert.LastSeen+ }metricStatus := buildMetricAlertStatus(metricStatusInput{value: value,triggered: triggered,@@ -292,6 +301,7 @@ func (m *Manager) evaluateCanonicalMetricAlert(spec alertspecs.ResourceAlertSpecwindow: windowed,incident: incident,observedAt: observedAt,+ lastBreachAt: lastBreachAt,})if !exists {diff --git a/internal/alerts/metric_breach_time_test.go b/internal/alerts/metric_breach_time_test.gonew file mode 100644index 0000000000..8690adb7e1--- /dev/null+++ b/internal/alerts/metric_breach_time_test.go@@ -0,0 +1,176 @@+package alerts++import (+ "bytes"+ "encoding/json"+ "os"+ "path/filepath"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/models"+)++// Decode the public field rather than sharing the evaluator's choice of date.+// These controls also compile against a source which does not expose it yet.+func metricBreachTime(t *testing.T, status *models.MetricAlertStatus) time.Time {+ t.Helper()+ data, err := json.Marshal(status)+ if err != nil {+ t.Fatal(err)+ }+ var wire struct {+ LastBreachAt time.Time `json:"lastBreachAt"`+ }+ if err := json.Unmarshal(data, &wire); err != nil {+ t.Fatal(err)+ }+ return wire.LastBreachAt+}++func TestMetricBreachTimeTracksTheEvaluatedBreach(t *testing.T) {+ m, clock := newMetricStatusTestManager(t)+ fired := fireNodeTemperatureAlert(t, m, clock, 85)+ lastBreach := fired.LastSeen+ for _, step := range []struct {+ name string+ value float64+ phase string+ breaches bool+ }{+ {"held", 78, models.MetricAlertPhaseLatched, false},+ {"recovery starts", 72, models.MetricAlertPhaseRecovering, false},+ {"recovery advances", 75, models.MetricAlertPhaseRecovering, false},+ {"recovery resets", 77, models.MetricAlertPhaseLatched, false},+ {"exact trigger breaches", 80, models.MetricAlertPhaseBreaching, true},+ {"held after new breach", 78, models.MetricAlertPhaseLatched, false},+ {"higher breach", 90, models.MetricAlertPhaseBreaching, true},+ {"recovering after new breach", 74, models.MetricAlertPhaseRecovering, false},+ } {+ t.Run(step.name, func(t *testing.T) {+ clock.advance(30 * time.Second)+ checkMetricStatusNode(m, step.value)+ if step.breaches {+ lastBreach = clock.now+ }+ alert := activeNodeTemperatureAlert(t, m)+ if alert == nil || alert.MetricStatus == nil || alert.MetricStatus.Phase != step.phase {+ t.Fatalf("alert = %+v, want phase %s", alert, step.phase)+ }+ if got := metricBreachTime(t, alert.MetricStatus); !got.Equal(lastBreach) {+ t.Errorf("lastBreachAt = %v, want last evaluated breach %v", got, lastBreach)+ }+ if !alert.LastSeen.Equal(lastBreach) || !alert.MetricStatus.ObservedAt.Equal(clock.now) {+ t.Errorf("breach snapshot and live observation diverged: %+v", alert)+ }+ })+ }+}++func TestMetricBreachTimeRestoredWithoutLiveStatus(t *testing.T) {+ for _, durable := range []bool{false, true} {+ name := "recovery mirror"+ var options []ManagerOption+ if durable {+ name = "durable active state"+ options = []ManagerOption{WithDurableAlertStore()}+ }+ t.Run(name, func(t *testing.T) {+ dir := t.TempDir()+ m := NewManagerWithDataDir(dir, options...)+ t.Cleanup(m.Stop)+ clock := &metricStatusClock{now: time.Now().UTC()}+ m.now = func() time.Time { return clock.now }+ m.intentClock = func() time.Duration { return clock.tick }+ cfg := m.GetConfig()+ cfg.NodeDefaults.Temperature = &HysteresisThreshold{Trigger: 80, Clear: 75}+ m.UpdateConfig(cfg)+ fired := fireNodeTemperatureAlert(t, m, clock, 85)+ if err := m.SaveActiveAlerts(); err != nil {+ t.Fatal(err)+ }+ mirror := filepath.Join(dir, "alerts", "active-alerts.json")+ before, err := os.ReadFile(mirror)+ if err != nil {+ t.Fatal(err)+ }+ beforeInfo, err := os.Stat(mirror)+ if err != nil {+ t.Fatal(err)+ }++ clock.advance(time.Minute)+ checkMetricStatusNode(m, 78)+ if err := m.SaveActiveAlerts(); err != nil {+ t.Fatal(err)+ }+ after, err := os.ReadFile(mirror)+ if err != nil {+ t.Fatal(err)+ }+ afterInfo, err := os.Stat(mirror)+ if err != nil {+ t.Fatal(err)+ }+ if !bytes.Equal(before, after) || !beforeInfo.ModTime().Equal(afterInfo.ModTime()) {+ t.Fatal("holding an alert rewrote its durable recovery mirror")+ }+ if bytes.Contains(after, []byte("metricStatus")) || bytes.Contains(after, []byte("lastBreachAt")) {+ t.Fatal("volatile metric status reached durable alert storage")+ }+ m.Stop()++ restored := NewManagerWithDataDir(dir, options...)+ t.Cleanup(restored.Stop)+ restored.UpdateConfig(cfg)+ clock.advance(2 * time.Minute)+ restored.now = func() time.Time { return clock.now }+ restored.intentClock = func() time.Duration { return clock.tick }+ alert := activeNodeTemperatureAlert(t, restored)+ if alert == nil || alert.MetricStatus != nil || !alert.LastSeen.Equal(fired.LastSeen) {+ t.Fatalf("restart must retain the breach, not a live reading: %+v", alert)+ }+ checkMetricStatusNode(restored, 78)+ alert = activeNodeTemperatureAlert(t, restored)+ if alert == nil || alert.MetricStatus == nil || alert.MetricStatus.Phase != models.MetricAlertPhaseLatched {+ t.Fatalf("first restored hold = %+v", alert)+ }+ if got := metricBreachTime(t, alert.MetricStatus); !got.Equal(fired.LastSeen) {+ t.Fatalf("restored hold dated breach %v, want %v, not restart time %v", got, fired.LastSeen, clock.now)+ }+ if !alert.MetricStatus.ObservedAt.Equal(clock.now) {+ t.Fatal("restored live status kept the pre-restart observation")+ }+ })+ }+}++func TestMetricBreachTimeUnknownUntilANewBreach(t *testing.T) {+ m, clock := newMetricStatusTestManager(t)+ fired := fireNodeTemperatureAlert(t, m, clock, 85)+ // An older checkpoint can have the incident and value but no date. The+ // occurrence's start or the next healthy reading cannot date that value.+ fired.LastSeen = time.Time{}+ fired.MetricStatus = nil+ if err := m.restoreActiveAlertSnapshots([]*Alert{fired}, "legacy checkpoint without breach time", true); err != nil {+ t.Fatal(err)+ }+ clock.advance(time.Minute)+ checkMetricStatusNode(m, 78)+ held := activeNodeTemperatureAlert(t, m)+ if held == nil || held.MetricStatus == nil || held.MetricStatus.Phase != models.MetricAlertPhaseLatched {+ t.Fatalf("restored unknown-date hold = %+v", held)+ }+ if got := metricBreachTime(t, held.MetricStatus); !got.IsZero() {+ t.Fatalf("unknown breach date became %v", got)+ }+ data, err := json.Marshal(held.MetricStatus)+ if err != nil || bytes.Contains(data, []byte("lastBreachAt")) {+ t.Fatalf("unknown date reached wire: %s / %v", data, err)+ }+ clock.advance(time.Minute)+ checkMetricStatusNode(m, 80)+ if got := metricBreachTime(t, activeNodeTemperatureAlert(t, m).MetricStatus); !got.Equal(clock.now) {+ t.Fatalf("new breach date = %v, want %v", got, clock.now)+ }+}diff --git a/internal/alerts/metric_status.go b/internal/alerts/metric_status.goindex 7bc19204b1..d9a547ddd4 100644--- a/internal/alerts/metric_status.go+++ b/internal/alerts/metric_status.go@@ -20,6 +20,7 @@ type metricStatusInput struct {window metricWindowObservationincident reducer.IncidentobservedAt time.Time+ lastBreachAt time.Time}// metricStatusUnit is the unit of an evaluated metric value. CPU and memory@@ -52,6 +53,7 @@ func buildMetricAlertStatus(input metricStatusInput) *models.MetricAlertStatus {Trigger: input.trigger,Recovery: recovery,RecoveryDelaySeconds: input.recoveryDelaySeconds,+ LastBreachAt: input.lastBreachAt,}if input.window.WindowSeconds > 0 {raw := input.window.CurrentValuediff --git a/internal/alerts/unified_eval_test.go b/internal/alerts/unified_eval_test.goindex 09b4e176c2..ab533b8003 100644--- a/internal/alerts/unified_eval_test.go+++ b/internal/alerts/unified_eval_test.go@@ -1751,7 +1751,8 @@ func TestOpenTemperatureAlertReportsLiveReadingWhileItHolds(t *testing.T) {fired := fireNodeTemperatureAlert(t, m, clock, 85)status := fired.MetricStatusif status == nil || status.Phase != models.MetricAlertPhaseBreaching || status.Value != 85 ||- status.Trigger != 80 || status.Recovery != 75 || status.Unit != "°C" {+ status.Trigger != 80 || status.Recovery != 75 || status.Unit != "°C" ||+ !metricBreachTime(t, status).Equal(fired.LastSeen) {t.Fatalf("breaching status = %+v", status)}if status.RecoveryDelaySeconds <= 0 {@@ -1772,7 +1773,8 @@ func TestOpenTemperatureAlertReportsLiveReadingWhileItHolds(t *testing.T) {t.Fatalf("hold rewrote the breach snapshot: value=%v message=%q lastSeen=%v", held.Value, held.Message, held.LastSeen)}if got := held.MetricStatus; got == nil || got.Phase != models.MetricAlertPhaseLatched || got.Value != 78 ||- !got.ObservedAt.Equal(clock.now) || got.RecoveryStartedAt != nil {+ !got.ObservedAt.Equal(clock.now) || got.RecoveryStartedAt != nil ||+ !metricBreachTime(t, got).Equal(firedLastSeen) {t.Fatalf("latched status = %+v", got)}@@ -1782,13 +1784,15 @@ func TestOpenTemperatureAlertReportsLiveReadingWhileItHolds(t *testing.T) {checkMetricStatusNode(m, 72)if got := activeNodeTemperatureAlert(t, m).MetricStatus; got == nil ||got.Phase != models.MetricAlertPhaseRecovering || got.Value != 72 ||- got.RecoveryStartedAt == nil || !got.RecoveryStartedAt.Equal(recoveryStart) || got.RecoveryElapsedSeconds != 0 {+ got.RecoveryStartedAt == nil || !got.RecoveryStartedAt.Equal(recoveryStart) || got.RecoveryElapsedSeconds != 0 ||+ !metricBreachTime(t, got).Equal(firedLastSeen) {t.Fatalf("recovery start status = %+v", got)}clock.advance(delay / 2)checkMetricStatusNode(m, 75)if got := activeNodeTemperatureAlert(t, m).MetricStatus; got == nil ||- got.Phase != models.MetricAlertPhaseRecovering || got.RecoveryElapsedSeconds != int(delay/2/time.Second) {+ got.Phase != models.MetricAlertPhaseRecovering || got.RecoveryElapsedSeconds != int(delay/2/time.Second) ||+ !metricBreachTime(t, got).Equal(firedLastSeen) {t.Fatalf("recovery progress status = %+v", got)}@@ -1796,7 +1800,8 @@ func TestOpenTemperatureAlertReportsLiveReadingWhileItHolds(t *testing.T) {clock.advance(30 * time.Second)checkMetricStatusNode(m, 77)if got := activeNodeTemperatureAlert(t, m).MetricStatus; got == nil ||- got.Phase != models.MetricAlertPhaseLatched || got.RecoveryStartedAt != nil || got.RecoveryElapsedSeconds != 0 {+ got.Phase != models.MetricAlertPhaseLatched || got.RecoveryStartedAt != nil || got.RecoveryElapsedSeconds != 0 ||+ !metricBreachTime(t, got).Equal(firedLastSeen) {t.Fatalf("status after recovery reset = %+v", got)}if got := len(notified); got != notifications {diff --git a/internal/models/metrics_types_test.go b/internal/models/metrics_types_test.goindex de1973b924..a381fc6b03 100644--- a/internal/models/metrics_types_test.go+++ b/internal/models/metrics_types_test.go@@ -550,3 +550,47 @@ func TestMetricAlertStatusCloneIsDeep(t *testing.T) {t.Fatal("nil status must clone to nil")}}++// Check the encoded additive contract as an older client would see it; missing+// dates stay missing, including Go's zero time, without changing other fields.+func TestMetricAlertStatusBreachTimeWireAndClone(t *testing.T) {+ for _, tc := range []struct{ name, date string }{+ {"known", "2026-10-09T19:25:16.123456789Z"},+ {"offset", "2026-10-09T20:25:16+01:00"},+ {"unknown", ""},+ {"zero", "0001-01-01T00:00:00Z"},+ } {+ t.Run(tc.name, func(t *testing.T) {+ input := map[string]any{"phase": "latched", "value": 78, "trigger": 80, "recovery": 75}+ if tc.date != "" {+ input["lastBreachAt"] = tc.date+ }+ data, err := json.Marshal(input)+ if err != nil {+ t.Fatal(err)+ }+ var status MetricAlertStatus+ if err := json.Unmarshal(data, &status); err != nil {+ t.Fatal(err)+ }+ for _, copy := range []*MetricAlertStatus{&status, status.Clone()} {+ data, err = json.Marshal(copy)+ if err != nil {+ t.Fatal(err)+ }+ var wire map[string]any+ if err := json.Unmarshal(data, &wire); err != nil {+ t.Fatal(err)+ }+ got, present := wire["lastBreachAt"]+ wantDate := tc.date != "" && tc.date != "0001-01-01T00:00:00Z"+ if present != wantDate || wantDate && got != tc.date {+ t.Errorf("lastBreachAt = %v, present %v; want %q present %v", got, present, tc.date, wantDate)+ }+ if wire["value"] != float64(78) || wire["phase"] != "latched" {+ t.Fatalf("new date changed legacy reading: %s", data)+ }+ }+ })+ }+}diff --git a/internal/models/models.go b/internal/models/models.goindex 71c8db5262..b1ee9cd09e 100644--- a/internal/models/models.go+++ b/internal/models/models.go@@ -165,6 +165,10 @@ type MetricAlertStatus struct {// Recovery progress, present only in the recovering phase.RecoveryStartedAt *time.Time `json:"recoveryStartedAt,omitempty"`RecoveryElapsedSeconds int `json:"recoveryElapsedSeconds,omitempty"`+ // LastBreachAt dates the alert's Value: the latest evaluated reading that+ // met the trigger, not the newer observation of a hold or recovery run.+ // State/websocket alerts omit LastSeen; zero means the date is unknown.+ LastBreachAt time.Time `json:"lastBreachAt,omitzero"`}// Clone returns a deep copy.diff --git a/internal/monitoring/monitor_alert_intent_test.go b/internal/monitoring/monitor_alert_intent_test.goindex 2ec3c86355..49ca8f7e65 100644--- a/internal/monitoring/monitor_alert_intent_test.go+++ b/internal/monitoring/monitor_alert_intent_test.go@@ -1,6 +1,7 @@package monitoringimport (+ "encoding/json""testing""time"@@ -278,6 +279,15 @@ func TestSyncAlertsToStateCarriesLiveMetricStatusOfHeldAlert(t *testing.T) {Temperature: &models.Temperature{Available: true, CPUPackage: 85},}m.alertManager.CheckNode(node)+ var breachedAt time.Time+ for _, alert := range m.alertManager.GetActiveAlerts() {+ if alert.Type == "temperature" {+ breachedAt = alert.LastSeen+ }+ }+ if breachedAt.IsZero() {+ t.Fatal("breaching temperature did not open the alert")+ }node.Temperature = &models.Temperature{Available: true, CPUPackage: 78}m.alertManager.CheckNode(node)@@ -303,6 +313,44 @@ func TestSyncAlertsToStateCarriesLiveMetricStatusOfHeldAlert(t *testing.T) {if status.Phase != models.MetricAlertPhaseLatched || status.Value != 78 || status.Recovery != 75 {t.Fatalf("live status = %+v, want latched at 78 clearing at 75", status)}+ // The concrete /api/state and websocket projection has no LastSeen. The+ // live status must date the held breach without advancing it to this poll.+ data, err := json.Marshal(m.state.ToFrontend())+ if err != nil {+ t.Fatal(err)+ }+ var wire struct {+ ActiveAlerts []struct {+ Type string `json:"type"`+ LastSeen *time.Time `json:"lastSeen"`+ MetricStatus *struct {+ LastBreachAt time.Time `json:"lastBreachAt"`+ ObservedAt time.Time `json:"observedAt"`+ } `json:"metricStatus"`+ } `json:"activeAlerts"`+ }+ if err := json.Unmarshal(data, &wire); err != nil {+ t.Fatal(err)+ }+ found := false+ for _, alert := range wire.ActiveAlerts {+ if alert.Type != "temperature" {+ continue+ }+ found = true+ if alert.LastSeen != nil {+ t.Fatal("projection added poll-varying legacy LastSeen")+ }+ if alert.MetricStatus == nil || !alert.MetricStatus.LastBreachAt.Equal(breachedAt) {+ t.Fatalf("projected breach time = %+v, want %v", alert.MetricStatus, breachedAt)+ }+ if !alert.MetricStatus.ObservedAt.After(breachedAt) {+ t.Fatal("hold did not carry a newer independent observation")+ }+ }+ if !found {+ t.Fatal("frontend projection lost the held temperature alert")+ }}// After a restart, a host that has not reported again exists only throughdiff --git a/internal/websocket/state_delta_test.go b/internal/websocket/state_delta_test.goindex 29d30408b8..e2c3f233cb 100644--- a/internal/websocket/state_delta_test.go+++ b/internal/websocket/state_delta_test.go@@ -511,3 +511,82 @@ func TestClientStateBaselineAdvancesOnlyAfterDeltaIsQueued(t *testing.T) {t.Fatal("client baseline did not advance after the delta was queued")}}++// A receiver retains an unchanged breach date across held/recovery patches,+// advances it only with a new breach, and drops it when the producer has none.+func TestMetricBreachTimeStateDeltasPreserveAndWithdraw(t *testing.T) {+ state := models.EmptyStateFrontend()+ state.ActiveAlerts = []models.Alert{{ID: "alert-temperature", Type: "temperature", Value: 85}}+ setStatus := func(body string) {+ t.Helper()+ state.ActiveAlerts[0].MetricStatus = nil+ if body != "" {+ var status models.MetricAlertStatus+ if err := json.Unmarshal([]byte(body), &status); err != nil {+ t.Fatal(err)+ }+ state.ActiveAlerts[0].MetricStatus = &status+ }+ }+ setStatus(`{"phase":"breaching","value":85,"trigger":80,"recovery":75,"observedAt":"2026-10-09T19:00:00Z","lastBreachAt":"2026-10-09T19:00:00Z"}`)+ previous, err := buildClientStateSnapshot(state)+ if err != nil {+ t.Fatal(err)+ }+ var received map[string]any+ if err := json.Unmarshal(previous.keyed[activeAlertsField].entries["alert-temperature"], &received); err != nil {+ t.Fatal(err)+ }+ checkDate := func(want string) {+ t.Helper()+ status, _ := received["metricStatus"].(map[string]any)+ got, present := status["lastBreachAt"]+ if present != (want != "") || want != "" && got != want {+ t.Errorf("receiver date = %v present %v, want %q", got, present, want)+ }+ if _, present := received["lastSeen"]; present {+ t.Fatal("state dated lifecycle alerts through LastSeen")+ }+ }+ checkDate("2026-10-09T19:00:00Z")+ for _, step := range []struct{ name, body, date string }{+ {"hold", `{"phase":"latched","value":78,"trigger":80,"recovery":75,"observedAt":"2026-10-09T19:01:00Z","lastBreachAt":"2026-10-09T19:00:00Z"}`, "2026-10-09T19:00:00Z"},+ {"recovery", `{"phase":"recovering","value":72,"trigger":80,"recovery":75,"observedAt":"2026-10-09T19:02:00Z","lastBreachAt":"2026-10-09T19:00:00Z","recoveryStartedAt":"2026-10-09T19:02:00Z"}`, "2026-10-09T19:00:00Z"},+ {"new breach", `{"phase":"breaching","value":90,"trigger":80,"recovery":75,"observedAt":"2026-10-09T19:03:00Z","lastBreachAt":"2026-10-09T19:03:00Z"}`, "2026-10-09T19:03:00Z"},+ {"unknown date", `{"phase":"latched","value":78,"trigger":80,"recovery":75,"observedAt":"2026-10-09T19:04:00Z"}`, ""},+ {"no live status", "", ""},+ } {+ t.Run(step.name, func(t *testing.T) {+ setStatus(step.body)+ current, err := buildClientStateSnapshot(state)+ if err != nil {+ t.Fatal(err)+ }+ delta, err := buildClientStateDelta(previous, current)+ if err != nil {+ t.Fatal(err)+ }+ payload, ok := delta[activeAlertsDeltaField].(resourceDeltaPayload)+ if !ok || len(payload.Upserts) != 1 || len(payload.Removed) != 0 {+ t.Fatalf("alert transition did not produce one keyed patch: %+v", delta)+ }+ if _, full := delta[activeAlertsField]; full {+ t.Fatal("transition re-shipped all active alerts")+ }+ var patch map[string]any+ if err := json.Unmarshal(payload.Upserts[0], &patch); err != nil {+ t.Fatal(err)+ }+ received = applyLabelTagMergePatch(received, patch)+ var complete map[string]any+ if err := json.Unmarshal(current.keyed[activeAlertsField].entries["alert-temperature"], &complete); err != nil {+ t.Fatal(err)+ }+ if !reflect.DeepEqual(received, complete) {+ t.Fatalf("receiver differs from complete state: %+v vs %+v", received, complete)+ }+ checkDate(step.date)+ previous = current+ })+ }+}