Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Preserve held threshold alerts’ breach dates in live state

pulse-triage[bot]5709c6073c2d
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 841ac9578c..e0fdf6ff73 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -141,6 +141,11 @@ the alerts subsystem. Agent report admission, host continuity and
re-enrollment never read or write it, and it is not persisted with agent or
state records.
+`MetricAlertStatus.lastBreachAt` is an optional server-evaluated breach date,
+not an agent report timestamp. It stays with the volatile alert status and
+neither changes agent admission nor adds persistent agent/state fields. The
+model wire/clone controls cover known, offset, unknown and Go-zero dates.
+
### Source-record lookup preserves report admission
Host and Docker report admission still performs canonical host-view matching
diff --git a/docs/release-control/v6/internal/subsystems/alerts.md b/docs/release-control/v6/internal/subsystems/alerts.md
index 61fb8cae35..80d99f4d91 100644
--- a/docs/release-control/v6/internal/subsystems/alerts.md
+++ b/docs/release-control/v6/internal/subsystems/alerts.md
@@ -102,6 +102,20 @@ day is not auto-resolved as unmonitored. Frontend surfaces format the status
through `features/alerts/metricAlertPresentation.ts` and never re-derive the
phase or recovery timing.
+The optional `metricStatus.lastBreachAt` dates the evaluated value retained in
+`Alert.Value`, not the newer `observedAt` of a hold or recovery. A breaching
+sample (including equality at the trigger) advances it; a held or recovering
+sample uses the existing alert's `LastSeen`. Unknown/zero dates are omitted,
+never replaced by the occurrence start, restart or poll time. It remains
+volatile with the status, so restart rebuilds it from the saved breach and a
+hold does not rewrite the recovery mirror. Resolution drops the whole status.
+
+`TestMetricBreachTimeTracksTheEvaluatedBreach` checks holds, recovery progress
+and reset, exact-trigger and renewed breaches. The existing live-status and
+checkpoint controls also assert the breach date; restart controls cover both
+durable active state and the JSON mirror, unchanged mirror bytes/mtime, and
+legacy unknown-date recovery through a new breach.
+
### Alert card and open incident lead with the live reading — issue #2068
The Alerts overview card leads an open threshold alert with the presentation's
diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md
index 05ea050f9e..2b60032623 100644
--- a/docs/release-control/v6/internal/subsystems/api-contracts.md
+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md
@@ -127,6 +127,18 @@ while recovering, `recoveryStartedAt` and `recoveryElapsedSeconds`. Legacy
trigger. The field is absent for non-threshold alerts and until the first
evaluation after a restart, and clients then fall back to `message`.
+`metricStatus.lastBreachAt` optionally dates the alert's retained evaluated
+breach value. Holds and recovery do not advance it to `observedAt`, and an
+unknown/zero date is omitted. The websocket and `/api/state` projection still
+omit legacy `lastSeen`; `/api/alerts/active` retains it. Old clients may ignore
+the additive field; consumers without it may use a valid legacy `lastSeen`
+but must not fabricate the date from `startTime` or the current observation.
+After restart no live status exists until evaluation; resolved rows have none.
+
+`TestMetricBreachTimeStateDeltasPreserveAndWithdraw` reconstructs the client
+baseline through held, recovery, renewed-breach, unknown-date and no-status
+keyed patches, including removal rather than retention of an obsolete date.
+
### Organisation deletion retains data after incomplete monitoring shutdown
Authenticated organisation-owner deletion waits for tenant-loop exit and sealed
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index c53a8c63d7..7ec82afccf 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -243,6 +243,13 @@ lists carry the reading being evaluated now for a threshold alert held below
its trigger, while `value` and `message` keep the last breach. The projection
adds no copy of its own: `GetActiveAlerts` already returns deep clones.
+The optional status field `lastBreachAt` carries the held breach's original
+`LastSeen` through that same projection, without publishing the poll-varying
+`LastSeen` of other lifecycle alerts. `observedAt` still dates the current
+comparison. `TestSyncAlertsToStateCarriesLiveMetricStatusOfHeldAlert` checks
+the concrete frontend JSON as well as the in-memory held reading, with an
+independent breach date and omitted legacy `lastSeen`.
+
### Fresh broadcast projection owns one resource/target capture
Canonical live, fallback and standalone-host continuity broadcast reads prefer
diff --git a/internal/alerts/alerts_test.go b/internal/alerts/alerts_test.go
index fbdea21e4b..18f1a96a94 100644
--- a/internal/alerts/alerts_test.go
+++ b/internal/alerts/alerts_test.go
@@ -22472,7 +22472,7 @@ func TestCleanupRetentionRequiresObservationInactivity(t *testing.T) {
func TestMetricStatusStaysOutOfDurableAlertSnapshots(t *testing.T) {
m, clock := newMetricStatusTestManager(t)
- fireNodeTemperatureAlert(t, m, clock, 85)
+ fired := fireNodeTemperatureAlert(t, m, clock, 85)
clock.advance(30 * time.Second)
checkMetricStatusNode(m, 78)
@@ -22480,6 +22480,9 @@ func TestMetricStatusStaysOutOfDurableAlertSnapshots(t *testing.T) {
if len(live) != 1 || live[0].MetricStatus == nil || live[0].MetricStatus.Value != 78 {
t.Fatalf("live alerts do not carry the current reading: %+v", live)
}
+ if got := metricBreachTime(t, live[0].MetricStatus); !got.Equal(fired.LastSeen) {
+ t.Fatalf("live status lost the breach's date: %v, want %v", got, fired.LastSeen)
+ }
for _, alert := range m.snapshotActiveAlerts() {
if alert.MetricStatus != nil {
t.Fatalf("durable checkpoint carries volatile status for %s: %+v", alert.ID, alert.MetricStatus)
diff --git a/internal/alerts/canonical_metric.go b/internal/alerts/canonical_metric.go
index 8d29198075..d1d5bf40ff 100644
--- a/internal/alerts/canonical_metric.go
+++ b/internal/alerts/canonical_metric.go
@@ -282,6 +282,15 @@ func (m *Manager) evaluateCanonicalMetricAlert(spec alertspecs.ResourceAlertSpec
alertMetadata[k] = v
}
}
+ // Date the evaluated breach, not a newer hold/recovery observation.
+ // A restored alert retains its breach's LastSeen; without that evidence
+ // the date stays unknown rather than becoming the restart or poll time.
+ var lastBreachAt time.Time
+ if triggered {
+ lastBreachAt = observedAt
+ } else if exists && existingAlert != nil {
+ lastBreachAt = existingAlert.LastSeen
+ }
metricStatus := buildMetricAlertStatus(metricStatusInput{
value: value,
triggered: triggered,
@@ -292,6 +301,7 @@ func (m *Manager) evaluateCanonicalMetricAlert(spec alertspecs.ResourceAlertSpec
window: windowed,
incident: incident,
observedAt: observedAt,
+ lastBreachAt: lastBreachAt,
})
if !exists {
diff --git a/internal/alerts/metric_breach_time_test.go b/internal/alerts/metric_breach_time_test.go
new file mode 100644
index 0000000000..8690adb7e1
--- /dev/null
+++ b/internal/alerts/metric_breach_time_test.go
@@ -0,0 +1,176 @@
+package alerts
+
+import (
+ "bytes"
+ "encoding/json"
+ "os"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/models"
+)
+
+// Decode the public field rather than sharing the evaluator's choice of date.
+// These controls also compile against a source which does not expose it yet.
+func metricBreachTime(t *testing.T, status *models.MetricAlertStatus) time.Time {
+ t.Helper()
+ data, err := json.Marshal(status)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var wire struct {
+ LastBreachAt time.Time `json:"lastBreachAt"`
+ }
+ if err := json.Unmarshal(data, &wire); err != nil {
+ t.Fatal(err)
+ }
+ return wire.LastBreachAt
+}
+
+func TestMetricBreachTimeTracksTheEvaluatedBreach(t *testing.T) {
+ m, clock := newMetricStatusTestManager(t)
+ fired := fireNodeTemperatureAlert(t, m, clock, 85)
+ lastBreach := fired.LastSeen
+ for _, step := range []struct {
+ name string
+ value float64
+ phase string
+ breaches bool
+ }{
+ {"held", 78, models.MetricAlertPhaseLatched, false},
+ {"recovery starts", 72, models.MetricAlertPhaseRecovering, false},
+ {"recovery advances", 75, models.MetricAlertPhaseRecovering, false},
+ {"recovery resets", 77, models.MetricAlertPhaseLatched, false},
+ {"exact trigger breaches", 80, models.MetricAlertPhaseBreaching, true},
+ {"held after new breach", 78, models.MetricAlertPhaseLatched, false},
+ {"higher breach", 90, models.MetricAlertPhaseBreaching, true},
+ {"recovering after new breach", 74, models.MetricAlertPhaseRecovering, false},
+ } {
+ t.Run(step.name, func(t *testing.T) {
+ clock.advance(30 * time.Second)
+ checkMetricStatusNode(m, step.value)
+ if step.breaches {
+ lastBreach = clock.now
+ }
+ alert := activeNodeTemperatureAlert(t, m)
+ if alert == nil || alert.MetricStatus == nil || alert.MetricStatus.Phase != step.phase {
+ t.Fatalf("alert = %+v, want phase %s", alert, step.phase)
+ }
+ if got := metricBreachTime(t, alert.MetricStatus); !got.Equal(lastBreach) {
+ t.Errorf("lastBreachAt = %v, want last evaluated breach %v", got, lastBreach)
+ }
+ if !alert.LastSeen.Equal(lastBreach) || !alert.MetricStatus.ObservedAt.Equal(clock.now) {
+ t.Errorf("breach snapshot and live observation diverged: %+v", alert)
+ }
+ })
+ }
+}
+
+func TestMetricBreachTimeRestoredWithoutLiveStatus(t *testing.T) {
+ for _, durable := range []bool{false, true} {
+ name := "recovery mirror"
+ var options []ManagerOption
+ if durable {
+ name = "durable active state"
+ options = []ManagerOption{WithDurableAlertStore()}
+ }
+ t.Run(name, func(t *testing.T) {
+ dir := t.TempDir()
+ m := NewManagerWithDataDir(dir, options...)
+ t.Cleanup(m.Stop)
+ clock := &metricStatusClock{now: time.Now().UTC()}
+ m.now = func() time.Time { return clock.now }
+ m.intentClock = func() time.Duration { return clock.tick }
+ cfg := m.GetConfig()
+ cfg.NodeDefaults.Temperature = &HysteresisThreshold{Trigger: 80, Clear: 75}
+ m.UpdateConfig(cfg)
+ fired := fireNodeTemperatureAlert(t, m, clock, 85)
+ if err := m.SaveActiveAlerts(); err != nil {
+ t.Fatal(err)
+ }
+ mirror := filepath.Join(dir, "alerts", "active-alerts.json")
+ before, err := os.ReadFile(mirror)
+ if err != nil {
+ t.Fatal(err)
+ }
+ beforeInfo, err := os.Stat(mirror)
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ clock.advance(time.Minute)
+ checkMetricStatusNode(m, 78)
+ if err := m.SaveActiveAlerts(); err != nil {
+ t.Fatal(err)
+ }
+ after, err := os.ReadFile(mirror)
+ if err != nil {
+ t.Fatal(err)
+ }
+ afterInfo, err := os.Stat(mirror)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !bytes.Equal(before, after) || !beforeInfo.ModTime().Equal(afterInfo.ModTime()) {
+ t.Fatal("holding an alert rewrote its durable recovery mirror")
+ }
+ if bytes.Contains(after, []byte("metricStatus")) || bytes.Contains(after, []byte("lastBreachAt")) {
+ t.Fatal("volatile metric status reached durable alert storage")
+ }
+ m.Stop()
+
+ restored := NewManagerWithDataDir(dir, options...)
+ t.Cleanup(restored.Stop)
+ restored.UpdateConfig(cfg)
+ clock.advance(2 * time.Minute)
+ restored.now = func() time.Time { return clock.now }
+ restored.intentClock = func() time.Duration { return clock.tick }
+ alert := activeNodeTemperatureAlert(t, restored)
+ if alert == nil || alert.MetricStatus != nil || !alert.LastSeen.Equal(fired.LastSeen) {
+ t.Fatalf("restart must retain the breach, not a live reading: %+v", alert)
+ }
+ checkMetricStatusNode(restored, 78)
+ alert = activeNodeTemperatureAlert(t, restored)
+ if alert == nil || alert.MetricStatus == nil || alert.MetricStatus.Phase != models.MetricAlertPhaseLatched {
+ t.Fatalf("first restored hold = %+v", alert)
+ }
+ if got := metricBreachTime(t, alert.MetricStatus); !got.Equal(fired.LastSeen) {
+ t.Fatalf("restored hold dated breach %v, want %v, not restart time %v", got, fired.LastSeen, clock.now)
+ }
+ if !alert.MetricStatus.ObservedAt.Equal(clock.now) {
+ t.Fatal("restored live status kept the pre-restart observation")
+ }
+ })
+ }
+}
+
+func TestMetricBreachTimeUnknownUntilANewBreach(t *testing.T) {
+ m, clock := newMetricStatusTestManager(t)
+ fired := fireNodeTemperatureAlert(t, m, clock, 85)
+ // An older checkpoint can have the incident and value but no date. The
+ // occurrence's start or the next healthy reading cannot date that value.
+ fired.LastSeen = time.Time{}
+ fired.MetricStatus = nil
+ if err := m.restoreActiveAlertSnapshots([]*Alert{fired}, "legacy checkpoint without breach time", true); err != nil {
+ t.Fatal(err)
+ }
+ clock.advance(time.Minute)
+ checkMetricStatusNode(m, 78)
+ held := activeNodeTemperatureAlert(t, m)
+ if held == nil || held.MetricStatus == nil || held.MetricStatus.Phase != models.MetricAlertPhaseLatched {
+ t.Fatalf("restored unknown-date hold = %+v", held)
+ }
+ if got := metricBreachTime(t, held.MetricStatus); !got.IsZero() {
+ t.Fatalf("unknown breach date became %v", got)
+ }
+ data, err := json.Marshal(held.MetricStatus)
+ if err != nil || bytes.Contains(data, []byte("lastBreachAt")) {
+ t.Fatalf("unknown date reached wire: %s / %v", data, err)
+ }
+ clock.advance(time.Minute)
+ checkMetricStatusNode(m, 80)
+ if got := metricBreachTime(t, activeNodeTemperatureAlert(t, m).MetricStatus); !got.Equal(clock.now) {
+ t.Fatalf("new breach date = %v, want %v", got, clock.now)
+ }
+}
diff --git a/internal/alerts/metric_status.go b/internal/alerts/metric_status.go
index 7bc19204b1..d9a547ddd4 100644
--- a/internal/alerts/metric_status.go
+++ b/internal/alerts/metric_status.go
@@ -20,6 +20,7 @@ type metricStatusInput struct {
window metricWindowObservation
incident reducer.Incident
observedAt time.Time
+ lastBreachAt time.Time
}
// metricStatusUnit is the unit of an evaluated metric value. CPU and memory
@@ -52,6 +53,7 @@ func buildMetricAlertStatus(input metricStatusInput) *models.MetricAlertStatus {
Trigger: input.trigger,
Recovery: recovery,
RecoveryDelaySeconds: input.recoveryDelaySeconds,
+ LastBreachAt: input.lastBreachAt,
}
if input.window.WindowSeconds > 0 {
raw := input.window.CurrentValue
diff --git a/internal/alerts/unified_eval_test.go b/internal/alerts/unified_eval_test.go
index 09b4e176c2..ab533b8003 100644
--- a/internal/alerts/unified_eval_test.go
+++ b/internal/alerts/unified_eval_test.go
@@ -1751,7 +1751,8 @@ func TestOpenTemperatureAlertReportsLiveReadingWhileItHolds(t *testing.T) {
fired := fireNodeTemperatureAlert(t, m, clock, 85)
status := fired.MetricStatus
if status == nil || status.Phase != models.MetricAlertPhaseBreaching || status.Value != 85 ||
- status.Trigger != 80 || status.Recovery != 75 || status.Unit != "°C" {
+ status.Trigger != 80 || status.Recovery != 75 || status.Unit != "°C" ||
+ !metricBreachTime(t, status).Equal(fired.LastSeen) {
t.Fatalf("breaching status = %+v", status)
}
if status.RecoveryDelaySeconds <= 0 {
@@ -1772,7 +1773,8 @@ func TestOpenTemperatureAlertReportsLiveReadingWhileItHolds(t *testing.T) {
t.Fatalf("hold rewrote the breach snapshot: value=%v message=%q lastSeen=%v", held.Value, held.Message, held.LastSeen)
}
if got := held.MetricStatus; got == nil || got.Phase != models.MetricAlertPhaseLatched || got.Value != 78 ||
- !got.ObservedAt.Equal(clock.now) || got.RecoveryStartedAt != nil {
+ !got.ObservedAt.Equal(clock.now) || got.RecoveryStartedAt != nil ||
+ !metricBreachTime(t, got).Equal(firedLastSeen) {
t.Fatalf("latched status = %+v", got)
}
@@ -1782,13 +1784,15 @@ func TestOpenTemperatureAlertReportsLiveReadingWhileItHolds(t *testing.T) {
checkMetricStatusNode(m, 72)
if got := activeNodeTemperatureAlert(t, m).MetricStatus; got == nil ||
got.Phase != models.MetricAlertPhaseRecovering || got.Value != 72 ||
- got.RecoveryStartedAt == nil || !got.RecoveryStartedAt.Equal(recoveryStart) || got.RecoveryElapsedSeconds != 0 {
+ got.RecoveryStartedAt == nil || !got.RecoveryStartedAt.Equal(recoveryStart) || got.RecoveryElapsedSeconds != 0 ||
+ !metricBreachTime(t, got).Equal(firedLastSeen) {
t.Fatalf("recovery start status = %+v", got)
}
clock.advance(delay / 2)
checkMetricStatusNode(m, 75)
if got := activeNodeTemperatureAlert(t, m).MetricStatus; got == nil ||
- got.Phase != models.MetricAlertPhaseRecovering || got.RecoveryElapsedSeconds != int(delay/2/time.Second) {
+ got.Phase != models.MetricAlertPhaseRecovering || got.RecoveryElapsedSeconds != int(delay/2/time.Second) ||
+ !metricBreachTime(t, got).Equal(firedLastSeen) {
t.Fatalf("recovery progress status = %+v", got)
}
@@ -1796,7 +1800,8 @@ func TestOpenTemperatureAlertReportsLiveReadingWhileItHolds(t *testing.T) {
clock.advance(30 * time.Second)
checkMetricStatusNode(m, 77)
if got := activeNodeTemperatureAlert(t, m).MetricStatus; got == nil ||
- got.Phase != models.MetricAlertPhaseLatched || got.RecoveryStartedAt != nil || got.RecoveryElapsedSeconds != 0 {
+ got.Phase != models.MetricAlertPhaseLatched || got.RecoveryStartedAt != nil || got.RecoveryElapsedSeconds != 0 ||
+ !metricBreachTime(t, got).Equal(firedLastSeen) {
t.Fatalf("status after recovery reset = %+v", got)
}
if got := len(notified); got != notifications {
diff --git a/internal/models/metrics_types_test.go b/internal/models/metrics_types_test.go
index de1973b924..a381fc6b03 100644
--- a/internal/models/metrics_types_test.go
+++ b/internal/models/metrics_types_test.go
@@ -550,3 +550,47 @@ func TestMetricAlertStatusCloneIsDeep(t *testing.T) {
t.Fatal("nil status must clone to nil")
}
}
+
+// Check the encoded additive contract as an older client would see it; missing
+// dates stay missing, including Go's zero time, without changing other fields.
+func TestMetricAlertStatusBreachTimeWireAndClone(t *testing.T) {
+ for _, tc := range []struct{ name, date string }{
+ {"known", "2026-10-09T19:25:16.123456789Z"},
+ {"offset", "2026-10-09T20:25:16+01:00"},
+ {"unknown", ""},
+ {"zero", "0001-01-01T00:00:00Z"},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ input := map[string]any{"phase": "latched", "value": 78, "trigger": 80, "recovery": 75}
+ if tc.date != "" {
+ input["lastBreachAt"] = tc.date
+ }
+ data, err := json.Marshal(input)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var status MetricAlertStatus
+ if err := json.Unmarshal(data, &status); err != nil {
+ t.Fatal(err)
+ }
+ for _, copy := range []*MetricAlertStatus{&status, status.Clone()} {
+ data, err = json.Marshal(copy)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var wire map[string]any
+ if err := json.Unmarshal(data, &wire); err != nil {
+ t.Fatal(err)
+ }
+ got, present := wire["lastBreachAt"]
+ wantDate := tc.date != "" && tc.date != "0001-01-01T00:00:00Z"
+ if present != wantDate || wantDate && got != tc.date {
+ t.Errorf("lastBreachAt = %v, present %v; want %q present %v", got, present, tc.date, wantDate)
+ }
+ if wire["value"] != float64(78) || wire["phase"] != "latched" {
+ t.Fatalf("new date changed legacy reading: %s", data)
+ }
+ }
+ })
+ }
+}
diff --git a/internal/models/models.go b/internal/models/models.go
index 71c8db5262..b1ee9cd09e 100644
--- a/internal/models/models.go
+++ b/internal/models/models.go
@@ -165,6 +165,10 @@ type MetricAlertStatus struct {
// Recovery progress, present only in the recovering phase.
RecoveryStartedAt *time.Time `json:"recoveryStartedAt,omitempty"`
RecoveryElapsedSeconds int `json:"recoveryElapsedSeconds,omitempty"`
+ // LastBreachAt dates the alert's Value: the latest evaluated reading that
+ // met the trigger, not the newer observation of a hold or recovery run.
+ // State/websocket alerts omit LastSeen; zero means the date is unknown.
+ LastBreachAt time.Time `json:"lastBreachAt,omitzero"`
}
// Clone returns a deep copy.
diff --git a/internal/monitoring/monitor_alert_intent_test.go b/internal/monitoring/monitor_alert_intent_test.go
index 2ec3c86355..49ca8f7e65 100644
--- a/internal/monitoring/monitor_alert_intent_test.go
+++ b/internal/monitoring/monitor_alert_intent_test.go
@@ -1,6 +1,7 @@
package monitoring
import (
+ "encoding/json"
"testing"
"time"
@@ -278,6 +279,15 @@ func TestSyncAlertsToStateCarriesLiveMetricStatusOfHeldAlert(t *testing.T) {
Temperature: &models.Temperature{Available: true, CPUPackage: 85},
}
m.alertManager.CheckNode(node)
+ var breachedAt time.Time
+ for _, alert := range m.alertManager.GetActiveAlerts() {
+ if alert.Type == "temperature" {
+ breachedAt = alert.LastSeen
+ }
+ }
+ if breachedAt.IsZero() {
+ t.Fatal("breaching temperature did not open the alert")
+ }
node.Temperature = &models.Temperature{Available: true, CPUPackage: 78}
m.alertManager.CheckNode(node)
@@ -303,6 +313,44 @@ func TestSyncAlertsToStateCarriesLiveMetricStatusOfHeldAlert(t *testing.T) {
if status.Phase != models.MetricAlertPhaseLatched || status.Value != 78 || status.Recovery != 75 {
t.Fatalf("live status = %+v, want latched at 78 clearing at 75", status)
}
+ // The concrete /api/state and websocket projection has no LastSeen. The
+ // live status must date the held breach without advancing it to this poll.
+ data, err := json.Marshal(m.state.ToFrontend())
+ if err != nil {
+ t.Fatal(err)
+ }
+ var wire struct {
+ ActiveAlerts []struct {
+ Type string `json:"type"`
+ LastSeen *time.Time `json:"lastSeen"`
+ MetricStatus *struct {
+ LastBreachAt time.Time `json:"lastBreachAt"`
+ ObservedAt time.Time `json:"observedAt"`
+ } `json:"metricStatus"`
+ } `json:"activeAlerts"`
+ }
+ if err := json.Unmarshal(data, &wire); err != nil {
+ t.Fatal(err)
+ }
+ found := false
+ for _, alert := range wire.ActiveAlerts {
+ if alert.Type != "temperature" {
+ continue
+ }
+ found = true
+ if alert.LastSeen != nil {
+ t.Fatal("projection added poll-varying legacy LastSeen")
+ }
+ if alert.MetricStatus == nil || !alert.MetricStatus.LastBreachAt.Equal(breachedAt) {
+ t.Fatalf("projected breach time = %+v, want %v", alert.MetricStatus, breachedAt)
+ }
+ if !alert.MetricStatus.ObservedAt.After(breachedAt) {
+ t.Fatal("hold did not carry a newer independent observation")
+ }
+ }
+ if !found {
+ t.Fatal("frontend projection lost the held temperature alert")
+ }
}
// After a restart, a host that has not reported again exists only through
diff --git a/internal/websocket/state_delta_test.go b/internal/websocket/state_delta_test.go
index 29d30408b8..e2c3f233cb 100644
--- a/internal/websocket/state_delta_test.go
+++ b/internal/websocket/state_delta_test.go
@@ -511,3 +511,82 @@ func TestClientStateBaselineAdvancesOnlyAfterDeltaIsQueued(t *testing.T) {
t.Fatal("client baseline did not advance after the delta was queued")
}
}
+
+// A receiver retains an unchanged breach date across held/recovery patches,
+// advances it only with a new breach, and drops it when the producer has none.
+func TestMetricBreachTimeStateDeltasPreserveAndWithdraw(t *testing.T) {
+ state := models.EmptyStateFrontend()
+ state.ActiveAlerts = []models.Alert{{ID: "alert-temperature", Type: "temperature", Value: 85}}
+ setStatus := func(body string) {
+ t.Helper()
+ state.ActiveAlerts[0].MetricStatus = nil
+ if body != "" {
+ var status models.MetricAlertStatus
+ if err := json.Unmarshal([]byte(body), &status); err != nil {
+ t.Fatal(err)
+ }
+ state.ActiveAlerts[0].MetricStatus = &status
+ }
+ }
+ setStatus(`{"phase":"breaching","value":85,"trigger":80,"recovery":75,"observedAt":"2026-10-09T19:00:00Z","lastBreachAt":"2026-10-09T19:00:00Z"}`)
+ previous, err := buildClientStateSnapshot(state)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var received map[string]any
+ if err := json.Unmarshal(previous.keyed[activeAlertsField].entries["alert-temperature"], &received); err != nil {
+ t.Fatal(err)
+ }
+ checkDate := func(want string) {
+ t.Helper()
+ status, _ := received["metricStatus"].(map[string]any)
+ got, present := status["lastBreachAt"]
+ if present != (want != "") || want != "" && got != want {
+ t.Errorf("receiver date = %v present %v, want %q", got, present, want)
+ }
+ if _, present := received["lastSeen"]; present {
+ t.Fatal("state dated lifecycle alerts through LastSeen")
+ }
+ }
+ checkDate("2026-10-09T19:00:00Z")
+ for _, step := range []struct{ name, body, date string }{
+ {"hold", `{"phase":"latched","value":78,"trigger":80,"recovery":75,"observedAt":"2026-10-09T19:01:00Z","lastBreachAt":"2026-10-09T19:00:00Z"}`, "2026-10-09T19:00:00Z"},
+ {"recovery", `{"phase":"recovering","value":72,"trigger":80,"recovery":75,"observedAt":"2026-10-09T19:02:00Z","lastBreachAt":"2026-10-09T19:00:00Z","recoveryStartedAt":"2026-10-09T19:02:00Z"}`, "2026-10-09T19:00:00Z"},
+ {"new breach", `{"phase":"breaching","value":90,"trigger":80,"recovery":75,"observedAt":"2026-10-09T19:03:00Z","lastBreachAt":"2026-10-09T19:03:00Z"}`, "2026-10-09T19:03:00Z"},
+ {"unknown date", `{"phase":"latched","value":78,"trigger":80,"recovery":75,"observedAt":"2026-10-09T19:04:00Z"}`, ""},
+ {"no live status", "", ""},
+ } {
+ t.Run(step.name, func(t *testing.T) {
+ setStatus(step.body)
+ current, err := buildClientStateSnapshot(state)
+ if err != nil {
+ t.Fatal(err)
+ }
+ delta, err := buildClientStateDelta(previous, current)
+ if err != nil {
+ t.Fatal(err)
+ }
+ payload, ok := delta[activeAlertsDeltaField].(resourceDeltaPayload)
+ if !ok || len(payload.Upserts) != 1 || len(payload.Removed) != 0 {
+ t.Fatalf("alert transition did not produce one keyed patch: %+v", delta)
+ }
+ if _, full := delta[activeAlertsField]; full {
+ t.Fatal("transition re-shipped all active alerts")
+ }
+ var patch map[string]any
+ if err := json.Unmarshal(payload.Upserts[0], &patch); err != nil {
+ t.Fatal(err)
+ }
+ received = applyLabelTagMergePatch(received, patch)
+ var complete map[string]any
+ if err := json.Unmarshal(current.keyed[activeAlertsField].entries["alert-temperature"], &complete); err != nil {
+ t.Fatal(err)
+ }
+ if !reflect.DeepEqual(received, complete) {
+ t.Fatalf("receiver differs from complete state: %+v vs %+v", received, complete)
+ }
+ checkDate(step.date)
+ previous = current
+ })
+ }
+}