Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Push alert and status changes to the agent event stream
rcourtman182067027f26
diff --git a/cmd/pulse-mcp/README.md b/cmd/pulse-mcp/README.mdindex d9c6f36064..6bd8353cda 100644--- a/cmd/pulse-mcp/README.md+++ b/cmd/pulse-mcp/README.md@@ -218,6 +218,9 @@ notification on stdout for every non-transport event:{ "jsonrpc": "2.0", "method": "notifications/finding.created", "params": { ... } }{ "jsonrpc": "2.0", "method": "notifications/approval.pending", "params": { ... } }{ "jsonrpc": "2.0", "method": "notifications/action.completed", "params": { ... } }+{ "jsonrpc": "2.0", "method": "notifications/alert.fired", "params": { ... } }+{ "jsonrpc": "2.0", "method": "notifications/alert.resolved", "params": { ... } }+{ "jsonrpc": "2.0", "method": "notifications/resource.status_changed", "params": { ... } }```The `params` object is the SSE event's `data` payload verbatim, so an@@ -225,6 +228,17 @@ agent that already knows the substrate's wire shape sees identicalcontent to what an HTTP SSE consumer would. Transport plumbing(`stream.connected`, `heartbeat`) is filtered out.+`alert.fired`, `alert.resolved` and `resource.status_changed` carry the+canonical `resourceId`, resource name and type, and the `changeId` and `kind`+that `list_recent_changes` returns for the same change, so an agent can read+the surrounding history. They carry the alert identifier, type and level, or+the status `from` and `to`, never alert message or command text. A resource's+status changes within 30 seconds of the last announced one fold into a single+event with `coalescedTransitions`; a flap that ends where it started is not+announced. Pulse announces changes as it records them; it does not replay old+alert history on restart, and an event dropped for a slow client is still+readable through `list_recent_changes`.+`action.completed` also covers failed or refused terminal actions; its namealone is not a successful dispatch or recovery receipt. Inspect the result andverification evidence. Missing checks or `ran: false` do not establish thediff --git a/docs/AGENT_SUBSTRATE.md b/docs/AGENT_SUBSTRATE.mdindex 391d242da2..47afb3cec5 100644--- a/docs/AGENT_SUBSTRATE.md+++ b/docs/AGENT_SUBSTRATE.md@@ -110,7 +110,13 @@ actor. Validation failures emit the `operator_state_invalid` andthe specific conflict instead of retrying blindly.**Push.** `/api/agent/events` is an SSE stream that fires `finding.created`,-`approval.pending`, and `action.completed` as state changes. `action.completed`+`approval.pending`, `action.completed`, `alert.fired`, `alert.resolved` and+`resource.status_changed` as state changes. The alert and status events are+doorbells for changes Pulse just recorded: each carries the canonical+`resourceId` and the `changeId` that `list_recent_changes` returns, so read the+history there rather than treating the event as the whole record. Status flaps+on one resource are coalesced into at most one event per 30 seconds.+`action.completed`means a terminal action record, **including failed or refused actions**, notnecessarily a successful dispatch. Inspect its result and verification evidence;the event name alone does not confirm the intended outcome. An idle stream isdiff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 99244550fd..44f7ec9c1f 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -2749,6 +2749,14 @@ agents may read what changed across the fleet, but `command_executed` and`commandRedacted:true`, the command text, output, and caller details dropped,and a command's reason cut to its status. A change record is history, never commanddisclosure, install-token scope, or execution authority.++The `alert.fired`, `alert.resolved` and `resource.status_changed` events on+`/api/agent/events` are the push form of the same records and the same+boundary. Their payload carries only identifiers, the alert type and level, and+the status `from`/`to`; it never carries alert messages, change reasons, command+metadata or anything a monitoring-read token could not already read. An event is+a doorbell for a recorded change, never an instruction to act, install-token+scope, or execution authority.Agent-facing provisioning capabilities declared by`/api/agent/capabilities` are also API-owned projections over the canonicalnode lifecycle and discovery APIs. Lifecycle and infrastructure onboardingdiff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.mdindex 0deb2bad8a..589756684b 100644--- a/docs/release-control/v6/internal/subsystems/ai-runtime.md+++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md@@ -1984,8 +1984,12 @@ The Pulse Intelligence event vocabulary is part of that same sharedtransport-event filtering, MCP notification method projection, shared SSErecord parsing, shared SSE-to-MCP notification bridging, and reference probesmust consume the shared event kind constants/helpers instead of maintaining local copies of-`finding.created`, `approval.pending`, `action.completed`, `stream.connected`,-or `heartbeat`.+`finding.created`, `approval.pending`, `action.completed`, `alert.fired`,+`alert.resolved`, `resource.status_changed`, `stream.connected`, or+`heartbeat`. `AgentActionableEventKinds` lists the three change kinds, so MCP+`--emit-notifications` advertises and forwards them without a local list, and+`SubscribeEventsDescription` derives their wording, including the+`ResourceStatusEventCoalesceWindow` duration, from the same constants.MCP `tools/list` must use the shared`agentcapabilities.ProjectManifestSurfaceTools` projection for the Pulse MCPsurface. The manifest-owned `surfaceToolContracts` field is the allowlist fordiff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.mdindex a0c8b68ab1..13bd1ff6f2 100644--- a/docs/release-control/v6/internal/subsystems/api-contracts.md+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md@@ -6262,6 +6262,46 @@ by `internal/agentcapabilities/events.go`; API constants may expose typedaliases, but API handlers, manifest descriptions, MCP adapters, and probes mustnot re-declare the vocabulary independently.+The stream also carries `alert.fired`, `alert.resolved` and+`resource.status_changed`, the push half of `list_recent_changes`. They are+published from `internal/api/agent_change_events.go` at the moment a monitor's+`unifiedresources.MonitorAdapter` records the matching canonical resource+change: the adapter's `SetChangeObserver` hears registry-diff changes after the+new generation is swapped in, and alert lifecycle changes recorded through+`RecordLiveChange`, with the resource ID already resolved to the canonical+identity the row was stored under. Nothing is read back from the change store+and nothing new is persisted, so the events add no disk writes. Only the live+alert lifecycle callback records through `RecordLiveChange`; lifecycle replay,+active-alert reconciliation and imported history use `RecordChange`, so a+restart never re-announces old alerts. The payload+(`AgentEventResourceChangePayload`) mirrors the change record the timeline read+returns: `changeId` is its `id`, `kind` its `kind`, plus `resourceId`, current+`resourceName` and `resourceType`, `from`/`to` for status changes, and+`alertIdentifier`, `alertType`, `alertLevel` and `alertResolution` for alerts.+It never carries the alert message, change reason or command metadata. A+`state_transition` counts as `resource.status_changed` only when its+`changedFields` include `status` and the status differs; discovery, removal,+Docker command and Proxmox lifecycle transitions stay timeline-only. Status+events are coalesced per organization and resource over+`agentcapabilities.ResourceStatusEventCoalesceWindow` (30s): the first change+in a quiet period publishes at once, later changes fold into one event at the+window's end carrying `coalescedTransitions` (from the last announced status to+the latest), and a flap that returns to the announced status publishes nothing.+Volume is otherwise bounded by the broadcaster's existing per-subscriber drop.+These events are tenant-scoped: `AgentEvent` carries an unexported+organization tag set from the recording monitor, and `HandleAgentEvents`+delivers a tagged event only to subscribers whose request organization matches.+Untagged kinds keep reaching every subscriber. Proofs:+`TestAgentChangeEvents*` in `internal/api/agent_change_events_test.go`,+`TestContract_AgentChangeEventsStayInsideTheirOrganization` in+`internal/api/contract_test.go`,+`TestAgentSubstrate_LiveChangeEventsReachSubscribeEventsStream` in+`internal/api/agent_substrate_e2e_test.go`,+`TestMonitorAdapterChangeObserverHearsOnlyRecordedLiveChanges` in+`internal/unifiedresources/history_identity_test.go`, and+`TestAlertTimelineAnnouncesOnlyLiveLifecycleChanges` in+`internal/monitoring/monitor_alert_handling_test.go`.+The approval store exposes `SetOnApprovalCreated(cb)` so the APIlayer can install a fire-and-forget callback that runs after everysuccessful `CreateApproval`. The callback fires on its owndiff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 00d2282c9f..9140144a51 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -8765,3 +8765,17 @@ recovery and confirmed scoped absence. Their opt-in producer feeds the existingproduction Backups consumer for browser proof. Retained evidence is not a freshread, available archive, installed recovery or a diagnosis of #3248; no guest orbackup operation, thaw or filesystem-liveness assertion follows.++### Only live alert lifecycle changes are announced to agents (29 October 2026)++The alert manager's live lifecycle callback reaches+`Monitor.handleAlertLifecycleEvent`, which projects the event with+`live=true`; lifecycle replay and active-alert reconciliation call+`projectAlertLifecycleEvent(event, false)`, and imported history always+projects as not live. A live projection records its canonical resource change+through the resource store's `RecordLiveChange` when the store offers it, so+the agent event stream hears `alert.fired` and `alert.resolved` once, when the+alert engine emits them; every other projection uses `RecordChange` and stays+silent. Both paths write the same deterministic change row. Proof:+`TestAlertTimelineAnnouncesOnlyLiveLifecycleChanges` in+`internal/monitoring/monitor_alert_handling_test.go`.diff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.mdindex e217673167..c96cb0c2f2 100644--- a/docs/release-control/v6/internal/subsystems/security-privacy.md+++ b/docs/release-control/v6/internal/subsystems/security-privacy.md@@ -3757,3 +3757,16 @@ the session router's default branch, which still refuses tokens without`ai:chat` before answering `404`(`TestRelayMobileAccessScopeDeniesAdjacentAIRoutes`). The undo, redo, and steernotes above are history.++### Agent change events stay inside their organization (29 October 2026)++`alert.fired`, `alert.resolved` and `resource.status_changed` events carry an+unexported organization tag from the monitor that recorded the change, and+`HandleAgentEvents` delivers a tagged event only to subscribers whose request+organization matches (the legacy empty organization is `default`). A tenant+therefore never hears another tenant's alerts or outages on+`/api/agent/events`. The payload excludes alert messages, change reasons and+command metadata, so a monitoring-read token learns nothing from an event that+it could not already read through `list_recent_changes`. Proof:+`TestContract_AgentChangeEventsStayInsideTheirOrganization` in+`internal/api/contract_test.go`.diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex f25f6714e2..1b67c5e3f2 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -4712,6 +4712,16 @@ readable without a name, so the timeline's recovery posture is exactly the`resource_changes` table's: whatever survives a restart or restore is whatagents read, with no timeline-specific rehydration or recovery artifact.+The live `alert.fired`, `alert.resolved` and `resource.status_changed` agent+events add no persistence. They are published in memory after the monitor+adapter has recorded the matching `resource_changes` row, and per-resource+status coalescing lives only in process memory. An event dropped for a slow+subscriber, or lost to a restart, is not replayed; the durable row it pointed+at stays readable through `list_recent_changes`. Alert lifecycle replay,+active-alert reconciliation and imported history after a restore record their+rows through the non-announcing `RecordChange` path, so recovery never+re-announces old alerts.+The Patrol-control status endpoint at`/api/agent/patrol-control/status` introduces no new persistence. The legacy`/api/agent/operations-loop/status` URL remains a compatibility alias. It reads thediff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.mdindex 9ed78f5406..89cba4c4e0 100644--- a/docs/release-control/v6/internal/subsystems/unified-resources.md+++ b/docs/release-control/v6/internal/subsystems/unified-resources.md@@ -9042,3 +9042,18 @@ protect grouped metadata, responsive columns and bounded rendering. Browserproof must exercise the consuming list at desktop/narrow widths, inspect actualpixels, identity, state/age fit, touch/keyboard disclosure, filters and liveupdates. This is safer evidence inspection, not a restore or installed recovery.++### Monitor adapter announces the changes it recorded (29 October 2026)++`MonitorAdapter.SetChangeObserver` installs one non-blocking observer that+hears each change the adapter just persisted, with its resource ID resolved to+the canonical identity the row was stored under. Registry diffs from+`replaceRegistry` and `PopulateSupplementalRecords` are announced after the new+generation is swapped in, so an observer reading the adapter sees the+generation that produced the change; `recordRegistryChanges` and+`recordRegistryChangesBetweenGenerations` return only the changes the store+accepted. Alert lifecycle rows are announced only through `RecordLiveChange`;+`RecordChange` stays silent so replayed and imported history is not presented+as new. The API agent event stream is the only observer today. Proof:+`TestMonitorAdapterChangeObserverHearsOnlyRecordedLiveChanges` in+`internal/unifiedresources/history_identity_test.go`.diff --git a/frontend-modern/public/docs/AGENT_SUBSTRATE.md b/frontend-modern/public/docs/AGENT_SUBSTRATE.mdindex 391d242da2..47afb3cec5 100644--- a/frontend-modern/public/docs/AGENT_SUBSTRATE.md+++ b/frontend-modern/public/docs/AGENT_SUBSTRATE.md@@ -110,7 +110,13 @@ actor. Validation failures emit the `operator_state_invalid` andthe specific conflict instead of retrying blindly.**Push.** `/api/agent/events` is an SSE stream that fires `finding.created`,-`approval.pending`, and `action.completed` as state changes. `action.completed`+`approval.pending`, `action.completed`, `alert.fired`, `alert.resolved` and+`resource.status_changed` as state changes. The alert and status events are+doorbells for changes Pulse just recorded: each carries the canonical+`resourceId` and the `changeId` that `list_recent_changes` returns, so read the+history there rather than treating the event as the whole record. Status flaps+on one resource are coalesced into at most one event per 30 seconds.+`action.completed`means a terminal action record, **including failed or refused actions**, notnecessarily a successful dispatch. Inspect its result and verification evidence;the event name alone does not confirm the intended outcome. An idle stream isdiff --git a/internal/agentcapabilities/events.go b/internal/agentcapabilities/events.goindex 4787089604..f66e604842 100644--- a/internal/agentcapabilities/events.go+++ b/internal/agentcapabilities/events.go@@ -1,5 +1,7 @@package agentcapabilities+import "time"+// EventKind is the stable Pulse Intelligence event identifier used by API SSE// streams and external-agent adapters.type EventKind string@@ -16,6 +18,18 @@ const (// state.EventKindActionCompleted EventKind = "action.completed"+ // EventKindAlertFired fires when Pulse records an alert firing (or+ // re-firing) on a resource.+ EventKindAlertFired EventKind = "alert.fired"++ // EventKindAlertResolved fires when Pulse records an alert clearing.+ EventKindAlertResolved EventKind = "alert.resolved"++ // EventKindResourceStatusChanged fires when Pulse records a resource's+ // status changing, coalesced per resource so flaps cannot flood the+ // stream.+ EventKindResourceStatusChanged EventKind = "resource.status_changed"+// EventKindStreamConnected is the one-shot SSE synchronization marker for a// newly opened stream.EventKindStreamConnected EventKind = "stream.connected"@@ -28,8 +42,16 @@ var agentActionableEventKinds = []string{string(EventKindFindingCreated),string(EventKindApprovalPending),string(EventKindActionCompleted),+ string(EventKindAlertFired),+ string(EventKindAlertResolved),+ string(EventKindResourceStatusChanged),}+// ResourceStatusEventCoalesceWindow bounds resource.status_changed volume: a+// resource's first status change is announced at once, and further changes+// inside the window fold into at most one event at the window's end.+const ResourceStatusEventCoalesceWindow = 30 * time.Second+// AgentActionableEventKinds returns the non-transport event kinds external// agents should expect to handle as product notifications.func AgentActionableEventKinds() []string {@@ -54,6 +76,8 @@ func SubscribeEventsDescription() string {return "Subscribe to the SSE event stream for real-time notifications: " +string(EventKindFindingCreated) + " when a new finding is raised, " +string(EventKindApprovalPending) + " when a remediation request enters StatusPending and waits on operator decision, " +- string(EventKindActionCompleted) + " when an action audit reaches a terminal state (Completed or Failed, including refused-before-dispatch failures with stable error-token prefixes; carries a verification block with the read-after-write probe outcome so agents close the certainty loop without polling /api/actions/{id}), stream-local " ++ string(EventKindActionCompleted) + " when an action audit reaches a terminal state (Completed or Failed, including refused-before-dispatch failures with stable error-token prefixes; carries a verification block with the read-after-write probe outcome so agents close the certainty loop without polling /api/actions/{id}), " ++ string(EventKindAlertFired) + " and " + string(EventKindAlertResolved) + " when an alert fires or clears on a resource (alert identifier, type and level), " ++ string(EventKindResourceStatusChanged) + " when a resource's status changes (from/to; further changes within " + ResourceStatusEventCoalesceWindow.String() + " fold into one event carrying coalescedTransitions). Change events carry the canonical resourceId, resource name and type, and the changeId and kind that " + ListRecentChangesCapabilityName + " returns, so agents can read the surrounding history; replayed or imported history is not announced. Stream-local " +string(EventKindHeartbeat) + " every 15s. Command fields are redacted for monitoring-read API tokens unless the token also has ai:execute. Long-lived connection; agents listen instead of polling."}diff --git a/internal/agentcapabilities/events_test.go b/internal/agentcapabilities/events_test.goindex b5155f4e89..95e2a34d94 100644--- a/internal/agentcapabilities/events_test.go+++ b/internal/agentcapabilities/events_test.go@@ -12,6 +12,9 @@ func TestAgentActionableEventKindsReturnsDetachedKnownKinds(t *testing.T) {string(EventKindFindingCreated),string(EventKindApprovalPending),string(EventKindActionCompleted),+ string(EventKindAlertFired),+ string(EventKindAlertResolved),+ string(EventKindResourceStatusChanged),}if !reflect.DeepEqual(got, want) {t.Fatalf("AgentActionableEventKinds() = %v, want %v", got, want)@@ -44,4 +47,8 @@ func TestSubscribeEventsDescriptionUsesSharedEventKinds(t *testing.T) {t.Fatalf("subscribe_events description missing %q: %s", kind, description)}}+ // Change events point agents at the timeline read for surrounding history.+ if !strings.Contains(description, ListRecentChangesCapabilityName) {+ t.Fatalf("subscribe_events description must name %s: %s", ListRecentChangesCapabilityName, description)+ }}diff --git a/internal/api/agent_change_events.go b/internal/api/agent_change_events.gonew file mode 100644index 0000000000..69b74c7a97--- /dev/null+++ b/internal/api/agent_change_events.go@@ -0,0 +1,238 @@+package api++import (+ "strings"+ "sync"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/agentcapabilities"+ unified "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"+)++// AgentEventResourceChangePayload is the payload shape for alert.fired,+// alert.resolved and resource.status_changed events. It mirrors the change+// record list_recent_changes returns (changeId is that record's id, kind its+// kind) so an agent can read the surrounding history, and carries no alert+// message, reason text or command. ResourceName and ResourceType are the+// resource's current values and are empty when Pulse no longer knows it.+//+// CoalescedTransitions is set only on a resource.status_changed event that+// was held back by the per-resource coalescing window. It counts the status+// changes Pulse recorded in that window; From is the status last announced+// and To the status at the window's end. Every individual transition stays+// readable through list_recent_changes.+type AgentEventResourceChangePayload struct {+ ChangeID string `json:"changeId"`+ ResourceID string `json:"resourceId"`+ ResourceName string `json:"resourceName,omitempty"`+ ResourceType string `json:"resourceType,omitempty"`+ Kind string `json:"kind"`+ From string `json:"from,omitempty"`+ To string `json:"to,omitempty"`+ AlertIdentifier string `json:"alertIdentifier,omitempty"`+ AlertType string `json:"alertType,omitempty"`+ AlertLevel string `json:"alertLevel,omitempty"`+ AlertResolution string `json:"alertResolution,omitempty"`+ ObservedAt time.Time `json:"observedAt"`+ OccurredAt *time.Time `json:"occurredAt,omitempty"`+ CoalescedTransitions int `json:"coalescedTransitions,omitempty"`+}++// agentResourceLookup resolves a resource reference to the current resource;+// MonitorAdapter.GetByReference satisfies it.+type agentResourceLookup func(reference string) (*unified.Resource, string, bool)++type agentStatusKey struct {+ orgID string+ resourceID string+}++// agentStatusWindow is one resource's open coalescing window: the status last+// announced and the latest status change folded since then.+type agentStatusWindow struct {+ announced string+ pending *AgentEventResourceChangePayload+ folded int+}++// agentChangeEventProducer turns resource changes the monitor adapters just+// recorded into agent events. It reads nothing back from the change store and+// writes nothing, so it adds no disk load. Alert events publish as recorded;+// the alert engine's own delays and hysteresis already bound them. Status+// changes are coalesced per resource: the first change in a quiet period is+// announced at once, later changes inside the window fold into at most one+// event when it closes, and a flap that ends on the status last announced+// publishes nothing.+type agentChangeEventProducer struct {+ broadcaster *AgentEventBroadcaster+ window time.Duration+ afterFunc func(time.Duration, func())++ mu sync.Mutex+ windows map[agentStatusKey]*agentStatusWindow+}++func newAgentChangeEventProducer(broadcaster *AgentEventBroadcaster, window time.Duration) *agentChangeEventProducer {+ return &agentChangeEventProducer{+ broadcaster: broadcaster,+ window: window,+ afterFunc: func(d time.Duration, f func()) {+ time.AfterFunc(d, f)+ },+ windows: make(map[agentStatusKey]*agentStatusWindow),+ }+}++// observer returns the change observer for one organization's monitor+// adapter. Events it publishes reach only that organization's subscribers.+func (p *agentChangeEventProducer) observer(orgID string, lookup agentResourceLookup) unified.ResourceChangeObserver {+ orgID = normalizeAgentEventOrgID(orgID)+ return func(change unified.ResourceChange) {+ p.observe(orgID, change, lookup)+ }+}++func (p *agentChangeEventProducer) observe(orgID string, change unified.ResourceChange, lookup agentResourceLookup) {+ if p == nil || p.broadcaster == nil {+ return+ }+ kind, ok := agentEventKindForChange(change)+ if !ok {+ return+ }+ payload := agentResourceChangePayload(change)+ if kind != AgentEventResourceStatusChanged {+ p.publish(orgID, kind, payload, lookup)+ return+ }++ key := agentStatusKey{orgID: orgID, resourceID: payload.ResourceID}+ p.mu.Lock()+ if window, open := p.windows[key]; open {+ window.folded+++ payload.From = window.announced+ window.pending = &payload+ p.mu.Unlock()+ return+ }+ p.openStatusWindowLocked(key, payload.To, lookup)+ p.mu.Unlock()+ p.publish(orgID, kind, payload, lookup)+}++func (p *agentChangeEventProducer) openStatusWindowLocked(key agentStatusKey, announced string, lookup agentResourceLookup) {+ p.windows[key] = &agentStatusWindow{announced: announced}+ p.afterFunc(p.window, func() {+ p.closeStatusWindow(key, lookup)+ })+}++// closeStatusWindow publishes the folded change, if it left the resource in a+// different status from the one last announced, and opens a new window for+// it so continued flapping stays at one event per window.+func (p *agentChangeEventProducer) closeStatusWindow(key agentStatusKey, lookup agentResourceLookup) {+ p.mu.Lock()+ window := p.windows[key]+ delete(p.windows, key)+ if window == nil || window.pending == nil || window.pending.To == window.announced {+ p.mu.Unlock()+ return+ }+ payload := *window.pending+ payload.CoalescedTransitions = window.folded+ p.openStatusWindowLocked(key, payload.To, lookup)+ p.mu.Unlock()+ p.publish(key.orgID, AgentEventResourceStatusChanged, payload, lookup)+}++func (p *agentChangeEventProducer) publish(orgID string, kind AgentEventKind, payload AgentEventResourceChangePayload, lookup agentResourceLookup) {+ if lookup != nil {+ if resource, _, ok := lookup(payload.ResourceID); ok && resource != nil {+ payload.ResourceName = resource.Name+ payload.ResourceType = string(resource.Type)+ }+ }+ p.broadcaster.Publish(AgentEvent{+ Kind: kind,+ Payload: payload,+ orgID: orgID,+ })+}++// agentEventKindForChange maps a recorded change to the event agents hear.+// A state_transition counts as a status change only when the status field+// changed: discovery, removal, Docker command and Proxmox lifecycle+// transitions stay in list_recent_changes.+func agentEventKindForChange(change unified.ResourceChange) (AgentEventKind, bool) {+ switch change.Kind {+ case unified.ChangeAlertFired:+ return AgentEventAlertFired, true+ case unified.ChangeAlertResolved:+ return AgentEventAlertResolved, true+ case unified.ChangeStateTransition:+ if changeTouchesField(change, "status") && change.From != change.To {+ return AgentEventResourceStatusChanged, true+ }+ }+ return "", false+}++func changeTouchesField(change unified.ResourceChange, field string) bool {+ switch fields := change.Metadata["changedFields"].(type) {+ case []string:+ for _, candidate := range fields {+ if candidate == field {+ return true+ }+ }+ case []any:+ for _, candidate := range fields {+ if candidate == field {+ return true+ }+ }+ }+ return false+}++func agentResourceChangePayload(change unified.ResourceChange) AgentEventResourceChangePayload {+ payload := AgentEventResourceChangePayload{+ ChangeID: change.ID,+ ResourceID: unified.CanonicalResourceID(change.ResourceID),+ Kind: string(change.Kind),+ ObservedAt: change.ObservedAt,+ }+ if change.OccurredAt != nil {+ occurredAt := *change.OccurredAt+ payload.OccurredAt = &occurredAt+ }+ switch change.Kind {+ case unified.ChangeAlertFired, unified.ChangeAlertResolved:+ payload.AlertIdentifier = changeMetadataString(change, unified.MetadataAlertIdentifier)+ payload.AlertType = changeMetadataString(change, unified.MetadataAlertType)+ payload.AlertLevel = changeMetadataString(change, unified.MetadataAlertLevel)+ payload.AlertResolution = changeMetadataString(change, unified.MetadataAlertResolution)+ default:+ payload.From = change.From+ payload.To = change.To+ }+ return payload+}++func changeMetadataString(change unified.ResourceChange, key string) string {+ value, _ := change.Metadata[key].(string)+ return strings.TrimSpace(value)+}++// normalizeAgentEventOrgID maps the legacy empty organization to the default+// one, matching how request contexts name it.+func normalizeAgentEventOrgID(orgID string) string {+ orgID = strings.TrimSpace(orgID)+ if orgID == "" {+ return "default"+ }+ return orgID+}++// defaultResourceStatusEventWindow is the manifest-declared coalescing window.+const defaultResourceStatusEventWindow = agentcapabilities.ResourceStatusEventCoalesceWindowdiff --git a/internal/api/agent_change_events_test.go b/internal/api/agent_change_events_test.gonew file mode 100644index 0000000000..a8d22f4844--- /dev/null+++ b/internal/api/agent_change_events_test.go@@ -0,0 +1,224 @@+package api++import (+ "encoding/json"+ "strings"+ "testing"+ "time"++ unified "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"+)++// manualStatusWindows replaces the producer's timers so tests close windows+// deterministically.+type manualStatusWindows struct {+ pending []func()+}++func (m *manualStatusWindows) after(_ time.Duration, f func()) {+ m.pending = append(m.pending, f)+}++func (m *manualStatusWindows) closeAll() {+ pending := m.pending+ m.pending = nil+ for _, f := range pending {+ f()+ }+}++func newTestAgentChangeEventProducer(b *AgentEventBroadcaster) (*agentChangeEventProducer, *manualStatusWindows) {+ producer := newAgentChangeEventProducer(b, time.Minute)+ windows := &manualStatusWindows{}+ producer.afterFunc = windows.after+ return producer, windows+}++func testStatusChange(id, resourceID, from, to string) unified.ResourceChange {+ return unified.ResourceChange{+ ID: id,+ ResourceID: resourceID,+ Kind: unified.ChangeStateTransition,+ From: from,+ To: to,+ ObservedAt: time.Date(2026, 10, 29, 6, 0, 0, 0, time.UTC),+ Metadata: map[string]any{"changedFields": []string{"status"}},+ }+}++func drainAgentEvents(events <-chan AgentEvent) []AgentEvent {+ var out []AgentEvent+ for {+ select {+ case event := <-events:+ out = append(out, event)+ default:+ return out+ }+ }+}++func TestAgentChangeEventsPublishAlertLifecycleWithoutContent(t *testing.T) {+ b := NewAgentEventBroadcaster()+ events, unsubscribe := b.Subscribe()+ defer unsubscribe()+ producer, _ := newTestAgentChangeEventProducer(b)+ lookup := func(ref string) (*unified.Resource, string, bool) {+ if ref != "vm-web" {+ return nil, "", false+ }+ return &unified.Resource{ID: "vm-web", Name: "web", Type: unified.ResourceTypeVM}, "vm-web", true+ }+ observe := producer.observer("", lookup)+ occurredAt := time.Date(2026, 10, 29, 5, 59, 0, 0, time.UTC)+ observe(unified.ResourceChange{+ ID: "change-fired",+ ResourceID: "vm-web",+ Kind: unified.ChangeAlertFired,+ ObservedAt: occurredAt,+ OccurredAt: &occurredAt,+ Reason: "alert fired: CPU at 97% on web",+ Metadata: map[string]any{+ unified.MetadataAlertIdentifier: "vm-web-cpu",+ unified.MetadataAlertType: "cpu",+ unified.MetadataAlertLevel: "critical",+ unified.MetadataAlertMessage: "CPU at 97% on web",+ unified.MetadataCommand: "systemctl restart web",+ },+ })+ observe(unified.ResourceChange{+ ID: "change-resolved",+ ResourceID: "vm-web",+ Kind: unified.ChangeAlertResolved,+ ObservedAt: occurredAt.Add(time.Minute),+ Metadata: map[string]any{+ unified.MetadataAlertIdentifier: "vm-web-cpu",+ unified.MetadataAlertType: "cpu",+ unified.MetadataAlertLevel: "critical",+ unified.MetadataAlertResolution: "moved_to_agent",+ },+ })++ got := drainAgentEvents(events)+ if len(got) != 2 {+ t.Fatalf("events = %+v, want fired and resolved", got)+ }+ if got[0].Kind != AgentEventAlertFired || got[1].Kind != AgentEventAlertResolved {+ t.Fatalf("kinds = %s, %s", got[0].Kind, got[1].Kind)+ }+ fired := got[0].Payload.(AgentEventResourceChangePayload)+ want := AgentEventResourceChangePayload{+ ChangeID: "change-fired",+ ResourceID: "vm-web",+ ResourceName: "web",+ ResourceType: string(unified.ResourceTypeVM),+ Kind: string(unified.ChangeAlertFired),+ AlertIdentifier: "vm-web-cpu",+ AlertType: "cpu",+ AlertLevel: "critical",+ ObservedAt: occurredAt,+ OccurredAt: &occurredAt,+ }+ if fired.OccurredAt == nil || !fired.OccurredAt.Equal(occurredAt) {+ t.Fatalf("occurredAt = %v", fired.OccurredAt)+ }+ fired.OccurredAt = want.OccurredAt+ if fired != want {+ t.Fatalf("fired payload = %+v\nwant %+v", fired, want)+ }+ encoded, err := json.Marshal(got[0])+ if err != nil {+ t.Fatal(err)+ }+ for _, forbidden := range []string{"97%", "systemctl", "orgID", "default"} {+ if strings.Contains(string(encoded), forbidden) {+ t.Fatalf("alert event must not carry %q: %s", forbidden, encoded)+ }+ }+ if resolved := got[1].Payload.(AgentEventResourceChangePayload); resolved.AlertResolution != "moved_to_agent" {+ t.Fatalf("resolved payload = %+v", resolved)+ }+}++func TestAgentChangeEventsIgnoreChangesThatAreNotStatusOrAlerts(t *testing.T) {+ b := NewAgentEventBroadcaster()+ events, unsubscribe := b.Subscribe()+ defer unsubscribe()+ producer, _ := newTestAgentChangeEventProducer(b)+ observe := producer.observer("default", nil)++ discovered := testStatusChange("discovered", "vm-new", "absent", "online")+ discovered.Metadata = map[string]any{"changeType": "resource_created"}+ lifecycle := testStatusChange("lock", "vm-web", "online", "online")+ lifecycle.Metadata = map[string]any{"changedFields": []string{"proxmox.lifecycle"}}+ for _, change := range []unified.ResourceChange{+ discovered,+ lifecycle,+ {ID: "ack", ResourceID: "vm-web", Kind: unified.ChangeAlertAcknowledged},+ {ID: "restart", ResourceID: "vm-web", Kind: unified.ChangeRestart},+ {ID: "cmd", ResourceID: "vm-web", Kind: unified.ChangeCommandExecuted, Metadata: map[string]any{unified.MetadataCommand: "reboot"}},+ } {+ observe(change)+ }+ if got := drainAgentEvents(events); len(got) != 0 {+ t.Fatalf("events = %+v, want none", got)+ }+}++// A flapping resource yields its first change at once and at most one more+// event per window; a flap that ends where it started publishes nothing.+func TestAgentChangeEventsCoalesceStatusFlapsPerResource(t *testing.T) {+ b := NewAgentEventBroadcaster()+ events, unsubscribe := b.Subscribe()+ defer unsubscribe()+ producer, windows := newTestAgentChangeEventProducer(b)+ observe := producer.observer("default", nil)++ observe(testStatusChange("c1", "vm-web", "online", "offline"))+ observe(testStatusChange("other", "vm-db", "online", "offline"))+ for i, to := range []string{"online", "offline", "online", "offline", "online"} {+ from := map[string]string{"online": "offline", "offline": "online"}[to]+ observe(testStatusChange("flap-"+string(rune('a'+i)), "vm-web", from, to))+ }+ got := drainAgentEvents(events)+ if len(got) != 2 {+ t.Fatalf("immediate events = %+v, want one per resource", got)+ }+ first := got[0].Payload.(AgentEventResourceChangePayload)+ if got[0].Kind != AgentEventResourceStatusChanged || first.ChangeID != "c1" || first.From != "online" || first.To != "offline" || first.CoalescedTransitions != 0 {+ t.Fatalf("first event = %s %+v", got[0].Kind, first)+ }++ windows.closeAll()+ got = drainAgentEvents(events)+ if len(got) != 1 {+ t.Fatalf("window close events = %+v, want one coalesced event for vm-web", got)+ }+ coalesced := got[0].Payload.(AgentEventResourceChangePayload)+ if coalesced.ResourceID != "vm-web" || coalesced.ChangeID != "flap-e" || coalesced.From != "offline" || coalesced.To != "online" || coalesced.CoalescedTransitions != 5 {+ t.Fatalf("coalesced = %+v", coalesced)+ }++ // The coalesced event opened a new window. A flap inside it that returns+ // to the announced status publishes nothing when it closes.+ observe(testStatusChange("g", "vm-web", "online", "offline"))+ observe(testStatusChange("h", "vm-web", "offline", "online"))+ windows.closeAll()+ if got := drainAgentEvents(events); len(got) != 0 {+ t.Fatalf("round-trip flap events = %+v, want none", got)+ }++ // With every window closed, the next change is announced at once again.+ observe(testStatusChange("i", "vm-web", "online", "offline"))+ got = drainAgentEvents(events)+ if len(got) != 1 || got[0].Payload.(AgentEventResourceChangePayload).ChangeID != "i" {+ t.Fatalf("post-quiet events = %+v", got)+ }+ windows.closeAll()+ producer.mu.Lock()+ open := len(producer.windows)+ producer.mu.Unlock()+ if open != 0 {+ t.Fatalf("closed windows must be forgotten, %d still open", open)+ }+}diff --git a/internal/api/agent_events.go b/internal/api/agent_events.goindex d8a0a67ed7..179988af36 100644--- a/internal/api/agent_events.go+++ b/internal/api/agent_events.go@@ -47,6 +47,14 @@ const (// the audit endpoint.AgentEventActionCompleted AgentEventKind = AgentEventKind(agentcapabilities.EventKindActionCompleted)+ // AgentEventAlertFired, AgentEventAlertResolved and+ // AgentEventResourceStatusChanged fire when a monitor adapter records+ // the matching live resource change. Payload is+ // AgentEventResourceChangePayload; see agent_change_events.go.+ AgentEventAlertFired AgentEventKind = AgentEventKind(agentcapabilities.EventKindAlertFired)+ AgentEventAlertResolved AgentEventKind = AgentEventKind(agentcapabilities.EventKindAlertResolved)+ AgentEventResourceStatusChanged AgentEventKind = AgentEventKind(agentcapabilities.EventKindResourceStatusChanged)+// AgentEventStreamConnected is emitted once when an SSE subscriber connects// so clients can distinguish a healthy stream from an idle one.AgentEventStreamConnected AgentEventKind = AgentEventKind(agentcapabilities.EventKindStreamConnected)@@ -66,6 +74,15 @@ type AgentEvent struct {ID uint64 `json:"id"`At time.Time `json:"at"`Payload any `json:"payload,omitempty"`++ // orgID scopes the event to one organization's subscribers. Empty means+ // the event is not tenant-scoped and reaches every subscriber.+ orgID string+}++// visibleToOrg reports whether a subscriber in orgID may receive the event.+func (e AgentEvent) visibleToOrg(orgID string) bool {+ return e.orgID == "" || e.orgID == normalizeAgentEventOrgID(orgID)}// AgentEventFindingCreatedPayload is the payload shape for@@ -267,6 +284,7 @@ func (b *AgentEventBroadcaster) HandleAgentEvents(w http.ResponseWriter, r *httpevents, unsubscribe := b.Subscribe()defer unsubscribe()+ orgID := GetOrgID(r.Context())// Initial sync event so agents that just connected can confirm// the stream is open before the first real event lands. This is@@ -289,6 +307,9 @@ func (b *AgentEventBroadcaster) HandleAgentEvents(w http.ResponseWriter, r *httpif !open {return}+ if !event.visibleToOrg(orgID) {+ continue+ }writeAgentSSEEvent(w, redactAgentEventCommandsForRequest(event, r))flusher.Flush()case <-heartbeat.C:diff --git a/internal/api/agent_substrate_e2e_test.go b/internal/api/agent_substrate_e2e_test.goindex 5f68a234a5..044c16ef07 100644--- a/internal/api/agent_substrate_e2e_test.go+++ b/internal/api/agent_substrate_e2e_test.go@@ -1,16 +1,21 @@package apiimport (+ "bufio""bytes"+ "context""encoding/json""net/http""net/http/httptest""strings""testing"+ "time""github.com/rcourtman/pulse-go-rewrite/internal/agentcapabilities""github.com/rcourtman/pulse-go-rewrite/internal/config"+ "github.com/rcourtman/pulse-go-rewrite/internal/models""github.com/rcourtman/pulse-go-rewrite/internal/monitoring"+ unifiedresources "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources")// TestAgentSubstrate_DiscoveryToTriageToDepthFlowsThroughHTTPBoundary is@@ -778,3 +783,142 @@ func assertAgentStableEnvelope(t *testing.T, label, body, wantCode string) {}}}++// TestAgentSubstrate_LiveChangeEventsReachSubscribeEventsStream proves the+// push half of the change substrate through the HTTP boundary: a status+// change and a live alert that the monitor's adapter records reach a+// monitoring-read subscriber of /api/agent/events as resource.status_changed+// and alert.fired, with the canonical resource ID, current name and the same+// change IDs list_recent_changes returns for them.+func TestAgentSubstrate_LiveChangeEventsReachSubscribeEventsStream(t *testing.T) {+ rawToken := "agent-live-events-e2e.12345678"+ cfg := newTestConfigWithTokens(t, newTokenRecord(t, rawToken, []string{config.ScopeMonitoringRead}, nil))+ monitor, err := monitoring.New(cfg)+ if err != nil {+ t.Fatalf("new monitor: %v", err)+ }+ t.Cleanup(monitor.Stop)+ router := NewRouter(cfg, monitor, nil, nil, func() error { return nil }, "1.0.0")+ cleanupTestRouter(t, router)+ adapter := router.monitorAdapterForMonitor(monitor)+ if adapter == nil {+ t.Fatal("router must wire a resource adapter for the monitor")+ }+ now := time.Now().UTC().Truncate(time.Second)+ snapshot := models.StateSnapshot{+ Nodes: []models.Node{{ID: "lab-pve1", Name: "pve1", Instance: "lab", Host: "https://pve1.lab:8006", Status: "online", LastSeen: now}},+ VMs: []models.VM{{ID: "lab:pve1:101", VMID: 101, Name: "web", Node: "pve1", Instance: "lab", Status: "running", Type: "qemu", LastSeen: now}},+ }+ adapter.PopulateFromSnapshot(snapshot)++ server := httptest.NewServer(router.Handler())+ defer server.Close()+ ctx, cancel := context.WithCancel(context.Background())+ defer cancel()+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, server.URL+agentcapabilities.AgentEventsPath, nil)+ if err != nil {+ t.Fatal(err)+ }+ req.Header.Set("X-API-Token", rawToken)+ resp, err := http.DefaultClient.Do(req)+ if err != nil {+ t.Fatalf("subscribe: %v", err)+ }+ defer resp.Body.Close()+ if resp.StatusCode != http.StatusOK {+ t.Fatalf("subscribe status = %d", resp.StatusCode)+ }+ deadline := time.Now().Add(2 * time.Second)+ for time.Now().Before(deadline) && router.agentEventBroadcaster.SubscriberCount() == 0 {+ time.Sleep(10 * time.Millisecond)+ }++ snapshot.VMs[0].Status = "stopped"+ snapshot.VMs[0].LastSeen = now.Add(time.Minute)+ adapter.PopulateFromSnapshot(snapshot)+ if err := adapter.RecordLiveChange(unifiedresources.ResourceChange{+ ID: "e2e-alert-fired",+ ResourceID: "lab:pve1:101",+ Kind: unifiedresources.ChangeAlertFired,+ SourceType: unifiedresources.SourceHeuristic,+ ObservedAt: now.Add(time.Minute),+ Metadata: map[string]any{+ unifiedresources.MetadataAlertIdentifier: "lab:pve1:101-powered-off",+ unifiedresources.MetadataAlertType: "powered-off",+ unifiedresources.MetadataAlertLevel: "critical",+ unifiedresources.MetadataAlertMessage: "VM web is stopped",+ },+ }); err != nil {+ t.Fatalf("record live alert: %v", err)+ }++ type frame struct {+ Kind string `json:"kind"`+ Payload AgentEventResourceChangePayload `json:"payload"`+ }+ frames := make(chan []frame, 1)+ go func() {+ scanner := bufio.NewScanner(resp.Body)+ var seen []frame+ for scanner.Scan() {+ line := scanner.Text()+ if !strings.HasPrefix(line, "data: ") {+ continue+ }+ var f frame+ if json.Unmarshal([]byte(strings.TrimPrefix(line, "data: ")), &f) != nil {+ continue+ }+ if f.Kind == string(agentcapabilities.EventKindResourceStatusChanged) || f.Kind == string(agentcapabilities.EventKindAlertFired) {+ seen = append(seen, f)+ }+ if len(seen) == 2 {+ frames <- seen+ return+ }+ }+ }()+ var got []frame+ select {+ case got = <-frames:+ case <-time.After(5 * time.Second):+ t.Fatal("timed out waiting for change events on /api/agent/events")+ }+ cancel()++ status, alert := got[0], got[1]+ if status.Kind != string(agentcapabilities.EventKindResourceStatusChanged) || alert.Kind != string(agentcapabilities.EventKindAlertFired) {+ t.Fatalf("event order = %s, %s", status.Kind, alert.Kind)+ }+ vmID := status.Payload.ResourceID+ if vmID == "" || vmID == "lab:pve1:101" || status.Payload.ResourceName != "web" || status.Payload.ResourceType != string(unifiedresources.ResourceTypeVM) {+ t.Fatalf("status payload must carry the canonical VM identity: %+v", status.Payload)+ }+ if status.Payload.From != "online" || status.Payload.To != "offline" {+ t.Fatalf("status payload from/to = %+v", status.Payload)+ }+ if alert.Payload.ResourceID != vmID || alert.Payload.ResourceName != "web" || alert.Payload.ChangeID != "e2e-alert-fired" ||+ alert.Payload.AlertLevel != "critical" || alert.Payload.AlertType != "powered-off" {+ t.Fatalf("alert payload = %+v", alert.Payload)+ }++ // The events point at rows the timeline read returns.+ listReq := httptest.NewRequest(http.MethodGet, agentcapabilities.RecentChangesCapabilityPath+"?resourceId="+vmID, nil)+ listReq.Header.Set("X-API-Token", rawToken)+ listRec := httptest.NewRecorder()+ router.Handler().ServeHTTP(listRec, listReq)+ if listRec.Code != http.StatusOK {+ t.Fatalf("list_recent_changes status = %d, body = %s", listRec.Code, listRec.Body.String())+ }+ var listed AgentRecentChanges+ if err := json.NewDecoder(listRec.Body).Decode(&listed); err != nil {+ t.Fatal(err)+ }+ ids := map[string]bool{}+ for _, change := range listed.Changes {+ ids[change.ID] = true+ }+ if !ids[status.Payload.ChangeID] || !ids[alert.Payload.ChangeID] {+ t.Fatalf("event change IDs %q and %q must be readable through list_recent_changes: %+v", status.Payload.ChangeID, alert.Payload.ChangeID, listed.Changes)+ }+}diff --git a/internal/api/contract_test.go b/internal/api/contract_test.goindex 1e14be14d1..43b18e39d7 100644--- a/internal/api/contract_test.go+++ b/internal/api/contract_test.go@@ -1,6 +1,7 @@package apiimport (+ "bufio""bytes""context""crypto/ed25519"@@ -45,6 +46,7 @@ import ("github.com/rcourtman/pulse-go-rewrite/internal/api/agentbinding""github.com/rcourtman/pulse-go-rewrite/internal/api/agenttokens""github.com/rcourtman/pulse-go-rewrite/internal/api/alerting"+ "github.com/rcourtman/pulse-go-rewrite/internal/api/apicontext""github.com/rcourtman/pulse-go-rewrite/internal/api/chartapi""github.com/rcourtman/pulse-go-rewrite/internal/api/configapi""github.com/rcourtman/pulse-go-rewrite/internal/api/resourceapi"@@ -28675,3 +28677,62 @@ func TestContract_PulseMCPPublishesRecentChangesAsReadOnlyTool(t *testing.T) {}t.Fatalf("Pulse MCP surface does not publish %s", agentcapabilities.ListRecentChangesCapabilityName)}++// Change events reach only subscribers in the organization whose monitor+// recorded them; untagged events still reach everyone.+func TestContract_AgentChangeEventsStayInsideTheirOrganization(t *testing.T) {+ b := NewAgentEventBroadcaster()+ producer, _ := newTestAgentChangeEventProducer(b)+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ ctx := context.WithValue(r.Context(), apicontext.OrgIDContextKey, "tenant-a")+ b.HandleAgentEvents(w, r.WithContext(ctx))+ }))+ defer server.Close()+ ctx, cancel := context.WithCancel(context.Background())+ defer cancel()+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, server.URL, nil)+ if err != nil {+ t.Fatal(err)+ }+ resp, err := http.DefaultClient.Do(req)+ if err != nil {+ t.Fatal(err)+ }+ defer resp.Body.Close()+ deadline := time.Now().Add(2 * time.Second)+ for time.Now().Before(deadline) && b.SubscriberCount() == 0 {+ time.Sleep(10 * time.Millisecond)+ }++ producer.observer("tenant-b", nil)(unifiedresources.ResourceChange{ID: "foreign", ResourceID: "vm-b", Kind: unifiedresources.ChangeAlertFired})+ producer.observer("", nil)(unifiedresources.ResourceChange{ID: "default-org", ResourceID: "vm-d", Kind: unifiedresources.ChangeAlertFired})+ producer.observer("tenant-a", nil)(unifiedresources.ResourceChange{ID: "own", ResourceID: "vm-a", Kind: unifiedresources.ChangeAlertFired})+ b.PublishFindingCreated(AgentEventFindingCreatedPayload{FindingID: "untagged-finding"})++ got := make(chan string, 1)+ go func() {+ scanner := bufio.NewScanner(resp.Body)+ var collected strings.Builder+ for scanner.Scan() {+ collected.WriteString(scanner.Text())+ collected.WriteString("\n")+ if strings.Contains(collected.String(), "untagged-finding") {+ got <- collected.String()+ return+ }+ }+ }()+ select {+ case stream := <-got:+ if !strings.Contains(stream, "event: alert.fired") || !strings.Contains(stream, `"changeId":"own"`) {+ t.Fatalf("own organization's alert missing from stream:\n%s", stream)+ }+ for _, foreign := range []string{`"changeId":"foreign"`, `"changeId":"default-org"`} {+ if strings.Contains(stream, foreign) {+ t.Fatalf("stream leaked %s:\n%s", foreign, stream)+ }+ }+ case <-time.After(3 * time.Second):+ t.Fatal("timed out waiting for the stream")+ }+}diff --git a/internal/api/router.go b/internal/api/router.goindex 6087f062ef..f0f86c532c 100644--- a/internal/api/router.go+++ b/internal/api/router.go@@ -104,6 +104,7 @@ type Router struct {maintenanceSentinel *maintenancesentinel.SentinelagentContextHandler *AgentContextHandleragentEventBroadcaster *AgentEventBroadcaster+ agentChangeEvents *agentChangeEventProducerresourceRegistry *unifiedresources.ResourceRegistrytrueNASPoller *monitoring.TrueNASPollervmwarePoller *monitoring.VMwarePoller@@ -569,6 +570,7 @@ func (r *Router) setupRoutes() {// read for the wrong org.r.agentContextHandler.SetApprovalsProvider(agentApprovalStoreProvider{})r.agentEventBroadcaster = NewAgentEventBroadcaster()+ r.agentChangeEvents = newAgentChangeEventProducer(r.agentEventBroadcaster, defaultResourceStatusEventWindow)if r.resourceHandlers != nil && r.agentEventBroadcaster != nil {r.resourceHandlers.SetActionCompletedPublisher(r.agentEventBroadcaster.PublishActionCompletedRecord)}@@ -1831,6 +1833,11 @@ func (r *Router) configureMonitorDependenciesLocked(m *monitoring.Monitor) {// Apply saved notification policy before the first publication, which can// evaluate alerts. The combined boundary avoids one refresh per source.if adapter := r.monitorAdapterForMonitor(m); adapter != nil {+ // Live alert and status changes this adapter records reach the+ // agent event stream for the monitor's organization only.+ if r.agentChangeEvents != nil {+ adapter.SetChangeObserver(r.agentChangeEvents.observer(m.GetOrgID(), adapter.GetByReference))+ }m.SetResourceStoreAndProviders(adapter, r.monitorSupplementalRecords)} else {m.SetSupplementalRecordsProviders(r.monitorSupplementalRecords)diff --git a/internal/monitoring/monitor_alert_handling_test.go b/internal/monitoring/monitor_alert_handling_test.goindex 6f68e3b4e1..4b97a6d4f1 100644--- a/internal/monitoring/monitor_alert_handling_test.go+++ b/internal/monitoring/monitor_alert_handling_test.go@@ -7,6 +7,7 @@ import ("net/http""net/http/httptest""strings"+ "sync""sync/atomic""testing""time"@@ -1950,3 +1951,89 @@ func TestActiveAlertsSnapshotCarriesSubjectFreeSummary(t *testing.T) {}require.True(t, found, "expected the container health alert in the state snapshot")}++type observedAlertChanges struct {+ mu sync.Mutex+ changes []unifiedresources.ResourceChange+}++func (o *observedAlertChanges) observe(change unifiedresources.ResourceChange) {+ o.mu.Lock()+ defer o.mu.Unlock()+ if change.Kind == unifiedresources.ChangeAlertFired || change.Kind == unifiedresources.ChangeAlertResolved {+ o.changes = append(o.changes, change)+ }+}++func (o *observedAlertChanges) take() []unifiedresources.ResourceChange {+ o.mu.Lock()+ defer o.mu.Unlock()+ out := o.changes+ o.changes = nil+ return out+}++// Live alert lifecycle events reach the adapter's change observer (and so the+// agent event stream) with the canonical resource ID they were stored under.+// Replay, reconciliation and imported history record the same rows without+// announcing them again.+func TestAlertTimelineAnnouncesOnlyLiveLifecycleChanges(t *testing.T) {+ store, err := unifiedresources.NewSQLiteResourceStore(t.TempDir(), "default")+ require.NoError(t, err)+ t.Cleanup(func() { require.NoError(t, store.Close()) })+ manager := alerts.NewManagerWithDataDir(t.TempDir())+ t.Cleanup(manager.Stop)+ config := manager.GetConfig()+ config.Enabled = true+ config.ActivationState = alerts.ActivationPending+ config.TimeThresholds = map[string]int{}+ config.SuppressionWindow = 0+ manager.UpdateConfig(config)++ containerID := strings.Repeat("e", 64)+ host := models.DockerHost{ID: "live-host", Hostname: "live-host", LastSeen: time.Now(), Containers: []models.DockerContainer{+ {ID: containerID, Name: "api", State: "running", Health: "unhealthy"},+ }}+ registry := unifiedresources.NewRegistry(store)+ registry.IngestSnapshot(models.StateSnapshot{DockerHosts: []models.DockerHost{host}})+ adapter := unifiedresources.NewMonitorAdapter(registry)+ observed := &observedAlertChanges{}+ adapter.SetChangeObserver(observed.observe)+ monitor := &Monitor{alertManager: manager, resourceStore: adapter}+ manager.SubscribeLifecycleCallback(monitor.handleAlertLifecycleEvent)+ canonicalID := unifiedresources.SourceSpecificID(unifiedresources.ResourceTypeAppContainer, unifiedresources.SourceDocker, host.ID+"/container/"+containerID)++ manager.CheckDockerHost(host)+ fired := observed.take()+ require.Len(t, fired, 1)+ require.Equal(t, unifiedresources.ChangeAlertFired, fired[0].Kind)+ require.Equal(t, canonicalID, fired[0].ResourceID)+ require.Equal(t, "docker-container-health", fired[0].Metadata[unifiedresources.MetadataAlertType])++ host.Containers[0].Health = "healthy"+ manager.CheckDockerHost(host)+ resolved := observed.take()+ require.Len(t, resolved, 1)+ require.Equal(t, unifiedresources.ChangeAlertResolved, resolved[0].Kind)+ require.Equal(t, canonicalID, resolved[0].ResourceID)++ // The announced changes are the stored rows, so agents can look them up.+ stored, err := store.GetRecentChangesFiltered(canonicalID, time.Time{}, 10, unifiedresources.ResourceChangeFilters{+ Kinds: []unifiedresources.ChangeKind{unifiedresources.ChangeAlertFired, unifiedresources.ChangeAlertResolved},+ })+ require.NoError(t, err)+ require.Len(t, stored, 2)+ require.ElementsMatch(t, []string{fired[0].ID, resolved[0].ID}, []string{stored[0].ID, stored[1].ID})++ // Replaying the whole lifecycle log projects the same events again; none+ // of them may be announced as new.+ monitor.replayAlertLifecycleProjections(context.Background())+ require.Empty(t, observed.take(), "replayed alert history must not reach the live change observer")++ alert := &alerts.Alert{ID: "imported-alert", Type: "cpu", Level: alerts.AlertLevelWarning, ResourceID: canonicalID, StartTime: time.Now().Add(-time.Hour)}+ monitor.recordAlertTimelineChange(alert, unifiedresources.ChangeAlertFired, alert.StartTime, "")+ require.Empty(t, observed.take(), "history projected outside the live callback must not be announced")+ recorded, err := store.GetRecentChangesFiltered(canonicalID, time.Time{}, 10, unifiedresources.ResourceChangeFilters{AlertIdentifiers: []string{alert.ID}})+ require.NoError(t, err)+ require.Len(t, recorded, 1, "the non-live projection still records the row")+}diff --git a/internal/monitoring/monitor_alerts.go b/internal/monitoring/monitor_alerts.goindex bdf77b0fc1..13bfb29ff1 100644--- a/internal/monitoring/monitor_alerts.go+++ b/internal/monitoring/monitor_alerts.go@@ -23,6 +23,13 @@ type canonicalResourceChangeRecorder interface {RecordChange(change unifiedresources.ResourceChange) error}+// liveResourceChangeRecorder records a change that is happening now and+// announces it to live subscribers (the agent event stream). Replayed and+// imported alert history uses RecordChange instead, so it is never announced.+type liveResourceChangeRecorder interface {+ RecordLiveChange(change unifiedresources.ResourceChange) error+}+// GetAlertManager returns the alert managerfunc (m *Monitor) GetAlertManager() *alerts.Manager {return m.alertManager@@ -328,7 +335,16 @@ func (m *Monitor) handleAlertEscalated(hub *websocket.Hub, alert *alerts.Alert,m.broadcastEscalatedAlert(hub, alert)}+// handleAlertLifecycleEvent projects a live lifecycle event from the alert+// manager's callback bus.func (m *Monitor) handleAlertLifecycleEvent(event alerts.LifecycleEvent) {+ m.projectAlertLifecycleEvent(event, true)+}++// projectAlertLifecycleEvent writes the incident timeline and canonical+// resource change for one lifecycle event. live is false for replay and+// reconciliation, which project events that already happened.+func (m *Monitor) projectAlertLifecycleEvent(event alerts.LifecycleEvent, live bool) {alert := event.Alertif m == nil || alert == nil {return@@ -356,7 +372,7 @@ func (m *Monitor) handleAlertLifecycleEvent(event alerts.LifecycleEvent) {if event.Type == eventlog.TypeFired && !alert.StartTime.IsZero() {occurredAt = alert.StartTime}- m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertFired, occurredAt, "")+ m.recordAlertTimelineChangeWithLiveness(timelineAlert, unifiedresources.ChangeAlertFired, occurredAt, "", live)case eventlog.TypeAcknowledged:if m.incidentStore != nil {m.incidentStore.RecordAlertAcknowledged(timelineAlert, actor)@@ -365,12 +381,12 @@ func (m *Monitor) handleAlertLifecycleEvent(event alerts.LifecycleEvent) {if alert.AckTime != nil && !alert.AckTime.IsZero() {occurredAt = *alert.AckTime}- m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertAcknowledged, occurredAt, actor)+ m.recordAlertTimelineChangeWithLiveness(timelineAlert, unifiedresources.ChangeAlertAcknowledged, occurredAt, actor, live)case eventlog.TypeUnacknowledged:if m.incidentStore != nil {m.incidentStore.RecordAlertUnacknowledged(timelineAlert, actor)}- m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertUnacknowledged, event.OccurredAt, actor)+ m.recordAlertTimelineChangeWithLiveness(timelineAlert, unifiedresources.ChangeAlertUnacknowledged, event.OccurredAt, actor, live)case eventlog.TypeSnoozed:timelineAlert = timelineAlert.Clone()if timelineAlert.Metadata == nil {@@ -379,14 +395,14 @@ func (m *Monitor) handleAlertLifecycleEvent(event alerts.LifecycleEvent) {if until := strings.TrimSpace(event.Details["until"]); until != "" {timelineAlert.Metadata["snoozedUntil"] = until}- m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertSnoozed, event.OccurredAt, actor)+ m.recordAlertTimelineChangeWithLiveness(timelineAlert, unifiedresources.ChangeAlertSnoozed, event.OccurredAt, actor, live)case eventlog.TypeUnsnoozed:- m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertUnsnoozed, event.OccurredAt, actor)+ m.recordAlertTimelineChangeWithLiveness(timelineAlert, unifiedresources.ChangeAlertUnsnoozed, event.OccurredAt, actor, live)case eventlog.TypeResolved:if m.incidentStore != nil {m.incidentStore.RecordAlertResolved(timelineAlert, event.OccurredAt)}- m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertResolved, event.OccurredAt, "")+ m.recordAlertTimelineChangeWithLiveness(timelineAlert, unifiedresources.ChangeAlertResolved, event.OccurredAt, "", live)case eventlog.TypeHistoryImported:m.materializeImportedAlertTimeline(timelineAlert, event.OccurredAt)}@@ -539,7 +555,7 @@ func (m *Monitor) replayAlertLifecycleProjections(ctx context.Context) {// Apply the whole occurrence (including imported acknowledgement and// resolution) before advancing its watermark, even if shutdown arrives// during a canonical write. Cancellation only stops the next event.- m.handleAlertLifecycleEvent(event)+ m.projectAlertLifecycleEvent(event, false)if eventID > maxApplied {maxApplied = eventID}@@ -603,26 +619,35 @@ func (m *Monitor) reconcileActiveAlertTimelines(ctx context.Context) {continue}}- m.handleAlertLifecycleEvent(alerts.LifecycleEvent{+ m.projectAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeFired,OccurredAt: alert.StartTime,Alert: alert,- })+ }, false)}}func (m *Monitor) recordAlertTimelineChange(alert *alerts.Alert, kind unifiedresources.ChangeKind, occurredAt time.Time, actor string) {+ m.recordAlertTimelineChangeWithLiveness(alert, kind, occurredAt, actor, false)+}++func (m *Monitor) recordAlertTimelineChangeWithLiveness(alert *alerts.Alert, kind unifiedresources.ChangeKind, occurredAt time.Time, actor string, live bool) {if alert == nil || m == nil {return}// Background catch-up replay runs concurrently with SetResourceStore, so// the store handle must be read under the monitor lock.m.mu.RLock()- recorder, ok := m.resourceStore.(canonicalResourceChangeRecorder)+ store := m.resourceStorem.mu.RUnlock()+ recorder, ok := store.(canonicalResourceChangeRecorder)if !ok || recorder == nil {return}+ record := recorder.RecordChange+ if liveRecorder, ok := store.(liveResourceChangeRecorder); ok && live {+ record = liveRecorder.RecordLiveChange+ }timelineChange := unifiedresources.AlertTimelineChange{AlertIdentifier: alert.ID,@@ -649,7 +674,7 @@ func (m *Monitor) recordAlertTimelineChange(alert *alerts.Alert, kind unifiedresstring(kind),occurredAt.UTC().Format(time.RFC3339Nano),}, "\x00"))).String()- if err := recorder.RecordChange(*change); err != nil {+ if err := record(*change); err != nil {log.Warn().Err(err).Str("resource_id", alert.ResourceID).diff --git a/internal/unifiedresources/change_emission.go b/internal/unifiedresources/change_emission.goindex ab3df74291..260eb1b826 100644--- a/internal/unifiedresources/change_emission.go+++ b/internal/unifiedresources/change_emission.go@@ -11,9 +11,12 @@ import ("github.com/google/uuid")-func recordRegistryChanges(store ResourceStore, before, after []Resource, observedAt time.Time, occurredAt *time.Time, sourceType ChangeSourceType, sourceAdapterHint ChangeSourceAdapter) {+// recordRegistryChanges persists the changes between two resource lists and+// returns the ones the store accepted, so the caller can announce exactly what+// was recorded.+func recordRegistryChanges(store ResourceStore, before, after []Resource, observedAt time.Time, occurredAt *time.Time, sourceType ChangeSourceType, sourceAdapterHint ChangeSourceAdapter) []ResourceChange {if store == nil {- return+ return nil}beforeByID := make(map[string]Resource, len(before))@@ -40,6 +43,7 @@ func recordRegistryChanges(store ResourceStore, before, after []Resource, observseen[id] = struct{}{}}+ var recorded []ResourceChangefor id := range seen {beforeResource, beforeOK := beforeByID[id]afterResource, afterOK := afterByID[id]@@ -49,8 +53,11 @@ func recordRegistryChanges(store ResourceStore, before, after []Resource, observ}if err := store.RecordChange(*change); err != nil {log.Printf("unifiedresources: failed to record change for %s: %v", change.ResourceID, err)+ continue}+ recorded = append(recorded, *change)}+ return recorded}// recordRegistryChangesBetweenGenerations compares two complete registry@@ -58,10 +65,11 @@ func recordRegistryChanges(store ResourceStore, before, after []Resource, observ// readers. The resources are read only while both registry locks are held;// persistence happens after releasing them so a slow store cannot block// registry readers. The caller must keep the old generation stable until this-// returns (MonitorAdapter's mutationMu does so during a replacement).-func recordRegistryChangesBetweenGenerations(before, after *ResourceRegistry, observedAt time.Time, occurredAt *time.Time, sourceType ChangeSourceType, sourceAdapterHint ChangeSourceAdapter) {+// returns (MonitorAdapter's mutationMu does so during a replacement). It+// returns the changes the store accepted.+func recordRegistryChangesBetweenGenerations(before, after *ResourceRegistry, observedAt time.Time, occurredAt *time.Time, sourceType ChangeSourceType, sourceAdapterHint ChangeSourceAdapter) []ResourceChange {if before == nil || after == nil || before.store == nil {- return+ return nil}var pending []ResourceChange@@ -100,11 +108,15 @@ func recordRegistryChangesBetweenGenerations(before, after *ResourceRegistry, obafter.mu.RUnlock()before.mu.RUnlock()+ recorded := pending[:0]for _, change := range pending {if err := before.store.RecordChange(change); err != nil {log.Printf("unifiedresources: failed to record change for %s: %v", change.ResourceID, err)+ continue}+ recorded = append(recorded, change)}+ return recorded}func buildResourceChange(before Resource, beforeOK bool, after Resource, afterOK bool, observedAt time.Time, occurredAt *time.Time, sourceType ChangeSourceType, sourceAdapterHint ChangeSourceAdapter) *ResourceChange {diff --git a/internal/unifiedresources/history_identity_test.go b/internal/unifiedresources/history_identity_test.goindex 95f12c8e32..f3d22ac01e 100644--- a/internal/unifiedresources/history_identity_test.go+++ b/internal/unifiedresources/history_identity_test.go@@ -1012,3 +1012,61 @@ func TestHistoryIdentityLinkedAgentDiskUsesCanonicalOwner(t *testing.T) {require.Empty(t, ownerID, ref)}}++// The change observer hears the changes a registry rebuild recorded, after+// the new generation is readable, and the alert rows recorded live through+// RecordLiveChange under their resolved canonical ID. RecordChange, the path+// replayed and imported history takes, stays silent.+func TestMonitorAdapterChangeObserverHearsOnlyRecordedLiveChanges(t *testing.T) {+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")+ require.NoError(t, err)+ t.Cleanup(func() { require.NoError(t, store.Close()) })+ now := time.Now().UTC().Truncate(time.Second)+ adapter := NewMonitorAdapter(NewRegistry(store))+ snapshot := proxmoxHistoryIdentitySnapshot(now)+ adapter.PopulateFromSnapshot(snapshot)+ vmID := historyIdentityResourceID(t, adapter.currentRegistry(), ResourceTypeVM, "web")++ var heard []ResourceChange+ var statusAtNotify string+ adapter.SetChangeObserver(func(change ResourceChange) {+ heard = append(heard, change)+ if change.ResourceID == vmID {+ if resource, _, ok := adapter.GetByReference(change.ResourceID); ok {+ statusAtNotify = string(resource.Status)+ }+ }+ })++ snapshot.VMs[0].Status = "stopped"+ snapshot.VMs[0].LastSeen = now.Add(time.Minute)+ adapter.PopulateFromSnapshot(snapshot)+ var transition *ResourceChange+ for i := range heard {+ if heard[i].ResourceID == vmID && heard[i].Kind == ChangeStateTransition {+ transition = &heard[i]+ }+ }+ require.NotNil(t, transition, "status change must reach the observer: %+v", heard)+ require.Equal(t, "online", transition.From)+ require.Equal(t, "offline", transition.To)+ require.Contains(t, transition.Metadata["changedFields"], "status")+ require.Equal(t, string(StatusOffline), statusAtNotify, "observer must read the generation that produced the change")+ stored, err := store.GetRecentChangesFiltered(vmID, time.Time{}, 10, ResourceChangeFilters{Kinds: []ChangeKind{ChangeStateTransition}})+ require.NoError(t, err)+ require.NotEmpty(t, stored)+ require.Equal(t, transition.ID, stored[0].ID)++ heard = nil+ require.NoError(t, adapter.RecordChange(ResourceChange{ID: "replayed", ResourceID: "lab:pve1:101", Kind: ChangeAlertFired, SourceType: SourceHeuristic, ObservedAt: now}))+ require.Empty(t, heard, "RecordChange must not announce")++ require.NoError(t, adapter.RecordLiveChange(ResourceChange{ID: "live", ResourceID: "lab:pve1:101", Kind: ChangeAlertFired, SourceType: SourceHeuristic, ObservedAt: now}))+ require.Len(t, heard, 1)+ require.Equal(t, "live", heard[0].ID)+ require.Equal(t, vmID, heard[0].ResourceID, "live alert changes carry the canonical ID they were stored under")++ adapter.SetChangeObserver(nil)+ require.NoError(t, adapter.RecordLiveChange(ResourceChange{ID: "unobserved", ResourceID: "lab:pve1:101", Kind: ChangeAlertResolved, SourceType: SourceHeuristic, ObservedAt: now}))+ require.Len(t, heard, 1)+}diff --git a/internal/unifiedresources/monitor_adapter.go b/internal/unifiedresources/monitor_adapter.goindex 732e1de1c9..03903492b9 100644--- a/internal/unifiedresources/monitor_adapter.go+++ b/internal/unifiedresources/monitor_adapter.go@@ -33,6 +33,43 @@ type MonitorAdapter struct {legacyHistory legacyHistoryBackfilloverlays overlayReadStateCache++ // changeObserver hears each live change after the store accepted it.+ // Guarded by mu.+ changeObserver ResourceChangeObserver+}++// ResourceChangeObserver is told about a change the adapter just recorded,+// with its resource ID already resolved to the canonical identity it was+// stored under. It runs on the recording goroutine, so it must not block.+// Replayed or imported history never reaches it: only registry diffs and+// changes recorded through RecordLiveChange do.+type ResourceChangeObserver func(change ResourceChange)++// SetChangeObserver installs the observer for live changes; nil removes it.+func (a *MonitorAdapter) SetChangeObserver(observer ResourceChangeObserver) {+ if a == nil {+ return+ }+ a.mu.Lock()+ a.changeObserver = observer+ a.mu.Unlock()+}++func (a *MonitorAdapter) notifyChanges(changes []ResourceChange) {+ if a == nil || len(changes) == 0 {+ return+ }+ a.mu.RLock()+ observer := a.changeObserver+ a.mu.RUnlock()+ if observer == nil {+ return+ }+ for _, change := range changes {+ change.ResourceID = CanonicalResourceID(change.ResourceID)+ observer(change)+ }}// overlayReadStateMaxAge bounds how long an overlay built from one registry@@ -245,9 +282,28 @@ func (a *MonitorAdapter) GetByReference(reference string) (*Resource, string, bo// resolve here so history reads by canonical ID include them. PVE disk alert// rows are owned row by row instead (proxmoxDiskAlertOwner).func (a *MonitorAdapter) RecordChange(change ResourceChange) error {+ _, _, err := a.recordChange(change)+ return err+}++// RecordLiveChange records a change that is happening now, as RecordChange+// does, then tells the change observer. Callers replaying or importing history+// use RecordChange so old events are not announced as new.+func (a *MonitorAdapter) RecordLiveChange(change ResourceChange) error {+ recorded, ok, err := a.recordChange(change)+ if err != nil || !ok {+ return err+ }+ a.notifyChanges([]ResourceChange{recorded})+ return nil+}++// recordChange persists the change and returns it as stored, with the+// resource ID it resolved to. ok is false when no durable store is attached.+func (a *MonitorAdapter) recordChange(change ResourceChange) (ResourceChange, bool, error) {registry := a.currentRegistry()if registry == nil || registry.store == nil {- return nil+ return change, false, nil}sourceRef := change.ResourceIDif ref, ok := proxmoxDiskAlertRowReference(change); ok {@@ -258,7 +314,8 @@ func (a *MonitorAdapter) RecordChange(change ResourceChange) error {change.Metadata[MetadataAlertResourceID] = refchange.ResourceID = owner}- return registry.store.RecordChange(change)+ err := registry.store.RecordChange(change)+ return change, err == nil, err}history, hasHistory := registry.store.(resourceHistoryIdentityWriter)unbound := false@@ -275,7 +332,7 @@ func (a *MonitorAdapter) RecordChange(change ResourceChange) error {if hasHistory {id, found, err := history.ResolveHistorySourceIdentity(sourceRef)if err != nil {- return err+ return change, false, err}if found {change.ResourceID = id@@ -291,7 +348,7 @@ func (a *MonitorAdapter) RecordChange(change ResourceChange) error {// No resource answers to it: keep the binding an earlier event recorded.id, found, err := history.ResolveHistorySourceIdentity(CanonicalResourceID(sourceRef))if err != nil {- return err+ return change, false, err}if found {change.ResourceID = id@@ -300,16 +357,17 @@ func (a *MonitorAdapter) RecordChange(change ResourceChange) error {}}if hasHistory && CanonicalResourceID(sourceRef) != CanonicalResourceID(change.ResourceID) {- return history.RecordChangeWithSourceIdentity(change, sourceRef)+ err := history.RecordChangeWithSourceIdentity(change, sourceRef)+ return change, err == nil, err}if err := registry.store.RecordChange(change); err != nil {- return err+ return change, false, err}// Inventory may name the resource in a later generation; bind the row then.if _, ok := registry.store.(legacyHistoryBinder); unbound && ok {a.legacyHistory.add(CanonicalResourceID(sourceRef), time.Now())}- return nil+ return change, true, nil}// GetRecentChanges forwards canonical resource-history reads into the@@ -545,7 +603,7 @@ func (a *MonitorAdapter) replaceRegistryLocked(snapshot models.StateSnapshot, reocc := snapshot.LastUpdateoccurredAt = &occ}- recordRegistryChangesBetweenGenerations(registry, rebuilt, rebuiltAt, occurredAt, SourcePulseDiff, "")+ recorded := recordRegistryChangesBetweenGenerations(registry, rebuilt, rebuiltAt, occurredAt, SourcePulseDiff, "")rebuilt.PersistIdentityPins()a.mu.Lock()@@ -555,6 +613,9 @@ func (a *MonitorAdapter) replaceRegistryLocked(snapshot models.StateSnapshot, rea.lastRebuiltAt = rebuiltAta.mu.Unlock()a.bindLegacyHistory(rebuilt, time.Now())+ // After the swap, so an observer reading the adapter sees the generation+ // that produced the change.+ a.notifyChanges(recorded)}// ShouldSkipAPIPolling returns true when agent coverage indicates API@@ -761,11 +822,12 @@ func (a *MonitorAdapter) PopulateSupplementalRecords(source DataSource, recordsbefore := registry.List()registry.IngestRecordsWithStaleThresholds(source, records, thresholds)rebuiltAt := time.Now().UTC()- recordRegistryChanges(registry.store, before, registry.List(), rebuiltAt, nil, SourcePlatformEvent, changeSourceAdapterForDataSource(source))+ recorded := recordRegistryChanges(registry.store, before, registry.List(), rebuiltAt, nil, SourcePlatformEvent, changeSourceAdapterForDataSource(source))registry.PersistIdentityPins()a.mu.Lock()a.lastRebuiltAt = rebuiltAta.mu.Unlock()+ a.notifyChanges(recorded)}// MetricsTargetForResource resolves the history/metrics target for a canonical