Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Push alert and status changes to the agent event stream

rcourtman182067027f26
diff --git a/cmd/pulse-mcp/README.md b/cmd/pulse-mcp/README.md
index d9c6f36064..6bd8353cda 100644
--- a/cmd/pulse-mcp/README.md
+++ b/cmd/pulse-mcp/README.md
@@ -218,6 +218,9 @@ notification on stdout for every non-transport event:
{ "jsonrpc": "2.0", "method": "notifications/finding.created", "params": { ... } }
{ "jsonrpc": "2.0", "method": "notifications/approval.pending", "params": { ... } }
{ "jsonrpc": "2.0", "method": "notifications/action.completed", "params": { ... } }
+{ "jsonrpc": "2.0", "method": "notifications/alert.fired", "params": { ... } }
+{ "jsonrpc": "2.0", "method": "notifications/alert.resolved", "params": { ... } }
+{ "jsonrpc": "2.0", "method": "notifications/resource.status_changed", "params": { ... } }
```
The `params` object is the SSE event's `data` payload verbatim, so an
@@ -225,6 +228,17 @@ agent that already knows the substrate's wire shape sees identical
content to what an HTTP SSE consumer would. Transport plumbing
(`stream.connected`, `heartbeat`) is filtered out.
+`alert.fired`, `alert.resolved` and `resource.status_changed` carry the
+canonical `resourceId`, resource name and type, and the `changeId` and `kind`
+that `list_recent_changes` returns for the same change, so an agent can read
+the surrounding history. They carry the alert identifier, type and level, or
+the status `from` and `to`, never alert message or command text. A resource's
+status changes within 30 seconds of the last announced one fold into a single
+event with `coalescedTransitions`; a flap that ends where it started is not
+announced. Pulse announces changes as it records them; it does not replay old
+alert history on restart, and an event dropped for a slow client is still
+readable through `list_recent_changes`.
+
`action.completed` also covers failed or refused terminal actions; its name
alone is not a successful dispatch or recovery receipt. Inspect the result and
verification evidence. Missing checks or `ran: false` do not establish the
diff --git a/docs/AGENT_SUBSTRATE.md b/docs/AGENT_SUBSTRATE.md
index 391d242da2..47afb3cec5 100644
--- a/docs/AGENT_SUBSTRATE.md
+++ b/docs/AGENT_SUBSTRATE.md
@@ -110,7 +110,13 @@ actor. Validation failures emit the `operator_state_invalid` and
the specific conflict instead of retrying blindly.
**Push.** `/api/agent/events` is an SSE stream that fires `finding.created`,
-`approval.pending`, and `action.completed` as state changes. `action.completed`
+`approval.pending`, `action.completed`, `alert.fired`, `alert.resolved` and
+`resource.status_changed` as state changes. The alert and status events are
+doorbells for changes Pulse just recorded: each carries the canonical
+`resourceId` and the `changeId` that `list_recent_changes` returns, so read the
+history there rather than treating the event as the whole record. Status flaps
+on one resource are coalesced into at most one event per 30 seconds.
+`action.completed`
means a terminal action record, **including failed or refused actions**, not
necessarily a successful dispatch. Inspect its result and verification evidence;
the event name alone does not confirm the intended outcome. An idle stream is
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 99244550fd..44f7ec9c1f 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -2749,6 +2749,14 @@ agents may read what changed across the fleet, but `command_executed` and
`commandRedacted:true`, the command text, output, and caller details dropped,
and a command's reason cut to its status. A change record is history, never command
disclosure, install-token scope, or execution authority.
+
+The `alert.fired`, `alert.resolved` and `resource.status_changed` events on
+`/api/agent/events` are the push form of the same records and the same
+boundary. Their payload carries only identifiers, the alert type and level, and
+the status `from`/`to`; it never carries alert messages, change reasons, command
+metadata or anything a monitoring-read token could not already read. An event is
+a doorbell for a recorded change, never an instruction to act, install-token
+scope, or execution authority.
Agent-facing provisioning capabilities declared by
`/api/agent/capabilities` are also API-owned projections over the canonical
node lifecycle and discovery APIs. Lifecycle and infrastructure onboarding
diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.md
index 0deb2bad8a..589756684b 100644
--- a/docs/release-control/v6/internal/subsystems/ai-runtime.md
+++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md
@@ -1984,8 +1984,12 @@ The Pulse Intelligence event vocabulary is part of that same shared
transport-event filtering, MCP notification method projection, shared SSE
record parsing, shared SSE-to-MCP notification bridging, and reference probes
must consume the shared event kind constants/helpers instead of maintaining local copies of
-`finding.created`, `approval.pending`, `action.completed`, `stream.connected`,
-or `heartbeat`.
+`finding.created`, `approval.pending`, `action.completed`, `alert.fired`,
+`alert.resolved`, `resource.status_changed`, `stream.connected`, or
+`heartbeat`. `AgentActionableEventKinds` lists the three change kinds, so MCP
+`--emit-notifications` advertises and forwards them without a local list, and
+`SubscribeEventsDescription` derives their wording, including the
+`ResourceStatusEventCoalesceWindow` duration, from the same constants.
MCP `tools/list` must use the shared
`agentcapabilities.ProjectManifestSurfaceTools` projection for the Pulse MCP
surface. The manifest-owned `surfaceToolContracts` field is the allowlist for
diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md
index a0c8b68ab1..13bd1ff6f2 100644
--- a/docs/release-control/v6/internal/subsystems/api-contracts.md
+++ b/docs/release-control/v6/internal/subsystems/api-contracts.md
@@ -6262,6 +6262,46 @@ by `internal/agentcapabilities/events.go`; API constants may expose typed
aliases, but API handlers, manifest descriptions, MCP adapters, and probes must
not re-declare the vocabulary independently.
+The stream also carries `alert.fired`, `alert.resolved` and
+`resource.status_changed`, the push half of `list_recent_changes`. They are
+published from `internal/api/agent_change_events.go` at the moment a monitor's
+`unifiedresources.MonitorAdapter` records the matching canonical resource
+change: the adapter's `SetChangeObserver` hears registry-diff changes after the
+new generation is swapped in, and alert lifecycle changes recorded through
+`RecordLiveChange`, with the resource ID already resolved to the canonical
+identity the row was stored under. Nothing is read back from the change store
+and nothing new is persisted, so the events add no disk writes. Only the live
+alert lifecycle callback records through `RecordLiveChange`; lifecycle replay,
+active-alert reconciliation and imported history use `RecordChange`, so a
+restart never re-announces old alerts. The payload
+(`AgentEventResourceChangePayload`) mirrors the change record the timeline read
+returns: `changeId` is its `id`, `kind` its `kind`, plus `resourceId`, current
+`resourceName` and `resourceType`, `from`/`to` for status changes, and
+`alertIdentifier`, `alertType`, `alertLevel` and `alertResolution` for alerts.
+It never carries the alert message, change reason or command metadata. A
+`state_transition` counts as `resource.status_changed` only when its
+`changedFields` include `status` and the status differs; discovery, removal,
+Docker command and Proxmox lifecycle transitions stay timeline-only. Status
+events are coalesced per organization and resource over
+`agentcapabilities.ResourceStatusEventCoalesceWindow` (30s): the first change
+in a quiet period publishes at once, later changes fold into one event at the
+window's end carrying `coalescedTransitions` (from the last announced status to
+the latest), and a flap that returns to the announced status publishes nothing.
+Volume is otherwise bounded by the broadcaster's existing per-subscriber drop.
+These events are tenant-scoped: `AgentEvent` carries an unexported
+organization tag set from the recording monitor, and `HandleAgentEvents`
+delivers a tagged event only to subscribers whose request organization matches.
+Untagged kinds keep reaching every subscriber. Proofs:
+`TestAgentChangeEvents*` in `internal/api/agent_change_events_test.go`,
+`TestContract_AgentChangeEventsStayInsideTheirOrganization` in
+`internal/api/contract_test.go`,
+`TestAgentSubstrate_LiveChangeEventsReachSubscribeEventsStream` in
+`internal/api/agent_substrate_e2e_test.go`,
+`TestMonitorAdapterChangeObserverHearsOnlyRecordedLiveChanges` in
+`internal/unifiedresources/history_identity_test.go`, and
+`TestAlertTimelineAnnouncesOnlyLiveLifecycleChanges` in
+`internal/monitoring/monitor_alert_handling_test.go`.
+
The approval store exposes `SetOnApprovalCreated(cb)` so the API
layer can install a fire-and-forget callback that runs after every
successful `CreateApproval`. The callback fires on its own
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 00d2282c9f..9140144a51 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -8765,3 +8765,17 @@ recovery and confirmed scoped absence. Their opt-in producer feeds the existing
production Backups consumer for browser proof. Retained evidence is not a fresh
read, available archive, installed recovery or a diagnosis of #3248; no guest or
backup operation, thaw or filesystem-liveness assertion follows.
+
+### Only live alert lifecycle changes are announced to agents (29 October 2026)
+
+The alert manager's live lifecycle callback reaches
+`Monitor.handleAlertLifecycleEvent`, which projects the event with
+`live=true`; lifecycle replay and active-alert reconciliation call
+`projectAlertLifecycleEvent(event, false)`, and imported history always
+projects as not live. A live projection records its canonical resource change
+through the resource store's `RecordLiveChange` when the store offers it, so
+the agent event stream hears `alert.fired` and `alert.resolved` once, when the
+alert engine emits them; every other projection uses `RecordChange` and stays
+silent. Both paths write the same deterministic change row. Proof:
+`TestAlertTimelineAnnouncesOnlyLiveLifecycleChanges` in
+`internal/monitoring/monitor_alert_handling_test.go`.
diff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.md
index e217673167..c96cb0c2f2 100644
--- a/docs/release-control/v6/internal/subsystems/security-privacy.md
+++ b/docs/release-control/v6/internal/subsystems/security-privacy.md
@@ -3757,3 +3757,16 @@ the session router's default branch, which still refuses tokens without
`ai:chat` before answering `404`
(`TestRelayMobileAccessScopeDeniesAdjacentAIRoutes`). The undo, redo, and steer
notes above are history.
+
+### Agent change events stay inside their organization (29 October 2026)
+
+`alert.fired`, `alert.resolved` and `resource.status_changed` events carry an
+unexported organization tag from the monitor that recorded the change, and
+`HandleAgentEvents` delivers a tagged event only to subscribers whose request
+organization matches (the legacy empty organization is `default`). A tenant
+therefore never hears another tenant's alerts or outages on
+`/api/agent/events`. The payload excludes alert messages, change reasons and
+command metadata, so a monitoring-read token learns nothing from an event that
+it could not already read through `list_recent_changes`. Proof:
+`TestContract_AgentChangeEventsStayInsideTheirOrganization` in
+`internal/api/contract_test.go`.
diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md
index f25f6714e2..1b67c5e3f2 100644
--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md
+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md
@@ -4712,6 +4712,16 @@ readable without a name, so the timeline's recovery posture is exactly the
`resource_changes` table's: whatever survives a restart or restore is what
agents read, with no timeline-specific rehydration or recovery artifact.
+The live `alert.fired`, `alert.resolved` and `resource.status_changed` agent
+events add no persistence. They are published in memory after the monitor
+adapter has recorded the matching `resource_changes` row, and per-resource
+status coalescing lives only in process memory. An event dropped for a slow
+subscriber, or lost to a restart, is not replayed; the durable row it pointed
+at stays readable through `list_recent_changes`. Alert lifecycle replay,
+active-alert reconciliation and imported history after a restore record their
+rows through the non-announcing `RecordChange` path, so recovery never
+re-announces old alerts.
+
The Patrol-control status endpoint at
`/api/agent/patrol-control/status` introduces no new persistence. The legacy
`/api/agent/operations-loop/status` URL remains a compatibility alias. It reads the
diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.md
index 9ed78f5406..89cba4c4e0 100644
--- a/docs/release-control/v6/internal/subsystems/unified-resources.md
+++ b/docs/release-control/v6/internal/subsystems/unified-resources.md
@@ -9042,3 +9042,18 @@ protect grouped metadata, responsive columns and bounded rendering. Browser
proof must exercise the consuming list at desktop/narrow widths, inspect actual
pixels, identity, state/age fit, touch/keyboard disclosure, filters and live
updates. This is safer evidence inspection, not a restore or installed recovery.
+
+### Monitor adapter announces the changes it recorded (29 October 2026)
+
+`MonitorAdapter.SetChangeObserver` installs one non-blocking observer that
+hears each change the adapter just persisted, with its resource ID resolved to
+the canonical identity the row was stored under. Registry diffs from
+`replaceRegistry` and `PopulateSupplementalRecords` are announced after the new
+generation is swapped in, so an observer reading the adapter sees the
+generation that produced the change; `recordRegistryChanges` and
+`recordRegistryChangesBetweenGenerations` return only the changes the store
+accepted. Alert lifecycle rows are announced only through `RecordLiveChange`;
+`RecordChange` stays silent so replayed and imported history is not presented
+as new. The API agent event stream is the only observer today. Proof:
+`TestMonitorAdapterChangeObserverHearsOnlyRecordedLiveChanges` in
+`internal/unifiedresources/history_identity_test.go`.
diff --git a/frontend-modern/public/docs/AGENT_SUBSTRATE.md b/frontend-modern/public/docs/AGENT_SUBSTRATE.md
index 391d242da2..47afb3cec5 100644
--- a/frontend-modern/public/docs/AGENT_SUBSTRATE.md
+++ b/frontend-modern/public/docs/AGENT_SUBSTRATE.md
@@ -110,7 +110,13 @@ actor. Validation failures emit the `operator_state_invalid` and
the specific conflict instead of retrying blindly.
**Push.** `/api/agent/events` is an SSE stream that fires `finding.created`,
-`approval.pending`, and `action.completed` as state changes. `action.completed`
+`approval.pending`, `action.completed`, `alert.fired`, `alert.resolved` and
+`resource.status_changed` as state changes. The alert and status events are
+doorbells for changes Pulse just recorded: each carries the canonical
+`resourceId` and the `changeId` that `list_recent_changes` returns, so read the
+history there rather than treating the event as the whole record. Status flaps
+on one resource are coalesced into at most one event per 30 seconds.
+`action.completed`
means a terminal action record, **including failed or refused actions**, not
necessarily a successful dispatch. Inspect its result and verification evidence;
the event name alone does not confirm the intended outcome. An idle stream is
diff --git a/internal/agentcapabilities/events.go b/internal/agentcapabilities/events.go
index 4787089604..f66e604842 100644
--- a/internal/agentcapabilities/events.go
+++ b/internal/agentcapabilities/events.go
@@ -1,5 +1,7 @@
package agentcapabilities
+import "time"
+
// EventKind is the stable Pulse Intelligence event identifier used by API SSE
// streams and external-agent adapters.
type EventKind string
@@ -16,6 +18,18 @@ const (
// state.
EventKindActionCompleted EventKind = "action.completed"
+ // EventKindAlertFired fires when Pulse records an alert firing (or
+ // re-firing) on a resource.
+ EventKindAlertFired EventKind = "alert.fired"
+
+ // EventKindAlertResolved fires when Pulse records an alert clearing.
+ EventKindAlertResolved EventKind = "alert.resolved"
+
+ // EventKindResourceStatusChanged fires when Pulse records a resource's
+ // status changing, coalesced per resource so flaps cannot flood the
+ // stream.
+ EventKindResourceStatusChanged EventKind = "resource.status_changed"
+
// EventKindStreamConnected is the one-shot SSE synchronization marker for a
// newly opened stream.
EventKindStreamConnected EventKind = "stream.connected"
@@ -28,8 +42,16 @@ var agentActionableEventKinds = []string{
string(EventKindFindingCreated),
string(EventKindApprovalPending),
string(EventKindActionCompleted),
+ string(EventKindAlertFired),
+ string(EventKindAlertResolved),
+ string(EventKindResourceStatusChanged),
}
+// ResourceStatusEventCoalesceWindow bounds resource.status_changed volume: a
+// resource's first status change is announced at once, and further changes
+// inside the window fold into at most one event at the window's end.
+const ResourceStatusEventCoalesceWindow = 30 * time.Second
+
// AgentActionableEventKinds returns the non-transport event kinds external
// agents should expect to handle as product notifications.
func AgentActionableEventKinds() []string {
@@ -54,6 +76,8 @@ func SubscribeEventsDescription() string {
return "Subscribe to the SSE event stream for real-time notifications: " +
string(EventKindFindingCreated) + " when a new finding is raised, " +
string(EventKindApprovalPending) + " when a remediation request enters StatusPending and waits on operator decision, " +
- string(EventKindActionCompleted) + " when an action audit reaches a terminal state (Completed or Failed, including refused-before-dispatch failures with stable error-token prefixes; carries a verification block with the read-after-write probe outcome so agents close the certainty loop without polling /api/actions/{id}), stream-local " +
+ string(EventKindActionCompleted) + " when an action audit reaches a terminal state (Completed or Failed, including refused-before-dispatch failures with stable error-token prefixes; carries a verification block with the read-after-write probe outcome so agents close the certainty loop without polling /api/actions/{id}), " +
+ string(EventKindAlertFired) + " and " + string(EventKindAlertResolved) + " when an alert fires or clears on a resource (alert identifier, type and level), " +
+ string(EventKindResourceStatusChanged) + " when a resource's status changes (from/to; further changes within " + ResourceStatusEventCoalesceWindow.String() + " fold into one event carrying coalescedTransitions). Change events carry the canonical resourceId, resource name and type, and the changeId and kind that " + ListRecentChangesCapabilityName + " returns, so agents can read the surrounding history; replayed or imported history is not announced. Stream-local " +
string(EventKindHeartbeat) + " every 15s. Command fields are redacted for monitoring-read API tokens unless the token also has ai:execute. Long-lived connection; agents listen instead of polling."
}
diff --git a/internal/agentcapabilities/events_test.go b/internal/agentcapabilities/events_test.go
index b5155f4e89..95e2a34d94 100644
--- a/internal/agentcapabilities/events_test.go
+++ b/internal/agentcapabilities/events_test.go
@@ -12,6 +12,9 @@ func TestAgentActionableEventKindsReturnsDetachedKnownKinds(t *testing.T) {
string(EventKindFindingCreated),
string(EventKindApprovalPending),
string(EventKindActionCompleted),
+ string(EventKindAlertFired),
+ string(EventKindAlertResolved),
+ string(EventKindResourceStatusChanged),
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("AgentActionableEventKinds() = %v, want %v", got, want)
@@ -44,4 +47,8 @@ func TestSubscribeEventsDescriptionUsesSharedEventKinds(t *testing.T) {
t.Fatalf("subscribe_events description missing %q: %s", kind, description)
}
}
+ // Change events point agents at the timeline read for surrounding history.
+ if !strings.Contains(description, ListRecentChangesCapabilityName) {
+ t.Fatalf("subscribe_events description must name %s: %s", ListRecentChangesCapabilityName, description)
+ }
}
diff --git a/internal/api/agent_change_events.go b/internal/api/agent_change_events.go
new file mode 100644
index 0000000000..69b74c7a97
--- /dev/null
+++ b/internal/api/agent_change_events.go
@@ -0,0 +1,238 @@
+package api
+
+import (
+ "strings"
+ "sync"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/agentcapabilities"
+ unified "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+)
+
+// AgentEventResourceChangePayload is the payload shape for alert.fired,
+// alert.resolved and resource.status_changed events. It mirrors the change
+// record list_recent_changes returns (changeId is that record's id, kind its
+// kind) so an agent can read the surrounding history, and carries no alert
+// message, reason text or command. ResourceName and ResourceType are the
+// resource's current values and are empty when Pulse no longer knows it.
+//
+// CoalescedTransitions is set only on a resource.status_changed event that
+// was held back by the per-resource coalescing window. It counts the status
+// changes Pulse recorded in that window; From is the status last announced
+// and To the status at the window's end. Every individual transition stays
+// readable through list_recent_changes.
+type AgentEventResourceChangePayload struct {
+ ChangeID string `json:"changeId"`
+ ResourceID string `json:"resourceId"`
+ ResourceName string `json:"resourceName,omitempty"`
+ ResourceType string `json:"resourceType,omitempty"`
+ Kind string `json:"kind"`
+ From string `json:"from,omitempty"`
+ To string `json:"to,omitempty"`
+ AlertIdentifier string `json:"alertIdentifier,omitempty"`
+ AlertType string `json:"alertType,omitempty"`
+ AlertLevel string `json:"alertLevel,omitempty"`
+ AlertResolution string `json:"alertResolution,omitempty"`
+ ObservedAt time.Time `json:"observedAt"`
+ OccurredAt *time.Time `json:"occurredAt,omitempty"`
+ CoalescedTransitions int `json:"coalescedTransitions,omitempty"`
+}
+
+// agentResourceLookup resolves a resource reference to the current resource;
+// MonitorAdapter.GetByReference satisfies it.
+type agentResourceLookup func(reference string) (*unified.Resource, string, bool)
+
+type agentStatusKey struct {
+ orgID string
+ resourceID string
+}
+
+// agentStatusWindow is one resource's open coalescing window: the status last
+// announced and the latest status change folded since then.
+type agentStatusWindow struct {
+ announced string
+ pending *AgentEventResourceChangePayload
+ folded int
+}
+
+// agentChangeEventProducer turns resource changes the monitor adapters just
+// recorded into agent events. It reads nothing back from the change store and
+// writes nothing, so it adds no disk load. Alert events publish as recorded;
+// the alert engine's own delays and hysteresis already bound them. Status
+// changes are coalesced per resource: the first change in a quiet period is
+// announced at once, later changes inside the window fold into at most one
+// event when it closes, and a flap that ends on the status last announced
+// publishes nothing.
+type agentChangeEventProducer struct {
+ broadcaster *AgentEventBroadcaster
+ window time.Duration
+ afterFunc func(time.Duration, func())
+
+ mu sync.Mutex
+ windows map[agentStatusKey]*agentStatusWindow
+}
+
+func newAgentChangeEventProducer(broadcaster *AgentEventBroadcaster, window time.Duration) *agentChangeEventProducer {
+ return &agentChangeEventProducer{
+ broadcaster: broadcaster,
+ window: window,
+ afterFunc: func(d time.Duration, f func()) {
+ time.AfterFunc(d, f)
+ },
+ windows: make(map[agentStatusKey]*agentStatusWindow),
+ }
+}
+
+// observer returns the change observer for one organization's monitor
+// adapter. Events it publishes reach only that organization's subscribers.
+func (p *agentChangeEventProducer) observer(orgID string, lookup agentResourceLookup) unified.ResourceChangeObserver {
+ orgID = normalizeAgentEventOrgID(orgID)
+ return func(change unified.ResourceChange) {
+ p.observe(orgID, change, lookup)
+ }
+}
+
+func (p *agentChangeEventProducer) observe(orgID string, change unified.ResourceChange, lookup agentResourceLookup) {
+ if p == nil || p.broadcaster == nil {
+ return
+ }
+ kind, ok := agentEventKindForChange(change)
+ if !ok {
+ return
+ }
+ payload := agentResourceChangePayload(change)
+ if kind != AgentEventResourceStatusChanged {
+ p.publish(orgID, kind, payload, lookup)
+ return
+ }
+
+ key := agentStatusKey{orgID: orgID, resourceID: payload.ResourceID}
+ p.mu.Lock()
+ if window, open := p.windows[key]; open {
+ window.folded++
+ payload.From = window.announced
+ window.pending = &payload
+ p.mu.Unlock()
+ return
+ }
+ p.openStatusWindowLocked(key, payload.To, lookup)
+ p.mu.Unlock()
+ p.publish(orgID, kind, payload, lookup)
+}
+
+func (p *agentChangeEventProducer) openStatusWindowLocked(key agentStatusKey, announced string, lookup agentResourceLookup) {
+ p.windows[key] = &agentStatusWindow{announced: announced}
+ p.afterFunc(p.window, func() {
+ p.closeStatusWindow(key, lookup)
+ })
+}
+
+// closeStatusWindow publishes the folded change, if it left the resource in a
+// different status from the one last announced, and opens a new window for
+// it so continued flapping stays at one event per window.
+func (p *agentChangeEventProducer) closeStatusWindow(key agentStatusKey, lookup agentResourceLookup) {
+ p.mu.Lock()
+ window := p.windows[key]
+ delete(p.windows, key)
+ if window == nil || window.pending == nil || window.pending.To == window.announced {
+ p.mu.Unlock()
+ return
+ }
+ payload := *window.pending
+ payload.CoalescedTransitions = window.folded
+ p.openStatusWindowLocked(key, payload.To, lookup)
+ p.mu.Unlock()
+ p.publish(key.orgID, AgentEventResourceStatusChanged, payload, lookup)
+}
+
+func (p *agentChangeEventProducer) publish(orgID string, kind AgentEventKind, payload AgentEventResourceChangePayload, lookup agentResourceLookup) {
+ if lookup != nil {
+ if resource, _, ok := lookup(payload.ResourceID); ok && resource != nil {
+ payload.ResourceName = resource.Name
+ payload.ResourceType = string(resource.Type)
+ }
+ }
+ p.broadcaster.Publish(AgentEvent{
+ Kind: kind,
+ Payload: payload,
+ orgID: orgID,
+ })
+}
+
+// agentEventKindForChange maps a recorded change to the event agents hear.
+// A state_transition counts as a status change only when the status field
+// changed: discovery, removal, Docker command and Proxmox lifecycle
+// transitions stay in list_recent_changes.
+func agentEventKindForChange(change unified.ResourceChange) (AgentEventKind, bool) {
+ switch change.Kind {
+ case unified.ChangeAlertFired:
+ return AgentEventAlertFired, true
+ case unified.ChangeAlertResolved:
+ return AgentEventAlertResolved, true
+ case unified.ChangeStateTransition:
+ if changeTouchesField(change, "status") && change.From != change.To {
+ return AgentEventResourceStatusChanged, true
+ }
+ }
+ return "", false
+}
+
+func changeTouchesField(change unified.ResourceChange, field string) bool {
+ switch fields := change.Metadata["changedFields"].(type) {
+ case []string:
+ for _, candidate := range fields {
+ if candidate == field {
+ return true
+ }
+ }
+ case []any:
+ for _, candidate := range fields {
+ if candidate == field {
+ return true
+ }
+ }
+ }
+ return false
+}
+
+func agentResourceChangePayload(change unified.ResourceChange) AgentEventResourceChangePayload {
+ payload := AgentEventResourceChangePayload{
+ ChangeID: change.ID,
+ ResourceID: unified.CanonicalResourceID(change.ResourceID),
+ Kind: string(change.Kind),
+ ObservedAt: change.ObservedAt,
+ }
+ if change.OccurredAt != nil {
+ occurredAt := *change.OccurredAt
+ payload.OccurredAt = &occurredAt
+ }
+ switch change.Kind {
+ case unified.ChangeAlertFired, unified.ChangeAlertResolved:
+ payload.AlertIdentifier = changeMetadataString(change, unified.MetadataAlertIdentifier)
+ payload.AlertType = changeMetadataString(change, unified.MetadataAlertType)
+ payload.AlertLevel = changeMetadataString(change, unified.MetadataAlertLevel)
+ payload.AlertResolution = changeMetadataString(change, unified.MetadataAlertResolution)
+ default:
+ payload.From = change.From
+ payload.To = change.To
+ }
+ return payload
+}
+
+func changeMetadataString(change unified.ResourceChange, key string) string {
+ value, _ := change.Metadata[key].(string)
+ return strings.TrimSpace(value)
+}
+
+// normalizeAgentEventOrgID maps the legacy empty organization to the default
+// one, matching how request contexts name it.
+func normalizeAgentEventOrgID(orgID string) string {
+ orgID = strings.TrimSpace(orgID)
+ if orgID == "" {
+ return "default"
+ }
+ return orgID
+}
+
+// defaultResourceStatusEventWindow is the manifest-declared coalescing window.
+const defaultResourceStatusEventWindow = agentcapabilities.ResourceStatusEventCoalesceWindow
diff --git a/internal/api/agent_change_events_test.go b/internal/api/agent_change_events_test.go
new file mode 100644
index 0000000000..a8d22f4844
--- /dev/null
+++ b/internal/api/agent_change_events_test.go
@@ -0,0 +1,224 @@
+package api
+
+import (
+ "encoding/json"
+ "strings"
+ "testing"
+ "time"
+
+ unified "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+)
+
+// manualStatusWindows replaces the producer's timers so tests close windows
+// deterministically.
+type manualStatusWindows struct {
+ pending []func()
+}
+
+func (m *manualStatusWindows) after(_ time.Duration, f func()) {
+ m.pending = append(m.pending, f)
+}
+
+func (m *manualStatusWindows) closeAll() {
+ pending := m.pending
+ m.pending = nil
+ for _, f := range pending {
+ f()
+ }
+}
+
+func newTestAgentChangeEventProducer(b *AgentEventBroadcaster) (*agentChangeEventProducer, *manualStatusWindows) {
+ producer := newAgentChangeEventProducer(b, time.Minute)
+ windows := &manualStatusWindows{}
+ producer.afterFunc = windows.after
+ return producer, windows
+}
+
+func testStatusChange(id, resourceID, from, to string) unified.ResourceChange {
+ return unified.ResourceChange{
+ ID: id,
+ ResourceID: resourceID,
+ Kind: unified.ChangeStateTransition,
+ From: from,
+ To: to,
+ ObservedAt: time.Date(2026, 10, 29, 6, 0, 0, 0, time.UTC),
+ Metadata: map[string]any{"changedFields": []string{"status"}},
+ }
+}
+
+func drainAgentEvents(events <-chan AgentEvent) []AgentEvent {
+ var out []AgentEvent
+ for {
+ select {
+ case event := <-events:
+ out = append(out, event)
+ default:
+ return out
+ }
+ }
+}
+
+func TestAgentChangeEventsPublishAlertLifecycleWithoutContent(t *testing.T) {
+ b := NewAgentEventBroadcaster()
+ events, unsubscribe := b.Subscribe()
+ defer unsubscribe()
+ producer, _ := newTestAgentChangeEventProducer(b)
+ lookup := func(ref string) (*unified.Resource, string, bool) {
+ if ref != "vm-web" {
+ return nil, "", false
+ }
+ return &unified.Resource{ID: "vm-web", Name: "web", Type: unified.ResourceTypeVM}, "vm-web", true
+ }
+ observe := producer.observer("", lookup)
+ occurredAt := time.Date(2026, 10, 29, 5, 59, 0, 0, time.UTC)
+ observe(unified.ResourceChange{
+ ID: "change-fired",
+ ResourceID: "vm-web",
+ Kind: unified.ChangeAlertFired,
+ ObservedAt: occurredAt,
+ OccurredAt: &occurredAt,
+ Reason: "alert fired: CPU at 97% on web",
+ Metadata: map[string]any{
+ unified.MetadataAlertIdentifier: "vm-web-cpu",
+ unified.MetadataAlertType: "cpu",
+ unified.MetadataAlertLevel: "critical",
+ unified.MetadataAlertMessage: "CPU at 97% on web",
+ unified.MetadataCommand: "systemctl restart web",
+ },
+ })
+ observe(unified.ResourceChange{
+ ID: "change-resolved",
+ ResourceID: "vm-web",
+ Kind: unified.ChangeAlertResolved,
+ ObservedAt: occurredAt.Add(time.Minute),
+ Metadata: map[string]any{
+ unified.MetadataAlertIdentifier: "vm-web-cpu",
+ unified.MetadataAlertType: "cpu",
+ unified.MetadataAlertLevel: "critical",
+ unified.MetadataAlertResolution: "moved_to_agent",
+ },
+ })
+
+ got := drainAgentEvents(events)
+ if len(got) != 2 {
+ t.Fatalf("events = %+v, want fired and resolved", got)
+ }
+ if got[0].Kind != AgentEventAlertFired || got[1].Kind != AgentEventAlertResolved {
+ t.Fatalf("kinds = %s, %s", got[0].Kind, got[1].Kind)
+ }
+ fired := got[0].Payload.(AgentEventResourceChangePayload)
+ want := AgentEventResourceChangePayload{
+ ChangeID: "change-fired",
+ ResourceID: "vm-web",
+ ResourceName: "web",
+ ResourceType: string(unified.ResourceTypeVM),
+ Kind: string(unified.ChangeAlertFired),
+ AlertIdentifier: "vm-web-cpu",
+ AlertType: "cpu",
+ AlertLevel: "critical",
+ ObservedAt: occurredAt,
+ OccurredAt: &occurredAt,
+ }
+ if fired.OccurredAt == nil || !fired.OccurredAt.Equal(occurredAt) {
+ t.Fatalf("occurredAt = %v", fired.OccurredAt)
+ }
+ fired.OccurredAt = want.OccurredAt
+ if fired != want {
+ t.Fatalf("fired payload = %+v\nwant %+v", fired, want)
+ }
+ encoded, err := json.Marshal(got[0])
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, forbidden := range []string{"97%", "systemctl", "orgID", "default"} {
+ if strings.Contains(string(encoded), forbidden) {
+ t.Fatalf("alert event must not carry %q: %s", forbidden, encoded)
+ }
+ }
+ if resolved := got[1].Payload.(AgentEventResourceChangePayload); resolved.AlertResolution != "moved_to_agent" {
+ t.Fatalf("resolved payload = %+v", resolved)
+ }
+}
+
+func TestAgentChangeEventsIgnoreChangesThatAreNotStatusOrAlerts(t *testing.T) {
+ b := NewAgentEventBroadcaster()
+ events, unsubscribe := b.Subscribe()
+ defer unsubscribe()
+ producer, _ := newTestAgentChangeEventProducer(b)
+ observe := producer.observer("default", nil)
+
+ discovered := testStatusChange("discovered", "vm-new", "absent", "online")
+ discovered.Metadata = map[string]any{"changeType": "resource_created"}
+ lifecycle := testStatusChange("lock", "vm-web", "online", "online")
+ lifecycle.Metadata = map[string]any{"changedFields": []string{"proxmox.lifecycle"}}
+ for _, change := range []unified.ResourceChange{
+ discovered,
+ lifecycle,
+ {ID: "ack", ResourceID: "vm-web", Kind: unified.ChangeAlertAcknowledged},
+ {ID: "restart", ResourceID: "vm-web", Kind: unified.ChangeRestart},
+ {ID: "cmd", ResourceID: "vm-web", Kind: unified.ChangeCommandExecuted, Metadata: map[string]any{unified.MetadataCommand: "reboot"}},
+ } {
+ observe(change)
+ }
+ if got := drainAgentEvents(events); len(got) != 0 {
+ t.Fatalf("events = %+v, want none", got)
+ }
+}
+
+// A flapping resource yields its first change at once and at most one more
+// event per window; a flap that ends where it started publishes nothing.
+func TestAgentChangeEventsCoalesceStatusFlapsPerResource(t *testing.T) {
+ b := NewAgentEventBroadcaster()
+ events, unsubscribe := b.Subscribe()
+ defer unsubscribe()
+ producer, windows := newTestAgentChangeEventProducer(b)
+ observe := producer.observer("default", nil)
+
+ observe(testStatusChange("c1", "vm-web", "online", "offline"))
+ observe(testStatusChange("other", "vm-db", "online", "offline"))
+ for i, to := range []string{"online", "offline", "online", "offline", "online"} {
+ from := map[string]string{"online": "offline", "offline": "online"}[to]
+ observe(testStatusChange("flap-"+string(rune('a'+i)), "vm-web", from, to))
+ }
+ got := drainAgentEvents(events)
+ if len(got) != 2 {
+ t.Fatalf("immediate events = %+v, want one per resource", got)
+ }
+ first := got[0].Payload.(AgentEventResourceChangePayload)
+ if got[0].Kind != AgentEventResourceStatusChanged || first.ChangeID != "c1" || first.From != "online" || first.To != "offline" || first.CoalescedTransitions != 0 {
+ t.Fatalf("first event = %s %+v", got[0].Kind, first)
+ }
+
+ windows.closeAll()
+ got = drainAgentEvents(events)
+ if len(got) != 1 {
+ t.Fatalf("window close events = %+v, want one coalesced event for vm-web", got)
+ }
+ coalesced := got[0].Payload.(AgentEventResourceChangePayload)
+ if coalesced.ResourceID != "vm-web" || coalesced.ChangeID != "flap-e" || coalesced.From != "offline" || coalesced.To != "online" || coalesced.CoalescedTransitions != 5 {
+ t.Fatalf("coalesced = %+v", coalesced)
+ }
+
+ // The coalesced event opened a new window. A flap inside it that returns
+ // to the announced status publishes nothing when it closes.
+ observe(testStatusChange("g", "vm-web", "online", "offline"))
+ observe(testStatusChange("h", "vm-web", "offline", "online"))
+ windows.closeAll()
+ if got := drainAgentEvents(events); len(got) != 0 {
+ t.Fatalf("round-trip flap events = %+v, want none", got)
+ }
+
+ // With every window closed, the next change is announced at once again.
+ observe(testStatusChange("i", "vm-web", "online", "offline"))
+ got = drainAgentEvents(events)
+ if len(got) != 1 || got[0].Payload.(AgentEventResourceChangePayload).ChangeID != "i" {
+ t.Fatalf("post-quiet events = %+v", got)
+ }
+ windows.closeAll()
+ producer.mu.Lock()
+ open := len(producer.windows)
+ producer.mu.Unlock()
+ if open != 0 {
+ t.Fatalf("closed windows must be forgotten, %d still open", open)
+ }
+}
diff --git a/internal/api/agent_events.go b/internal/api/agent_events.go
index d8a0a67ed7..179988af36 100644
--- a/internal/api/agent_events.go
+++ b/internal/api/agent_events.go
@@ -47,6 +47,14 @@ const (
// the audit endpoint.
AgentEventActionCompleted AgentEventKind = AgentEventKind(agentcapabilities.EventKindActionCompleted)
+ // AgentEventAlertFired, AgentEventAlertResolved and
+ // AgentEventResourceStatusChanged fire when a monitor adapter records
+ // the matching live resource change. Payload is
+ // AgentEventResourceChangePayload; see agent_change_events.go.
+ AgentEventAlertFired AgentEventKind = AgentEventKind(agentcapabilities.EventKindAlertFired)
+ AgentEventAlertResolved AgentEventKind = AgentEventKind(agentcapabilities.EventKindAlertResolved)
+ AgentEventResourceStatusChanged AgentEventKind = AgentEventKind(agentcapabilities.EventKindResourceStatusChanged)
+
// AgentEventStreamConnected is emitted once when an SSE subscriber connects
// so clients can distinguish a healthy stream from an idle one.
AgentEventStreamConnected AgentEventKind = AgentEventKind(agentcapabilities.EventKindStreamConnected)
@@ -66,6 +74,15 @@ type AgentEvent struct {
ID uint64 `json:"id"`
At time.Time `json:"at"`
Payload any `json:"payload,omitempty"`
+
+ // orgID scopes the event to one organization's subscribers. Empty means
+ // the event is not tenant-scoped and reaches every subscriber.
+ orgID string
+}
+
+// visibleToOrg reports whether a subscriber in orgID may receive the event.
+func (e AgentEvent) visibleToOrg(orgID string) bool {
+ return e.orgID == "" || e.orgID == normalizeAgentEventOrgID(orgID)
}
// AgentEventFindingCreatedPayload is the payload shape for
@@ -267,6 +284,7 @@ func (b *AgentEventBroadcaster) HandleAgentEvents(w http.ResponseWriter, r *http
events, unsubscribe := b.Subscribe()
defer unsubscribe()
+ orgID := GetOrgID(r.Context())
// Initial sync event so agents that just connected can confirm
// the stream is open before the first real event lands. This is
@@ -289,6 +307,9 @@ func (b *AgentEventBroadcaster) HandleAgentEvents(w http.ResponseWriter, r *http
if !open {
return
}
+ if !event.visibleToOrg(orgID) {
+ continue
+ }
writeAgentSSEEvent(w, redactAgentEventCommandsForRequest(event, r))
flusher.Flush()
case <-heartbeat.C:
diff --git a/internal/api/agent_substrate_e2e_test.go b/internal/api/agent_substrate_e2e_test.go
index 5f68a234a5..044c16ef07 100644
--- a/internal/api/agent_substrate_e2e_test.go
+++ b/internal/api/agent_substrate_e2e_test.go
@@ -1,16 +1,21 @@
package api
import (
+ "bufio"
"bytes"
+ "context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
+ "time"
"github.com/rcourtman/pulse-go-rewrite/internal/agentcapabilities"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
+ "github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
+ unifiedresources "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
)
// TestAgentSubstrate_DiscoveryToTriageToDepthFlowsThroughHTTPBoundary is
@@ -778,3 +783,142 @@ func assertAgentStableEnvelope(t *testing.T, label, body, wantCode string) {
}
}
}
+
+// TestAgentSubstrate_LiveChangeEventsReachSubscribeEventsStream proves the
+// push half of the change substrate through the HTTP boundary: a status
+// change and a live alert that the monitor's adapter records reach a
+// monitoring-read subscriber of /api/agent/events as resource.status_changed
+// and alert.fired, with the canonical resource ID, current name and the same
+// change IDs list_recent_changes returns for them.
+func TestAgentSubstrate_LiveChangeEventsReachSubscribeEventsStream(t *testing.T) {
+ rawToken := "agent-live-events-e2e.12345678"
+ cfg := newTestConfigWithTokens(t, newTokenRecord(t, rawToken, []string{config.ScopeMonitoringRead}, nil))
+ monitor, err := monitoring.New(cfg)
+ if err != nil {
+ t.Fatalf("new monitor: %v", err)
+ }
+ t.Cleanup(monitor.Stop)
+ router := NewRouter(cfg, monitor, nil, nil, func() error { return nil }, "1.0.0")
+ cleanupTestRouter(t, router)
+ adapter := router.monitorAdapterForMonitor(monitor)
+ if adapter == nil {
+ t.Fatal("router must wire a resource adapter for the monitor")
+ }
+ now := time.Now().UTC().Truncate(time.Second)
+ snapshot := models.StateSnapshot{
+ Nodes: []models.Node{{ID: "lab-pve1", Name: "pve1", Instance: "lab", Host: "https://pve1.lab:8006", Status: "online", LastSeen: now}},
+ VMs: []models.VM{{ID: "lab:pve1:101", VMID: 101, Name: "web", Node: "pve1", Instance: "lab", Status: "running", Type: "qemu", LastSeen: now}},
+ }
+ adapter.PopulateFromSnapshot(snapshot)
+
+ server := httptest.NewServer(router.Handler())
+ defer server.Close()
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, server.URL+agentcapabilities.AgentEventsPath, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ req.Header.Set("X-API-Token", rawToken)
+ resp, err := http.DefaultClient.Do(req)
+ if err != nil {
+ t.Fatalf("subscribe: %v", err)
+ }
+ defer resp.Body.Close()
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("subscribe status = %d", resp.StatusCode)
+ }
+ deadline := time.Now().Add(2 * time.Second)
+ for time.Now().Before(deadline) && router.agentEventBroadcaster.SubscriberCount() == 0 {
+ time.Sleep(10 * time.Millisecond)
+ }
+
+ snapshot.VMs[0].Status = "stopped"
+ snapshot.VMs[0].LastSeen = now.Add(time.Minute)
+ adapter.PopulateFromSnapshot(snapshot)
+ if err := adapter.RecordLiveChange(unifiedresources.ResourceChange{
+ ID: "e2e-alert-fired",
+ ResourceID: "lab:pve1:101",
+ Kind: unifiedresources.ChangeAlertFired,
+ SourceType: unifiedresources.SourceHeuristic,
+ ObservedAt: now.Add(time.Minute),
+ Metadata: map[string]any{
+ unifiedresources.MetadataAlertIdentifier: "lab:pve1:101-powered-off",
+ unifiedresources.MetadataAlertType: "powered-off",
+ unifiedresources.MetadataAlertLevel: "critical",
+ unifiedresources.MetadataAlertMessage: "VM web is stopped",
+ },
+ }); err != nil {
+ t.Fatalf("record live alert: %v", err)
+ }
+
+ type frame struct {
+ Kind string `json:"kind"`
+ Payload AgentEventResourceChangePayload `json:"payload"`
+ }
+ frames := make(chan []frame, 1)
+ go func() {
+ scanner := bufio.NewScanner(resp.Body)
+ var seen []frame
+ for scanner.Scan() {
+ line := scanner.Text()
+ if !strings.HasPrefix(line, "data: ") {
+ continue
+ }
+ var f frame
+ if json.Unmarshal([]byte(strings.TrimPrefix(line, "data: ")), &f) != nil {
+ continue
+ }
+ if f.Kind == string(agentcapabilities.EventKindResourceStatusChanged) || f.Kind == string(agentcapabilities.EventKindAlertFired) {
+ seen = append(seen, f)
+ }
+ if len(seen) == 2 {
+ frames <- seen
+ return
+ }
+ }
+ }()
+ var got []frame
+ select {
+ case got = <-frames:
+ case <-time.After(5 * time.Second):
+ t.Fatal("timed out waiting for change events on /api/agent/events")
+ }
+ cancel()
+
+ status, alert := got[0], got[1]
+ if status.Kind != string(agentcapabilities.EventKindResourceStatusChanged) || alert.Kind != string(agentcapabilities.EventKindAlertFired) {
+ t.Fatalf("event order = %s, %s", status.Kind, alert.Kind)
+ }
+ vmID := status.Payload.ResourceID
+ if vmID == "" || vmID == "lab:pve1:101" || status.Payload.ResourceName != "web" || status.Payload.ResourceType != string(unifiedresources.ResourceTypeVM) {
+ t.Fatalf("status payload must carry the canonical VM identity: %+v", status.Payload)
+ }
+ if status.Payload.From != "online" || status.Payload.To != "offline" {
+ t.Fatalf("status payload from/to = %+v", status.Payload)
+ }
+ if alert.Payload.ResourceID != vmID || alert.Payload.ResourceName != "web" || alert.Payload.ChangeID != "e2e-alert-fired" ||
+ alert.Payload.AlertLevel != "critical" || alert.Payload.AlertType != "powered-off" {
+ t.Fatalf("alert payload = %+v", alert.Payload)
+ }
+
+ // The events point at rows the timeline read returns.
+ listReq := httptest.NewRequest(http.MethodGet, agentcapabilities.RecentChangesCapabilityPath+"?resourceId="+vmID, nil)
+ listReq.Header.Set("X-API-Token", rawToken)
+ listRec := httptest.NewRecorder()
+ router.Handler().ServeHTTP(listRec, listReq)
+ if listRec.Code != http.StatusOK {
+ t.Fatalf("list_recent_changes status = %d, body = %s", listRec.Code, listRec.Body.String())
+ }
+ var listed AgentRecentChanges
+ if err := json.NewDecoder(listRec.Body).Decode(&listed); err != nil {
+ t.Fatal(err)
+ }
+ ids := map[string]bool{}
+ for _, change := range listed.Changes {
+ ids[change.ID] = true
+ }
+ if !ids[status.Payload.ChangeID] || !ids[alert.Payload.ChangeID] {
+ t.Fatalf("event change IDs %q and %q must be readable through list_recent_changes: %+v", status.Payload.ChangeID, alert.Payload.ChangeID, listed.Changes)
+ }
+}
diff --git a/internal/api/contract_test.go b/internal/api/contract_test.go
index 1e14be14d1..43b18e39d7 100644
--- a/internal/api/contract_test.go
+++ b/internal/api/contract_test.go
@@ -1,6 +1,7 @@
package api
import (
+ "bufio"
"bytes"
"context"
"crypto/ed25519"
@@ -45,6 +46,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/api/agentbinding"
"github.com/rcourtman/pulse-go-rewrite/internal/api/agenttokens"
"github.com/rcourtman/pulse-go-rewrite/internal/api/alerting"
+ "github.com/rcourtman/pulse-go-rewrite/internal/api/apicontext"
"github.com/rcourtman/pulse-go-rewrite/internal/api/chartapi"
"github.com/rcourtman/pulse-go-rewrite/internal/api/configapi"
"github.com/rcourtman/pulse-go-rewrite/internal/api/resourceapi"
@@ -28675,3 +28677,62 @@ func TestContract_PulseMCPPublishesRecentChangesAsReadOnlyTool(t *testing.T) {
}
t.Fatalf("Pulse MCP surface does not publish %s", agentcapabilities.ListRecentChangesCapabilityName)
}
+
+// Change events reach only subscribers in the organization whose monitor
+// recorded them; untagged events still reach everyone.
+func TestContract_AgentChangeEventsStayInsideTheirOrganization(t *testing.T) {
+ b := NewAgentEventBroadcaster()
+ producer, _ := newTestAgentChangeEventProducer(b)
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ ctx := context.WithValue(r.Context(), apicontext.OrgIDContextKey, "tenant-a")
+ b.HandleAgentEvents(w, r.WithContext(ctx))
+ }))
+ defer server.Close()
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, server.URL, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ resp, err := http.DefaultClient.Do(req)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer resp.Body.Close()
+ deadline := time.Now().Add(2 * time.Second)
+ for time.Now().Before(deadline) && b.SubscriberCount() == 0 {
+ time.Sleep(10 * time.Millisecond)
+ }
+
+ producer.observer("tenant-b", nil)(unifiedresources.ResourceChange{ID: "foreign", ResourceID: "vm-b", Kind: unifiedresources.ChangeAlertFired})
+ producer.observer("", nil)(unifiedresources.ResourceChange{ID: "default-org", ResourceID: "vm-d", Kind: unifiedresources.ChangeAlertFired})
+ producer.observer("tenant-a", nil)(unifiedresources.ResourceChange{ID: "own", ResourceID: "vm-a", Kind: unifiedresources.ChangeAlertFired})
+ b.PublishFindingCreated(AgentEventFindingCreatedPayload{FindingID: "untagged-finding"})
+
+ got := make(chan string, 1)
+ go func() {
+ scanner := bufio.NewScanner(resp.Body)
+ var collected strings.Builder
+ for scanner.Scan() {
+ collected.WriteString(scanner.Text())
+ collected.WriteString("\n")
+ if strings.Contains(collected.String(), "untagged-finding") {
+ got <- collected.String()
+ return
+ }
+ }
+ }()
+ select {
+ case stream := <-got:
+ if !strings.Contains(stream, "event: alert.fired") || !strings.Contains(stream, `"changeId":"own"`) {
+ t.Fatalf("own organization's alert missing from stream:\n%s", stream)
+ }
+ for _, foreign := range []string{`"changeId":"foreign"`, `"changeId":"default-org"`} {
+ if strings.Contains(stream, foreign) {
+ t.Fatalf("stream leaked %s:\n%s", foreign, stream)
+ }
+ }
+ case <-time.After(3 * time.Second):
+ t.Fatal("timed out waiting for the stream")
+ }
+}
diff --git a/internal/api/router.go b/internal/api/router.go
index 6087f062ef..f0f86c532c 100644
--- a/internal/api/router.go
+++ b/internal/api/router.go
@@ -104,6 +104,7 @@ type Router struct {
maintenanceSentinel *maintenancesentinel.Sentinel
agentContextHandler *AgentContextHandler
agentEventBroadcaster *AgentEventBroadcaster
+ agentChangeEvents *agentChangeEventProducer
resourceRegistry *unifiedresources.ResourceRegistry
trueNASPoller *monitoring.TrueNASPoller
vmwarePoller *monitoring.VMwarePoller
@@ -569,6 +570,7 @@ func (r *Router) setupRoutes() {
// read for the wrong org.
r.agentContextHandler.SetApprovalsProvider(agentApprovalStoreProvider{})
r.agentEventBroadcaster = NewAgentEventBroadcaster()
+ r.agentChangeEvents = newAgentChangeEventProducer(r.agentEventBroadcaster, defaultResourceStatusEventWindow)
if r.resourceHandlers != nil && r.agentEventBroadcaster != nil {
r.resourceHandlers.SetActionCompletedPublisher(r.agentEventBroadcaster.PublishActionCompletedRecord)
}
@@ -1831,6 +1833,11 @@ func (r *Router) configureMonitorDependenciesLocked(m *monitoring.Monitor) {
// Apply saved notification policy before the first publication, which can
// evaluate alerts. The combined boundary avoids one refresh per source.
if adapter := r.monitorAdapterForMonitor(m); adapter != nil {
+ // Live alert and status changes this adapter records reach the
+ // agent event stream for the monitor's organization only.
+ if r.agentChangeEvents != nil {
+ adapter.SetChangeObserver(r.agentChangeEvents.observer(m.GetOrgID(), adapter.GetByReference))
+ }
m.SetResourceStoreAndProviders(adapter, r.monitorSupplementalRecords)
} else {
m.SetSupplementalRecordsProviders(r.monitorSupplementalRecords)
diff --git a/internal/monitoring/monitor_alert_handling_test.go b/internal/monitoring/monitor_alert_handling_test.go
index 6f68e3b4e1..4b97a6d4f1 100644
--- a/internal/monitoring/monitor_alert_handling_test.go
+++ b/internal/monitoring/monitor_alert_handling_test.go
@@ -7,6 +7,7 @@ import (
"net/http"
"net/http/httptest"
"strings"
+ "sync"
"sync/atomic"
"testing"
"time"
@@ -1950,3 +1951,89 @@ func TestActiveAlertsSnapshotCarriesSubjectFreeSummary(t *testing.T) {
}
require.True(t, found, "expected the container health alert in the state snapshot")
}
+
+type observedAlertChanges struct {
+ mu sync.Mutex
+ changes []unifiedresources.ResourceChange
+}
+
+func (o *observedAlertChanges) observe(change unifiedresources.ResourceChange) {
+ o.mu.Lock()
+ defer o.mu.Unlock()
+ if change.Kind == unifiedresources.ChangeAlertFired || change.Kind == unifiedresources.ChangeAlertResolved {
+ o.changes = append(o.changes, change)
+ }
+}
+
+func (o *observedAlertChanges) take() []unifiedresources.ResourceChange {
+ o.mu.Lock()
+ defer o.mu.Unlock()
+ out := o.changes
+ o.changes = nil
+ return out
+}
+
+// Live alert lifecycle events reach the adapter's change observer (and so the
+// agent event stream) with the canonical resource ID they were stored under.
+// Replay, reconciliation and imported history record the same rows without
+// announcing them again.
+func TestAlertTimelineAnnouncesOnlyLiveLifecycleChanges(t *testing.T) {
+ store, err := unifiedresources.NewSQLiteResourceStore(t.TempDir(), "default")
+ require.NoError(t, err)
+ t.Cleanup(func() { require.NoError(t, store.Close()) })
+ manager := alerts.NewManagerWithDataDir(t.TempDir())
+ t.Cleanup(manager.Stop)
+ config := manager.GetConfig()
+ config.Enabled = true
+ config.ActivationState = alerts.ActivationPending
+ config.TimeThresholds = map[string]int{}
+ config.SuppressionWindow = 0
+ manager.UpdateConfig(config)
+
+ containerID := strings.Repeat("e", 64)
+ host := models.DockerHost{ID: "live-host", Hostname: "live-host", LastSeen: time.Now(), Containers: []models.DockerContainer{
+ {ID: containerID, Name: "api", State: "running", Health: "unhealthy"},
+ }}
+ registry := unifiedresources.NewRegistry(store)
+ registry.IngestSnapshot(models.StateSnapshot{DockerHosts: []models.DockerHost{host}})
+ adapter := unifiedresources.NewMonitorAdapter(registry)
+ observed := &observedAlertChanges{}
+ adapter.SetChangeObserver(observed.observe)
+ monitor := &Monitor{alertManager: manager, resourceStore: adapter}
+ manager.SubscribeLifecycleCallback(monitor.handleAlertLifecycleEvent)
+ canonicalID := unifiedresources.SourceSpecificID(unifiedresources.ResourceTypeAppContainer, unifiedresources.SourceDocker, host.ID+"/container/"+containerID)
+
+ manager.CheckDockerHost(host)
+ fired := observed.take()
+ require.Len(t, fired, 1)
+ require.Equal(t, unifiedresources.ChangeAlertFired, fired[0].Kind)
+ require.Equal(t, canonicalID, fired[0].ResourceID)
+ require.Equal(t, "docker-container-health", fired[0].Metadata[unifiedresources.MetadataAlertType])
+
+ host.Containers[0].Health = "healthy"
+ manager.CheckDockerHost(host)
+ resolved := observed.take()
+ require.Len(t, resolved, 1)
+ require.Equal(t, unifiedresources.ChangeAlertResolved, resolved[0].Kind)
+ require.Equal(t, canonicalID, resolved[0].ResourceID)
+
+ // The announced changes are the stored rows, so agents can look them up.
+ stored, err := store.GetRecentChangesFiltered(canonicalID, time.Time{}, 10, unifiedresources.ResourceChangeFilters{
+ Kinds: []unifiedresources.ChangeKind{unifiedresources.ChangeAlertFired, unifiedresources.ChangeAlertResolved},
+ })
+ require.NoError(t, err)
+ require.Len(t, stored, 2)
+ require.ElementsMatch(t, []string{fired[0].ID, resolved[0].ID}, []string{stored[0].ID, stored[1].ID})
+
+ // Replaying the whole lifecycle log projects the same events again; none
+ // of them may be announced as new.
+ monitor.replayAlertLifecycleProjections(context.Background())
+ require.Empty(t, observed.take(), "replayed alert history must not reach the live change observer")
+
+ alert := &alerts.Alert{ID: "imported-alert", Type: "cpu", Level: alerts.AlertLevelWarning, ResourceID: canonicalID, StartTime: time.Now().Add(-time.Hour)}
+ monitor.recordAlertTimelineChange(alert, unifiedresources.ChangeAlertFired, alert.StartTime, "")
+ require.Empty(t, observed.take(), "history projected outside the live callback must not be announced")
+ recorded, err := store.GetRecentChangesFiltered(canonicalID, time.Time{}, 10, unifiedresources.ResourceChangeFilters{AlertIdentifiers: []string{alert.ID}})
+ require.NoError(t, err)
+ require.Len(t, recorded, 1, "the non-live projection still records the row")
+}
diff --git a/internal/monitoring/monitor_alerts.go b/internal/monitoring/monitor_alerts.go
index bdf77b0fc1..13bfb29ff1 100644
--- a/internal/monitoring/monitor_alerts.go
+++ b/internal/monitoring/monitor_alerts.go
@@ -23,6 +23,13 @@ type canonicalResourceChangeRecorder interface {
RecordChange(change unifiedresources.ResourceChange) error
}
+// liveResourceChangeRecorder records a change that is happening now and
+// announces it to live subscribers (the agent event stream). Replayed and
+// imported alert history uses RecordChange instead, so it is never announced.
+type liveResourceChangeRecorder interface {
+ RecordLiveChange(change unifiedresources.ResourceChange) error
+}
+
// GetAlertManager returns the alert manager
func (m *Monitor) GetAlertManager() *alerts.Manager {
return m.alertManager
@@ -328,7 +335,16 @@ func (m *Monitor) handleAlertEscalated(hub *websocket.Hub, alert *alerts.Alert,
m.broadcastEscalatedAlert(hub, alert)
}
+// handleAlertLifecycleEvent projects a live lifecycle event from the alert
+// manager's callback bus.
func (m *Monitor) handleAlertLifecycleEvent(event alerts.LifecycleEvent) {
+ m.projectAlertLifecycleEvent(event, true)
+}
+
+// projectAlertLifecycleEvent writes the incident timeline and canonical
+// resource change for one lifecycle event. live is false for replay and
+// reconciliation, which project events that already happened.
+func (m *Monitor) projectAlertLifecycleEvent(event alerts.LifecycleEvent, live bool) {
alert := event.Alert
if m == nil || alert == nil {
return
@@ -356,7 +372,7 @@ func (m *Monitor) handleAlertLifecycleEvent(event alerts.LifecycleEvent) {
if event.Type == eventlog.TypeFired && !alert.StartTime.IsZero() {
occurredAt = alert.StartTime
}
- m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertFired, occurredAt, "")
+ m.recordAlertTimelineChangeWithLiveness(timelineAlert, unifiedresources.ChangeAlertFired, occurredAt, "", live)
case eventlog.TypeAcknowledged:
if m.incidentStore != nil {
m.incidentStore.RecordAlertAcknowledged(timelineAlert, actor)
@@ -365,12 +381,12 @@ func (m *Monitor) handleAlertLifecycleEvent(event alerts.LifecycleEvent) {
if alert.AckTime != nil && !alert.AckTime.IsZero() {
occurredAt = *alert.AckTime
}
- m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertAcknowledged, occurredAt, actor)
+ m.recordAlertTimelineChangeWithLiveness(timelineAlert, unifiedresources.ChangeAlertAcknowledged, occurredAt, actor, live)
case eventlog.TypeUnacknowledged:
if m.incidentStore != nil {
m.incidentStore.RecordAlertUnacknowledged(timelineAlert, actor)
}
- m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertUnacknowledged, event.OccurredAt, actor)
+ m.recordAlertTimelineChangeWithLiveness(timelineAlert, unifiedresources.ChangeAlertUnacknowledged, event.OccurredAt, actor, live)
case eventlog.TypeSnoozed:
timelineAlert = timelineAlert.Clone()
if timelineAlert.Metadata == nil {
@@ -379,14 +395,14 @@ func (m *Monitor) handleAlertLifecycleEvent(event alerts.LifecycleEvent) {
if until := strings.TrimSpace(event.Details["until"]); until != "" {
timelineAlert.Metadata["snoozedUntil"] = until
}
- m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertSnoozed, event.OccurredAt, actor)
+ m.recordAlertTimelineChangeWithLiveness(timelineAlert, unifiedresources.ChangeAlertSnoozed, event.OccurredAt, actor, live)
case eventlog.TypeUnsnoozed:
- m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertUnsnoozed, event.OccurredAt, actor)
+ m.recordAlertTimelineChangeWithLiveness(timelineAlert, unifiedresources.ChangeAlertUnsnoozed, event.OccurredAt, actor, live)
case eventlog.TypeResolved:
if m.incidentStore != nil {
m.incidentStore.RecordAlertResolved(timelineAlert, event.OccurredAt)
}
- m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertResolved, event.OccurredAt, "")
+ m.recordAlertTimelineChangeWithLiveness(timelineAlert, unifiedresources.ChangeAlertResolved, event.OccurredAt, "", live)
case eventlog.TypeHistoryImported:
m.materializeImportedAlertTimeline(timelineAlert, event.OccurredAt)
}
@@ -539,7 +555,7 @@ func (m *Monitor) replayAlertLifecycleProjections(ctx context.Context) {
// Apply the whole occurrence (including imported acknowledgement and
// resolution) before advancing its watermark, even if shutdown arrives
// during a canonical write. Cancellation only stops the next event.
- m.handleAlertLifecycleEvent(event)
+ m.projectAlertLifecycleEvent(event, false)
if eventID > maxApplied {
maxApplied = eventID
}
@@ -603,26 +619,35 @@ func (m *Monitor) reconcileActiveAlertTimelines(ctx context.Context) {
continue
}
}
- m.handleAlertLifecycleEvent(alerts.LifecycleEvent{
+ m.projectAlertLifecycleEvent(alerts.LifecycleEvent{
Type: eventlog.TypeFired,
OccurredAt: alert.StartTime,
Alert: alert,
- })
+ }, false)
}
}
func (m *Monitor) recordAlertTimelineChange(alert *alerts.Alert, kind unifiedresources.ChangeKind, occurredAt time.Time, actor string) {
+ m.recordAlertTimelineChangeWithLiveness(alert, kind, occurredAt, actor, false)
+}
+
+func (m *Monitor) recordAlertTimelineChangeWithLiveness(alert *alerts.Alert, kind unifiedresources.ChangeKind, occurredAt time.Time, actor string, live bool) {
if alert == nil || m == nil {
return
}
// Background catch-up replay runs concurrently with SetResourceStore, so
// the store handle must be read under the monitor lock.
m.mu.RLock()
- recorder, ok := m.resourceStore.(canonicalResourceChangeRecorder)
+ store := m.resourceStore
m.mu.RUnlock()
+ recorder, ok := store.(canonicalResourceChangeRecorder)
if !ok || recorder == nil {
return
}
+ record := recorder.RecordChange
+ if liveRecorder, ok := store.(liveResourceChangeRecorder); ok && live {
+ record = liveRecorder.RecordLiveChange
+ }
timelineChange := unifiedresources.AlertTimelineChange{
AlertIdentifier: alert.ID,
@@ -649,7 +674,7 @@ func (m *Monitor) recordAlertTimelineChange(alert *alerts.Alert, kind unifiedres
string(kind),
occurredAt.UTC().Format(time.RFC3339Nano),
}, "\x00"))).String()
- if err := recorder.RecordChange(*change); err != nil {
+ if err := record(*change); err != nil {
log.Warn().
Err(err).
Str("resource_id", alert.ResourceID).
diff --git a/internal/unifiedresources/change_emission.go b/internal/unifiedresources/change_emission.go
index ab3df74291..260eb1b826 100644
--- a/internal/unifiedresources/change_emission.go
+++ b/internal/unifiedresources/change_emission.go
@@ -11,9 +11,12 @@ import (
"github.com/google/uuid"
)
-func recordRegistryChanges(store ResourceStore, before, after []Resource, observedAt time.Time, occurredAt *time.Time, sourceType ChangeSourceType, sourceAdapterHint ChangeSourceAdapter) {
+// recordRegistryChanges persists the changes between two resource lists and
+// returns the ones the store accepted, so the caller can announce exactly what
+// was recorded.
+func recordRegistryChanges(store ResourceStore, before, after []Resource, observedAt time.Time, occurredAt *time.Time, sourceType ChangeSourceType, sourceAdapterHint ChangeSourceAdapter) []ResourceChange {
if store == nil {
- return
+ return nil
}
beforeByID := make(map[string]Resource, len(before))
@@ -40,6 +43,7 @@ func recordRegistryChanges(store ResourceStore, before, after []Resource, observ
seen[id] = struct{}{}
}
+ var recorded []ResourceChange
for id := range seen {
beforeResource, beforeOK := beforeByID[id]
afterResource, afterOK := afterByID[id]
@@ -49,8 +53,11 @@ func recordRegistryChanges(store ResourceStore, before, after []Resource, observ
}
if err := store.RecordChange(*change); err != nil {
log.Printf("unifiedresources: failed to record change for %s: %v", change.ResourceID, err)
+ continue
}
+ recorded = append(recorded, *change)
}
+ return recorded
}
// recordRegistryChangesBetweenGenerations compares two complete registry
@@ -58,10 +65,11 @@ func recordRegistryChanges(store ResourceStore, before, after []Resource, observ
// readers. The resources are read only while both registry locks are held;
// persistence happens after releasing them so a slow store cannot block
// registry readers. The caller must keep the old generation stable until this
-// returns (MonitorAdapter's mutationMu does so during a replacement).
-func recordRegistryChangesBetweenGenerations(before, after *ResourceRegistry, observedAt time.Time, occurredAt *time.Time, sourceType ChangeSourceType, sourceAdapterHint ChangeSourceAdapter) {
+// returns (MonitorAdapter's mutationMu does so during a replacement). It
+// returns the changes the store accepted.
+func recordRegistryChangesBetweenGenerations(before, after *ResourceRegistry, observedAt time.Time, occurredAt *time.Time, sourceType ChangeSourceType, sourceAdapterHint ChangeSourceAdapter) []ResourceChange {
if before == nil || after == nil || before.store == nil {
- return
+ return nil
}
var pending []ResourceChange
@@ -100,11 +108,15 @@ func recordRegistryChangesBetweenGenerations(before, after *ResourceRegistry, ob
after.mu.RUnlock()
before.mu.RUnlock()
+ recorded := pending[:0]
for _, change := range pending {
if err := before.store.RecordChange(change); err != nil {
log.Printf("unifiedresources: failed to record change for %s: %v", change.ResourceID, err)
+ continue
}
+ recorded = append(recorded, change)
}
+ return recorded
}
func buildResourceChange(before Resource, beforeOK bool, after Resource, afterOK bool, observedAt time.Time, occurredAt *time.Time, sourceType ChangeSourceType, sourceAdapterHint ChangeSourceAdapter) *ResourceChange {
diff --git a/internal/unifiedresources/history_identity_test.go b/internal/unifiedresources/history_identity_test.go
index 95f12c8e32..f3d22ac01e 100644
--- a/internal/unifiedresources/history_identity_test.go
+++ b/internal/unifiedresources/history_identity_test.go
@@ -1012,3 +1012,61 @@ func TestHistoryIdentityLinkedAgentDiskUsesCanonicalOwner(t *testing.T) {
require.Empty(t, ownerID, ref)
}
}
+
+// The change observer hears the changes a registry rebuild recorded, after
+// the new generation is readable, and the alert rows recorded live through
+// RecordLiveChange under their resolved canonical ID. RecordChange, the path
+// replayed and imported history takes, stays silent.
+func TestMonitorAdapterChangeObserverHearsOnlyRecordedLiveChanges(t *testing.T) {
+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")
+ require.NoError(t, err)
+ t.Cleanup(func() { require.NoError(t, store.Close()) })
+ now := time.Now().UTC().Truncate(time.Second)
+ adapter := NewMonitorAdapter(NewRegistry(store))
+ snapshot := proxmoxHistoryIdentitySnapshot(now)
+ adapter.PopulateFromSnapshot(snapshot)
+ vmID := historyIdentityResourceID(t, adapter.currentRegistry(), ResourceTypeVM, "web")
+
+ var heard []ResourceChange
+ var statusAtNotify string
+ adapter.SetChangeObserver(func(change ResourceChange) {
+ heard = append(heard, change)
+ if change.ResourceID == vmID {
+ if resource, _, ok := adapter.GetByReference(change.ResourceID); ok {
+ statusAtNotify = string(resource.Status)
+ }
+ }
+ })
+
+ snapshot.VMs[0].Status = "stopped"
+ snapshot.VMs[0].LastSeen = now.Add(time.Minute)
+ adapter.PopulateFromSnapshot(snapshot)
+ var transition *ResourceChange
+ for i := range heard {
+ if heard[i].ResourceID == vmID && heard[i].Kind == ChangeStateTransition {
+ transition = &heard[i]
+ }
+ }
+ require.NotNil(t, transition, "status change must reach the observer: %+v", heard)
+ require.Equal(t, "online", transition.From)
+ require.Equal(t, "offline", transition.To)
+ require.Contains(t, transition.Metadata["changedFields"], "status")
+ require.Equal(t, string(StatusOffline), statusAtNotify, "observer must read the generation that produced the change")
+ stored, err := store.GetRecentChangesFiltered(vmID, time.Time{}, 10, ResourceChangeFilters{Kinds: []ChangeKind{ChangeStateTransition}})
+ require.NoError(t, err)
+ require.NotEmpty(t, stored)
+ require.Equal(t, transition.ID, stored[0].ID)
+
+ heard = nil
+ require.NoError(t, adapter.RecordChange(ResourceChange{ID: "replayed", ResourceID: "lab:pve1:101", Kind: ChangeAlertFired, SourceType: SourceHeuristic, ObservedAt: now}))
+ require.Empty(t, heard, "RecordChange must not announce")
+
+ require.NoError(t, adapter.RecordLiveChange(ResourceChange{ID: "live", ResourceID: "lab:pve1:101", Kind: ChangeAlertFired, SourceType: SourceHeuristic, ObservedAt: now}))
+ require.Len(t, heard, 1)
+ require.Equal(t, "live", heard[0].ID)
+ require.Equal(t, vmID, heard[0].ResourceID, "live alert changes carry the canonical ID they were stored under")
+
+ adapter.SetChangeObserver(nil)
+ require.NoError(t, adapter.RecordLiveChange(ResourceChange{ID: "unobserved", ResourceID: "lab:pve1:101", Kind: ChangeAlertResolved, SourceType: SourceHeuristic, ObservedAt: now}))
+ require.Len(t, heard, 1)
+}
diff --git a/internal/unifiedresources/monitor_adapter.go b/internal/unifiedresources/monitor_adapter.go
index 732e1de1c9..03903492b9 100644
--- a/internal/unifiedresources/monitor_adapter.go
+++ b/internal/unifiedresources/monitor_adapter.go
@@ -33,6 +33,43 @@ type MonitorAdapter struct {
legacyHistory legacyHistoryBackfill
overlays overlayReadStateCache
+
+ // changeObserver hears each live change after the store accepted it.
+ // Guarded by mu.
+ changeObserver ResourceChangeObserver
+}
+
+// ResourceChangeObserver is told about a change the adapter just recorded,
+// with its resource ID already resolved to the canonical identity it was
+// stored under. It runs on the recording goroutine, so it must not block.
+// Replayed or imported history never reaches it: only registry diffs and
+// changes recorded through RecordLiveChange do.
+type ResourceChangeObserver func(change ResourceChange)
+
+// SetChangeObserver installs the observer for live changes; nil removes it.
+func (a *MonitorAdapter) SetChangeObserver(observer ResourceChangeObserver) {
+ if a == nil {
+ return
+ }
+ a.mu.Lock()
+ a.changeObserver = observer
+ a.mu.Unlock()
+}
+
+func (a *MonitorAdapter) notifyChanges(changes []ResourceChange) {
+ if a == nil || len(changes) == 0 {
+ return
+ }
+ a.mu.RLock()
+ observer := a.changeObserver
+ a.mu.RUnlock()
+ if observer == nil {
+ return
+ }
+ for _, change := range changes {
+ change.ResourceID = CanonicalResourceID(change.ResourceID)
+ observer(change)
+ }
}
// overlayReadStateMaxAge bounds how long an overlay built from one registry
@@ -245,9 +282,28 @@ func (a *MonitorAdapter) GetByReference(reference string) (*Resource, string, bo
// resolve here so history reads by canonical ID include them. PVE disk alert
// rows are owned row by row instead (proxmoxDiskAlertOwner).
func (a *MonitorAdapter) RecordChange(change ResourceChange) error {
+ _, _, err := a.recordChange(change)
+ return err
+}
+
+// RecordLiveChange records a change that is happening now, as RecordChange
+// does, then tells the change observer. Callers replaying or importing history
+// use RecordChange so old events are not announced as new.
+func (a *MonitorAdapter) RecordLiveChange(change ResourceChange) error {
+ recorded, ok, err := a.recordChange(change)
+ if err != nil || !ok {
+ return err
+ }
+ a.notifyChanges([]ResourceChange{recorded})
+ return nil
+}
+
+// recordChange persists the change and returns it as stored, with the
+// resource ID it resolved to. ok is false when no durable store is attached.
+func (a *MonitorAdapter) recordChange(change ResourceChange) (ResourceChange, bool, error) {
registry := a.currentRegistry()
if registry == nil || registry.store == nil {
- return nil
+ return change, false, nil
}
sourceRef := change.ResourceID
if ref, ok := proxmoxDiskAlertRowReference(change); ok {
@@ -258,7 +314,8 @@ func (a *MonitorAdapter) RecordChange(change ResourceChange) error {
change.Metadata[MetadataAlertResourceID] = ref
change.ResourceID = owner
}
- return registry.store.RecordChange(change)
+ err := registry.store.RecordChange(change)
+ return change, err == nil, err
}
history, hasHistory := registry.store.(resourceHistoryIdentityWriter)
unbound := false
@@ -275,7 +332,7 @@ func (a *MonitorAdapter) RecordChange(change ResourceChange) error {
if hasHistory {
id, found, err := history.ResolveHistorySourceIdentity(sourceRef)
if err != nil {
- return err
+ return change, false, err
}
if found {
change.ResourceID = id
@@ -291,7 +348,7 @@ func (a *MonitorAdapter) RecordChange(change ResourceChange) error {
// No resource answers to it: keep the binding an earlier event recorded.
id, found, err := history.ResolveHistorySourceIdentity(CanonicalResourceID(sourceRef))
if err != nil {
- return err
+ return change, false, err
}
if found {
change.ResourceID = id
@@ -300,16 +357,17 @@ func (a *MonitorAdapter) RecordChange(change ResourceChange) error {
}
}
if hasHistory && CanonicalResourceID(sourceRef) != CanonicalResourceID(change.ResourceID) {
- return history.RecordChangeWithSourceIdentity(change, sourceRef)
+ err := history.RecordChangeWithSourceIdentity(change, sourceRef)
+ return change, err == nil, err
}
if err := registry.store.RecordChange(change); err != nil {
- return err
+ return change, false, err
}
// Inventory may name the resource in a later generation; bind the row then.
if _, ok := registry.store.(legacyHistoryBinder); unbound && ok {
a.legacyHistory.add(CanonicalResourceID(sourceRef), time.Now())
}
- return nil
+ return change, true, nil
}
// GetRecentChanges forwards canonical resource-history reads into the
@@ -545,7 +603,7 @@ func (a *MonitorAdapter) replaceRegistryLocked(snapshot models.StateSnapshot, re
occ := snapshot.LastUpdate
occurredAt = &occ
}
- recordRegistryChangesBetweenGenerations(registry, rebuilt, rebuiltAt, occurredAt, SourcePulseDiff, "")
+ recorded := recordRegistryChangesBetweenGenerations(registry, rebuilt, rebuiltAt, occurredAt, SourcePulseDiff, "")
rebuilt.PersistIdentityPins()
a.mu.Lock()
@@ -555,6 +613,9 @@ func (a *MonitorAdapter) replaceRegistryLocked(snapshot models.StateSnapshot, re
a.lastRebuiltAt = rebuiltAt
a.mu.Unlock()
a.bindLegacyHistory(rebuilt, time.Now())
+ // After the swap, so an observer reading the adapter sees the generation
+ // that produced the change.
+ a.notifyChanges(recorded)
}
// ShouldSkipAPIPolling returns true when agent coverage indicates API
@@ -761,11 +822,12 @@ func (a *MonitorAdapter) PopulateSupplementalRecords(source DataSource, records
before := registry.List()
registry.IngestRecordsWithStaleThresholds(source, records, thresholds)
rebuiltAt := time.Now().UTC()
- recordRegistryChanges(registry.store, before, registry.List(), rebuiltAt, nil, SourcePlatformEvent, changeSourceAdapterForDataSource(source))
+ recorded := recordRegistryChanges(registry.store, before, registry.List(), rebuiltAt, nil, SourcePlatformEvent, changeSourceAdapterForDataSource(source))
registry.PersistIdentityPins()
a.mu.Lock()
a.lastRebuiltAt = rebuiltAt
a.mu.Unlock()
+ a.notifyChanges(recorded)
}
// MetricsTargetForResource resolves the history/metrics target for a canonical