Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Record Docker host, service and sub-resource alert history
rcourtmanf4c104621580
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 01286836ea..e1b8d4a5e7 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -1052,6 +1052,20 @@ replay stays idempotent. The real alert-manager callback path is covered by`TestProxmoxAlertTimelineUsesCanonicalHistoryIdentity` in`internal/monitoring/monitor_alert_handling_test.go`.+Docker host and Swarm service alert lifecycle events, and sub-resource alert+events (ZFS pools and devices, host filesystems, disks, RAID arrays, sensors+and the Unraid array), pass through the same writer, which binds them to the+Docker host, Swarm service, storage or host named in the unified-resources+history identity clause. Container names and shortened container IDs still+bind nothing. `CheckDockerHost` gives a container without an ID its host's+reference, so that container's events join the host's history. A producer+that changes one of these reference shapes must change+`dockerHostHistoryReference` or `historySubResourceOwner` with it.+`TestOwnerAlertTimelinesUseCanonicalHistoryIdentity` in+`internal/monitoring/monitor_alert_handling_test.go` pins the Docker host,+Swarm service, ZFS pool and device, and host filesystem shapes against the+real alert manager.+TrueNAS native alert projection preserves the trimmed, uppercase provider level in ResourceIncident.NativeSeverity. INFO and NOTICE retain the same canonical monitor risk; consumers must not lose their distinct actionability when projecting provider evidence. Native CRITICAL, ALERT, and EMERGENCY all project to canonical critical severity; EMERGENCY must not be discarded as unknown or make a still-active condition appear recovered. WARNING remains warning, and INFO and NOTICE remain informational at this projection boundary.Verification: `TestIncidentProjectionPreservesNativeSeverity` in `internal/truenas/provider_pool_health_contract_test.go` covers all seven native levels and case/whitespace normalization. `TestTrueNASNativeSeverityDispatch` in `internal/alerts/truenas_native_dispatch_test.go` verifies downstream INFO suppression, NOTICE preservation, notification severity, duplicate-poll retention, and confirmed recovery callback identity. The TrueNAS lifecycle tests in `internal/alerts/unified_incidents_test.go` require repeated EMERGENCY evidence to interrupt recovery confirmation. These are fixture-based projection and manager checks, not appliance ingestion or external notification-provider receipt proof.diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.mdindex 8b80254b40..1b1e4d58d9 100644--- a/docs/release-control/v6/internal/subsystems/unified-resources.md+++ b/docs/release-control/v6/internal/subsystems/unified-resources.md@@ -5846,10 +5846,12 @@ and storage IDs), a node-scoped Proxmox guest reference after migration, or`ResolveReferenceID` used by alert policy, so a resource named like a systemreference cannot capture its events. A reference two resources answer to bindsto neither: its event keeps its own reference and creates or changes no-binding, though an earlier binding still covers that reference on read. A+binding, though an earlier binding still covers that reference on read. Like+any unbound reference it is retried, and once a later generation resolves it+to one resource it binds there, replacing an earlier binding. Areference no resourceanswers to follows its retained binding after inventory removal, or keeps its-own history. Docker references never take this path. Alert journal references+own history. Docker container references never take this path. Alert journal referenceswithout a binding (read once per process, plus any written before inventorynamed their resource) are retried on each published registry generation for`legacyHistoryBindWindow`. Bound rows keep their recorded resource ID, so a@@ -5861,6 +5863,46 @@ Assistant and Patrol resource contexts read it from the store. Proof:`TestHistoryIdentityMonitorAdapterResolvesProxmoxAlertReferences` and`TestHistoryIdentityBindsLegacyAlertRowsFromRegistryGenerations` in`internal/unifiedresources/history_identity_test.go`.+Docker host alerts (`docker:<host ID>`) and Swarm service alerts+(`docker:<host ID>/service/<service ID>`) resolve only through the registry's+Docker source identities: the host ID is the Docker host's source ID, and a+service reference names the service with that exact ID in the host's Swarm+cluster, whichever manager reported it. A hostname in the host position is+only ever looked up as a Docker host source ID. A service without an ID is+referenced by its normalized name in the same place, so while its cluster has+such a service every service reference there is a conflict. Alert references+do not record whether they came from an ID, so a row journaled by an ID-less+service can still join another service's history if a later generation or+event binds the same reference; Swarm always reports service IDs, so that needs a malformed+report naming a service exactly like another service's ID. Hostless+`docker-service:<name>` references, container names and shortened container+IDs never bind and are never retried against later inventory. A container+without an ID alerts under its host's reference (`alerts.DockerResourceID`), so+its events join that host's history, never another container's.+Sub-resource alert references name no resource of their own and bind to the+owner whose durable ID they carry (`historySubResourceOwner`): ZFS pool and+device alerts (`<storage ID>/zfs-pool:<pool>[/device:<device>]`) to the+storage, the Unraid array alert (`agent:<host ID>/storage:unraid-array`) to+the array storage with that agent source ID, and host filesystem, SMART disk,+disk temperature, RAID and custom-sensor alerts (`agent:<host ID>/disk:`,+`/disk_temp:`, `/raid:`, `/custom:`) to the host. The owner reference must+resolve to exactly one resource of the expected type; an ambiguous or+incompatible owner is a conflict, so the event keeps its own reference,+follows no retained binding and is retried. Pool, mount and kernel device labels are names,+and a device name can denote a different disk after a reboot, so these alerts+never bind a physical disk. Any other sub-resource shape stays unbound until+its producer and this list change together. The Docker, storage and platform+host-row drawers do not request per-resource history, so these bindings reach+the facets, timeline and intelligence APIs and Assistant resource context, not+a drawer. Patrol's scoped change feed matches the stored resource ID without+expanding bindings, so it includes events recorded under the owner's+canonical ID but not rows journaled before their binding.+Proof: `TestHistoryIdentityMonitorAdapterResolvesDockerHostAndServiceReferences`+and `TestHistoryIdentityMonitorAdapterResolvesSubResourceReferences` in+`internal/unifiedresources/history_identity_test.go`, and+`TestOwnerAlertTimelinesUseCanonicalHistoryIdentity` in+`internal/monitoring/monitor_alert_handling_test.go` against the real alert+producers.That same shared timeline vocabulary now includes the `activity` change kindfor provider-read breadcrumbs such as VMware tasks and events, plus the`vmware_adapter` source-adapter token for canonical provenance drill-down.diff --git a/internal/monitoring/monitor_alert_handling_test.go b/internal/monitoring/monitor_alert_handling_test.goindex bb65eb663a..a8b7454b6c 100644--- a/internal/monitoring/monitor_alert_handling_test.go+++ b/internal/monitoring/monitor_alert_handling_test.go@@ -150,6 +150,67 @@ func TestProxmoxAlertTimelineUsesCanonicalHistoryIdentity(t *testing.T) {require.Equal(t, canonical["pve1"], changes[0].ResourceID)}+// Docker host, Swarm service and sub-resource alerts (a ZFS pool and device,+// a host filesystem) carry references the registry has no resource for. Their+// lifecycle must reach the owner's canonical history, so the producers' ref+// shapes are pinned here against the real alert manager.+func TestOwnerAlertTimelinesUseCanonicalHistoryIdentity(t *testing.T) {+ store, err := unifiedresources.NewSQLiteResourceStore(t.TempDir(), "default")+ require.NoError(t, err)+ t.Cleanup(func() { require.NoError(t, store.Close()) })+ manager := alerts.NewManagerWithDataDir(t.TempDir())+ t.Cleanup(manager.Stop)+ config := manager.GetConfig()+ config.Enabled = true+ config.ActivationState = alerts.ActivationPending+ config.TimeThresholds = map[string]int{"all": 0}+ config.MetricTimeThresholds = map[string]map[string]int{"all": {"disk": 0}}+ config.SuppressionWindow = 0+ manager.UpdateConfig(config)+ now := time.Now()+ dockerHost := models.DockerHost{ID: "dh-7f3a", Hostname: "tower", DisplayName: "tower", Status: "online", LastSeen: now,+ Swarm: &models.DockerSwarmInfo{NodeID: "node-a", NodeRole: "manager", LocalState: "active", ControlAvailable: true, ClusterID: "swarm-1"},+ Services: []models.DockerService{{ID: "x7k2m9q4w1e8r5t3y6u0i2o4p", Name: "web", DesiredTasks: 2, RunningTasks: 0}}}+ storage := models.Storage{ID: "lab-pve1-local-zfs", Name: "local-zfs", Node: "pve1", Instance: "lab", Type: "zfspool", Status: "available", Total: 100, Used: 10, Usage: 10, Enabled: true, Active: true, LastSeen: now,+ ZFSPool: &models.ZFSPool{Name: "rpool", State: "DEGRADED", Status: "Degraded", Devices: []models.ZFSDevice{{Name: "sda2", Type: "disk", State: "FAULTED"}}}}+ agentHost := models.Host{ID: "host-nas", Hostname: "nas", DisplayName: "nas", MachineID: "fedcba9876543210", Status: "online", LastSeen: now,+ Disks: []models.Disk{{Mountpoint: "/var", Device: "/dev/sdb1", Type: "ext4", Total: 100, Used: 99, Free: 1, Usage: 99}}}+ registry := unifiedresources.NewRegistry(store)+ registry.IngestSnapshot(models.StateSnapshot{DockerHosts: []models.DockerHost{dockerHost}, Storage: []models.Storage{storage}, Hosts: []models.Host{agentHost}})+ monitor := &Monitor{alertManager: manager, resourceStore: unifiedresources.NewMonitorAdapter(registry)}+ manager.SubscribeLifecycleCallback(monitor.handleAlertLifecycleEvent)+ manager.CheckDockerHost(dockerHost)+ manager.CheckStorage(storage)+ manager.CheckHost(agentHost)+ for range 3 {+ manager.HandleDockerHostOffline(dockerHost)+ }++ owner := map[string]string{}+ for _, resource := range registry.List() {+ owner[string(resource.Type)+"/"+resource.Name] = resource.ID+ }+ filters := unifiedresources.ResourceChangeFilters{Kinds: []unifiedresources.ChangeKind{unifiedresources.ChangeAlertFired}}+ want := map[string]string{+ "docker-host-offline": owner["agent/tower"],+ "docker-service-health": owner["docker-service/web"],+ "zfs-pool-state": owner["storage/local-zfs"],+ "zfs-device": owner["storage/local-zfs"],+ "disk": owner["agent/nas"],+ }+ for alertType, canonicalID := range want {+ require.NotEmpty(t, canonicalID, alertType)+ changes, err := store.GetRecentChangesFiltered(canonicalID, time.Time{}, 10, filters)+ require.NoError(t, err)+ found := false+ for _, change := range changes {+ require.Equal(t, canonicalID, change.ResourceID)+ found = found || change.Metadata["alert_type"] == alertType+ }+ require.True(t, found, "%s alert missing from %s history: %+v", alertType, canonicalID, changes)+ }+}+func TestMonitor_HandleAlertFired_Extra(t *testing.T) {// 1. Alert is nilm1 := &Monitor{}diff --git a/internal/unifiedresources/history_identity.go b/internal/unifiedresources/history_identity.goindex 9dc69d886e..61ddc1ce1b 100644--- a/internal/unifiedresources/history_identity.go+++ b/internal/unifiedresources/history_identity.go@@ -32,35 +32,180 @@ func legacyDockerHistoryIdentity(ref string) (sourceID, canonicalID string, ok breturn sourceID, SourceSpecificID(ResourceTypeAppContainer, SourceDocker, sourceID), true}-// isDockerHistoryReference marks Docker alert source references, which only a-// complete container ID binds (legacyDockerHistoryIdentity). A container name-// or shortened ID must never join history through general reference matching.+// isDockerHistoryReference marks Docker alert source references, including the+// hostless "docker-service:<name>" form. They never join history through+// general reference matching: only a Docker host, a Swarm service ID+// (dockerHostHistoryReference) or a complete container ID+// (legacyDockerHistoryIdentity) binds.func isDockerHistoryReference(ref string) bool {- return strings.HasPrefix(strings.TrimSpace(ref), "docker:")+ ref = strings.TrimSpace(ref)+ return strings.HasPrefix(ref, "docker:") || strings.HasPrefix(ref, "docker-service:")+}++// isDockerNameHistoryReference marks Docker alert references that are not host+// or service references: container references, which only a complete+// container ID binds, and hostless service names. A name or shortened ID must+// never join history, not even through a retained binding.+func isDockerNameHistoryReference(ref string) bool {+ _, _, ok := dockerHostHistoryReference(ref)+ return isDockerHistoryReference(ref) && !ok+}++// dockerHostHistoryReference parses the Docker host alert reference+// ("docker:<host ID>", alerts.DockerResourceID without a container) and the+// Swarm service alert reference ("docker:<host ID>/service/<service ID>",+// alerts.DockerServiceResourceID). serviceID is empty for a host reference.+// "docker:unknown" is the shared fallback for alerts without a host.+func dockerHostHistoryReference(ref string) (hostID, serviceID string, ok bool) {+ rest, found := strings.CutPrefix(strings.TrimSpace(ref), "docker:")+ if !found {+ return "", "", false+ }+ hostID, serviceID, isService := strings.Cut(rest, "/service/")+ if hostID == "" || hostID == "unknown" || strings.TrimSpace(hostID) != hostID || strings.Contains(hostID, "/") {+ return "", "", false+ }+ if isService && (serviceID == "" || strings.TrimSpace(serviceID) != serviceID || strings.Contains(serviceID, "/")) {+ return "", "", false+ }+ return hostID, serviceID, true+}++// dockerHistoryOwnerLocked resolves a Docker host or Swarm service alert+// reference through the registry's own Docker source identities. The host ID+// is the Docker host's source ID. A service reference names the service with+// that ID in the host's Swarm cluster, which every manager of the cluster+// reports under its own host ID. A service without an ID is referenced by its+// normalized name in the same place, so while the cluster has such a service+// every service reference there is a conflict.+func (rr *ResourceRegistry) dockerHistoryOwnerLocked(ref string) (resourceID string, conflict bool) {+ hostID, serviceID, ok := dockerHostHistoryReference(ref)+ if !ok {+ return "", false+ }+ hostResourceID := rr.bySource[SourceDocker][hostID]+ host := rr.resources[hostResourceID]+ if host == nil || CanonicalResourceType(host.Type) != ResourceTypeAgent {+ return "", false+ }+ if serviceID == "" {+ return hostResourceID, false+ }+ if host.Docker == nil {+ return "", false+ }+ cluster := dockerSwarmClusterKeyFromMeta(host.Docker.Swarm)+ if cluster == "" {+ return "", false+ }+ for _, other := range rr.resources {+ if CanonicalResourceType(other.Type) == ResourceTypeDockerService && other.Docker != nil &&+ strings.TrimSpace(other.Docker.ServiceID) == "" && dockerSwarmClusterKeyFromMeta(other.Docker.Swarm) == cluster {+ return "", true+ }+ }+ serviceResourceID := rr.bySource[SourceDocker][normalizeSourceID(cluster+":service:"+serviceID)]+ service := rr.resources[serviceResourceID]+ if service == nil || CanonicalResourceType(service.Type) != ResourceTypeDockerService ||+ service.Docker == nil || strings.TrimSpace(service.Docker.ServiceID) != serviceID {+ return "", false+ }+ return serviceResourceID, false+}++// historySubResourceOwner names the owner of a sub-resource alert reference,+// one that names no resource of its own, and the type that owner must have.+// Only these producer shapes qualify, each carrying its owner's durable+// identity:+// - "<storage ID>/zfs-pool:<pool>[/device:<device>]" (ZFS pool state,+// errors and device health) belongs to the storage.+// - "agent:<host ID>/storage:<array>" (Unraid array) is the agent source ID+// "<host ID>/storage:<array>" of the host's array storage.+// - "agent:<host ID>/disk:<mount or device>", ".../disk_temp:<device>",+// ".../raid:<device>" and ".../custom:<sensor>" belong to the host.+//+// Pool, mount and kernel device labels are names: a device name can belong to+// a different disk after a reboot, so these never bind a physical disk.+func historySubResourceOwner(ref string) (ownerRef string, ownerType ResourceType, ok bool) {+ if rest, found := strings.CutPrefix(ref, "agent:"); found {+ hostID, child, _ := strings.Cut(rest, "/")+ kind, label, _ := strings.Cut(child, ":")+ if hostID == "" || label == "" {+ return "", "", false+ }+ switch kind {+ case "storage":+ return hostID + "/" + child, ResourceTypeStorage, true+ case "disk", "disk_temp", "raid", "custom":+ return "agent:" + hostID, ResourceTypeAgent, true+ }+ return "", "", false+ }+ if i := strings.LastIndex(ref, "/zfs-pool:"); i > 0 {+ pool, device, hasDevice := strings.Cut(ref[i+len("/zfs-pool:"):], "/device:")+ if pool == "" || strings.Contains(pool, "/") || (hasDevice && (device == "" || strings.Contains(device, "/"))) {+ return "", "", false+ }+ return ref[:i], ResourceTypeStorage, true+ }+ return "", "", false}-// resolveHistoryReference resolves a non-Docker alert reference to the-// resource whose history it joins. Only durable identities qualify: the-// canonical ID, a retired era, a source ID (Proxmox node, guest and storage-// IDs), a node-scoped Proxmox guest reference, or an agent alert reference-// ("agent:<host ID>"). Names and hostnames never bind history, so a resource-// named like a system reference cannot capture its events. claimed reports-// that inventory answers to the reference, possibly ambiguously; an ambiguous-// reference resolves to nothing and must not follow a retained binding.+// resolveHistoryReference resolves an alert reference to the resource whose+// history it joins. Only durable identities qualify: the canonical ID, a+// retired era, a source ID (Proxmox node, guest and storage IDs), a+// node-scoped Proxmox guest reference, an agent alert reference+// ("agent:<host ID>"), a Docker host or Swarm service ID, or the owner of a+// sub-resource reference (historySubResourceOwner). Names and hostnames never+// bind history, so a resource named like a system reference cannot capture+// its events. claimed reports that inventory answers to the reference,+// possibly ambiguously; an ambiguous or conflicting reference resolves to+// nothing and must not follow a retained binding.func (rr *ResourceRegistry) resolveHistoryReference(ref string) (resourceID string, claimed bool) {ref = CanonicalResourceID(ref)- if rr == nil || ref == "" || isDockerHistoryReference(ref) {+ if rr == nil || ref == "" {return "", false}rr.mu.RLock()defer rr.mu.RUnlock()+ if isDockerHistoryReference(ref) {+ resourceID, conflict := rr.dockerHistoryOwnerLocked(ref)+ return resourceID, resourceID != "" || conflict+ }+ matches := rr.durableHistoryMatchesLocked(ref)+ if ownerRef, ownerType, ok := historySubResourceOwner(ref); ok && len(matches) == 0 {+ // The owner must resolve uniquely and have the expected type. An owner+ // reference that names an ambiguous or incompatible resource is a+ // conflict: the event keeps its own reference.+ owners := rr.durableHistoryMatchesLocked(ownerRef)+ for id := range owners {+ if len(owners) != 1 || CanonicalResourceType(rr.resources[id].Type) != ownerType {+ return "", true+ }+ matches[id] = struct{}{}+ }+ }+ if len(matches) != 1 {+ return "", len(matches) > 1+ }+ for id := range matches {+ resourceID = id+ }+ return resourceID, true+}++// durableHistoryMatchesLocked lists the resources that answer to a reference+// by durable identity.+func (rr *ResourceRegistry) durableHistoryMatchesLocked(ref string) map[string]struct{} {+ matches := make(map[string]struct{}, 1)if rr.resources[ref] != nil {- return ref, true+ matches[ref] = struct{}{}+ return matches}if id := rr.supersededResourceIDLocked(ref); rr.resources[id] != nil {- return id, true+ matches[id] = struct{}{}+ return matches}- matches := make(map[string]struct{}, 1)for _, mapping := range rr.bySource {if id := mapping[ref]; rr.resources[id] != nil {matches[id] = struct{}{}@@ -78,13 +223,7 @@ func (rr *ResourceRegistry) resolveHistoryReference(ref string) (resourceID stri}}}- if len(matches) != 1 {- return "", len(matches) > 1- }- for id := range matches {- resourceID = id- }- return resourceID, true+ return matches}// resourceHistoryIdentityWriter binds a source reference to the canonical@@ -187,11 +326,12 @@ func (b *legacyHistoryBackfill) add(ref string, now time.Time) {}// unboundHistoryReferences lists alert journal references without a binding.-// Docker references are excluded: the store migration binds full container-// IDs, and names or shortened IDs must never bind.+// Docker container references and hostless service names are excluded: the+// store migration binds full container IDs, and names or shortened IDs must+// never bind.func (s *SQLiteResourceStore) unboundHistoryReferences() ([]string, error) {rows, err := s.db.Query(`SELECT DISTINCT canonical_id FROM resource_changes- WHERE kind GLOB 'alert_*' AND canonical_id NOT GLOB 'docker:*'+ WHERE kind GLOB 'alert_*'AND canonical_id NOT IN (SELECT source_id FROM resource_history_aliases)ORDER BY canonical_id`)if err != nil {@@ -204,7 +344,9 @@ func (s *SQLiteResourceStore) unboundHistoryReferences() ([]string, error) {if err := rows.Scan(&ref); err != nil {return nil, err}- refs = append(refs, ref)+ if !isDockerNameHistoryReference(ref) {+ refs = append(refs, ref)+ }}return refs, rows.Err()}diff --git a/internal/unifiedresources/history_identity_test.go b/internal/unifiedresources/history_identity_test.goindex 935e260fb4..4744e08503 100644--- a/internal/unifiedresources/history_identity_test.go+++ b/internal/unifiedresources/history_identity_test.go@@ -333,3 +333,246 @@ func TestHistoryIdentityBindsLegacyAlertRowsFromRegistryGenerations(t *testing.Trequire.NoError(t, err)require.Equal(t, []string{"lab:pve1:999"}, refs)}++func requireHistoryReferenceBindings(t *testing.T, store *SQLiteResourceStore, bound map[string]string, unbound []string) {+ t.Helper()+ byResource := make(map[string][]string)+ for ref, canonicalID := range bound {+ byResource[canonicalID] = append(byResource[canonicalID], "fired-"+ref)+ id, found, err := store.ResolveHistorySourceIdentity(ref)+ require.NoError(t, err)+ require.True(t, found, ref)+ require.Equal(t, canonicalID, id, ref)+ }+ for canonicalID, want := range byResource {+ got, err := store.GetRecentChangesFiltered(canonicalID, time.Time{}, 50, ResourceChangeFilters{Kinds: []ChangeKind{ChangeAlertFired}})+ require.NoError(t, err)+ ids := make([]string, 0, len(got))+ for _, change := range got {+ ids = append(ids, change.ID)+ // A row journaled before its binding keeps its recorded reference.+ require.Contains(t, []string{canonicalID, strings.TrimPrefix(change.ID, "fired-")}, change.ResourceID, change.ID)+ }+ require.ElementsMatch(t, want, ids, canonicalID)+ }+ for _, ref := range unbound {+ _, found, err := store.ResolveHistorySourceIdentity(ref)+ require.NoError(t, err)+ require.False(t, found, ref)+ got, err := store.GetRecentChanges(ref, time.Time{}, 10)+ require.NoError(t, err)+ require.Len(t, got, 1, ref)+ require.Equal(t, ref, got[0].ResourceID, ref)+ }+}++// Docker host alerts reference the host's source ID ("docker:<host ID>") and+// Swarm service alerts the service ID ("docker:<host ID>/service/<ID>") from+// every manager that reports the service. Both join the canonical host and+// service history, also for a host merged into its agent's machine. Hostnames,+// service names, container names and shortened container IDs never bind, not+// even to the history of what they name, and a cluster with a service that+// lacks an ID, whose alerts carry its normalized name, binds no service.+func TestHistoryIdentityMonitorAdapterResolvesDockerHostAndServiceReferences(t *testing.T) {+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")+ require.NoError(t, err)+ t.Cleanup(func() { require.NoError(t, store.Close()) })+ now := time.Now().UTC().Truncate(time.Second)+ container := strings.Repeat("f", 64)+ serviceID := "x7k2m9q4w1e8r5t3y6u0i2o4p"+ otherServiceID := "q1w2e3r4t5y6u7i8o9p0a1s2d"+ manager := func(id, hostname, cluster string, services ...models.DockerService) models.DockerHost {+ return models.DockerHost{ID: id, Hostname: hostname, Status: "online", LastSeen: now, Services: services,+ Swarm: &models.DockerSwarmInfo{NodeID: id + "-node", NodeRole: "manager", LocalState: "active", ControlAvailable: true, ClusterID: cluster}}+ }+ web := models.DockerService{ID: serviceID, Name: "web"}+ // hostA runs on a machine whose agent also reports, so the two merge.+ hostA := manager("dh-7f3a", "tower", "swarm-1", web)+ hostA.AgentID, hostA.MachineID = "host-tower", "0123456789abcdef"+ hostA.Containers = []models.DockerContainer{{ID: container, Name: "worker", State: "running"}}+ hostB := manager("dh-91c0", "rack", "swarm-1", web)+ // In swarm-2 a service without an ID is named like another service's ID.+ hostC := manager("dh-c4d2", "edge", "swarm-2", models.DockerService{ID: otherServiceID, Name: "api"}, models.DockerService{Name: strings.ToUpper(otherServiceID)})+ // Rows journaled before this process started are bound by a later generation.+ legacy := []string{"docker:dh-91c0", "docker:dh-91c0/service/" + serviceID, "docker:dh-7f3a/worker"}+ for i, ref := range legacy {+ require.NoError(t, store.RecordChange(ResourceChange{ID: "fired-" + ref, ResourceID: ref, Kind: ChangeAlertFired, SourceType: SourceHeuristic, ObservedAt: now.Add(time.Duration(i) * time.Second)}))+ }+ adapter := NewMonitorAdapter(NewRegistry(store))+ adapter.PopulateFromSnapshot(models.StateSnapshot{+ Hosts: []models.Host{{ID: "host-tower", Hostname: "tower", MachineID: "0123456789abcdef", Status: "online", LastSeen: now}},+ DockerHosts: []models.DockerHost{hostA, hostB, hostC},+ })+ registry := adapter.currentRegistry()+ hostAID := registry.sourceResourceID(SourceDocker, "dh-7f3a")+ hostBID := registry.sourceResourceID(SourceDocker, "dh-91c0")+ serviceResourceID := historyIdentityResourceID(t, registry, ResourceTypeDockerService, "web")+ containerID := registry.sourceResourceID(SourceDocker, "dh-7f3a/container/"+container)+ require.Equal(t, registry.sourceResourceID(SourceAgent, "host-tower"), hostAID, "the Docker host merges into its agent's machine")+ require.NotEqual(t, hostAID, hostBID)+ require.NotEmpty(t, containerID)++ bound := map[string]string{+ "docker:dh-91c0": hostBID,+ "docker:dh-91c0/service/" + serviceID: serviceResourceID,+ "docker:dh-7f3a": hostAID,+ "docker:dh-7f3a/service/" + serviceID: serviceResourceID,+ }+ unbound := []string{+ "docker:tower", // hostname+ "docker:dh-7f3a/service/web", // service name+ "docker:dh-c4d2/service/" + otherServiceID, // its cluster has a service without an ID+ "docker-service:web", // hostless service name+ "docker:dh-7f3a/worker", // container name+ "docker:dh-7f3a/" + container[:12], // shortened container ID+ "docker:unknown",+ }+ for i, ref := range append([]string{"docker:dh-7f3a", "docker:dh-7f3a/service/" + serviceID}, unbound...) {+ if ref == "docker:dh-7f3a/worker" {+ continue // journaled above+ }+ require.NoError(t, adapter.RecordChange(ResourceChange{ID: "fired-" + ref, ResourceID: ref, Kind: ChangeAlertFired, SourceType: SourceHeuristic, ObservedAt: now.Add(time.Minute + time.Duration(i)*time.Second)}))+ }+ requireHistoryReferenceBindings(t, store, bound, unbound)+ // Names and shortened IDs are never retried against later inventory.+ require.Contains(t, adapter.legacyHistory.pending, "docker:tower")+ for _, ref := range []string{"docker:dh-7f3a/worker", "docker:dh-7f3a/" + container[:12], "docker-service:web"} {+ require.NotContains(t, adapter.legacyHistory.pending, ref)+ }+ got, err := store.GetRecentChangesFiltered(containerID, time.Time{}, 10, ResourceChangeFilters{Kinds: []ChangeKind{ChangeAlertFired}})+ require.NoError(t, err)+ require.Empty(t, got, "a container name or short ID never joins the container's history")+ // While its cluster has a service without an ID, a service reference is a+ // conflict and must not follow a binding recorded earlier.+ collision := "docker:dh-c4d2/service/" + otherServiceID+ apiID := historyIdentityResourceID(t, registry, ResourceTypeDockerService, "api")+ require.NoError(t, store.RecordChangeWithSourceIdentity(ResourceChange{ID: "earlier-api", ResourceID: apiID, Kind: ChangeAlertFired, SourceType: SourceHeuristic, ObservedAt: now}, collision))+ require.NoError(t, adapter.RecordChange(ResourceChange{ID: "conflict-api", ResourceID: collision, Kind: ChangeAlertFired, SourceType: SourceHeuristic, ObservedAt: now.Add(2 * time.Minute)}))+ // The same holds once the service with that ID has left inventory.+ hostC.Services = hostC.Services[1:]+ adapter.PopulateFromSnapshot(models.StateSnapshot{+ Hosts: []models.Host{{ID: "host-tower", Hostname: "tower", MachineID: "0123456789abcdef", Status: "online", LastSeen: now}},+ DockerHosts: []models.DockerHost{hostA, hostB, hostC},+ })+ require.NoError(t, adapter.RecordChange(ResourceChange{ID: "conflict-api-removed", ResourceID: collision, Kind: ChangeAlertFired, SourceType: SourceHeuristic, ObservedAt: now.Add(3 * time.Minute)}))+ for _, id := range []string{"conflict-api", "conflict-api-removed"} {+ var recordedAs string+ require.NoError(t, store.db.QueryRow(`SELECT canonical_id FROM resource_changes WHERE id = ?`, id).Scan(&recordedAs))+ require.Equal(t, collision, recordedAs, id)+ }++ // After the service leaves inventory its alerts follow the retained binding;+ // a container name still binds nothing.+ removed := NewMonitorAdapter(NewRegistry(store))+ for _, ref := range []string{"docker:dh-91c0/service/" + serviceID, "docker:dh-7f3a/worker"} {+ require.NoError(t, removed.RecordChange(ResourceChange{ID: "resolved-" + ref, ResourceID: ref, Kind: ChangeAlertResolved, SourceType: SourceHeuristic, ObservedAt: now.Add(time.Hour)}))+ }+ got, err = store.GetRecentChangesFiltered(serviceResourceID, time.Time{}, 10, ResourceChangeFilters{Kinds: []ChangeKind{ChangeAlertFired, ChangeAlertResolved}})+ require.NoError(t, err)+ require.Len(t, got, 3)+ require.Equal(t, "resolved-docker:dh-91c0/service/"+serviceID, got[0].ID)+ require.Equal(t, serviceResourceID, got[0].ResourceID)+ got, err = store.GetRecentChanges("docker:dh-7f3a/worker", time.Time{}, 10)+ require.NoError(t, err)+ require.Len(t, got, 2)+ require.Equal(t, "docker:dh-7f3a/worker", got[0].ResourceID)+}++// Sub-resource alerts (ZFS pools and devices, host filesystems, disks, RAID+// arrays, sensors, the Unraid array) name no resource of their own. They join+// the history of the owner their reference carries a durable ID for: the+// storage, the host, or the Unraid array storage. Device labels never bind a+// physical disk, and an owner of the wrong type or a name binds nothing.+func TestHistoryIdentityMonitorAdapterResolvesSubResourceReferences(t *testing.T) {+ store, err := NewSQLiteResourceStore(t.TempDir(), "default")+ require.NoError(t, err)+ t.Cleanup(func() { require.NoError(t, store.Close()) })+ now := time.Now().UTC().Truncate(time.Second)+ snapshot := proxmoxHistoryIdentitySnapshot(now)+ snapshot.Storage = []models.Storage{{ID: "lab-pve1-local-zfs", Name: "local-zfs", Node: "pve1", Instance: "lab", Type: "zfspool", Status: "available", Total: 100, Used: 10, LastSeen: now}}+ snapshot.Hosts[0].Sensors.SMART = []models.HostDiskSMART{{Device: "sda", Serial: "S3Z1NB0K", Model: "Samsung SSD", Health: "PASSED", Temperature: 34}}+ snapshot.Hosts = append(snapshot.Hosts, models.Host{ID: "host-nas", Hostname: "nas", MachineID: "fedcba9876543210", Status: "online", LastSeen: now,+ Unraid: &models.HostUnraidStorage{ArrayStarted: true, ArrayState: "STARTED", Disks: []models.HostUnraidDisk{{Name: "disk1", Device: "sdb", Role: "data", Status: "DISK_OK", Serial: "WD-1"}}}})+ adapter := NewMonitorAdapter(NewRegistry(store))+ adapter.PopulateFromSnapshot(snapshot)+ registry := adapter.currentRegistry()+ nodeID := historyIdentityResourceID(t, registry, ResourceTypeAgent, "pve1")+ storageID := registry.sourceResourceID(SourceProxmox, "lab-pve1-local-zfs")+ unraidID := registry.sourceResourceID(SourceAgent, "host-nas/storage:unraid-array")+ diskID := registry.sourceResourceID(SourceAgent, HostSMARTDiskSourceID(snapshot.Hosts[0], snapshot.Hosts[0].Sensors.SMART[0]))+ require.NotEmpty(t, storageID)+ require.NotEmpty(t, unraidID)+ require.NotEmpty(t, diskID)++ bound := map[string]string{+ "lab-pve1-local-zfs/zfs-pool:local-zfs": storageID,+ "lab-pve1-local-zfs/zfs-pool:local-zfs/device:sda2": storageID,+ "agent:host-pve1/disk:var": nodeID,+ "agent:host-pve1/disk:sda": nodeID,+ "agent:host-pve1/disk_temp:sda": nodeID,+ "agent:host-pve1/raid:md0": nodeID,+ "agent:host-pve1/custom:fan1": nodeID,+ "agent:host-nas/storage:unraid-array": unraidID,+ }+ unbound := []string{+ "lab-pve1/zfs-pool:local-zfs", // the owner part names a node, not a storage+ "local-zfs/zfs-pool:local-zfs", // a storage name, not its ID+ "agent:pve1/disk:var", // a hostname, not the host ID+ "agent:host-pve1/service:sshd", // not a sub-resource shape any producer emits+ "agent:host-pve1/disk:",+ }+ refs := append([]string{}, unbound...)+ for ref := range bound {+ refs = append(refs, ref)+ }+ for i, ref := range refs {+ require.NoError(t, adapter.RecordChange(ResourceChange{ID: "fired-" + ref, ResourceID: ref, Kind: ChangeAlertFired, SourceType: SourceHeuristic, ObservedAt: now.Add(time.Duration(i) * time.Second)}))+ }+ requireHistoryReferenceBindings(t, store, bound, unbound)+ got, err := store.GetRecentChangesFiltered(diskID, time.Time{}, 10, ResourceChangeFilters{Kinds: []ChangeKind{ChangeAlertFired}})+ require.NoError(t, err)+ require.Empty(t, got, "a kernel device label never binds a physical disk")++ // Recovery after the storage left inventory follows the retained binding.+ pool := "lab-pve1-local-zfs/zfs-pool:local-zfs"+ removed := NewMonitorAdapter(NewRegistry(store))+ require.NoError(t, removed.RecordChange(ResourceChange{ID: "resolved-" + pool, ResourceID: pool, Kind: ChangeAlertResolved, SourceType: SourceHeuristic, ObservedAt: now.Add(time.Hour)}))+ got, err = store.GetRecentChangesFiltered(storageID, time.Time{}, 10, ResourceChangeFilters{Kinds: []ChangeKind{ChangeAlertResolved}})+ require.NoError(t, err)+ require.Len(t, got, 1)+ require.Equal(t, storageID, got[0].ResourceID)++ // An owner reference that two resources answer to, or that names a resource+ // of another type, binds nothing and must not follow the retained binding.+ vmID := historyIdentityResourceID(t, registry, ResourceTypeVM, "web")+ for _, conflict := range []struct {+ name string+ source DataSource+ }{{"ambiguous", SourcePBS}, {"wrong-type", SourceProxmox}} {+ registry.mu.Lock()+ delete(registry.bySource[SourcePBS], "lab-pve1-local-zfs")+ if registry.bySource[conflict.source] == nil {+ registry.bySource[conflict.source] = make(map[string]string)+ }+ registry.bySource[conflict.source]["lab-pve1-local-zfs"] = vmID+ registry.mu.Unlock()+ id := conflict.name + "-" + pool+ require.NoError(t, adapter.RecordChange(ResourceChange{ID: id, ResourceID: pool, Kind: ChangeAlertFired, SourceType: SourceHeuristic, ObservedAt: now.Add(2 * time.Hour)}))+ var recordedAs string+ require.NoError(t, store.db.QueryRow(`SELECT canonical_id FROM resource_changes WHERE id = ?`, id).Scan(&recordedAs))+ require.Equal(t, pool, recordedAs, conflict.name)+ bound, _, err := store.ResolveHistorySourceIdentity(pool)+ require.NoError(t, err)+ require.Equal(t, storageID, bound, conflict.name)+ }+ // A conflicting reference without a binding is retried, like one found at+ // startup, and binds once a generation names its owner unambiguously.+ device := "lab-pve1-local-zfs/zfs-pool:local-zfs/device:sdc2"+ require.NoError(t, adapter.RecordChange(ResourceChange{ID: "fired-" + device, ResourceID: device, Kind: ChangeAlertFired, SourceType: SourceHeuristic, ObservedAt: now.Add(3 * time.Hour)}))+ require.Contains(t, adapter.legacyHistory.pending, device)+ adapter.PopulateFromSnapshot(snapshot)+ deviceBinding, found, err := store.ResolveHistorySourceIdentity(device)+ require.NoError(t, err)+ require.True(t, found)+ require.Equal(t, storageID, deviceBinding)+}diff --git a/internal/unifiedresources/monitor_adapter.go b/internal/unifiedresources/monitor_adapter.goindex 94795237b4..26a39e0958 100644--- a/internal/unifiedresources/monitor_adapter.go+++ b/internal/unifiedresources/monitor_adapter.go@@ -198,7 +198,8 @@ func (a *MonitorAdapter) currentRegistry() *ResourceRegistry {// RecordChange forwards canonical resource-history events into the underlying// resource store when monitoring has a durable store attached. Alert lifecycle// events carry source-native references (Proxmox node and guest IDs, agent-// IDs); they resolve here so history reads by canonical ID include them.+// IDs, Docker host and Swarm service IDs, sub-resource references); they+// resolve here so history reads by canonical ID include them.func (a *MonitorAdapter) RecordChange(change ResourceChange) error {registry := a.currentRegistry()if registry == nil || registry.store == nil {@@ -229,16 +230,19 @@ func (a *MonitorAdapter) RecordChange(change ResourceChange) error {}} else if resolvedID, claimed := registry.resolveHistoryReference(sourceRef); resolvedID != "" {change.ResourceID = resolvedID- } else if hasHistory && !claimed && !isDockerHistoryReference(sourceRef) {- // No resource answers to it: keep the binding an earlier event recorded.- id, found, err := history.ResolveHistorySourceIdentity(CanonicalResourceID(sourceRef))- if err != nil {- return err- }- if found {- change.ResourceID = id- } else {- unbound = true+ } else if hasHistory && !isDockerNameHistoryReference(sourceRef) {+ // A later generation may name the resource or settle a conflict.+ unbound = true+ if !claimed {+ // No resource answers to it: keep the binding an earlier event recorded.+ id, found, err := history.ResolveHistorySourceIdentity(CanonicalResourceID(sourceRef))+ if err != nil {+ return err+ }+ if found {+ change.ResourceID = id+ unbound = false+ }}}if hasHistory && CanonicalResourceID(sourceRef) != CanonicalResourceID(change.ResourceID) {