Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Bind guest memory samples to the active read lifecycle
pulse-triage[bot]6d7054cde247
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.mdindex 341147242a..f40d77c368 100644--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md@@ -143,6 +143,17 @@ late replies, both invalidation routes, direct/cluster HTTP cancellation,independent keys and ordinary new-lifecycle recovery. It is not installedremoval, native guest recovery or evidence that a cancelled guest command stopped.+Guest memory read ownership is also internal monitor state, not agent protocol+or enrolment. Retiring a PVE connection removes only its QGA memory samples and+backoff and cancels its outstanding memory reads without releasing their+transport uncertainty. Independent Pulse-agent reports, machine/token identity+and linked-agent memory selection are unchanged. An unavailable or stopped QGA+guest cannot carry its old successful/negative sample into a new start.+`TestRetiredQGAMemoryCannotAlterIndependentAgentEvidence` in+`internal/monitoring/monitor_host_agents_test.go` verifies this separation and+complete connection-name matching; memory lifecycle HTTP controls preserve the+consumed-command fence. This is not installed removal or native recovery proof.+The shared VM model's `GuestAgentEvidence` is monitor-owned source-timecoordination, not agent input or a public field. It prevents retained guestidentity from renewing fallback QGA eligibility. Agent enrollment, token anddiff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 500b1e0351..ce4760aa7f 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -993,6 +993,34 @@ cooldown and independent VM discovery, without shortening or replaying thatfence. These are source lifecycle/safety controls, not native thaw, restart orevery-volume/History recovery evidence.+Guest memory fallback has its own per-instance/node/VM read owner, protected+by the same lock as its short-lived cache. An overlapping refresh defers rather+than dispatching a second read. Explicit metadata invalidation, known agent+unavailability (even while display metadata is fresh), an observed non-running+VM and PVE connection retirement cancel an outstanding memory context and+discard both successful memory samples and negative backoff. A new start or+re-added connection must obtain its own sample. Retirement matches the complete+instance component, not a connection-name prefix. Ordinary backup-lock and+transport deferral retain the original successful sample and its age.++The obsolete read still owns its key until it actually returns. Its late+success, empty result or error cannot recreate a sample/backoff or escape as a+new observation. Cancelled callers likewise do not create a negative guest+sample. Publication and invalidation are serialized; completion releases the+owner with no persistent generation tombstones. None of these operations+releases, shortens or retries the Proxmox transport's consumed-command fence.+Independent VMs, PVE/status memory and linked Pulse-agent evidence remain usable.++`guest_memory_lifecycle_test.go` covers successful/negative cache invalidation,+late replies, overlapping reads, cancelled admission and ordinary recovery.+`guest_memory_lifecycle_http_test.go` covers consumed-body cancellation through+direct/cluster clients, preserved diagnostic uncertainty and independent VMs;+both ordinary collectors publish new restart samples through canonical JSON+and memory History. `TestRetiredQGAMemoryCannotAlterIndependentAgentEvidence`+checks connection-scoped removal and unchanged independent agent reports and+linked memory selection. These are synthetic source lifecycle controls, not+native OS/disks/History recovery, backup thaw or reporter #2762 attribution.+`guest_metadata_reservation_test.go` exercises real HTTP enrichment with coldand retained caches, active and expired scheduling holds, early unverified-lockand cancelled-slot exits, independent VMs and ordinary resumption. The existing@@ -2846,6 +2874,8 @@ cleanup so readers cannot retain orphaned runtime or alert projections.22. `pkg/proxmox/zfs.go`22a. `pkg/pbs/client.go`23. `internal/monitoring/guest_memory_sources.go`+23a. `internal/monitoring/guest_memory_agent.go`+23b. `internal/monitoring/guest_memory_lifecycle.go`24. `internal/monitoring/guest_memory_stability.go`25. `internal/monitoring/monitor_polling_vm.go`26. `internal/monitoring/monitor_pve_guest_builders.go`diff --git a/internal/monitoring/guest_memory_agent.go b/internal/monitoring/guest_memory_agent.goindex 4196aabb7a..e07959d37e 100644--- a/internal/monitoring/guest_memory_agent.go+++ b/internal/monitoring/guest_memory_agent.go@@ -86,6 +86,9 @@ func (m *Monitor) getVMAgentMemoryAvailability(ctx context.Context, client PVEClif node == "" || vmid <= 0 {return proxmox.LinuxMemoryAvailability{}, fmt.Errorf("invalid arguments for guest agent meminfo lookup")}+ if ctx.Err() != nil {+ return proxmox.LinuxMemoryAvailability{}, proxmox.ErrGuestAgentDeferred+ }cacheKey := guestMemoryCacheKey(instanceName, node, vmid)now := time.Now()@@ -97,14 +100,20 @@ func (m *Monitor) getVMAgentMemoryAvailability(ctx context.Context, client PVEClreturn proxmox.LinuxMemoryAvailability{}, fmt.Errorf("guest agent Linux meminfo unavailable for Windows")}- m.rrdCacheMu.RLock()+ // Cache admission and read ownership share the invalidation lock. A miss+ // never grants permission to overlap another read, even after cancellation.+ m.rrdCacheMu.Lock()+ if m.vmAgentMemInFlight[cacheKey] != nil || ctx.Err() != nil {+ m.rrdCacheMu.Unlock()+ return proxmox.LinuxMemoryAvailability{}, proxmox.ErrGuestAgentDeferred+ }if entry, ok := m.vmAgentMemCache[cacheKey]; ok {ttl := vmAgentMemCacheTTLif entry.negative {ttl = m.vmAgentMemNegativeCacheTTL(instanceName, node, vmid)}if now.Sub(entry.fetchedAt) < ttl {- m.rrdCacheMu.RUnlock()+ m.rrdCacheMu.Unlock()if entry.negative {return proxmox.LinuxMemoryAvailability{}, fmt.Errorf("guest agent meminfo fallback unavailable")}@@ -118,10 +127,22 @@ func (m *Monitor) getVMAgentMemoryAvailability(ctx context.Context, client PVECl}, nil}}- m.rrdCacheMu.RUnlock()-requestCtx, cancel := context.WithTimeout(ctx, vmAgentMemRequestTTL)- defer cancel()+ fetch := &guestMemoryFetch{cancel: cancel}+ if m.vmAgentMemInFlight == nil {+ m.vmAgentMemInFlight = make(map[string]*guestMemoryFetch)+ }+ m.vmAgentMemInFlight[cacheKey] = fetch+ m.rrdCacheMu.Unlock()+ defer func() {+ cancel()+ m.rrdCacheMu.Lock()+ delete(m.vmAgentMemInFlight, cacheKey)+ m.rrdCacheMu.Unlock()+ }()+ if requestCtx.Err() != nil {+ return proxmox.LinuxMemoryAvailability{}, proxmox.ErrGuestAgentDeferred+ }var info proxmox.LinuxMemoryAvailabilityvar err error@@ -141,14 +162,19 @@ func (m *Monitor) getVMAgentMemoryAvailability(ctx context.Context, client PVEClreturn proxmox.LinuxMemoryAvailability{}, fmt.Errorf("guest agent meminfo fallback unsupported")}- if errors.Is(err, proxmox.ErrGuestAgentDeferred) {+ m.rrdCacheMu.Lock()+ defer m.rrdCacheMu.Unlock()+ if fetch.invalidated || requestCtx.Err() != nil || errors.Is(err, proxmox.ErrGuestAgentDeferred) {// A coordination pause is not a failed guest observation. Preserve the// last successful cache and its original timestamp for normal resumption.- return proxmox.LinuxMemoryAvailability{}, err+ // Cancellation cannot establish completion of a consumed guest command,+ // or let its old reply/backoff repopulate a replacement lifecycle.+ if errors.Is(err, proxmox.ErrGuestAgentDeferred) && !fetch.invalidated && requestCtx.Err() == nil {+ return proxmox.LinuxMemoryAvailability{}, err+ }+ return proxmox.LinuxMemoryAvailability{}, proxmox.ErrGuestAgentDeferred}- m.rrdCacheMu.Lock()- defer m.rrdCacheMu.Unlock()if m.vmAgentMemCache == nil {m.vmAgentMemCache = make(map[string]agentMemCacheEntry)}diff --git a/internal/monitoring/guest_memory_lifecycle.go b/internal/monitoring/guest_memory_lifecycle.gonew file mode 100644index 0000000000..e94c8c10f9--- /dev/null+++ b/internal/monitoring/guest_memory_lifecycle.go@@ -0,0 +1,58 @@+package monitoring++import (+ "context"+ "strings"+)++// Cancellation is not completion. The obsolete owner remains in the map until+// its call returns; neither a janitor nor a new lifecycle releases its fence.+// Guarded by rrdCacheMu, including publication in getVMAgentMemoryAvailability.+type guestMemoryFetch struct {+ cancel context.CancelFunc+ invalidated bool+}++func (m *Monitor) invalidateGuestMemoryFetchLocked(key string) {+ if fetch := m.vmAgentMemInFlight[key]; fetch != nil {+ fetch.invalidated = true+ fetch.cancel()+ }+}++func (m *Monitor) clearGuestMemoryCache(instanceName, node string, vmid int) {+ if m == nil {+ return+ }+ key := guestMemoryCacheKey(instanceName, node, vmid)+ m.rrdCacheMu.Lock()+ m.invalidateGuestMemoryFetchLocked(key)+ delete(m.vmAgentMemCache, key)+ m.rrdCacheMu.Unlock()+}++// PVE node names do not contain '/'. Split the last two components, not an+// instance prefix: a connection named "site/other" is not owned by "site".+func guestMemoryCacheBelongsToInstance(key, instanceName string) bool {+ end := strings.LastIndexByte(key, '/')+ if end < 0 {+ return false+ }+ end = strings.LastIndexByte(key[:end], '/')+ return end >= 0 && key[:end] == instanceName+}++func (m *Monitor) retireGuestMemoryCache(instanceName string) {+ m.rrdCacheMu.Lock()+ defer m.rrdCacheMu.Unlock()+ for key := range m.vmAgentMemInFlight {+ if guestMemoryCacheBelongsToInstance(key, instanceName) {+ m.invalidateGuestMemoryFetchLocked(key)+ }+ }+ for key := range m.vmAgentMemCache {+ if guestMemoryCacheBelongsToInstance(key, instanceName) {+ delete(m.vmAgentMemCache, key)+ }+ }+}diff --git a/internal/monitoring/guest_memory_lifecycle_http_test.go b/internal/monitoring/guest_memory_lifecycle_http_test.gonew file mode 100644index 0000000000..a85e5dea94--- /dev/null+++ b/internal/monitoring/guest_memory_lifecycle_http_test.go@@ -0,0 +1,198 @@+package monitoring++import (+ "context"+ "encoding/json"+ "errors"+ "fmt"+ "net/http"+ "net/http/httptest"+ "strings"+ "sync"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/models"+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"+ "github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"+)++func TestGuestMemoryLifecycleHTTPCancellationPreservesSharedFence(t *testing.T) {+ for _, kind := range []string{"client", "cluster-client"} {+ for _, disposition := range []string{"guest", "connection", "unavailable", "stopped"} {+ t.Run(kind+"/"+disposition, func(t *testing.T) {+ entered, release := make(chan struct{}, 1), make(chan struct{})+ var once sync.Once+ unblock := func() { once.Do(func() { close(release) }) }+ var mu sync.Mutex+ var commands []string+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.Header().Set("Content-Type", "application/json")+ switch {+ case strings.HasSuffix(r.URL.Path, "/nodes"):+ fmt.Fprint(w, `{"data":[]}`)+ case strings.HasSuffix(r.URL.Path, "/config"):+ fmt.Fprint(w, `{"data":{}}`)+ default:+ mu.Lock()+ commands = append(commands, r.URL.Path)+ mu.Unlock()+ if strings.Contains(r.URL.Path, "/qemu/105/agent/") {+ // Consumed headers do not establish body/command completion.+ w.WriteHeader(http.StatusOK)+ w.(http.Flusher).Flush()+ entered <- struct{}{}+ <-release+ } else {+ fmt.Fprint(w, `{"data":{"content":"MemTotal: 8192 kB\nMemAvailable: 6144 kB\nMemFree: 1024 kB\n","truncated":false}}`)+ }+ }+ }))+ defer server.Close()+ defer unblock()+ client := windowsBootstrapClient(t, server.URL, kind)+ m := &Monitor{}+ done := make(chan guestMemoryReply, 1)+ go func() {+ info, err := m.getVMAgentMemoryAvailability(context.Background(), client, "http-memory", "node", 105)+ done <- guestMemoryReply{info, err}+ }()+ select {+ case <-entered:+ case <-time.After(5 * time.Second):+ t.Fatal("synthetic file-read body did not start")+ }+ invalidateMemoryLifecycle(m, "http-memory", disposition)+ if got := readGuestMemoryReply(t, done); !errors.Is(got.err, proxmox.ErrGuestAgentDeferred) || got.info != (proxmox.LinuxMemoryAvailability{}) {+ t.Fatal("cancelled HTTP read published successful/failed guest memory")+ }+ // A new client and the replacement monitor lifecycle must both+ // respect the original consumed command's transport uncertainty.+ diagnostic := windowsBootstrapClient(t, server.URL, "client")+ if _, err := diagnostic.GetVMAgentInfo(context.Background(), "node", 105); proxmox.GuestAgentDeferredReason(err) != "agent-cooldown" {+ t.Errorf("invalidation erased shared diagnostic uncertainty: %v", err)+ }+ if _, err := m.getVMAgentMemoryAvailability(context.Background(), client, "http-memory", "node", 105); !errors.Is(err, proxmox.ErrGuestAgentDeferred) {+ t.Error("replacement memory lifecycle bypassed shared uncertainty")+ }+ if len(m.vmAgentMemCache) != 0 || len(m.vmAgentMemInFlight) != 0 {+ t.Error("cancelled/deferred memory retained a negative sample or leaked ownership")+ }+ mu.Lock()+ before := append([]string(nil), commands...)+ mu.Unlock()+ if len(before) != 1 || !strings.HasSuffix(before[0], "/file-read") {+ t.Fatal("invalidation/cooldown issued another command")+ }+ if info, err := m.getVMAgentMemoryAvailability(context.Background(), client, "http-memory", "node", 106); err != nil || info.EffectiveAvailable != 6144*1024 {+ t.Errorf("cancelled read blocked independent VM: %+v / %v", info, err)+ }+ mu.Lock()+ if len(commands) != 2 || !strings.Contains(commands[1], "/qemu/106/agent/file-read") {+ t.Error("independent VM added traffic to the cancelled VM")+ }+ mu.Unlock()+ })+ }+ }+}++func TestGuestMemoryLifecycleRestartPublishesNewSample(t *testing.T) {+ for _, kind := range []string{"client", "cluster-client"} {+ for _, collector := range []string{"cluster", "node"} {+ for _, disposition := range []string{"guest", "connection", "unavailable", "stopped"} {+ t.Run(kind+"/"+collector+"/"+disposition, func(t *testing.T) {+ f, server := newWindowsBootstrapFixture(t)+ f.osName = "Debian GNU/Linux"+ client := windowsBootstrapClient(t, server.URL, kind)+ m := windowsBootstrapMonitor(t)+ initial := windowsBootstrapPoll(t, m, client, collector, "bootstrap", "node", 105, 1)+ key := guestMemoryCacheKey("bootstrap", "node", 105)+ original := m.vmAgentMemCache[key]+ if initial.Memory.Used != int64(2*bootstrapMiB) || initial.Memory.Observation.Source != "guest-agent-meminfo" || original.fetchedAt.IsZero() {+ t.Fatal("initial real-client Linux memory sample missing")+ }+ invalidateMemoryLifecycle(m, "bootstrap", disposition)+ f.mu.Lock()+ f.linuxAvailable = 4 * bootstrapMiB+ f.mu.Unlock()+ f.poll.Store(2)+ resumed := windowsBootstrapPoll(t, m, client, collector, "bootstrap", "node", 105, 2)+ if resumed.ID != initial.ID || resumed.Memory.Used != int64(4*bootstrapMiB) || resumed.Memory.Usage != 50 || resumed.Memory.Observation.State != "current" || resumed.Memory.Observation.Source != "guest-agent-meminfo" || !resumed.Memory.Observation.ObservedAt.After(original.fetchedAt) {+ t.Errorf("restart reused pre-transition memory: %+v", resumed.Memory)+ }+ var fileReads int+ for _, command := range f.trace() {+ if command == "file-read" {+ fileReads+++ }+ }+ if fileReads != 2 {+ t.Errorf("restart file reads = %d, want two separate lifecycle samples", fileReads)+ }+ front := m.buildBroadcastFrontendStateFromSnapshot(models.StateSnapshot{VMs: []models.VM{resumed}}, m.mockModeFence.begin())+ data, err := json.Marshal(front.Resources)+ if err != nil {+ t.Fatal(err)+ }+ var served []struct{ Proxmox unifiedresources.ProxmoxData }+ if err := json.Unmarshal(data, &served); err != nil || len(served) != 1 || served[0].Proxmox.Memory.Used != resumed.Memory.Used || served[0].Proxmox.Memory.Observation != resumed.Memory.Observation {+ t.Errorf("served memory lost the new sample: %s / %v", data, err)+ }+ points := m.metricsHistory.GetGuestMetrics(initial.ID, "memory", time.Hour)+ if len(points) != 2 || points[0].Value != 25 || points[1].Value != 50 {+ t.Errorf("restart History retained old numeric memory: %+v", points)+ }+ })+ }+ }+ }+}++func TestGuestMemoryLifecycleCancelledRefreshPreservesOriginalSample(t *testing.T) {+ for _, outcome := range []string{"success", "failure", "empty"} {+ t.Run(outcome, func(t *testing.T) {+ key := guestMemoryCacheKey("cancel-refresh", "node", 105)+ original := agentMemCacheEntry{info: memoryLifecycleSample(4096), fetchedAt: time.Now().Add(-2 * vmAgentMemCacheTTL)}+ m := &Monitor{vmAgentMemCache: map[string]agentMemCacheEntry{key: original}}+ ctx, cancel := context.WithCancel(context.Background())+ defer cancel()+ old := &blockedGuestMemoryClient{stubPVEClient: &stubPVEClient{}, entered: make(chan context.Context, 1), release: make(chan struct{}), info: memoryLifecycleSample(6144)}+ if outcome != "success" {+ old.info = proxmox.LinuxMemoryAvailability{}+ }+ if outcome == "failure" {+ old.err = context.Canceled+ }+ var once sync.Once+ unblock := func() { once.Do(func() { close(old.release) }) }+ defer unblock()+ done := make(chan guestMemoryReply, 1)+ go func() {+ info, err := m.getVMAgentMemoryAvailability(ctx, old, "cancel-refresh", "node", 105)+ done <- guestMemoryReply{info, err}+ }()+ select {+ case <-old.entered:+ case <-time.After(5 * time.Second):+ t.Fatal("cancelled refresh did not start")+ }+ cancel()+ nextInfo := memoryLifecycleSample(5120)+ next := &guestMemoryAgentTestClient{stubPVEClient: &stubPVEClient{}, memInfo: &nextInfo}+ if _, err := m.getVMAgentMemoryAvailability(context.Background(), next, "cancel-refresh", "node", 105); !errors.Is(err, proxmox.ErrGuestAgentDeferred) || next.memCalls != 0 {+ t.Error("parent cancellation released an unfinished memory read")+ }+ unblock()+ if got := readGuestMemoryReply(t, done); !errors.Is(got.err, proxmox.ErrGuestAgentDeferred) || got.info != (proxmox.LinuxMemoryAvailability{}) {+ t.Error("cancelled refresh returned an unsupported new observation")+ }+ if m.vmAgentMemCache[key] != original || len(m.vmAgentMemInFlight) != 0 {+ t.Error("caller cancellation renewed/erased the original sample or created backoff")+ }+ if got, err := m.getVMAgentMemoryAvailability(context.Background(), next, "cancel-refresh", "node", 105); err != nil || got != nextInfo || next.memCalls != 1 {+ t.Error("ordinary post-cancellation refresh did not obtain new evidence")+ }+ })+ }+}diff --git a/internal/monitoring/guest_metadata.go b/internal/monitoring/guest_metadata.goindex d17133f990..bcdbf8bff2 100644--- a/internal/monitoring/guest_metadata.go+++ b/internal/monitoring/guest_metadata.go@@ -343,6 +343,9 @@ func (m *Monitor) fetchGuestAgentMetadata(ctx context.Context, client PVEClientIagentAvailable := client != nil && ((vmStatus != nil && vmStatus.Agent.IsAvailable()) || allowWithoutStatus)if !agentAvailable {+ // Display identity can remain useful, but a known-unavailable agent ends+ // the memory observation lifecycle even while metadata is still fresh.+ m.clearGuestMemoryCache(instanceName, nodeName, vmid)if ok && now.Sub(cached.fetchedAt) < guestMetadataCacheEntryTTL(cached) {return cloneStringSlice(cached.ipAddresses), cloneGuestNetworkInterfaces(cached.networkInterfaces), cached.osName, cached.osVersion, cached.agentVersion, false}@@ -617,6 +620,7 @@ func (m *Monitor) clearGuestMetadataCache(instanceName, nodeName string, vmid inif m == nil {return}+ m.clearGuestMemoryCache(instanceName, nodeName, vmid)key := guestMetadataCacheKey(instanceName, nodeName, vmid)// Keep the old owner until completion, but cancel its context and exclude alldiff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.goindex 3dd13e6d1e..4c98450f7e 100644--- a/internal/monitoring/monitor.go+++ b/internal/monitoring/monitor.go@@ -1387,6 +1387,7 @@ type Monitor struct {rrdCacheMu sync.RWMutex // Protects short-lived guest memory caches.nodeRRDMemCache map[string]rrdMemCacheEntryvmAgentMemCache map[string]agentMemCacheEntry+ vmAgentMemInFlight map[string]*guestMemoryFetch // Guarded by rrdCacheMu; retained until the read returns, even after invalidation.removedDockerHosts map[string]time.Time // Track deliberately removed Docker hosts (ID -> removal time)dockerTokenBindings map[string]string // Track token ID -> Docker host identity bindings to enforce uniquenessdockerIdentityFlaps map[string]*identityFlapTracker // Track per-host identity flapping (cloned VMs sharing machine-id)diff --git a/internal/monitoring/monitor_cluster_dedupe.go b/internal/monitoring/monitor_cluster_dedupe.goindex 798572ce60..cc98b554d0 100644--- a/internal/monitoring/monitor_cluster_dedupe.go+++ b/internal/monitoring/monitor_cluster_dedupe.go@@ -76,6 +76,7 @@ func (m *Monitor) retirePVEInstanceRuntime(instanceName string) {if m == nil || instanceName == "" {return}+ m.retireGuestMemoryCache(instanceName)if m.taskQueue != nil {m.taskQueue.Remove(InstanceTypePVE, instanceName)diff --git a/internal/monitoring/monitor_host_agents_test.go b/internal/monitoring/monitor_host_agents_test.goindex 8c5b891047..5b13f45ced 100644--- a/internal/monitoring/monitor_host_agents_test.go+++ b/internal/monitoring/monitor_host_agents_test.go@@ -8456,3 +8456,28 @@ func TestGuestMetadataActivityConcurrentCleanupContract(t *testing.T) {t.Error("concurrent cleanup lost suppression or renewed stale useful evidence")}}++func TestRetiredQGAMemoryCannotAlterIndependentAgentEvidence(t *testing.T) {+ const instance = "site"+ entry := agentMemCacheEntry{info: memoryLifecycleSample(4096), fetchedAt: time.Now()}+ host := models.Host{ID: "independent-host", LastSeen: time.Now(), Memory: models.Memory{Total: 8192, Used: 1024, Free: 7168, Usage: 12.5}}+ m := &Monitor{state: models.NewState(), vmAgentMemCache: map[string]agentMemCacheEntry{+ guestMemoryCacheKey(instance, "node", 105): entry,+ guestMemoryCacheKey(instance, "other", 106): entry,+ guestMemoryCacheKey("site/other", "node", 105): entry,+ guestMemoryCacheKey("site-other", "node", 105): entry,+ }}+ m.state.UpdateHosts([]models.Host{host})+ m.retirePVEInstanceRuntime(instance)+ if len(m.vmAgentMemCache) != 2 || m.vmAgentMemCache[guestMemoryCacheKey("site/other", "node", 105)] != entry || m.vmAgentMemCache[guestMemoryCacheKey("site-other", "node", 105)] != entry {+ t.Error("connection retirement retained its own QGA memory or erased a different connection")+ }+ if got := m.state.GetSnapshot().Hosts; !reflect.DeepEqual(got, []models.Host{host}) {+ t.Error("QGA memory retirement changed independent Pulse-agent reports")+ }+ id := makeGuestID(instance, "node", 105)+ total, used, source, deferred := m.resolveGuestStatusMemory(context.Background(), nil, instance, "guest", "node", 105, id, &proxmox.VMStatus{MaxMem: 8192, Mem: 8192}, map[string]models.Host{id: host}, 8192, "", nil)+ if total != 8192 || used != 1024 || source != "agent" || deferred {+ t.Errorf("independent linked-agent fallback changed after QGA retirement: %d/%d/%s/%t", total, used, source, deferred)+ }+}diff --git a/internal/monitoring/monitor_pve_guest_builders.go b/internal/monitoring/monitor_pve_guest_builders.goindex 4860bc91b2..e17178b5ac 100644--- a/internal/monitoring/monitor_pve_guest_builders.go+++ b/internal/monitoring/monitor_pve_guest_builders.go@@ -266,9 +266,14 @@ func (m *Monitor) buildVMFromClusterResource(if res.Type == "qemu" && state.diskUsed == 0 && state.diskTotal > 0 && res.Status == "running" {state.diskUsage = -1}- if res.Status != "running" && state.diskTotal > 0 {- state.diskUsage = -1- state.diskStatusReason = "vm-stopped"+ if res.Status != "running" {+ // A subsequent start must obtain its own guest memory observation rather+ // than inherit a success or failure from before this stopped poll.+ m.clearGuestMemoryCache(instanceName, res.Node, res.VMID)+ if state.diskTotal > 0 {+ state.diskUsage = -1+ state.diskStatusReason = "vm-stopped"+ }}// For running VMs, always try to get filesystem info from guest agent