Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Bind guest memory samples to the active read lifecycle

pulse-triage[bot]6d7054cde247
diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
index 341147242a..f40d77c368 100644
--- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md
@@ -143,6 +143,17 @@ late replies, both invalidation routes, direct/cluster HTTP cancellation,
independent keys and ordinary new-lifecycle recovery. It is not installed
removal, native guest recovery or evidence that a cancelled guest command stopped.
+Guest memory read ownership is also internal monitor state, not agent protocol
+or enrolment. Retiring a PVE connection removes only its QGA memory samples and
+backoff and cancels its outstanding memory reads without releasing their
+transport uncertainty. Independent Pulse-agent reports, machine/token identity
+and linked-agent memory selection are unchanged. An unavailable or stopped QGA
+guest cannot carry its old successful/negative sample into a new start.
+`TestRetiredQGAMemoryCannotAlterIndependentAgentEvidence` in
+`internal/monitoring/monitor_host_agents_test.go` verifies this separation and
+complete connection-name matching; memory lifecycle HTTP controls preserve the
+consumed-command fence. This is not installed removal or native recovery proof.
+
The shared VM model's `GuestAgentEvidence` is monitor-owned source-time
coordination, not agent input or a public field. It prevents retained guest
identity from renewing fallback QGA eligibility. Agent enrollment, token and
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 500b1e0351..ce4760aa7f 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -993,6 +993,34 @@ cooldown and independent VM discovery, without shortening or replaying that
fence. These are source lifecycle/safety controls, not native thaw, restart or
every-volume/History recovery evidence.
+Guest memory fallback has its own per-instance/node/VM read owner, protected
+by the same lock as its short-lived cache. An overlapping refresh defers rather
+than dispatching a second read. Explicit metadata invalidation, known agent
+unavailability (even while display metadata is fresh), an observed non-running
+VM and PVE connection retirement cancel an outstanding memory context and
+discard both successful memory samples and negative backoff. A new start or
+re-added connection must obtain its own sample. Retirement matches the complete
+instance component, not a connection-name prefix. Ordinary backup-lock and
+transport deferral retain the original successful sample and its age.
+
+The obsolete read still owns its key until it actually returns. Its late
+success, empty result or error cannot recreate a sample/backoff or escape as a
+new observation. Cancelled callers likewise do not create a negative guest
+sample. Publication and invalidation are serialized; completion releases the
+owner with no persistent generation tombstones. None of these operations
+releases, shortens or retries the Proxmox transport's consumed-command fence.
+Independent VMs, PVE/status memory and linked Pulse-agent evidence remain usable.
+
+`guest_memory_lifecycle_test.go` covers successful/negative cache invalidation,
+late replies, overlapping reads, cancelled admission and ordinary recovery.
+`guest_memory_lifecycle_http_test.go` covers consumed-body cancellation through
+direct/cluster clients, preserved diagnostic uncertainty and independent VMs;
+both ordinary collectors publish new restart samples through canonical JSON
+and memory History. `TestRetiredQGAMemoryCannotAlterIndependentAgentEvidence`
+checks connection-scoped removal and unchanged independent agent reports and
+linked memory selection. These are synthetic source lifecycle controls, not
+native OS/disks/History recovery, backup thaw or reporter #2762 attribution.
+
`guest_metadata_reservation_test.go` exercises real HTTP enrichment with cold
and retained caches, active and expired scheduling holds, early unverified-lock
and cancelled-slot exits, independent VMs and ordinary resumption. The existing
@@ -2846,6 +2874,8 @@ cleanup so readers cannot retain orphaned runtime or alert projections.
22. `pkg/proxmox/zfs.go`
22a. `pkg/pbs/client.go`
23. `internal/monitoring/guest_memory_sources.go`
+23a. `internal/monitoring/guest_memory_agent.go`
+23b. `internal/monitoring/guest_memory_lifecycle.go`
24. `internal/monitoring/guest_memory_stability.go`
25. `internal/monitoring/monitor_polling_vm.go`
26. `internal/monitoring/monitor_pve_guest_builders.go`
diff --git a/internal/monitoring/guest_memory_agent.go b/internal/monitoring/guest_memory_agent.go
index 4196aabb7a..e07959d37e 100644
--- a/internal/monitoring/guest_memory_agent.go
+++ b/internal/monitoring/guest_memory_agent.go
@@ -86,6 +86,9 @@ func (m *Monitor) getVMAgentMemoryAvailability(ctx context.Context, client PVECl
if node == "" || vmid <= 0 {
return proxmox.LinuxMemoryAvailability{}, fmt.Errorf("invalid arguments for guest agent meminfo lookup")
}
+ if ctx.Err() != nil {
+ return proxmox.LinuxMemoryAvailability{}, proxmox.ErrGuestAgentDeferred
+ }
cacheKey := guestMemoryCacheKey(instanceName, node, vmid)
now := time.Now()
@@ -97,14 +100,20 @@ func (m *Monitor) getVMAgentMemoryAvailability(ctx context.Context, client PVECl
return proxmox.LinuxMemoryAvailability{}, fmt.Errorf("guest agent Linux meminfo unavailable for Windows")
}
- m.rrdCacheMu.RLock()
+ // Cache admission and read ownership share the invalidation lock. A miss
+ // never grants permission to overlap another read, even after cancellation.
+ m.rrdCacheMu.Lock()
+ if m.vmAgentMemInFlight[cacheKey] != nil || ctx.Err() != nil {
+ m.rrdCacheMu.Unlock()
+ return proxmox.LinuxMemoryAvailability{}, proxmox.ErrGuestAgentDeferred
+ }
if entry, ok := m.vmAgentMemCache[cacheKey]; ok {
ttl := vmAgentMemCacheTTL
if entry.negative {
ttl = m.vmAgentMemNegativeCacheTTL(instanceName, node, vmid)
}
if now.Sub(entry.fetchedAt) < ttl {
- m.rrdCacheMu.RUnlock()
+ m.rrdCacheMu.Unlock()
if entry.negative {
return proxmox.LinuxMemoryAvailability{}, fmt.Errorf("guest agent meminfo fallback unavailable")
}
@@ -118,10 +127,22 @@ func (m *Monitor) getVMAgentMemoryAvailability(ctx context.Context, client PVECl
}, nil
}
}
- m.rrdCacheMu.RUnlock()
-
requestCtx, cancel := context.WithTimeout(ctx, vmAgentMemRequestTTL)
- defer cancel()
+ fetch := &guestMemoryFetch{cancel: cancel}
+ if m.vmAgentMemInFlight == nil {
+ m.vmAgentMemInFlight = make(map[string]*guestMemoryFetch)
+ }
+ m.vmAgentMemInFlight[cacheKey] = fetch
+ m.rrdCacheMu.Unlock()
+ defer func() {
+ cancel()
+ m.rrdCacheMu.Lock()
+ delete(m.vmAgentMemInFlight, cacheKey)
+ m.rrdCacheMu.Unlock()
+ }()
+ if requestCtx.Err() != nil {
+ return proxmox.LinuxMemoryAvailability{}, proxmox.ErrGuestAgentDeferred
+ }
var info proxmox.LinuxMemoryAvailability
var err error
@@ -141,14 +162,19 @@ func (m *Monitor) getVMAgentMemoryAvailability(ctx context.Context, client PVECl
return proxmox.LinuxMemoryAvailability{}, fmt.Errorf("guest agent meminfo fallback unsupported")
}
- if errors.Is(err, proxmox.ErrGuestAgentDeferred) {
+ m.rrdCacheMu.Lock()
+ defer m.rrdCacheMu.Unlock()
+ if fetch.invalidated || requestCtx.Err() != nil || errors.Is(err, proxmox.ErrGuestAgentDeferred) {
// A coordination pause is not a failed guest observation. Preserve the
// last successful cache and its original timestamp for normal resumption.
- return proxmox.LinuxMemoryAvailability{}, err
+ // Cancellation cannot establish completion of a consumed guest command,
+ // or let its old reply/backoff repopulate a replacement lifecycle.
+ if errors.Is(err, proxmox.ErrGuestAgentDeferred) && !fetch.invalidated && requestCtx.Err() == nil {
+ return proxmox.LinuxMemoryAvailability{}, err
+ }
+ return proxmox.LinuxMemoryAvailability{}, proxmox.ErrGuestAgentDeferred
}
- m.rrdCacheMu.Lock()
- defer m.rrdCacheMu.Unlock()
if m.vmAgentMemCache == nil {
m.vmAgentMemCache = make(map[string]agentMemCacheEntry)
}
diff --git a/internal/monitoring/guest_memory_lifecycle.go b/internal/monitoring/guest_memory_lifecycle.go
new file mode 100644
index 0000000000..e94c8c10f9
--- /dev/null
+++ b/internal/monitoring/guest_memory_lifecycle.go
@@ -0,0 +1,58 @@
+package monitoring
+
+import (
+ "context"
+ "strings"
+)
+
+// Cancellation is not completion. The obsolete owner remains in the map until
+// its call returns; neither a janitor nor a new lifecycle releases its fence.
+// Guarded by rrdCacheMu, including publication in getVMAgentMemoryAvailability.
+type guestMemoryFetch struct {
+ cancel context.CancelFunc
+ invalidated bool
+}
+
+func (m *Monitor) invalidateGuestMemoryFetchLocked(key string) {
+ if fetch := m.vmAgentMemInFlight[key]; fetch != nil {
+ fetch.invalidated = true
+ fetch.cancel()
+ }
+}
+
+func (m *Monitor) clearGuestMemoryCache(instanceName, node string, vmid int) {
+ if m == nil {
+ return
+ }
+ key := guestMemoryCacheKey(instanceName, node, vmid)
+ m.rrdCacheMu.Lock()
+ m.invalidateGuestMemoryFetchLocked(key)
+ delete(m.vmAgentMemCache, key)
+ m.rrdCacheMu.Unlock()
+}
+
+// PVE node names do not contain '/'. Split the last two components, not an
+// instance prefix: a connection named "site/other" is not owned by "site".
+func guestMemoryCacheBelongsToInstance(key, instanceName string) bool {
+ end := strings.LastIndexByte(key, '/')
+ if end < 0 {
+ return false
+ }
+ end = strings.LastIndexByte(key[:end], '/')
+ return end >= 0 && key[:end] == instanceName
+}
+
+func (m *Monitor) retireGuestMemoryCache(instanceName string) {
+ m.rrdCacheMu.Lock()
+ defer m.rrdCacheMu.Unlock()
+ for key := range m.vmAgentMemInFlight {
+ if guestMemoryCacheBelongsToInstance(key, instanceName) {
+ m.invalidateGuestMemoryFetchLocked(key)
+ }
+ }
+ for key := range m.vmAgentMemCache {
+ if guestMemoryCacheBelongsToInstance(key, instanceName) {
+ delete(m.vmAgentMemCache, key)
+ }
+ }
+}
diff --git a/internal/monitoring/guest_memory_lifecycle_http_test.go b/internal/monitoring/guest_memory_lifecycle_http_test.go
new file mode 100644
index 0000000000..a85e5dea94
--- /dev/null
+++ b/internal/monitoring/guest_memory_lifecycle_http_test.go
@@ -0,0 +1,198 @@
+package monitoring
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/models"
+ "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
+)
+
+func TestGuestMemoryLifecycleHTTPCancellationPreservesSharedFence(t *testing.T) {
+ for _, kind := range []string{"client", "cluster-client"} {
+ for _, disposition := range []string{"guest", "connection", "unavailable", "stopped"} {
+ t.Run(kind+"/"+disposition, func(t *testing.T) {
+ entered, release := make(chan struct{}, 1), make(chan struct{})
+ var once sync.Once
+ unblock := func() { once.Do(func() { close(release) }) }
+ var mu sync.Mutex
+ var commands []string
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Header().Set("Content-Type", "application/json")
+ switch {
+ case strings.HasSuffix(r.URL.Path, "/nodes"):
+ fmt.Fprint(w, `{"data":[]}`)
+ case strings.HasSuffix(r.URL.Path, "/config"):
+ fmt.Fprint(w, `{"data":{}}`)
+ default:
+ mu.Lock()
+ commands = append(commands, r.URL.Path)
+ mu.Unlock()
+ if strings.Contains(r.URL.Path, "/qemu/105/agent/") {
+ // Consumed headers do not establish body/command completion.
+ w.WriteHeader(http.StatusOK)
+ w.(http.Flusher).Flush()
+ entered <- struct{}{}
+ <-release
+ } else {
+ fmt.Fprint(w, `{"data":{"content":"MemTotal: 8192 kB\nMemAvailable: 6144 kB\nMemFree: 1024 kB\n","truncated":false}}`)
+ }
+ }
+ }))
+ defer server.Close()
+ defer unblock()
+ client := windowsBootstrapClient(t, server.URL, kind)
+ m := &Monitor{}
+ done := make(chan guestMemoryReply, 1)
+ go func() {
+ info, err := m.getVMAgentMemoryAvailability(context.Background(), client, "http-memory", "node", 105)
+ done <- guestMemoryReply{info, err}
+ }()
+ select {
+ case <-entered:
+ case <-time.After(5 * time.Second):
+ t.Fatal("synthetic file-read body did not start")
+ }
+ invalidateMemoryLifecycle(m, "http-memory", disposition)
+ if got := readGuestMemoryReply(t, done); !errors.Is(got.err, proxmox.ErrGuestAgentDeferred) || got.info != (proxmox.LinuxMemoryAvailability{}) {
+ t.Fatal("cancelled HTTP read published successful/failed guest memory")
+ }
+ // A new client and the replacement monitor lifecycle must both
+ // respect the original consumed command's transport uncertainty.
+ diagnostic := windowsBootstrapClient(t, server.URL, "client")
+ if _, err := diagnostic.GetVMAgentInfo(context.Background(), "node", 105); proxmox.GuestAgentDeferredReason(err) != "agent-cooldown" {
+ t.Errorf("invalidation erased shared diagnostic uncertainty: %v", err)
+ }
+ if _, err := m.getVMAgentMemoryAvailability(context.Background(), client, "http-memory", "node", 105); !errors.Is(err, proxmox.ErrGuestAgentDeferred) {
+ t.Error("replacement memory lifecycle bypassed shared uncertainty")
+ }
+ if len(m.vmAgentMemCache) != 0 || len(m.vmAgentMemInFlight) != 0 {
+ t.Error("cancelled/deferred memory retained a negative sample or leaked ownership")
+ }
+ mu.Lock()
+ before := append([]string(nil), commands...)
+ mu.Unlock()
+ if len(before) != 1 || !strings.HasSuffix(before[0], "/file-read") {
+ t.Fatal("invalidation/cooldown issued another command")
+ }
+ if info, err := m.getVMAgentMemoryAvailability(context.Background(), client, "http-memory", "node", 106); err != nil || info.EffectiveAvailable != 6144*1024 {
+ t.Errorf("cancelled read blocked independent VM: %+v / %v", info, err)
+ }
+ mu.Lock()
+ if len(commands) != 2 || !strings.Contains(commands[1], "/qemu/106/agent/file-read") {
+ t.Error("independent VM added traffic to the cancelled VM")
+ }
+ mu.Unlock()
+ })
+ }
+ }
+}
+
+func TestGuestMemoryLifecycleRestartPublishesNewSample(t *testing.T) {
+ for _, kind := range []string{"client", "cluster-client"} {
+ for _, collector := range []string{"cluster", "node"} {
+ for _, disposition := range []string{"guest", "connection", "unavailable", "stopped"} {
+ t.Run(kind+"/"+collector+"/"+disposition, func(t *testing.T) {
+ f, server := newWindowsBootstrapFixture(t)
+ f.osName = "Debian GNU/Linux"
+ client := windowsBootstrapClient(t, server.URL, kind)
+ m := windowsBootstrapMonitor(t)
+ initial := windowsBootstrapPoll(t, m, client, collector, "bootstrap", "node", 105, 1)
+ key := guestMemoryCacheKey("bootstrap", "node", 105)
+ original := m.vmAgentMemCache[key]
+ if initial.Memory.Used != int64(2*bootstrapMiB) || initial.Memory.Observation.Source != "guest-agent-meminfo" || original.fetchedAt.IsZero() {
+ t.Fatal("initial real-client Linux memory sample missing")
+ }
+ invalidateMemoryLifecycle(m, "bootstrap", disposition)
+ f.mu.Lock()
+ f.linuxAvailable = 4 * bootstrapMiB
+ f.mu.Unlock()
+ f.poll.Store(2)
+ resumed := windowsBootstrapPoll(t, m, client, collector, "bootstrap", "node", 105, 2)
+ if resumed.ID != initial.ID || resumed.Memory.Used != int64(4*bootstrapMiB) || resumed.Memory.Usage != 50 || resumed.Memory.Observation.State != "current" || resumed.Memory.Observation.Source != "guest-agent-meminfo" || !resumed.Memory.Observation.ObservedAt.After(original.fetchedAt) {
+ t.Errorf("restart reused pre-transition memory: %+v", resumed.Memory)
+ }
+ var fileReads int
+ for _, command := range f.trace() {
+ if command == "file-read" {
+ fileReads++
+ }
+ }
+ if fileReads != 2 {
+ t.Errorf("restart file reads = %d, want two separate lifecycle samples", fileReads)
+ }
+ front := m.buildBroadcastFrontendStateFromSnapshot(models.StateSnapshot{VMs: []models.VM{resumed}}, m.mockModeFence.begin())
+ data, err := json.Marshal(front.Resources)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var served []struct{ Proxmox unifiedresources.ProxmoxData }
+ if err := json.Unmarshal(data, &served); err != nil || len(served) != 1 || served[0].Proxmox.Memory.Used != resumed.Memory.Used || served[0].Proxmox.Memory.Observation != resumed.Memory.Observation {
+ t.Errorf("served memory lost the new sample: %s / %v", data, err)
+ }
+ points := m.metricsHistory.GetGuestMetrics(initial.ID, "memory", time.Hour)
+ if len(points) != 2 || points[0].Value != 25 || points[1].Value != 50 {
+ t.Errorf("restart History retained old numeric memory: %+v", points)
+ }
+ })
+ }
+ }
+ }
+}
+
+func TestGuestMemoryLifecycleCancelledRefreshPreservesOriginalSample(t *testing.T) {
+ for _, outcome := range []string{"success", "failure", "empty"} {
+ t.Run(outcome, func(t *testing.T) {
+ key := guestMemoryCacheKey("cancel-refresh", "node", 105)
+ original := agentMemCacheEntry{info: memoryLifecycleSample(4096), fetchedAt: time.Now().Add(-2 * vmAgentMemCacheTTL)}
+ m := &Monitor{vmAgentMemCache: map[string]agentMemCacheEntry{key: original}}
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+ old := &blockedGuestMemoryClient{stubPVEClient: &stubPVEClient{}, entered: make(chan context.Context, 1), release: make(chan struct{}), info: memoryLifecycleSample(6144)}
+ if outcome != "success" {
+ old.info = proxmox.LinuxMemoryAvailability{}
+ }
+ if outcome == "failure" {
+ old.err = context.Canceled
+ }
+ var once sync.Once
+ unblock := func() { once.Do(func() { close(old.release) }) }
+ defer unblock()
+ done := make(chan guestMemoryReply, 1)
+ go func() {
+ info, err := m.getVMAgentMemoryAvailability(ctx, old, "cancel-refresh", "node", 105)
+ done <- guestMemoryReply{info, err}
+ }()
+ select {
+ case <-old.entered:
+ case <-time.After(5 * time.Second):
+ t.Fatal("cancelled refresh did not start")
+ }
+ cancel()
+ nextInfo := memoryLifecycleSample(5120)
+ next := &guestMemoryAgentTestClient{stubPVEClient: &stubPVEClient{}, memInfo: &nextInfo}
+ if _, err := m.getVMAgentMemoryAvailability(context.Background(), next, "cancel-refresh", "node", 105); !errors.Is(err, proxmox.ErrGuestAgentDeferred) || next.memCalls != 0 {
+ t.Error("parent cancellation released an unfinished memory read")
+ }
+ unblock()
+ if got := readGuestMemoryReply(t, done); !errors.Is(got.err, proxmox.ErrGuestAgentDeferred) || got.info != (proxmox.LinuxMemoryAvailability{}) {
+ t.Error("cancelled refresh returned an unsupported new observation")
+ }
+ if m.vmAgentMemCache[key] != original || len(m.vmAgentMemInFlight) != 0 {
+ t.Error("caller cancellation renewed/erased the original sample or created backoff")
+ }
+ if got, err := m.getVMAgentMemoryAvailability(context.Background(), next, "cancel-refresh", "node", 105); err != nil || got != nextInfo || next.memCalls != 1 {
+ t.Error("ordinary post-cancellation refresh did not obtain new evidence")
+ }
+ })
+ }
+}
diff --git a/internal/monitoring/guest_metadata.go b/internal/monitoring/guest_metadata.go
index d17133f990..bcdbf8bff2 100644
--- a/internal/monitoring/guest_metadata.go
+++ b/internal/monitoring/guest_metadata.go
@@ -343,6 +343,9 @@ func (m *Monitor) fetchGuestAgentMetadata(ctx context.Context, client PVEClientI
agentAvailable := client != nil && ((vmStatus != nil && vmStatus.Agent.IsAvailable()) || allowWithoutStatus)
if !agentAvailable {
+ // Display identity can remain useful, but a known-unavailable agent ends
+ // the memory observation lifecycle even while metadata is still fresh.
+ m.clearGuestMemoryCache(instanceName, nodeName, vmid)
if ok && now.Sub(cached.fetchedAt) < guestMetadataCacheEntryTTL(cached) {
return cloneStringSlice(cached.ipAddresses), cloneGuestNetworkInterfaces(cached.networkInterfaces), cached.osName, cached.osVersion, cached.agentVersion, false
}
@@ -617,6 +620,7 @@ func (m *Monitor) clearGuestMetadataCache(instanceName, nodeName string, vmid in
if m == nil {
return
}
+ m.clearGuestMemoryCache(instanceName, nodeName, vmid)
key := guestMetadataCacheKey(instanceName, nodeName, vmid)
// Keep the old owner until completion, but cancel its context and exclude all
diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.go
index 3dd13e6d1e..4c98450f7e 100644
--- a/internal/monitoring/monitor.go
+++ b/internal/monitoring/monitor.go
@@ -1387,6 +1387,7 @@ type Monitor struct {
rrdCacheMu sync.RWMutex // Protects short-lived guest memory caches.
nodeRRDMemCache map[string]rrdMemCacheEntry
vmAgentMemCache map[string]agentMemCacheEntry
+ vmAgentMemInFlight map[string]*guestMemoryFetch // Guarded by rrdCacheMu; retained until the read returns, even after invalidation.
removedDockerHosts map[string]time.Time // Track deliberately removed Docker hosts (ID -> removal time)
dockerTokenBindings map[string]string // Track token ID -> Docker host identity bindings to enforce uniqueness
dockerIdentityFlaps map[string]*identityFlapTracker // Track per-host identity flapping (cloned VMs sharing machine-id)
diff --git a/internal/monitoring/monitor_cluster_dedupe.go b/internal/monitoring/monitor_cluster_dedupe.go
index 798572ce60..cc98b554d0 100644
--- a/internal/monitoring/monitor_cluster_dedupe.go
+++ b/internal/monitoring/monitor_cluster_dedupe.go
@@ -76,6 +76,7 @@ func (m *Monitor) retirePVEInstanceRuntime(instanceName string) {
if m == nil || instanceName == "" {
return
}
+ m.retireGuestMemoryCache(instanceName)
if m.taskQueue != nil {
m.taskQueue.Remove(InstanceTypePVE, instanceName)
diff --git a/internal/monitoring/monitor_host_agents_test.go b/internal/monitoring/monitor_host_agents_test.go
index 8c5b891047..5b13f45ced 100644
--- a/internal/monitoring/monitor_host_agents_test.go
+++ b/internal/monitoring/monitor_host_agents_test.go
@@ -8456,3 +8456,28 @@ func TestGuestMetadataActivityConcurrentCleanupContract(t *testing.T) {
t.Error("concurrent cleanup lost suppression or renewed stale useful evidence")
}
}
+
+func TestRetiredQGAMemoryCannotAlterIndependentAgentEvidence(t *testing.T) {
+ const instance = "site"
+ entry := agentMemCacheEntry{info: memoryLifecycleSample(4096), fetchedAt: time.Now()}
+ host := models.Host{ID: "independent-host", LastSeen: time.Now(), Memory: models.Memory{Total: 8192, Used: 1024, Free: 7168, Usage: 12.5}}
+ m := &Monitor{state: models.NewState(), vmAgentMemCache: map[string]agentMemCacheEntry{
+ guestMemoryCacheKey(instance, "node", 105): entry,
+ guestMemoryCacheKey(instance, "other", 106): entry,
+ guestMemoryCacheKey("site/other", "node", 105): entry,
+ guestMemoryCacheKey("site-other", "node", 105): entry,
+ }}
+ m.state.UpdateHosts([]models.Host{host})
+ m.retirePVEInstanceRuntime(instance)
+ if len(m.vmAgentMemCache) != 2 || m.vmAgentMemCache[guestMemoryCacheKey("site/other", "node", 105)] != entry || m.vmAgentMemCache[guestMemoryCacheKey("site-other", "node", 105)] != entry {
+ t.Error("connection retirement retained its own QGA memory or erased a different connection")
+ }
+ if got := m.state.GetSnapshot().Hosts; !reflect.DeepEqual(got, []models.Host{host}) {
+ t.Error("QGA memory retirement changed independent Pulse-agent reports")
+ }
+ id := makeGuestID(instance, "node", 105)
+ total, used, source, deferred := m.resolveGuestStatusMemory(context.Background(), nil, instance, "guest", "node", 105, id, &proxmox.VMStatus{MaxMem: 8192, Mem: 8192}, map[string]models.Host{id: host}, 8192, "", nil)
+ if total != 8192 || used != 1024 || source != "agent" || deferred {
+ t.Errorf("independent linked-agent fallback changed after QGA retirement: %d/%d/%s/%t", total, used, source, deferred)
+ }
+}
diff --git a/internal/monitoring/monitor_pve_guest_builders.go b/internal/monitoring/monitor_pve_guest_builders.go
index 4860bc91b2..e17178b5ac 100644
--- a/internal/monitoring/monitor_pve_guest_builders.go
+++ b/internal/monitoring/monitor_pve_guest_builders.go
@@ -266,9 +266,14 @@ func (m *Monitor) buildVMFromClusterResource(
if res.Type == "qemu" && state.diskUsed == 0 && state.diskTotal > 0 && res.Status == "running" {
state.diskUsage = -1
}
- if res.Status != "running" && state.diskTotal > 0 {
- state.diskUsage = -1
- state.diskStatusReason = "vm-stopped"
+ if res.Status != "running" {
+ // A subsequent start must obtain its own guest memory observation rather
+ // than inherit a success or failure from before this stopped poll.
+ m.clearGuestMemoryCache(instanceName, res.Node, res.VMID)
+ if state.diskTotal > 0 {
+ state.diskUsage = -1
+ state.diskStatusReason = "vm-stopped"
+ }
}
// For running VMs, always try to get filesystem info from guest agent