Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep unread local backup files scoped to their storage node
pulse-triage[bot]e8c391665b4d
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 18cac33cc4..31c9177da0 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -17,6 +17,34 @@## Purpose+### Same-named local backup disks retain independent read origins++An accepted storage listing on one PVE node cannot erase last-known files on+another node's unread local disk, even when both are named `local`. Each accepted+non-shared row keeps its storage read node as runtime-only provenance; the+public `node` remains guest attribution and may differ after migration. Hard+content failures and unread node catalogues/offline nodes retain only their+known local rows. Older rows without a read origin are retained conservatively+until an ordinary complete read replaces them. An accepted shared-storage read+still covers that shared repository; disabled/node-restricted query admission+and its cross-node success handling are unchanged.++Carried rows keep their exact dates and IDs, do not enter the observed-point+batch and cannot renew point evidence. Partial scans remain partial and cannot+reconcile durable history. Accepted independent rows still update or disappear+on a successful empty listing. A complete empty scan can still establish+absence, and ordinary successful reads restore the list. There is no public+JSON/schema, guest/backup command, credential or policy change.++`pve_local_storage_retention_test.go` drives both node orders, repeated content+and catalogue failures/offline nodes, migrated guests, accepted empty siblings,+unknown old origins, connection isolation, shared-storage controls, complete+absence and recovery through the poller/ingest/posture journey. Its optional+producer supplies the production Coverage table's desktop/narrow browser proof.+Retained rows are not a fresh read, archive-availability or installed recovery+claim, and this does not establish #3248's original stale-timestamp cause.++### Missing host-agent disk readings are not recorded as idle`ApplyHostReport` applies the canonical first-filesystem disk admission beforediff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex 7990f8a9b3..b447ef0517 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -21,6 +21,34 @@## Purpose+### Same-named local backup disks retain independent read origins++An accepted storage listing on one PVE node cannot erase last-known files on+another node's unread local disk, even when both are named `local`. Each accepted+non-shared row keeps its storage read node as runtime-only provenance; the+public `node` remains guest attribution and may differ after migration. Hard+content failures and unread node catalogues/offline nodes retain only their+known local rows. Older rows without a read origin are retained conservatively+until an ordinary complete read replaces them. An accepted shared-storage read+still covers that shared repository; disabled/node-restricted query admission+and its cross-node success handling are unchanged.++Carried rows keep their exact dates and IDs, do not enter the observed-point+batch and cannot renew point evidence. Partial scans remain partial and cannot+reconcile durable history. Accepted independent rows still update or disappear+on a successful empty listing. A complete empty scan can still establish+absence, and ordinary successful reads restore the list. There is no public+JSON/schema, guest/backup command, credential or policy change.++`pve_local_storage_retention_test.go` drives both node orders, repeated content+and catalogue failures/offline nodes, migrated guests, accepted empty siblings,+unknown old origins, connection isolation, shared-storage controls, complete+absence and recovery through the poller/ingest/posture journey. Its optional+producer supplies the production Coverage table's desktop/narrow browser proof.+Retained rows are not a fresh read, archive-availability or installed recovery+claim, and this does not establish #3248's original stale-timestamp cause.++### Protection assessment batches share the consuming read's lifetimeEvery protection-posture batch carries a cancellable transport signal owned bydiff --git a/internal/models/models.go b/internal/models/models.goindex 9352cf9adb..3c97b24295 100644--- a/internal/models/models.go+++ b/internal/models/models.go@@ -3486,6 +3486,11 @@ type BackupTask struct {// StorageBackup represents a backup file in storagetype StorageBackup struct {+ // StorageNode is the node whose non-shared disk supplied this row. Node+ // remains the guest's current location and can change after migration.+ // This is runtime carry-forward provenance, not part of the public JSON.+ // Empty means shared storage or an older row with unknown read origin.+ StorageNode string `json:"-"`ID string `json:"id"`Storage string `json:"storage"`Node string `json:"node"`diff --git a/internal/monitoring/monitor_backups.go b/internal/monitoring/monitor_backups.goindex dbe75d1863..3f23b7cdfe 100644--- a/internal/monitoring/monitor_backups.go+++ b/internal/monitoring/monitor_backups.go@@ -189,7 +189,11 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceNamehadPermissionError := false // Track if any permission errors occurred this cyclepermissionFailureCount := 0 // Number of node/storage reads rejected by authorizationstoragePreserveNeeded := map[string]struct{}{}+ confirmationPreserveNeeded := map[string]struct{}{}+ unreadNodes := map[string]struct{}{}+ failedLocalStorages := map[pveLocalBackupStorage]struct{}{}storageSuccess := map[string]struct{}{}+ sharedStorageSuccess := map[string]struct{}{}unqueryableStorages := map[string]struct{}{}readState := m.backupReadStateForInstance(instanceName)@@ -221,8 +225,9 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName// For each node, get storage and check contentfor _, node := range nodes {if nodeEffectiveStatus[node.Node] != "online" {+ unreadNodes[node.Node] = struct{}{}for _, storageName := range storageNamesForNode(readState, instanceName, node.Node) {- storagePreserveNeeded[storageName] = struct{}{}+ confirmationPreserveNeeded[storageName] = struct{}{}}continue}@@ -263,8 +268,9 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceNamemonErr := errors.NewMonitorError(errors.ErrorTypeAPI, "get_storage_for_backups", instanceName, err).WithNode(node.Node)log.Warn().Err(monErr).Str("node", node.Node).Msg("failed to get storage for backups - skipping node")for _, storageName := range storageNamesForNode(readState, instanceName, node.Node) {- storagePreserveNeeded[storageName] = struct{}{}+ confirmationPreserveNeeded[storageName] = struct{}{}}+ unreadNodes[node.Node] = struct{}{}storageQueryErrors++continue}@@ -293,6 +299,13 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceNamecontinue}+ // A shared listing can cover several nodes. A same-named local+ // disk cannot: keep its read origin separate from guest location.+ storageNode := ""+ if storage.Shared == 0 && !isInherentlySharedStorageType(storage.Type) {+ storageNode = node.Node+ }+// Get storage contentcontents, err := client.GetStorageContent(ctx, node.Node, storage.Storage)if err != nil {@@ -308,7 +321,9 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceNameStr("storage", storage.Storage).Msg("Failed to get storage content")}- if _, ok := storageSuccess[storage.Storage]; !ok {+ if storageNode != "" {+ failedLocalStorages[pveLocalBackupStorage{node: storageNode, storage: storage.Storage}] = struct{}{}+ } else {storagePreserveNeeded[storage.Storage] = struct{}{}}contentFailures++@@ -317,6 +332,9 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceNamecontentSuccess++storageSuccess[storage.Storage] = struct{}{}+ if storageNode == "" {+ sharedStorageSuccess[storage.Storage] = struct{}{}+ }delete(storagePreserveNeeded, storage.Storage)// Convert to models@@ -422,6 +440,7 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName}backup := models.StorageBackup{+ StorageNode: storageNode,ID: fmt.Sprintf("%s-%s", instanceName, content.Volid),Storage: storage.Storage,Node: backupNode,@@ -459,10 +478,27 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName}}+ // Resolve name-wide skips/shared failures after every node, regardless of+ // order. A local disk failure remains scoped to its own node below.+ for storageName := range storageSuccess {+ delete(storagePreserveNeeded, storageName)+ }+ for storageName := range storagePreserveNeeded {+ confirmationPreserveNeeded[storageName] = struct{}{}+ }+ for storageName := range storageSuccess {+ delete(confirmationPreserveNeeded, storageName)+ }+// Carry-forward serves the live inventory, not a new provider observation.// Only rows actually returned this poll may renew point-level evidence.observedBackups := allBackups- allBackups, preservedStorages := preserveFailedStorageBackups(instanceName, snapshot, storagePreserveNeeded, allBackups)+ allBackups, preservedStorages := preserveFailedStorageBackups(instanceName, snapshot, pveBackupReadFailures{+ storages: storagePreserveNeeded,+ nodes: unreadNodes,+ local: failedLocalStorages,+ sharedSuccess: sharedStorageSuccess,+ }, allBackups)if len(preservedStorages) > 0 {log.Warn().Str("instance", instanceName).@@ -471,7 +507,7 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName}pbsConfirmations, preservedConfirmationStorages := preserveFailedPBSGuestConfirmations(- m.state.PBSGuestConfirmationsForInstance(instanceName), storagePreserveNeeded, pbsConfirmations)+ m.state.PBSGuestConfirmationsForInstance(instanceName), confirmationPreserveNeeded, pbsConfirmations)if len(preservedConfirmationStorages) > 0 {log.Warn().Str("instance", instanceName).@@ -744,8 +780,49 @@ func preserveFailedPBSGuestConfirmations(previous []models.PBSGuestConfirmation,return current, storages}-func preserveFailedStorageBackups(instanceName string, snapshot models.StateSnapshot, storagesToPreserve map[string]struct{}, current []models.StorageBackup) ([]models.StorageBackup, []string) {- if len(storagesToPreserve) == 0 {+type pveLocalBackupStorage struct {+ node string+ storage string+}++// Name-wide failures cover unread/disabled targets and shared storage. Actual+// local content failures and unread nodes must survive another node's success.+// Legacy rows without provenance are retained conservatively until a complete+// read replaces them; guest location is not evidence of storage read origin.+type pveBackupReadFailures struct {+ storages map[string]struct{}+ nodes map[string]struct{}+ local map[pveLocalBackupStorage]struct{}+ sharedSuccess map[string]struct{}+}++func (f pveBackupReadFailures) preserves(backup models.StorageBackup) bool {+ if _, ok := f.storages[backup.Storage]; ok {+ return true+ }+ if backup.StorageNode != "" {+ if _, ok := f.nodes[backup.StorageNode]; ok {+ return true+ }+ _, ok := f.local[pveLocalBackupStorage{node: backup.StorageNode, storage: backup.Storage}]+ return ok+ }+ if _, ok := f.sharedSuccess[backup.Storage]; ok {+ return false+ }+ if len(f.nodes) > 0 {+ return true+ }+ for scope := range f.local {+ if scope.storage == backup.Storage {+ return true+ }+ }+ return false+}++func preserveFailedStorageBackups(instanceName string, snapshot models.StateSnapshot, failures pveBackupReadFailures, current []models.StorageBackup) ([]models.StorageBackup, []string) {+ if len(failures.storages) == 0 && len(failures.nodes) == 0 && len(failures.local) == 0 {return current, nil}@@ -759,7 +836,7 @@ func preserveFailedStorageBackups(instanceName string, snapshot models.StateSnapif backup.Instance != instanceName {continue}- if _, ok := storagesToPreserve[backup.Storage]; !ok {+ if !failures.preserves(backup) {continue}if _, duplicate := existing[backup.ID]; duplicate {diff --git a/internal/monitoring/pve_local_storage_retention_test.go b/internal/monitoring/pve_local_storage_retention_test.gonew file mode 100644index 0000000000..8487b9d156--- /dev/null+++ b/internal/monitoring/pve_local_storage_retention_test.go@@ -0,0 +1,339 @@+package monitoring++import (+ "bytes"+ "context"+ "crypto/sha256"+ "encoding/json"+ "fmt"+ "os"+ "reflect"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/models"+ "github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"+ "github.com/rcourtman/pulse-go-rewrite/internal/recovery"+ "github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"+)++// The default local directory has the same storage ID on every PVE node,+// but a successful read on one node says nothing about the other's disk.+type pveSameNameStorageClient struct{ *pveReconciliationClient }++func (c *pveSameNameStorageClient) GetStorage(ctx context.Context, node string) ([]proxmox.Storage, error) {+ rows, err := c.pveReconciliationClient.GetStorage(ctx, node)+ for i := range rows {+ rows[i].Storage = "local"+ }+ return rows, err+}++func TestPVELocalStorageFailureRetainsOtherNodeHistory(t *testing.T) {+ for _, reverse := range []bool{false, true} {+ t.Run(fmt.Sprintf("reverse=%t", reverse), func(t *testing.T) {+ f := newPVEReconciliationFixture(t)+ if reverse {+ f.nodes = []proxmox.Node{f.nodes[1], f.nodes[0]}+ }+ c := &pveSameNameStorageClient{f.client}+ f.m.pollStorageBackupsWithNodes(context.Background(), "cluster", c, f.nodes, f.online)+ f.wait(t)+ before := f.m.state.GetSnapshot().PVEBackups.StorageBackups+ if len(before) != 2 {+ t.Fatalf("healthy local controls=%d want2", len(before))+ }+ f.client.contentErr = fmt.Errorf("503 unavailable")+ f.m.pollStorageBackupsWithNodes(context.Background(), "cluster", c, f.nodes, f.online)+ f.wait(t)+ rows := f.m.state.GetSnapshot().PVEBackups.StorageBackups+ var beforeB, afterB models.StorageBackup+ for _, r := range before {+ if r.VMID == 102 {+ beforeB = r+ }+ }+ for _, r := range rows {+ if r.VMID == 102 {+ afterB = r+ }+ }+ if !reflect.DeepEqual(beforeB, afterB) {+ t.Errorf("failed node's local history disappeared after another node read the same storage name: rows=%+v", rows)+ }+ f.assertHistory(t, recovery.ProtectionHistoryPartial, operationaltrust.EvidencePermissionsUnknown)+ })+ }+}++func TestPVELocalStorageReadScopeSurvivesPartialPoll(t *testing.T) {+ for _, failure := range []string{"content", "catalog", "offline"} {+ for _, reverse := range []bool{false, true} {+ for _, emptyReadable := range []bool{false, true} {+ for _, moved := range []bool{false, true} {+ t.Run(fmt.Sprintf("%s/reverse=%t/empty=%t/moved=%t", failure, reverse, emptyReadable, moved), func(t *testing.T) {+ f := newPVEReconciliationFixture(t)+ if reverse {+ f.nodes = []proxmox.Node{f.nodes[1], f.nodes[0]}+ }+ c := &pveSameNameStorageClient{f.client}+ poll := func() {+ f.m.pollStorageBackupsWithNodes(context.Background(), "cluster", c, f.nodes, f.online)+ f.wait(t)+ }+ if moved {+ // The guest has migrated, but its archive is still read from node-b.+ vms := f.m.state.GetSnapshot().VMs+ for i := range vms {+ if vms[i].VMID == 102 {+ vms[i].Node = "node-a"+ }+ }+ f.m.state.UpdateVMs(vms)+ }+ poll()+ before := f.points(t)+ var rawB models.StorageBackup+ for _, row := range f.m.state.GetSnapshot().PVEBackups.StorageBackups {+ if row.VMID == 102 {+ rawB = row+ }+ }+ if moved && rawB.Node != "node-a" {+ t.Fatalf("fixture failed to separate guest location: %+v", rawB)+ }+ if rawB.StorageNode != "node-b" {+ t.Fatalf("local storage origin=%q want node-b", rawB.StorageNode)+ }+ f.m.state.UpdateStorage([]models.Storage{+ {ID: "local-a", Name: "local", Node: "node-a", Instance: "cluster", Content: "backup", Type: "dir"},+ {ID: "local-b", Name: "local", Node: "node-b", Instance: "cluster", Content: "backup", Type: "dir"},+ })+ f.client.at = f.client.at.Add(15 * time.Minute)+ f.client.emptyAll = emptyReadable+ switch failure {+ case "content":+ f.client.contentErr = fmt.Errorf("503 unavailable")+ case "catalog":+ f.client.nodeErr = fmt.Errorf("503 unavailable")+ case "offline":+ f.online["node-b"] = "offline"+ }+ for range 2 {+ poll()+ var gotB models.StorageBackup+ rows := f.m.state.GetSnapshot().PVEBackups.StorageBackups+ for _, row := range rows {+ if row.VMID == 102 {+ gotB = row+ }+ }+ if !reflect.DeepEqual(rawB, gotB) {+ t.Errorf("unread local backup changed or disappeared: rows=%+v", rows)+ }+ want := 2+ if emptyReadable {+ want = 1+ }+ if len(rows) != want {+ t.Errorf("live rows=%d want%d; a successful empty sibling must not retain its own absent files", len(rows), want)+ }+ for id, p := range before {+ if p.SubjectResourceID == f.ids[1] && !reflect.DeepEqual(p, f.points(t)[id]) {+ t.Error("unread point evidence changed")+ }+ }+ f.assertHistory(t, recovery.ProtectionHistoryPartial, operationaltrust.EvidencePermissionsUnknown)+ }+ f.client.contentErr = nil+ f.client.nodeErr = nil+ f.online["node-b"] = "online"+ f.client.emptyAll = true+ poll()+ if len(f.m.state.GetSnapshot().PVEBackups.StorageBackups) != 0 || len(f.points(t)) != 0 {+ t.Fatal("complete empty recovery failed to clear confirmed-absent files")+ }+ f.client.emptyAll = false+ poll()+ if len(f.m.state.GetSnapshot().PVEBackups.StorageBackups) != 2 || len(f.points(t)) != 2 {+ t.Fatal("ordinary complete read did not restore both backups")+ }+ f.assertHistory(t, recovery.ProtectionHistoryComplete, operationaltrust.EvidencePermissionsSufficient)+ })+ }+ }+ }+ }+}++func TestPVELocalStorageCarryForwardKeepsUnknownOriginsConservative(t *testing.T) {+ failures := pveBackupReadFailures{local: map[pveLocalBackupStorage]struct{}{{node: "failed", storage: "local"}: {}}}+ previous := []models.StorageBackup{+ {ID: "legacy", Instance: "cluster", Storage: "local", Node: "guest-moved"},+ {ID: "independent", Instance: "cluster", Storage: "local", Node: "guest-moved", StorageNode: "healthy"},+ {ID: "affected", Instance: "cluster", Storage: "local", Node: "guest-moved", StorageNode: "failed"},+ {ID: "other", Instance: "other-cluster", Storage: "local", StorageNode: "failed"},+ }+ got, _ := preserveFailedStorageBackups("cluster", models.StateSnapshot{PVEBackups: models.PVEBackups{StorageBackups: previous}}, failures, nil)+ if len(got) != 2 || got[0].ID != "legacy" || got[1].ID != "affected" {+ t.Fatalf("scope/legacy fallback lost isolation: %+v", got)+ }+ // With no node catalog, unknown origins cannot be treated as absent.+ failures = pveBackupReadFailures{nodes: map[string]struct{}{"failed": {}}}+ got, _ = preserveFailedStorageBackups("cluster", models.StateSnapshot{PVEBackups: models.PVEBackups{StorageBackups: previous}}, failures, nil)+ if len(got) != 2 || got[0].ID != "legacy" || got[1].ID != "affected" {+ t.Fatalf("unread-node fallback lost isolation: %+v", got)+ }+ encoded, err := json.Marshal(previous)+ if err != nil || bytes.Contains(encoded, []byte("storageNode")) || bytes.Contains(encoded, []byte("StorageNode")) {+ t.Fatalf("runtime origin leaked into existing JSON contract: %s/%v", encoded, err)+ }+}++// The normal Go suite checks the full accepted/partial/empty lifecycle.+// Exporting these same frames for browser evidence is optional.+func TestPVELocalStorageAcceptedReadLifecycle(t *testing.T) {+ f := newPVEReconciliationFixture(t)+ c := &pveSameNameStorageClient{f.client}+ frames := map[string]any{}+ for _, phase := range []string{"observed", "partial", "recovered", "confirmed-empty"} {+ switch phase {+ case "partial":+ f.client.contentErr = fmt.Errorf("503 unavailable")+ f.client.at = f.client.at.Add(15 * time.Minute)+ case "recovered":+ f.client.contentErr = nil+ case "confirmed-empty":+ f.client.emptyAll = true+ }+ f.m.pollStorageBackupsWithNodes(context.Background(), "cluster", c, f.nodes, f.online)+ f.wait(t)+ rows, points, postures := f.m.state.GetSnapshot().PVEBackups, f.points(t), f.postures(t)+ want := 2+ if phase == "confirmed-empty" {+ want = 0+ }+ if len(rows.StorageBackups) != want || len(points) != want {+ t.Fatalf("phase %s live/recorded inventory = %d/%d, want %d", phase, len(rows.StorageBackups), len(points), want)+ }+ for _, p := range postures {+ if phase == "observed" || phase == "recovered" {+ if p.State != recovery.ProtectionStateProtected {+ t.Fatalf("phase %s accepted completed backup is not protected: %+v", phase, p)+ }+ } else if p.State == recovery.ProtectionStateProtected {+ t.Fatalf("phase %s claimed protection without complete current backup evidence", phase)+ }+ }+ frames[phase] = map[string]any{"pve": map[string]any{"data": rows}, "postures": map[string]any{"data": postures, "policy": recovery.DefaultProtectionPosturePolicy.Payload()}, "points": points}+ }+ if os.Getenv("PULSE_LOCAL_STORAGE_FRAMES") != "1" {+ return+ }+ source, err := os.ReadFile("monitor_backups.go")+ if err != nil {+ t.Fatal(err)+ }+ data, err := json.Marshal(map[string]any{"runtime_sha256": fmt.Sprintf("%x", sha256.Sum256(source)), "observed_at": time.Now().UTC(), "resource_ids": f.ids, "frames": frames})+ if err != nil {+ t.Fatal(err)+ }+ t.Log("LOCAL_STORAGE_FRAMES=" + string(data))+}++// Shared storage still has one authoritative listing, unlike two local disks.+type pveSharedStorageClient struct {+ *pveSameNameStorageClient+ byFlag bool+}++func (c *pveSharedStorageClient) GetStorage(ctx context.Context, node string) ([]proxmox.Storage, error) {+ rows, err := c.pveSameNameStorageClient.GetStorage(ctx, node)+ for i := range rows {+ if c.byFlag {+ rows[i].Shared = 1+ } else {+ rows[i].Type = "nfs"+ }+ }+ return rows, err+}++func TestPVELocalStorageScopeKeepsAcceptedSharedAbsence(t *testing.T) {+ for _, byFlag := range []bool{false, true} {+ for _, reverse := range []bool{false, true} {+ t.Run(fmt.Sprintf("flag=%t/reverse=%t", byFlag, reverse), func(t *testing.T) {+ f := newPVEReconciliationFixture(t)+ if reverse {+ f.nodes = []proxmox.Node{f.nodes[1], f.nodes[0]}+ }+ c := &pveSharedStorageClient{&pveSameNameStorageClient{f.client}, byFlag}+ poll := func() {+ f.m.pollStorageBackupsWithNodes(context.Background(), "cluster", c, f.nodes, f.online)+ f.wait(t)+ }+ poll()+ for _, row := range f.m.state.GetSnapshot().PVEBackups.StorageBackups {+ if row.StorageNode != "" {+ t.Fatal("shared listing acquired a local-disk origin")+ }+ }+ f.client.emptyAll = true+ f.client.nodeErr = fmt.Errorf("503 unavailable")+ poll()+ if len(f.m.state.GetSnapshot().PVEBackups.StorageBackups) != 0 {+ t.Fatal("unread peer overrode an accepted shared-storage empty listing")+ }+ // Durable reconciliation remains conservative for a partial+ // provider scan, regardless of the live shared listing.+ if len(f.points(t)) != 2 {+ t.Fatal("partial shared read deleted durable history")+ }+ f.assertHistory(t, recovery.ProtectionHistoryPartial, operationaltrust.EvidencePermissionsUnknown)+ })+ }+ }+}++func TestPVELocalStorageUnreadNodeDoesNotRetainRemovedSiblingStorage(t *testing.T) {+ for _, offline := range []bool{false, true} {+ for _, reverse := range []bool{false, true} {+ t.Run(fmt.Sprintf("offline=%t/reverse=%t", offline, reverse), func(t *testing.T) {+ f := newPVEReconciliationFixture(t)+ if reverse {+ f.nodes = []proxmox.Node{f.nodes[1], f.nodes[0]}+ }+ c := &pveSameNameStorageClient{f.client}+ poll := func() {+ f.m.pollStorageBackupsWithNodes(context.Background(), "cluster", c, f.nodes, f.online)+ f.wait(t)+ }+ poll()+ f.m.state.UpdateStorage([]models.Storage{+ {ID: "local-a", Name: "local", Node: "node-a", Instance: "cluster", Content: "backup", Type: "dir"},+ {ID: "local-b", Name: "local", Node: "node-b", Instance: "cluster", Content: "backup", Type: "dir"},+ })+ // One catalogue confirms its local target is gone. The other+ // node is unread and cannot establish absence on its own disk.+ f.client.noStorages = true+ if offline {+ f.online["node-b"] = "offline"+ } else {+ f.client.nodeErr = fmt.Errorf("503 unavailable")+ }+ poll()+ rows := f.m.state.GetSnapshot().PVEBackups.StorageBackups+ if len(rows) != 1 || rows[0].VMID != 102 {+ t.Fatalf("unread node retained an independently removed local target: %+v", rows)+ }+ f.assertHistory(t, recovery.ProtectionHistoryPartial, operationaltrust.EvidencePermissionsUnknown)+ f.client.nodeErr = nil+ f.online["node-b"] = "online"+ poll()+ if len(f.m.state.GetSnapshot().PVEBackups.StorageBackups) != 0 || len(f.points(t)) != 0 {+ t.Fatal("complete catalogue absence did not clear the remaining history")+ }+ })+ }+ }+}diff --git a/internal/monitoring/storage_backup_preserve_test.go b/internal/monitoring/storage_backup_preserve_test.goindex 737483709c..c17733aad0 100644--- a/internal/monitoring/storage_backup_preserve_test.go+++ b/internal/monitoring/storage_backup_preserve_test.go@@ -40,7 +40,7 @@ func TestPreserveFailedStorageBackups(t *testing.T) {"nas-share": {},}- merged, storages := preserveFailedStorageBackups(instance, snapshot, toPreserve, current)+ merged, storages := preserveFailedStorageBackups(instance, snapshot, pveBackupReadFailures{storages: toPreserve}, current)if len(merged) != 2 {t.Fatalf("expected 2 backups after merge, got %d", len(merged))@@ -83,7 +83,7 @@ func TestPreserveFailedStorageBackupsSkipsDuplicates(t *testing.T) {"nas-share": {},}- merged, storages := preserveFailedStorageBackups(instance, snapshot, toPreserve, current)+ merged, storages := preserveFailedStorageBackups(instance, snapshot, pveBackupReadFailures{storages: toPreserve}, current)if len(merged) != 1 {t.Fatalf("expected duplicate backup to be ignored, got %d entries", len(merged))@@ -106,7 +106,7 @@ func TestPreserveFailedStorageBackupsEmptyPreserveMap(t *testing.T) {},}- merged, storages := preserveFailedStorageBackups(instance, snapshot, nil, current)+ merged, storages := preserveFailedStorageBackups(instance, snapshot, pveBackupReadFailures{}, current)if len(merged) != 1 {t.Fatalf("expected current unchanged with 1 backup, got %d", len(merged))@@ -116,7 +116,7 @@ func TestPreserveFailedStorageBackupsEmptyPreserveMap(t *testing.T) {}// Also test empty map (not nil)- merged2, storages2 := preserveFailedStorageBackups(instance, snapshot, map[string]struct{}{}, current)+ merged2, storages2 := preserveFailedStorageBackups(instance, snapshot, pveBackupReadFailures{storages: map[string]struct{}{}}, current)if len(merged2) != 1 {t.Fatalf("expected current unchanged with 1 backup, got %d", len(merged2))}@@ -141,7 +141,7 @@ func TestPreserveFailedStorageBackupsNoMatchingBackups(t *testing.T) {"nas-share": {}, // Storage not in snapshot}- merged, storages := preserveFailedStorageBackups(instance, snapshot, toPreserve, current)+ merged, storages := preserveFailedStorageBackups(instance, snapshot, pveBackupReadFailures{storages: toPreserve}, current)if len(merged) != 1 {t.Fatalf("expected current unchanged with 1 backup, got %d", len(merged))@@ -167,7 +167,7 @@ func TestPreserveFailedStorageBackupsWrongInstance(t *testing.T) {"nas-share": {},}- merged, storages := preserveFailedStorageBackups(instance, snapshot, toPreserve, current)+ merged, storages := preserveFailedStorageBackups(instance, snapshot, pveBackupReadFailures{storages: toPreserve}, current)if len(merged) != 1 {t.Fatalf("expected current unchanged (wrong instance skipped), got %d backups", len(merged))@@ -194,7 +194,7 @@ func TestPreserveFailedStorageBackupsStorageNotInPreserveMap(t *testing.T) {"nas-share": {}, // Only nas-share should be preserved}- merged, storages := preserveFailedStorageBackups(instance, snapshot, toPreserve, current)+ merged, storages := preserveFailedStorageBackups(instance, snapshot, pveBackupReadFailures{storages: toPreserve}, current)if len(merged) != 2 {t.Fatalf("expected 2 backups (original + nas-share), got %d", len(merged))@@ -228,7 +228,7 @@ func TestPreserveFailedStorageBackupsSortedStorageNames(t *testing.T) {"middle-storage": {},}- merged, storages := preserveFailedStorageBackups(instance, snapshot, toPreserve, current)+ merged, storages := preserveFailedStorageBackups(instance, snapshot, pveBackupReadFailures{storages: toPreserve}, current)if len(merged) != 3 {t.Fatalf("expected 3 backups, got %d", len(merged))