Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Keep unread local backup files scoped to their storage node

pulse-triage[bot]e8c391665b4d
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 18cac33cc4..31c9177da0 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -17,6 +17,34 @@
## Purpose
+### Same-named local backup disks retain independent read origins
+
+An accepted storage listing on one PVE node cannot erase last-known files on
+another node's unread local disk, even when both are named `local`. Each accepted
+non-shared row keeps its storage read node as runtime-only provenance; the
+public `node` remains guest attribution and may differ after migration. Hard
+content failures and unread node catalogues/offline nodes retain only their
+known local rows. Older rows without a read origin are retained conservatively
+until an ordinary complete read replaces them. An accepted shared-storage read
+still covers that shared repository; disabled/node-restricted query admission
+and its cross-node success handling are unchanged.
+
+Carried rows keep their exact dates and IDs, do not enter the observed-point
+batch and cannot renew point evidence. Partial scans remain partial and cannot
+reconcile durable history. Accepted independent rows still update or disappear
+on a successful empty listing. A complete empty scan can still establish
+absence, and ordinary successful reads restore the list. There is no public
+JSON/schema, guest/backup command, credential or policy change.
+
+`pve_local_storage_retention_test.go` drives both node orders, repeated content
+and catalogue failures/offline nodes, migrated guests, accepted empty siblings,
+unknown old origins, connection isolation, shared-storage controls, complete
+absence and recovery through the poller/ingest/posture journey. Its optional
+producer supplies the production Coverage table's desktop/narrow browser proof.
+Retained rows are not a fresh read, archive-availability or installed recovery
+claim, and this does not establish #3248's original stale-timestamp cause.
+
+
### Missing host-agent disk readings are not recorded as idle
`ApplyHostReport` applies the canonical first-filesystem disk admission before
diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md
index 7990f8a9b3..b447ef0517 100644
--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md
+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md
@@ -21,6 +21,34 @@
## Purpose
+### Same-named local backup disks retain independent read origins
+
+An accepted storage listing on one PVE node cannot erase last-known files on
+another node's unread local disk, even when both are named `local`. Each accepted
+non-shared row keeps its storage read node as runtime-only provenance; the
+public `node` remains guest attribution and may differ after migration. Hard
+content failures and unread node catalogues/offline nodes retain only their
+known local rows. Older rows without a read origin are retained conservatively
+until an ordinary complete read replaces them. An accepted shared-storage read
+still covers that shared repository; disabled/node-restricted query admission
+and its cross-node success handling are unchanged.
+
+Carried rows keep their exact dates and IDs, do not enter the observed-point
+batch and cannot renew point evidence. Partial scans remain partial and cannot
+reconcile durable history. Accepted independent rows still update or disappear
+on a successful empty listing. A complete empty scan can still establish
+absence, and ordinary successful reads restore the list. There is no public
+JSON/schema, guest/backup command, credential or policy change.
+
+`pve_local_storage_retention_test.go` drives both node orders, repeated content
+and catalogue failures/offline nodes, migrated guests, accepted empty siblings,
+unknown old origins, connection isolation, shared-storage controls, complete
+absence and recovery through the poller/ingest/posture journey. Its optional
+producer supplies the production Coverage table's desktop/narrow browser proof.
+Retained rows are not a fresh read, archive-availability or installed recovery
+claim, and this does not establish #3248's original stale-timestamp cause.
+
+
### Protection assessment batches share the consuming read's lifetime
Every protection-posture batch carries a cancellable transport signal owned by
diff --git a/internal/models/models.go b/internal/models/models.go
index 9352cf9adb..3c97b24295 100644
--- a/internal/models/models.go
+++ b/internal/models/models.go
@@ -3486,6 +3486,11 @@ type BackupTask struct {
// StorageBackup represents a backup file in storage
type StorageBackup struct {
+ // StorageNode is the node whose non-shared disk supplied this row. Node
+ // remains the guest's current location and can change after migration.
+ // This is runtime carry-forward provenance, not part of the public JSON.
+ // Empty means shared storage or an older row with unknown read origin.
+ StorageNode string `json:"-"`
ID string `json:"id"`
Storage string `json:"storage"`
Node string `json:"node"`
diff --git a/internal/monitoring/monitor_backups.go b/internal/monitoring/monitor_backups.go
index dbe75d1863..3f23b7cdfe 100644
--- a/internal/monitoring/monitor_backups.go
+++ b/internal/monitoring/monitor_backups.go
@@ -189,7 +189,11 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
hadPermissionError := false // Track if any permission errors occurred this cycle
permissionFailureCount := 0 // Number of node/storage reads rejected by authorization
storagePreserveNeeded := map[string]struct{}{}
+ confirmationPreserveNeeded := map[string]struct{}{}
+ unreadNodes := map[string]struct{}{}
+ failedLocalStorages := map[pveLocalBackupStorage]struct{}{}
storageSuccess := map[string]struct{}{}
+ sharedStorageSuccess := map[string]struct{}{}
unqueryableStorages := map[string]struct{}{}
readState := m.backupReadStateForInstance(instanceName)
@@ -221,8 +225,9 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
// For each node, get storage and check content
for _, node := range nodes {
if nodeEffectiveStatus[node.Node] != "online" {
+ unreadNodes[node.Node] = struct{}{}
for _, storageName := range storageNamesForNode(readState, instanceName, node.Node) {
- storagePreserveNeeded[storageName] = struct{}{}
+ confirmationPreserveNeeded[storageName] = struct{}{}
}
continue
}
@@ -263,8 +268,9 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
monErr := errors.NewMonitorError(errors.ErrorTypeAPI, "get_storage_for_backups", instanceName, err).WithNode(node.Node)
log.Warn().Err(monErr).Str("node", node.Node).Msg("failed to get storage for backups - skipping node")
for _, storageName := range storageNamesForNode(readState, instanceName, node.Node) {
- storagePreserveNeeded[storageName] = struct{}{}
+ confirmationPreserveNeeded[storageName] = struct{}{}
}
+ unreadNodes[node.Node] = struct{}{}
storageQueryErrors++
continue
}
@@ -293,6 +299,13 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
continue
}
+ // A shared listing can cover several nodes. A same-named local
+ // disk cannot: keep its read origin separate from guest location.
+ storageNode := ""
+ if storage.Shared == 0 && !isInherentlySharedStorageType(storage.Type) {
+ storageNode = node.Node
+ }
+
// Get storage content
contents, err := client.GetStorageContent(ctx, node.Node, storage.Storage)
if err != nil {
@@ -308,7 +321,9 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
Str("storage", storage.Storage).
Msg("Failed to get storage content")
}
- if _, ok := storageSuccess[storage.Storage]; !ok {
+ if storageNode != "" {
+ failedLocalStorages[pveLocalBackupStorage{node: storageNode, storage: storage.Storage}] = struct{}{}
+ } else {
storagePreserveNeeded[storage.Storage] = struct{}{}
}
contentFailures++
@@ -317,6 +332,9 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
contentSuccess++
storageSuccess[storage.Storage] = struct{}{}
+ if storageNode == "" {
+ sharedStorageSuccess[storage.Storage] = struct{}{}
+ }
delete(storagePreserveNeeded, storage.Storage)
// Convert to models
@@ -422,6 +440,7 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
}
backup := models.StorageBackup{
+ StorageNode: storageNode,
ID: fmt.Sprintf("%s-%s", instanceName, content.Volid),
Storage: storage.Storage,
Node: backupNode,
@@ -459,10 +478,27 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
}
}
+ // Resolve name-wide skips/shared failures after every node, regardless of
+ // order. A local disk failure remains scoped to its own node below.
+ for storageName := range storageSuccess {
+ delete(storagePreserveNeeded, storageName)
+ }
+ for storageName := range storagePreserveNeeded {
+ confirmationPreserveNeeded[storageName] = struct{}{}
+ }
+ for storageName := range storageSuccess {
+ delete(confirmationPreserveNeeded, storageName)
+ }
+
// Carry-forward serves the live inventory, not a new provider observation.
// Only rows actually returned this poll may renew point-level evidence.
observedBackups := allBackups
- allBackups, preservedStorages := preserveFailedStorageBackups(instanceName, snapshot, storagePreserveNeeded, allBackups)
+ allBackups, preservedStorages := preserveFailedStorageBackups(instanceName, snapshot, pveBackupReadFailures{
+ storages: storagePreserveNeeded,
+ nodes: unreadNodes,
+ local: failedLocalStorages,
+ sharedSuccess: sharedStorageSuccess,
+ }, allBackups)
if len(preservedStorages) > 0 {
log.Warn().
Str("instance", instanceName).
@@ -471,7 +507,7 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
}
pbsConfirmations, preservedConfirmationStorages := preserveFailedPBSGuestConfirmations(
- m.state.PBSGuestConfirmationsForInstance(instanceName), storagePreserveNeeded, pbsConfirmations)
+ m.state.PBSGuestConfirmationsForInstance(instanceName), confirmationPreserveNeeded, pbsConfirmations)
if len(preservedConfirmationStorages) > 0 {
log.Warn().
Str("instance", instanceName).
@@ -744,8 +780,49 @@ func preserveFailedPBSGuestConfirmations(previous []models.PBSGuestConfirmation,
return current, storages
}
-func preserveFailedStorageBackups(instanceName string, snapshot models.StateSnapshot, storagesToPreserve map[string]struct{}, current []models.StorageBackup) ([]models.StorageBackup, []string) {
- if len(storagesToPreserve) == 0 {
+type pveLocalBackupStorage struct {
+ node string
+ storage string
+}
+
+// Name-wide failures cover unread/disabled targets and shared storage. Actual
+// local content failures and unread nodes must survive another node's success.
+// Legacy rows without provenance are retained conservatively until a complete
+// read replaces them; guest location is not evidence of storage read origin.
+type pveBackupReadFailures struct {
+ storages map[string]struct{}
+ nodes map[string]struct{}
+ local map[pveLocalBackupStorage]struct{}
+ sharedSuccess map[string]struct{}
+}
+
+func (f pveBackupReadFailures) preserves(backup models.StorageBackup) bool {
+ if _, ok := f.storages[backup.Storage]; ok {
+ return true
+ }
+ if backup.StorageNode != "" {
+ if _, ok := f.nodes[backup.StorageNode]; ok {
+ return true
+ }
+ _, ok := f.local[pveLocalBackupStorage{node: backup.StorageNode, storage: backup.Storage}]
+ return ok
+ }
+ if _, ok := f.sharedSuccess[backup.Storage]; ok {
+ return false
+ }
+ if len(f.nodes) > 0 {
+ return true
+ }
+ for scope := range f.local {
+ if scope.storage == backup.Storage {
+ return true
+ }
+ }
+ return false
+}
+
+func preserveFailedStorageBackups(instanceName string, snapshot models.StateSnapshot, failures pveBackupReadFailures, current []models.StorageBackup) ([]models.StorageBackup, []string) {
+ if len(failures.storages) == 0 && len(failures.nodes) == 0 && len(failures.local) == 0 {
return current, nil
}
@@ -759,7 +836,7 @@ func preserveFailedStorageBackups(instanceName string, snapshot models.StateSnap
if backup.Instance != instanceName {
continue
}
- if _, ok := storagesToPreserve[backup.Storage]; !ok {
+ if !failures.preserves(backup) {
continue
}
if _, duplicate := existing[backup.ID]; duplicate {
diff --git a/internal/monitoring/pve_local_storage_retention_test.go b/internal/monitoring/pve_local_storage_retention_test.go
new file mode 100644
index 0000000000..8487b9d156
--- /dev/null
+++ b/internal/monitoring/pve_local_storage_retention_test.go
@@ -0,0 +1,339 @@
+package monitoring
+
+import (
+ "bytes"
+ "context"
+ "crypto/sha256"
+ "encoding/json"
+ "fmt"
+ "os"
+ "reflect"
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/models"
+ "github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
+ "github.com/rcourtman/pulse-go-rewrite/internal/recovery"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
+)
+
+// The default local directory has the same storage ID on every PVE node,
+// but a successful read on one node says nothing about the other's disk.
+type pveSameNameStorageClient struct{ *pveReconciliationClient }
+
+func (c *pveSameNameStorageClient) GetStorage(ctx context.Context, node string) ([]proxmox.Storage, error) {
+ rows, err := c.pveReconciliationClient.GetStorage(ctx, node)
+ for i := range rows {
+ rows[i].Storage = "local"
+ }
+ return rows, err
+}
+
+func TestPVELocalStorageFailureRetainsOtherNodeHistory(t *testing.T) {
+ for _, reverse := range []bool{false, true} {
+ t.Run(fmt.Sprintf("reverse=%t", reverse), func(t *testing.T) {
+ f := newPVEReconciliationFixture(t)
+ if reverse {
+ f.nodes = []proxmox.Node{f.nodes[1], f.nodes[0]}
+ }
+ c := &pveSameNameStorageClient{f.client}
+ f.m.pollStorageBackupsWithNodes(context.Background(), "cluster", c, f.nodes, f.online)
+ f.wait(t)
+ before := f.m.state.GetSnapshot().PVEBackups.StorageBackups
+ if len(before) != 2 {
+ t.Fatalf("healthy local controls=%d want2", len(before))
+ }
+ f.client.contentErr = fmt.Errorf("503 unavailable")
+ f.m.pollStorageBackupsWithNodes(context.Background(), "cluster", c, f.nodes, f.online)
+ f.wait(t)
+ rows := f.m.state.GetSnapshot().PVEBackups.StorageBackups
+ var beforeB, afterB models.StorageBackup
+ for _, r := range before {
+ if r.VMID == 102 {
+ beforeB = r
+ }
+ }
+ for _, r := range rows {
+ if r.VMID == 102 {
+ afterB = r
+ }
+ }
+ if !reflect.DeepEqual(beforeB, afterB) {
+ t.Errorf("failed node's local history disappeared after another node read the same storage name: rows=%+v", rows)
+ }
+ f.assertHistory(t, recovery.ProtectionHistoryPartial, operationaltrust.EvidencePermissionsUnknown)
+ })
+ }
+}
+
+func TestPVELocalStorageReadScopeSurvivesPartialPoll(t *testing.T) {
+ for _, failure := range []string{"content", "catalog", "offline"} {
+ for _, reverse := range []bool{false, true} {
+ for _, emptyReadable := range []bool{false, true} {
+ for _, moved := range []bool{false, true} {
+ t.Run(fmt.Sprintf("%s/reverse=%t/empty=%t/moved=%t", failure, reverse, emptyReadable, moved), func(t *testing.T) {
+ f := newPVEReconciliationFixture(t)
+ if reverse {
+ f.nodes = []proxmox.Node{f.nodes[1], f.nodes[0]}
+ }
+ c := &pveSameNameStorageClient{f.client}
+ poll := func() {
+ f.m.pollStorageBackupsWithNodes(context.Background(), "cluster", c, f.nodes, f.online)
+ f.wait(t)
+ }
+ if moved {
+ // The guest has migrated, but its archive is still read from node-b.
+ vms := f.m.state.GetSnapshot().VMs
+ for i := range vms {
+ if vms[i].VMID == 102 {
+ vms[i].Node = "node-a"
+ }
+ }
+ f.m.state.UpdateVMs(vms)
+ }
+ poll()
+ before := f.points(t)
+ var rawB models.StorageBackup
+ for _, row := range f.m.state.GetSnapshot().PVEBackups.StorageBackups {
+ if row.VMID == 102 {
+ rawB = row
+ }
+ }
+ if moved && rawB.Node != "node-a" {
+ t.Fatalf("fixture failed to separate guest location: %+v", rawB)
+ }
+ if rawB.StorageNode != "node-b" {
+ t.Fatalf("local storage origin=%q want node-b", rawB.StorageNode)
+ }
+ f.m.state.UpdateStorage([]models.Storage{
+ {ID: "local-a", Name: "local", Node: "node-a", Instance: "cluster", Content: "backup", Type: "dir"},
+ {ID: "local-b", Name: "local", Node: "node-b", Instance: "cluster", Content: "backup", Type: "dir"},
+ })
+ f.client.at = f.client.at.Add(15 * time.Minute)
+ f.client.emptyAll = emptyReadable
+ switch failure {
+ case "content":
+ f.client.contentErr = fmt.Errorf("503 unavailable")
+ case "catalog":
+ f.client.nodeErr = fmt.Errorf("503 unavailable")
+ case "offline":
+ f.online["node-b"] = "offline"
+ }
+ for range 2 {
+ poll()
+ var gotB models.StorageBackup
+ rows := f.m.state.GetSnapshot().PVEBackups.StorageBackups
+ for _, row := range rows {
+ if row.VMID == 102 {
+ gotB = row
+ }
+ }
+ if !reflect.DeepEqual(rawB, gotB) {
+ t.Errorf("unread local backup changed or disappeared: rows=%+v", rows)
+ }
+ want := 2
+ if emptyReadable {
+ want = 1
+ }
+ if len(rows) != want {
+ t.Errorf("live rows=%d want%d; a successful empty sibling must not retain its own absent files", len(rows), want)
+ }
+ for id, p := range before {
+ if p.SubjectResourceID == f.ids[1] && !reflect.DeepEqual(p, f.points(t)[id]) {
+ t.Error("unread point evidence changed")
+ }
+ }
+ f.assertHistory(t, recovery.ProtectionHistoryPartial, operationaltrust.EvidencePermissionsUnknown)
+ }
+ f.client.contentErr = nil
+ f.client.nodeErr = nil
+ f.online["node-b"] = "online"
+ f.client.emptyAll = true
+ poll()
+ if len(f.m.state.GetSnapshot().PVEBackups.StorageBackups) != 0 || len(f.points(t)) != 0 {
+ t.Fatal("complete empty recovery failed to clear confirmed-absent files")
+ }
+ f.client.emptyAll = false
+ poll()
+ if len(f.m.state.GetSnapshot().PVEBackups.StorageBackups) != 2 || len(f.points(t)) != 2 {
+ t.Fatal("ordinary complete read did not restore both backups")
+ }
+ f.assertHistory(t, recovery.ProtectionHistoryComplete, operationaltrust.EvidencePermissionsSufficient)
+ })
+ }
+ }
+ }
+ }
+}
+
+func TestPVELocalStorageCarryForwardKeepsUnknownOriginsConservative(t *testing.T) {
+ failures := pveBackupReadFailures{local: map[pveLocalBackupStorage]struct{}{{node: "failed", storage: "local"}: {}}}
+ previous := []models.StorageBackup{
+ {ID: "legacy", Instance: "cluster", Storage: "local", Node: "guest-moved"},
+ {ID: "independent", Instance: "cluster", Storage: "local", Node: "guest-moved", StorageNode: "healthy"},
+ {ID: "affected", Instance: "cluster", Storage: "local", Node: "guest-moved", StorageNode: "failed"},
+ {ID: "other", Instance: "other-cluster", Storage: "local", StorageNode: "failed"},
+ }
+ got, _ := preserveFailedStorageBackups("cluster", models.StateSnapshot{PVEBackups: models.PVEBackups{StorageBackups: previous}}, failures, nil)
+ if len(got) != 2 || got[0].ID != "legacy" || got[1].ID != "affected" {
+ t.Fatalf("scope/legacy fallback lost isolation: %+v", got)
+ }
+ // With no node catalog, unknown origins cannot be treated as absent.
+ failures = pveBackupReadFailures{nodes: map[string]struct{}{"failed": {}}}
+ got, _ = preserveFailedStorageBackups("cluster", models.StateSnapshot{PVEBackups: models.PVEBackups{StorageBackups: previous}}, failures, nil)
+ if len(got) != 2 || got[0].ID != "legacy" || got[1].ID != "affected" {
+ t.Fatalf("unread-node fallback lost isolation: %+v", got)
+ }
+ encoded, err := json.Marshal(previous)
+ if err != nil || bytes.Contains(encoded, []byte("storageNode")) || bytes.Contains(encoded, []byte("StorageNode")) {
+ t.Fatalf("runtime origin leaked into existing JSON contract: %s/%v", encoded, err)
+ }
+}
+
+// The normal Go suite checks the full accepted/partial/empty lifecycle.
+// Exporting these same frames for browser evidence is optional.
+func TestPVELocalStorageAcceptedReadLifecycle(t *testing.T) {
+ f := newPVEReconciliationFixture(t)
+ c := &pveSameNameStorageClient{f.client}
+ frames := map[string]any{}
+ for _, phase := range []string{"observed", "partial", "recovered", "confirmed-empty"} {
+ switch phase {
+ case "partial":
+ f.client.contentErr = fmt.Errorf("503 unavailable")
+ f.client.at = f.client.at.Add(15 * time.Minute)
+ case "recovered":
+ f.client.contentErr = nil
+ case "confirmed-empty":
+ f.client.emptyAll = true
+ }
+ f.m.pollStorageBackupsWithNodes(context.Background(), "cluster", c, f.nodes, f.online)
+ f.wait(t)
+ rows, points, postures := f.m.state.GetSnapshot().PVEBackups, f.points(t), f.postures(t)
+ want := 2
+ if phase == "confirmed-empty" {
+ want = 0
+ }
+ if len(rows.StorageBackups) != want || len(points) != want {
+ t.Fatalf("phase %s live/recorded inventory = %d/%d, want %d", phase, len(rows.StorageBackups), len(points), want)
+ }
+ for _, p := range postures {
+ if phase == "observed" || phase == "recovered" {
+ if p.State != recovery.ProtectionStateProtected {
+ t.Fatalf("phase %s accepted completed backup is not protected: %+v", phase, p)
+ }
+ } else if p.State == recovery.ProtectionStateProtected {
+ t.Fatalf("phase %s claimed protection without complete current backup evidence", phase)
+ }
+ }
+ frames[phase] = map[string]any{"pve": map[string]any{"data": rows}, "postures": map[string]any{"data": postures, "policy": recovery.DefaultProtectionPosturePolicy.Payload()}, "points": points}
+ }
+ if os.Getenv("PULSE_LOCAL_STORAGE_FRAMES") != "1" {
+ return
+ }
+ source, err := os.ReadFile("monitor_backups.go")
+ if err != nil {
+ t.Fatal(err)
+ }
+ data, err := json.Marshal(map[string]any{"runtime_sha256": fmt.Sprintf("%x", sha256.Sum256(source)), "observed_at": time.Now().UTC(), "resource_ids": f.ids, "frames": frames})
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Log("LOCAL_STORAGE_FRAMES=" + string(data))
+}
+
+// Shared storage still has one authoritative listing, unlike two local disks.
+type pveSharedStorageClient struct {
+ *pveSameNameStorageClient
+ byFlag bool
+}
+
+func (c *pveSharedStorageClient) GetStorage(ctx context.Context, node string) ([]proxmox.Storage, error) {
+ rows, err := c.pveSameNameStorageClient.GetStorage(ctx, node)
+ for i := range rows {
+ if c.byFlag {
+ rows[i].Shared = 1
+ } else {
+ rows[i].Type = "nfs"
+ }
+ }
+ return rows, err
+}
+
+func TestPVELocalStorageScopeKeepsAcceptedSharedAbsence(t *testing.T) {
+ for _, byFlag := range []bool{false, true} {
+ for _, reverse := range []bool{false, true} {
+ t.Run(fmt.Sprintf("flag=%t/reverse=%t", byFlag, reverse), func(t *testing.T) {
+ f := newPVEReconciliationFixture(t)
+ if reverse {
+ f.nodes = []proxmox.Node{f.nodes[1], f.nodes[0]}
+ }
+ c := &pveSharedStorageClient{&pveSameNameStorageClient{f.client}, byFlag}
+ poll := func() {
+ f.m.pollStorageBackupsWithNodes(context.Background(), "cluster", c, f.nodes, f.online)
+ f.wait(t)
+ }
+ poll()
+ for _, row := range f.m.state.GetSnapshot().PVEBackups.StorageBackups {
+ if row.StorageNode != "" {
+ t.Fatal("shared listing acquired a local-disk origin")
+ }
+ }
+ f.client.emptyAll = true
+ f.client.nodeErr = fmt.Errorf("503 unavailable")
+ poll()
+ if len(f.m.state.GetSnapshot().PVEBackups.StorageBackups) != 0 {
+ t.Fatal("unread peer overrode an accepted shared-storage empty listing")
+ }
+ // Durable reconciliation remains conservative for a partial
+ // provider scan, regardless of the live shared listing.
+ if len(f.points(t)) != 2 {
+ t.Fatal("partial shared read deleted durable history")
+ }
+ f.assertHistory(t, recovery.ProtectionHistoryPartial, operationaltrust.EvidencePermissionsUnknown)
+ })
+ }
+ }
+}
+
+func TestPVELocalStorageUnreadNodeDoesNotRetainRemovedSiblingStorage(t *testing.T) {
+ for _, offline := range []bool{false, true} {
+ for _, reverse := range []bool{false, true} {
+ t.Run(fmt.Sprintf("offline=%t/reverse=%t", offline, reverse), func(t *testing.T) {
+ f := newPVEReconciliationFixture(t)
+ if reverse {
+ f.nodes = []proxmox.Node{f.nodes[1], f.nodes[0]}
+ }
+ c := &pveSameNameStorageClient{f.client}
+ poll := func() {
+ f.m.pollStorageBackupsWithNodes(context.Background(), "cluster", c, f.nodes, f.online)
+ f.wait(t)
+ }
+ poll()
+ f.m.state.UpdateStorage([]models.Storage{
+ {ID: "local-a", Name: "local", Node: "node-a", Instance: "cluster", Content: "backup", Type: "dir"},
+ {ID: "local-b", Name: "local", Node: "node-b", Instance: "cluster", Content: "backup", Type: "dir"},
+ })
+ // One catalogue confirms its local target is gone. The other
+ // node is unread and cannot establish absence on its own disk.
+ f.client.noStorages = true
+ if offline {
+ f.online["node-b"] = "offline"
+ } else {
+ f.client.nodeErr = fmt.Errorf("503 unavailable")
+ }
+ poll()
+ rows := f.m.state.GetSnapshot().PVEBackups.StorageBackups
+ if len(rows) != 1 || rows[0].VMID != 102 {
+ t.Fatalf("unread node retained an independently removed local target: %+v", rows)
+ }
+ f.assertHistory(t, recovery.ProtectionHistoryPartial, operationaltrust.EvidencePermissionsUnknown)
+ f.client.nodeErr = nil
+ f.online["node-b"] = "online"
+ poll()
+ if len(f.m.state.GetSnapshot().PVEBackups.StorageBackups) != 0 || len(f.points(t)) != 0 {
+ t.Fatal("complete catalogue absence did not clear the remaining history")
+ }
+ })
+ }
+ }
+}
diff --git a/internal/monitoring/storage_backup_preserve_test.go b/internal/monitoring/storage_backup_preserve_test.go
index 737483709c..c17733aad0 100644
--- a/internal/monitoring/storage_backup_preserve_test.go
+++ b/internal/monitoring/storage_backup_preserve_test.go
@@ -40,7 +40,7 @@ func TestPreserveFailedStorageBackups(t *testing.T) {
"nas-share": {},
}
- merged, storages := preserveFailedStorageBackups(instance, snapshot, toPreserve, current)
+ merged, storages := preserveFailedStorageBackups(instance, snapshot, pveBackupReadFailures{storages: toPreserve}, current)
if len(merged) != 2 {
t.Fatalf("expected 2 backups after merge, got %d", len(merged))
@@ -83,7 +83,7 @@ func TestPreserveFailedStorageBackupsSkipsDuplicates(t *testing.T) {
"nas-share": {},
}
- merged, storages := preserveFailedStorageBackups(instance, snapshot, toPreserve, current)
+ merged, storages := preserveFailedStorageBackups(instance, snapshot, pveBackupReadFailures{storages: toPreserve}, current)
if len(merged) != 1 {
t.Fatalf("expected duplicate backup to be ignored, got %d entries", len(merged))
@@ -106,7 +106,7 @@ func TestPreserveFailedStorageBackupsEmptyPreserveMap(t *testing.T) {
},
}
- merged, storages := preserveFailedStorageBackups(instance, snapshot, nil, current)
+ merged, storages := preserveFailedStorageBackups(instance, snapshot, pveBackupReadFailures{}, current)
if len(merged) != 1 {
t.Fatalf("expected current unchanged with 1 backup, got %d", len(merged))
@@ -116,7 +116,7 @@ func TestPreserveFailedStorageBackupsEmptyPreserveMap(t *testing.T) {
}
// Also test empty map (not nil)
- merged2, storages2 := preserveFailedStorageBackups(instance, snapshot, map[string]struct{}{}, current)
+ merged2, storages2 := preserveFailedStorageBackups(instance, snapshot, pveBackupReadFailures{storages: map[string]struct{}{}}, current)
if len(merged2) != 1 {
t.Fatalf("expected current unchanged with 1 backup, got %d", len(merged2))
}
@@ -141,7 +141,7 @@ func TestPreserveFailedStorageBackupsNoMatchingBackups(t *testing.T) {
"nas-share": {}, // Storage not in snapshot
}
- merged, storages := preserveFailedStorageBackups(instance, snapshot, toPreserve, current)
+ merged, storages := preserveFailedStorageBackups(instance, snapshot, pveBackupReadFailures{storages: toPreserve}, current)
if len(merged) != 1 {
t.Fatalf("expected current unchanged with 1 backup, got %d", len(merged))
@@ -167,7 +167,7 @@ func TestPreserveFailedStorageBackupsWrongInstance(t *testing.T) {
"nas-share": {},
}
- merged, storages := preserveFailedStorageBackups(instance, snapshot, toPreserve, current)
+ merged, storages := preserveFailedStorageBackups(instance, snapshot, pveBackupReadFailures{storages: toPreserve}, current)
if len(merged) != 1 {
t.Fatalf("expected current unchanged (wrong instance skipped), got %d backups", len(merged))
@@ -194,7 +194,7 @@ func TestPreserveFailedStorageBackupsStorageNotInPreserveMap(t *testing.T) {
"nas-share": {}, // Only nas-share should be preserved
}
- merged, storages := preserveFailedStorageBackups(instance, snapshot, toPreserve, current)
+ merged, storages := preserveFailedStorageBackups(instance, snapshot, pveBackupReadFailures{storages: toPreserve}, current)
if len(merged) != 2 {
t.Fatalf("expected 2 backups (original + nas-share), got %d", len(merged))
@@ -228,7 +228,7 @@ func TestPreserveFailedStorageBackupsSortedStorageNames(t *testing.T) {
"middle-storage": {},
}
- merged, storages := preserveFailedStorageBackups(instance, snapshot, toPreserve, current)
+ merged, storages := preserveFailedStorageBackups(instance, snapshot, pveBackupReadFailures{storages: toPreserve}, current)
if len(merged) != 3 {
t.Fatalf("expected 3 backups, got %d", len(merged))