Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Preserve accepted PBS listings through later collection expiry

pulse-triage[bot]623b5ce265f2
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 990ce413eb..c07a065cfe 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -17,6 +17,32 @@
## Purpose
+### Accepted PBS inventory survives its collection budget, not its owner
+
+The existing five-minute PBS backup budget bounds provider reads, not acceptance
+of earlier successful namespace/snapshot listings. Expiry classifies remaining
+scopes as unread without starting queued HTTP requests. Accepted snapshot rows
+and their actual read times publish together; failed/queued reads preserve live
+cache only, never renew durable point evidence. Incomplete enumeration remains
+partial/unavailable and cannot reconcile durable absence or assert protection.
+Explicit cancellation still withdraws the complete enumeration.
+
+A background scan captures its runtime and client at admission. Runtime/client
+replacement cancels that owner, invalidates its cadence claim, and fences local
+publication; old completion cannot release a replacement worker. Same-name client
+replacement discards old snapshot read times, requiring independent refresh.
+The core cycle's own context still does not cancel detached backup work. A spent
+budget never starts an already-expired backup-alert read. Ordinary cadence,
+five workers, retention bounds, auth, guest operations and policy are unchanged.
+
+The normal monitoring suite includes `pbs_backup_budget_test.go` through
+`TestPBSBudgetPublishesAcceptedListings`, and `pbs_backup_lifecycle_test.go`.
+They check VM/CT, datastore/namespace/queued-snapshot/live-task expiry, cold live
+state, accepted emptiness, nonrenewal and ordinary recovery, plus actual async
+admission, cancellation, runtime/client replacement and old-worker completion.
+Source-bound desktop/narrow Backups proof consumes produced partial/recovery
+frames. This is truthful collection, not original frozen-guest recovery.
+
### Docker image replacements invalidate older tag comparisons
Successful tag/platform results are cached for six hours. If an unseen local
diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md
index 0ba8567256..85b762f480 100644
--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md
+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md
@@ -21,6 +21,26 @@
## Purpose
+### PBS collection expiry preserves accepted scopes without inventing safety
+
+PBS publishes namespace and snapshot listings accepted before a later child
+budget expires. Unread datastores/namespaces and failed or queued snapshot groups
+retain only existing raw history and cache times. Late responses are unread;
+accepted empty scopes remove only their own raw inventory. Partial enumeration
+does not reconcile durable absence, renew unread evidence, or claim protection.
+Unknown live-writer visibility keeps incomplete artifacts non-successful, never
+invents a completion. Explicit cancellation or an ended runtime/client binding
+still prevents publication, even after dropping a spent child deadline.
+
+`pbs_backup_budget_test.go` and the registered protection-observation entry point
+exercise actual PBS HTTP collection through live state, mapper, durable ingest
+and posture for VM/CT, warm/cold state, every expiry phase and ordinary recovery.
+Lifecycle tests cover the background owner and same-name client replacement.
+Desktop/narrow browser proof consumes those emitted inventory/posture frames
+through existing Backups polling and disclosures; dates, partial protection and
+independent unread records must remain legible. No new API, auth, policy,
+retention, guest operation, release or installed-recovery claim is implied.
+
### Selected PBS datastore remains inspectable without hover
The existing PBS row disclosure is named by backup reader and datastore, rather
diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.go
index f5c7b7346a..52735ddb5d 100644
--- a/internal/monitoring/monitor.go
+++ b/internal/monitoring/monitor.go
@@ -1309,6 +1309,7 @@ type Monitor struct {
pveGuestPublicationMu sync.RWMutex // Orders guest/backup publication with connection/runtime replacement and retirement.
pveGuestPolls map[string]*pveGuestPoll // Guarded by m.mu; canceled cycles retain ownership until completion unless the binding ends.
pbsClients map[string]*pbs.Client
+ pbsBackupPublicationMu sync.RWMutex // Orders PBS backup publication with runtime/client replacement.
pmgClients map[string]*pmg.Client
availabilityStatuses map[string]AvailabilityProbeStatus
availabilityByLocation map[string]map[string]AvailabilityProbeStatus
@@ -1380,7 +1381,7 @@ type Monitor struct {
backupAlertEvalMu sync.Mutex // Coalesce global rollup reads and order failure/recovery publication.
persistence *config.ConfigPersistence // Add persistence for saving updated configs
instanceTokens APITokenInventory // Instance inventory holding agent tokens bound to this tenant org; see APITokenInventories
- pbsBackupPollers map[string]bool // Track PBS backup polling goroutines per instance
+ pbsBackupPollers map[string]*pbsBackupPoll // Runtime/client-owned PBS backup scans; guarded by mu
pbsBackupCacheTime map[string]map[pbsBackupGroupKey]time.Time // Track when each PBS backup group was last fetched
runtimePollingMu sync.RWMutex // Guards runtimePollingOverride; polling goroutines read it every cycle
runtimePollingOverride runtimePollingOverrides // Runtime polling-cadence overrides pushed by the settings API (#1619)
@@ -1511,6 +1512,8 @@ type Monitor struct {
func (m *Monitor) setRuntimeContext(ctx context.Context, hub *websocket.Hub) {
m.pveGuestPublicationMu.Lock()
defer m.pveGuestPublicationMu.Unlock()
+ m.pbsBackupPublicationMu.Lock()
+ defer m.pbsBackupPublicationMu.Unlock()
m.mu.Lock()
defer m.mu.Unlock()
for name := range m.pveGuestPolls {
@@ -1525,6 +1528,9 @@ func (m *Monitor) setRuntimeContext(ctx context.Context, hub *websocket.Hub) {
for name := range m.pveBackupPolls {
m.cancelPVEBackupPollLocked(name)
}
+ for name := range m.pbsBackupPollers {
+ m.cancelPBSBackupPollLocked(name)
+ }
m.runtimeCtx = ctx
m.wsHub = hub
}
@@ -2015,7 +2021,7 @@ func New(cfg *config.Config) (*Monitor, error) {
pveBackupTemplateSubjects: make(map[string]map[string]struct{}),
backupPermissionWarnings: make(map[string]string),
persistence: config.NewConfigPersistence(cfg.DataPath),
- pbsBackupPollers: make(map[string]bool),
+ pbsBackupPollers: make(map[string]*pbsBackupPoll),
pbsBackupCacheTime: make(map[string]map[pbsBackupGroupKey]time.Time),
nodeSnapshots: make(map[string]NodeMemorySnapshot),
guestSnapshots: make(map[string]GuestMemorySnapshot),
diff --git a/internal/monitoring/monitor_backups.go b/internal/monitoring/monitor_backups.go
index b11d70479d..72fa946c67 100644
--- a/internal/monitoring/monitor_backups.go
+++ b/internal/monitoring/monitor_backups.go
@@ -1918,13 +1918,6 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien
// Process each datastore
for _, ds := range datastores {
- if ctx.Err() != nil {
- log.Warn().
- Str("instance", instanceName).
- Msg("PBS backup polling cancelled before completion")
- return
- }
-
namespacePaths := namespacePathsForDatastore(ds)
log.Info().
@@ -1952,16 +1945,23 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien
groupsRequested := 0
for _, namespace := range namespacePaths {
+ namespaceKey := pbsBackupNamespaceKey{datastore: ds.Name, namespace: namespace}
if ctx.Err() != nil {
- log.Warn().
- Str("instance", instanceName).
- Msg("PBS backup polling cancelled mid-datastore")
- return
+ // Classify every unread namespace without starting an expired
+ // request. Earlier accepted namespaces remain independently useful.
+ if ctx.Err() != context.DeadlineExceeded {
+ return
+ }
+ datastoreNamespaceErrors++
+ transientNamespaces[namespaceKey] = struct{}{}
+ continue
}
-
groups, err := client.ListBackupGroups(ctx, ds.Name, namespace)
- namespaceKey := pbsBackupNamespaceKey{datastore: ds.Name, namespace: namespace}
queriedNamespaces[namespaceKey] = struct{}{}
+ if ctx.Err() != nil {
+ // A late nil-error response is not an accepted empty listing.
+ err = ctx.Err()
+ }
if err != nil {
datastoreNamespaceErrors++
if !shouldReuseCachedPBSBackups(err) {
@@ -2071,8 +2071,7 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien
// Stage successful read times until their matching rows are published.
// A later cancellation must not freshen the previously published rows
// when this poll's refreshed payload never reaches state.
- fetchedAt := time.Now()
- for _, key := range fetched.refreshed {
+ for key, fetchedAt := range fetched.refreshed {
observedGroups[key] = struct{}{}
refreshedGroups[key] = fetchedAt
}
@@ -2115,8 +2114,8 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien
}
}
- if ctx.Err() != nil {
- // An interrupted enumeration cannot authoritatively replace history.
+ if ctx.Err() != nil && ctx.Err() != context.DeadlineExceeded {
+ // Explicit cancellation still withdraws the enumeration.
return
}
@@ -2145,6 +2144,57 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien
Int("count", len(allBackups)).
Msg("PBS backups fetched")
+ // A manifestless snapshot is incomplete, but it is not necessarily still
+ // being written: failed backup and PBS-to-PBS sync tasks can leave that
+ // shape behind. Correlate incomplete artifacts with PBS's live task view
+ // before they feed guest "Backup Running" state. If task visibility is
+ // denied or transiently unavailable, keep the previous conservative
+ // inference rather than claiming that no write is active.
+ if hasIncompletePBSBackup(allBackups) {
+ // Write-activity evidence belongs to this poll, not to the snapshot
+ // cache. A failed snapshot refresh may have reused an incomplete
+ // artifact whose previous poll authoritatively found no writer. Clear
+ // that old observation before asking for the current task view so a
+ // task-query failure leaves visibility unknown instead of publishing a
+ // stale terminal-failure conclusion.
+ clearPBSWriteActivityObservation(allBackups)
+ var runningDataTasks []pbs.RunningDataTask
+ runningTasksErr := ctx.Err()
+ if runningTasksErr == nil {
+ runningDataTasks, runningTasksErr = client.ListRunningDataTasks(ctx)
+ if ctx.Err() != nil {
+ runningTasksErr = ctx.Err()
+ }
+ }
+ if runningTasksErr != nil {
+ log.Warn().Err(runningTasksErr).
+ Str("instance", instanceName).
+ Msg("Could not correlate incomplete PBS snapshots with running tasks")
+ } else {
+ applyPBSWriteActivity(allBackups, runningDataTasks)
+ }
+ }
+
+ if ctx.Err() != nil && ctx.Err() != context.DeadlineExceeded {
+ return
+ }
+ if ctx.Err() == context.DeadlineExceeded && datastoreErrors == 0 {
+ // The final live-writer read can spend the remaining budget too.
+ // Its unknown result cannot turn a partial collection into protection.
+ datastoreErrors = 1
+ }
+
+ publicationCtx := ctx
+ if ctx.Err() == context.DeadlineExceeded {
+ // Drop only the child budget, not its runtime/client owner value.
+ publicationCtx = context.WithoutCancel(ctx)
+ }
+ unlock, current := m.beginPBSBackupPublication(publicationCtx, instanceName)
+ if !current {
+ return
+ }
+ defer unlock()
+
// With no accepted group listing, a mixture of rejected and transiently
// unread namespaces does not establish that every datastore was denied.
// Partial successful collection still reports any actual access rejection.
@@ -2186,36 +2236,6 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien
return
}
- // A manifestless snapshot is incomplete, but it is not necessarily still
- // being written: failed backup and PBS-to-PBS sync tasks can leave that
- // shape behind. Correlate incomplete artifacts with PBS's live task view
- // before they feed guest "Backup Running" state. If task visibility is
- // denied or transiently unavailable, keep the previous conservative
- // inference rather than claiming that no write is active.
- if hasIncompletePBSBackup(allBackups) {
- // Write-activity evidence belongs to this poll, not to the snapshot
- // cache. A failed snapshot refresh may have reused an incomplete
- // artifact whose previous poll authoritatively found no writer. Clear
- // that old observation before asking for the current task view so a
- // task-query failure leaves visibility unknown instead of publishing a
- // stale terminal-failure conclusion.
- clearPBSWriteActivityObservation(allBackups)
- runningDataTasks, runningTasksErr := client.ListRunningDataTasks(ctx)
- if runningTasksErr != nil {
- log.Warn().Err(runningTasksErr).
- Str("instance", instanceName).
- Msg("Could not correlate incomplete PBS snapshots with running tasks")
- } else {
- applyPBSWriteActivity(allBackups, runningDataTasks)
- }
- }
-
- if ctx.Err() != nil {
- // Running-task correlation is the last HTTP read and can also be
- // interrupted. Do not publish its incomplete payload or renew cache.
- return
- }
-
// Publication is the commit boundary: no cancellation return may split a
// successful read time from its matching payload. Per-instance poller
// admission serializes this publication with the next ordinary poll.
@@ -2280,7 +2300,11 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien
// Sync backup times to VMs/Containers and republish them to canonical resources.
m.syncGuestBackupTimesAndResourceStore()
- m.checkBackupAlerts(context.Background())
+ if ctx.Err() == nil {
+ // Collection expiry is not a failed alert-store read or permission
+ // to clear an existing wanted alert. The next ordinary check remains.
+ m.checkBackupAlerts(ctx)
+ }
// Immediately broadcast the updated state so frontend sees new backups
m.broadcastStateUpdate()
@@ -2450,7 +2474,7 @@ type pbsBackupSnapshotFetchResult struct {
backups []models.PBSBackup
errors int
terminalErrors int
- refreshed []pbsBackupGroupKey
+ refreshed map[pbsBackupGroupKey]time.Time
}
func (m *Monitor) fetchPBSBackupSnapshots(ctx context.Context, client *pbs.Client, instanceName string, requests []pbsBackupFetchRequest) pbsBackupSnapshotFetchResult {
@@ -2481,7 +2505,15 @@ func (m *Monitor) fetchPBSBackupSnapshots(ctx context.Context, client *pbs.Clien
Str("id", req.group.BackupID).
Msg("Refreshing PBS backup group")
- snapshots, err := client.ListBackupSnapshots(ctx, req.datastore, req.namespace, req.group.BackupType, req.group.BackupID)
+ // Drain queued work as unread after expiry, without provider I/O.
+ var snapshots []pbs.BackupSnapshot
+ err := ctx.Err()
+ if err == nil {
+ snapshots, err = client.ListBackupSnapshots(ctx, req.datastore, req.namespace, req.group.BackupType, req.group.BackupID)
+ if ctx.Err() != nil {
+ err = ctx.Err()
+ }
+ }
if err != nil {
log.Error().
Err(err).
@@ -2503,17 +2535,14 @@ func (m *Monitor) fetchPBSBackupSnapshots(ctx context.Context, client *pbs.Clien
continue
}
+ fetchedAt := time.Now()
backups := convertPBSSnapshots(instanceName, req.datastore, req.namespace, snapshots)
- select {
- case results <- pbsBackupSnapshotFetchResult{
+ results <- pbsBackupSnapshotFetchResult{
backups: backups,
- refreshed: []pbsBackupGroupKey{{
+ refreshed: map[pbsBackupGroupKey]time.Time{{
datastore: req.datastore, namespace: req.namespace,
backupType: req.group.BackupType, backupID: req.group.BackupID,
- }},
- }:
- case <-ctx.Done():
- return
+ }: fetchedAt},
}
}
}()
@@ -2522,11 +2551,9 @@ func (m *Monitor) fetchPBSBackupSnapshots(ctx context.Context, client *pbs.Clien
go func() {
defer close(jobs)
for _, req := range requests {
- select {
- case jobs <- req:
- case <-ctx.Done():
- return
- }
+ // Every planned group receives a result, including cached rows for
+ // unread groups; only successful, on-time reads are refreshed.
+ jobs <- req
}
}()
@@ -2535,12 +2562,14 @@ func (m *Monitor) fetchPBSBackupSnapshots(ctx context.Context, client *pbs.Clien
close(results)
}()
- var combined pbsBackupSnapshotFetchResult
+ combined := pbsBackupSnapshotFetchResult{refreshed: make(map[pbsBackupGroupKey]time.Time)}
for result := range results {
combined.backups = appendPBSBackupsWithinLimit(combined.backups, result.backups)
combined.errors += result.errors
combined.terminalErrors += result.terminalErrors
- combined.refreshed = append(combined.refreshed, result.refreshed...)
+ for key, fetchedAt := range result.refreshed {
+ combined.refreshed[key] = fetchedAt
+ }
}
return combined
diff --git a/internal/monitoring/monitor_client_init.go b/internal/monitoring/monitor_client_init.go
index 24902da8a8..407fb29183 100644
--- a/internal/monitoring/monitor_client_init.go
+++ b/internal/monitoring/monitor_client_init.go
@@ -123,7 +123,7 @@ func (m *Monitor) initPBSClients(cfg *config.Config) {
}, monErr)
continue
}
- m.pbsClients[pbsInst.Name] = client
+ m.replacePBSClient(pbsInst.Name, client)
log.Info().
Str("instance", pbsInst.Name).
Msg("PBS client created; awaiting first connectivity poll")
diff --git a/internal/monitoring/monitor_client_reconnect.go b/internal/monitoring/monitor_client_reconnect.go
index 08cb8bf0f0..73df2656a6 100644
--- a/internal/monitoring/monitor_client_reconnect.go
+++ b/internal/monitoring/monitor_client_reconnect.go
@@ -186,9 +186,7 @@ func (m *Monitor) retryFailedConnections(ctx context.Context) {
continue
}
- m.mu.Lock()
- m.pbsClients[pbsInst.Name] = client
- m.mu.Unlock()
+ m.replacePBSClient(pbsInst.Name, client)
log.Info().
Str("instance", pbsInst.Name).
diff --git a/internal/monitoring/monitor_pbs_pmg.go b/internal/monitoring/monitor_pbs_pmg.go
index 1488592cee..3e368705d8 100644
--- a/internal/monitoring/monitor_pbs_pmg.go
+++ b/internal/monitoring/monitor_pbs_pmg.go
@@ -558,43 +558,37 @@ func (m *Monitor) pollPBSBackupsAsync(instanceName string, client *pbs.Client, i
}
datastoreSnapshot := append([]models.PBSDatastore(nil), datastores...)
- m.mu.Lock()
- if m.pbsBackupPollers[instanceName] {
- m.mu.Unlock()
- log.Debug().Str("instance", instanceName).Msg("PBS backup polling already in progress")
- return
+ for i := range datastoreSnapshot {
+ datastoreSnapshot[i].Namespaces = append([]models.PBSNamespace(nil), datastores[i].Namespaces...)
}
- if m.pbsBackupPollers == nil {
- m.pbsBackupPollers = make(map[string]bool)
+ namespaceErrorSnapshot := make(map[string]error, len(namespaceErrors))
+ for name, err := range namespaceErrors {
+ namespaceErrorSnapshot[name] = err
}
- if m.lastPBSBackupPoll == nil {
- m.lastPBSBackupPoll = make(map[string]time.Time)
+ owner := m.beginPBSBackupPoll(instanceName, client, lastPoll, newLast)
+ if owner == nil {
+ log.Debug().Str("instance", instanceName).Msg("PBS backup polling already in progress")
+ return
}
- m.pbsBackupPollers[instanceName] = true
- m.lastPBSBackupPoll[instanceName] = newLast
- m.mu.Unlock()
go func() {
defer recoverFromPanic(fmt.Sprintf("pollPBSBackups-%s", instanceName))
- defer func() {
- m.mu.Lock()
- delete(m.pbsBackupPollers, instanceName)
- m.lastPBSBackupPoll[instanceName] = time.Now()
- m.mu.Unlock()
- }()
+ defer m.finishPBSBackupPoll(instanceName, owner)
// The main polling loop cancels its per-cycle context after returning.
// Backup work keeps the existing independent, runtime-scoped deadline.
- parentCtx := m.getRuntimeContext()
- if parentCtx == nil {
- parentCtx = context.Background()
- }
+ parentCtx := context.WithValue(owner.ctx, pbsBackupPollContextKey{}, owner)
backupCtx, cancel := context.WithTimeout(parentCtx, 5*time.Minute)
defer cancel()
if backupCtx.Err() != nil {
return
}
if discoveryErr != nil {
+ unlock, current := m.beginPBSBackupPublication(backupCtx, instanceName)
+ if !current {
+ return
+ }
+ defer unlock()
// One unreadable discovery scope, not a guessed datastore count.
// Only actual authentication/authorization status establishes denial.
denied := 0
@@ -610,7 +604,7 @@ func (m *Monitor) pollPBSBackupsAsync(instanceName string, client *pbs.Client, i
} else {
// The explicit empty list is authoritative and reconciles only this
// connection; the ordinary poller also drops its live rows/cache.
- m.pollPBSBackups(backupCtx, instanceName, client, datastoreSnapshot, namespaceErrors)
+ m.pollPBSBackups(backupCtx, instanceName, client, datastoreSnapshot, namespaceErrorSnapshot)
}
log.Info().Str("instance", instanceName).Dur("duration", time.Since(now)).Msg("Completed background PBS backup polling")
}()
diff --git a/internal/monitoring/pbs_backup_budget_test.go b/internal/monitoring/pbs_backup_budget_test.go
new file mode 100644
index 0000000000..4f00725810
--- /dev/null
+++ b/internal/monitoring/pbs_backup_budget_test.go
@@ -0,0 +1,261 @@
+package monitoring
+
+import (
+ "context"
+ "crypto/sha256"
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "reflect"
+ "strconv"
+ "strings"
+ "sync/atomic"
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/models"
+ "github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
+ "github.com/rcourtman/pulse-go-rewrite/internal/recovery"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/pbs"
+)
+
+// These are ordinary inventory reads against loopback fixtures, not guest
+// operations, installed backup jobs or a reproduction of a frozen workload.
+type pbsBudgetFixture struct {
+ *pbsPosturePollFixture
+ phase, reads, lateReads atomic.Int32
+ at atomic.Int64
+ boundary string
+ empty bool
+ datastores []models.PBSDatastore
+ blocked chan struct{}
+}
+
+func newPBSBudgetFixture(t *testing.T, kind, boundary string) *pbsBudgetFixture {
+ t.Helper()
+ count := 3
+ if boundary == "snapshots" {
+ count = 10
+ } else if boundary == "tasks" {
+ count = 1
+ }
+ f := &pbsBudgetFixture{pbsPosturePollFixture: newPBSPosturePollFixture(t, count, kind), boundary: boundary, blocked: make(chan struct{}, 32)}
+ f.at.Store(f.backupAt)
+ f.datastores = []models.PBSDatastore{{Name: "archive"}, {Name: "slow"}, {Name: "never"}}
+ if boundary == "namespace" {
+ f.datastores = []models.PBSDatastore{{Name: "archive", Namespaces: []models.PBSNamespace{{Path: ""}, {Path: "slow"}, {Path: "never"}}}}
+ } else if boundary == "snapshots" || boundary == "tasks" {
+ f.datastores = []models.PBSDatastore{{Name: "archive"}}
+ }
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ f.reads.Add(1)
+ stage := r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]
+ id := r.URL.Query().Get("backup-id")
+ scope := ""
+ if boundary == "namespace" {
+ scope = r.URL.Query().Get("ns")
+ } else if strings.Contains(r.URL.Path, "/slow/") {
+ scope = "slow"
+ } else if strings.Contains(r.URL.Path, "/never/") {
+ scope = "never"
+ }
+ if f.phase.Load() == 1 {
+ if scope == "never" {
+ f.lateReads.Add(1)
+ }
+ blocked := scope == "slow" || boundary == "snapshots" && stage == "snapshots" && id != "100" || boundary == "tasks" && stage == "tasks"
+ if blocked {
+ f.blocked <- struct{}{}
+ <-r.Context().Done()
+ return
+ }
+ }
+ var data any = []any{}
+ switch stage {
+ case "groups":
+ ids := []int{100}
+ if boundary == "snapshots" {
+ // More groups than workers: unread queued groups must retain
+ // their cache without making a request after the deadline.
+ ids = []int{100, 101, 102, 103, 104, 105, 106, 107, 108, 109}
+ } else if scope == "slow" {
+ ids = []int{101}
+ } else if scope == "never" {
+ ids = []int{102}
+ }
+ groups := []map[string]any{}
+ for _, vmid := range ids {
+ if f.empty && f.phase.Load() == 1 && vmid == 100 {
+ continue
+ }
+ groups = append(groups, map[string]any{"backup-type": kind, "backup-id": strconv.Itoa(vmid), "last-backup": f.at.Load(), "backup-count": 1})
+ }
+ data = groups
+ case "snapshots":
+ row := map[string]any{"backup-type": kind, "backup-id": id, "backup-time": f.at.Load(), "size": 1024,
+ "files": []string{"index.json.blob"}, "verification": map[string]any{"state": "ok"}, "comment": "guest-" + id}
+ if boundary == "tasks" && f.phase.Load() == 1 {
+ row["size"], row["files"] = 0, []string{"guest.conf.blob"}
+ }
+ data = []map[string]any{row}
+ case "tasks":
+ default:
+ http.NotFound(w, r)
+ return
+ }
+ _ = json.NewEncoder(w).Encode(map[string]any{"data": data})
+ }))
+ t.Cleanup(server.Close)
+ var err error
+ f.client, err = pbs.NewClient(pbs.ClientConfig{Host: server.URL, TokenName: "fixture@pbs!reader", TokenValue: "fixture-only"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ return f
+}
+
+func (f *pbsBudgetFixture) collect(t *testing.T, ctx context.Context) {
+ t.Helper()
+ f.m.pollPBSBackups(ctx, "pbs-main", f.client, f.datastores, nil)
+ f.wait(t)
+}
+
+func (f *pbsBudgetFixture) frame(t *testing.T) map[string]any {
+ t.Helper()
+ return map[string]any{
+ "pbs": map[string]any{"data": f.m.state.GetSnapshot().PBSBackups},
+ "pve": map[string]any{"data": f.m.state.GetSnapshot().PVEBackups},
+ "postures": map[string]any{"data": f.postures(t), "policy": recovery.DefaultProtectionPosturePolicy.Payload()},
+ "points": f.points(t, "pbs-main"),
+ }
+}
+
+func exercisePBSBudgetAcceptedListings(t *testing.T) {
+ for _, kind := range []string{"vm", "ct"} {
+ for _, boundary := range []string{"datastore", "namespace", "snapshots", "tasks"} {
+ for _, cold := range []bool{false, true} {
+ t.Run(fmt.Sprintf("%s/%s/cold=%t", kind, boundary, cold), func(t *testing.T) {
+ f := newPBSBudgetFixture(t, kind, boundary)
+ f.collect(t, context.Background())
+ oldRaw := f.m.state.GetSnapshot().PBSBackups
+ oldPoints := f.points(t, "pbs-main")
+ frames := map[string]any{"observed": f.frame(t)}
+ f.m.mu.RLock()
+ cache := make(map[pbsBackupGroupKey]time.Time)
+ for key, at := range f.m.pbsBackupCacheTime["pbs-main"] {
+ cache[key] = at
+ }
+ f.m.mu.RUnlock()
+ if cold {
+ f.m.state.UpdatePBSBackups("pbs-main", nil)
+ }
+ f.at.Add(15 * 60)
+ f.phase.Store(1)
+ startReads := f.reads.Load()
+ ctx, cancel := context.WithTimeout(context.Background(), time.Second)
+ defer cancel()
+ f.collect(t, ctx)
+ if ctx.Err() != context.DeadlineExceeded || f.lateReads.Load() != 0 {
+ t.Fatalf("budget did not expire cleanly: err=%v queued late reads=%d", ctx.Err(), f.lateReads.Load())
+ }
+ if boundary == "snapshots" && f.reads.Load()-startReads > 2+pbsBackupSnapshotFetchWorkers {
+ t.Error("queued snapshot work started after expiry")
+ }
+ raw := f.m.state.GetSnapshot().PBSBackups
+ advanced := false
+ for _, row := range raw {
+ if row.VMID == "100" {
+ advanced = row.BackupTime.Unix() == f.at.Load()
+ if boundary == "tasks" && (!row.InProgress || row.WriteActivityObserved || row.WriteActive) {
+ t.Error("unread live tasks invented terminal or successful write evidence")
+ }
+ } else {
+ found := false
+ for _, old := range oldRaw {
+ found = found || reflect.DeepEqual(row, old)
+ }
+ if cold || !found {
+ t.Error("unread raw history was invented or freshened")
+ }
+ }
+ }
+ if !advanced || !cold && len(raw) != len(oldRaw) || cold && len(raw) != 1 {
+ t.Errorf("accepted snapshot discarded or unread membership changed: advanced=%t rows=%d prior=%d cold=%t", advanced, len(raw), len(oldRaw), cold)
+ }
+ for key, old := range cache {
+ at := f.m.pbsBackupCacheTimeFor("pbs-main", key)
+ if key.backupID == "100" {
+ if !at.After(old) || !at.Before(time.Now()) {
+ t.Error("accepted read time was not published with its snapshot")
+ }
+ } else if !at.Equal(old) {
+ t.Error("unread cached snapshot's read time changed")
+ }
+ }
+ points := f.points(t, "pbs-main")
+ for _, old := range oldPoints {
+ found := false
+ for _, now := range points {
+ found = found || reflect.DeepEqual(old, now)
+ }
+ if !found {
+ t.Error("partial enumeration changed/deleted previously accepted durable evidence")
+ }
+ }
+ for _, posture := range f.postures(t) {
+ if posture.State == recovery.ProtectionStateProtected || len(posture.ProviderStates) != 1 || posture.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || posture.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown {
+ t.Errorf("partial collection invented protection: %+v", posture)
+ }
+ }
+ frames["partial"] = f.frame(t)
+ f.phase.Store(2)
+ f.collect(t, context.Background())
+ for _, posture := range f.postures(t) {
+ if posture.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryComplete || posture.State != recovery.ProtectionStateProtected {
+ t.Errorf("ordinary complete poll failed to recover: %+v", posture)
+ }
+ }
+ frames["recovered"] = f.frame(t)
+ if !cold && (boundary == "datastore" || boundary == "tasks") {
+ if path := os.Getenv("PULSE_PBS_BUDGET_FRAMES"); path != "" {
+ data, err := json.Marshal(frames)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(path+"-"+kind+"-"+boundary+".json", data, 0600); err != nil {
+ t.Fatal(err)
+ }
+ t.Logf("PULSE_PBS_FRAME %s-%s %x %s", kind, boundary, sha256.Sum256(data), data)
+ }
+ }
+ t.Logf("kind=%s boundary=%s cold=%t: accepted snapshot advanced; unread raw/cache/durable evidence retained; partial posture and ordinary recovery proved", kind, boundary, cold)
+ })
+ }
+ }
+ }
+}
+
+func TestPBSBudgetAcceptedEmptyDoesNotProveUnreadAbsence(t *testing.T) {
+ for _, kind := range []string{"vm", "ct"} {
+ t.Run(kind, func(t *testing.T) {
+ f := newPBSBudgetFixture(t, kind, "datastore")
+ f.collect(t, context.Background())
+ points := f.points(t, "pbs-main")
+ f.empty = true
+ f.phase.Store(1)
+ ctx, cancel := context.WithTimeout(context.Background(), time.Second)
+ defer cancel()
+ f.collect(t, ctx)
+ if len(f.m.state.GetSnapshot().PBSBackups) != 2 || !reflect.DeepEqual(points, f.points(t, "pbs-main")) {
+ t.Error("accepted empty scope retained raw rows or partial scope deleted durable evidence")
+ }
+ for _, row := range f.m.state.GetSnapshot().PBSBackups {
+ if row.VMID == "100" {
+ t.Error("accepted empty listing failed to remove its raw row")
+ }
+ }
+ })
+ }
+}
diff --git a/internal/monitoring/pbs_backup_lifecycle_test.go b/internal/monitoring/pbs_backup_lifecycle_test.go
new file mode 100644
index 0000000000..66498873c0
--- /dev/null
+++ b/internal/monitoring/pbs_backup_lifecycle_test.go
@@ -0,0 +1,134 @@
+package monitoring
+
+import (
+ "context"
+ "reflect"
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/config"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/pbs"
+)
+
+func waitPBSBudget(t *testing.T, ch <-chan struct{}, phase string) {
+ t.Helper()
+ select {
+ case <-ch:
+ case <-time.After(10 * time.Second):
+ t.Fatalf("PBS worker did not reach %s", phase)
+ }
+}
+
+func TestPBSBudgetOwnerWithdrawsLatePublication(t *testing.T) {
+ for _, transition := range []string{"runtime cancellation", "runtime replacement", "client replacement", "client retirement"} {
+ t.Run(transition, func(t *testing.T) {
+ f := newPBSBudgetFixture(t, "vm", "datastore")
+ f.collect(t, context.Background())
+ before, raw := f.points(t, "pbs-main"), f.m.state.GetSnapshot().PBSBackups
+ f.m.replacePBSClient("pbs-main", f.client)
+ parent, cancel := context.WithCancel(context.Background())
+ defer cancel()
+ f.m.setRuntimeContext(parent, nil)
+ f.m.config = &config.Config{EnableBackupPolling: true, BackupPollingCycles: 1}
+ cfg := &config.PBSInstance{Name: "pbs-main", MonitorBackups: true, MonitorDatastores: true}
+ f.at.Add(15 * 60)
+ f.phase.Store(1)
+ f.m.pollPBSBackupsAsync("pbs-main", f.client, cfg, f.datastores, nil, nil)
+ waitPBSBudget(t, f.blocked, "later datastore read")
+ f.m.mu.RLock()
+ owner := f.m.pbsBackupPollers["pbs-main"]
+ f.m.mu.RUnlock()
+ if owner == nil || owner.ctx.Err() != nil {
+ t.Fatal("background work lost its live runtime owner")
+ }
+ // A second ordinary cycle is not a second worker, even if cadence
+ // says it is due while the first HTTP request is still in flight.
+ f.m.pollPBSBackupsAsync("pbs-main", f.client, cfg, f.datastores, nil, nil)
+ f.m.mu.RLock()
+ if f.m.pbsBackupPollers["pbs-main"] != owner {
+ t.Error("ordinary cycle stacked a backup worker")
+ }
+ f.m.mu.RUnlock()
+ switch transition {
+ case "runtime cancellation":
+ cancel()
+ case "runtime replacement":
+ f.m.setRuntimeContext(context.Background(), nil) // Old parent intentionally remains live.
+ case "client replacement":
+ f.m.replacePBSClient("pbs-main", &pbs.Client{})
+ case "client retirement":
+ f.m.replacePBSClient("pbs-main", nil)
+ }
+ waitPBSBudget(t, owner.done, "withdrawn worker completion")
+ f.wait(t)
+ if !reflect.DeepEqual(raw, f.m.state.GetSnapshot().PBSBackups) || !reflect.DeepEqual(before, f.points(t, "pbs-main")) {
+ t.Error("withdrawn owner published its earlier/late rows")
+ }
+ // Dropping an exhausted child deadline must never erase the owner.
+ ctx := context.WithValue(context.Background(), pbsBackupPollContextKey{}, owner)
+ if unlock, ok := f.m.beginPBSBackupPublication(context.WithoutCancel(ctx), "pbs-main"); ok {
+ unlock()
+ t.Error("late owner was admitted after its binding ended")
+ }
+ f.m.mu.RLock()
+ at := f.m.lastPBSBackupPoll["pbs-main"]
+ f.m.mu.RUnlock()
+ if !at.IsZero() {
+ t.Error("withdrawn worker postponed its replacement's first scan")
+ }
+ if transition != "runtime cancellation" {
+ f.phase.Store(2)
+ f.m.replacePBSClient("pbs-main", f.client)
+ f.m.pollPBSBackupsAsync("pbs-main", f.client, cfg, f.datastores, nil, nil)
+ deadline := time.Now().Add(10 * time.Second)
+ for {
+ f.m.mu.RLock()
+ active := f.m.pbsBackupPollers["pbs-main"]
+ f.m.mu.RUnlock()
+ if active == nil {
+ break
+ }
+ if time.Now().After(deadline) {
+ t.Fatal("replacement ordinary scan did not finish")
+ }
+ time.Sleep(time.Millisecond)
+ }
+ f.wait(t)
+ for _, row := range f.m.state.GetSnapshot().PBSBackups {
+ if row.BackupTime.Unix() != f.at.Load() {
+ t.Error("ordinary replacement borrowed former client's cached snapshots")
+ }
+ }
+ }
+ })
+ }
+}
+
+func TestPBSBudgetOldCompletionCannotReleaseNewClaim(t *testing.T) {
+ m := &Monitor{}
+ first, second := &pbs.Client{}, &pbs.Client{}
+ m.replacePBSClient("reader", first)
+ m.setRuntimeContext(context.Background(), nil)
+ old := m.beginPBSBackupPoll("reader", first, time.Time{}, time.Now())
+ if old == nil {
+ t.Fatal("first claim was not admitted")
+ }
+ m.replacePBSClient("reader", second)
+ next := time.Now()
+ current := m.beginPBSBackupPoll("reader", second, time.Time{}, next)
+ if current == nil {
+ t.Fatal("new binding did not get an independent first claim")
+ }
+ m.finishPBSBackupPoll("reader", old)
+ if m.pbsBackupPollers["reader"] != current || !m.lastPBSBackupPoll["reader"].Equal(next) {
+ t.Error("old completion removed the new worker or changed its cadence")
+ }
+ ctx, cancel := context.WithDeadline(context.WithValue(current.ctx, pbsBackupPollContextKey{}, current), time.Now().Add(-time.Second))
+ defer cancel()
+ unlock, ok := m.beginPBSBackupPublication(context.WithoutCancel(ctx), "reader")
+ if !ok {
+ t.Fatal("spent child budget blocked live owner's local publication")
+ }
+ unlock()
+ m.finishPBSBackupPoll("reader", current)
+}
diff --git a/internal/monitoring/pbs_backup_poll.go b/internal/monitoring/pbs_backup_poll.go
new file mode 100644
index 0000000000..8faa85cfb2
--- /dev/null
+++ b/internal/monitoring/pbs_backup_poll.go
@@ -0,0 +1,113 @@
+package monitoring
+
+import (
+ "context"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/pkg/pbs"
+)
+
+// A collection budget belongs to a single runtime/client binding. Spending
+// that budget may publish earlier accepted reads, but ending the binding may
+// not. Cancellation retains the admission until the worker has really left.
+type pbsBackupPoll struct {
+ ctx context.Context
+ cancel context.CancelFunc
+ client *pbs.Client
+ done chan struct{}
+}
+
+type pbsBackupPollContextKey struct{}
+
+func (m *Monitor) beginPBSBackupPoll(instance string, client *pbs.Client, last, next time.Time) *pbsBackupPoll {
+ m.pbsBackupPublicationMu.RLock()
+ defer m.pbsBackupPublicationMu.RUnlock()
+ m.mu.Lock()
+ defer m.mu.Unlock()
+ if client == nil || m.pbsClients[instance] != client || !m.lastPBSBackupPoll[instance].Equal(last) || m.pbsBackupPollers[instance] != nil {
+ return nil
+ }
+ parent := m.runtimeCtx
+ if parent == nil {
+ parent = context.Background()
+ }
+ if parent.Err() != nil {
+ return nil
+ }
+ ctx, cancel := context.WithCancel(parent)
+ owner := &pbsBackupPoll{ctx: ctx, cancel: cancel, client: client, done: make(chan struct{})}
+ if m.pbsBackupPollers == nil {
+ m.pbsBackupPollers = make(map[string]*pbsBackupPoll)
+ }
+ if m.lastPBSBackupPoll == nil {
+ m.lastPBSBackupPoll = make(map[string]time.Time)
+ }
+ m.pbsBackupPollers[instance] = owner
+ m.lastPBSBackupPoll[instance] = next
+ return owner
+}
+
+func (m *Monitor) finishPBSBackupPoll(instance string, owner *pbsBackupPoll) {
+ m.mu.Lock()
+ if m.pbsBackupPollers[instance] == owner {
+ delete(m.pbsBackupPollers, instance)
+ if owner.ctx.Err() == nil && m.pbsClients[instance] == owner.client {
+ m.lastPBSBackupPoll[instance] = time.Now()
+ } else {
+ delete(m.lastPBSBackupPoll, instance)
+ }
+ }
+ m.mu.Unlock()
+ owner.cancel()
+ close(owner.done)
+}
+
+// Called with the publication writer and m.mu held. An old worker cannot
+// publish into, or release, a later binding's claim even if its read returns late.
+func (m *Monitor) cancelPBSBackupPollLocked(instance string) {
+ if owner := m.pbsBackupPollers[instance]; owner != nil {
+ owner.cancel()
+ delete(m.pbsBackupPollers, instance)
+ }
+ delete(m.lastPBSBackupPoll, instance)
+}
+
+func (m *Monitor) replacePBSClient(instance string, client *pbs.Client) {
+ m.pbsBackupPublicationMu.Lock()
+ defer m.pbsBackupPublicationMu.Unlock()
+ m.mu.Lock()
+ defer m.mu.Unlock()
+ m.cancelPBSBackupPollLocked(instance)
+ // A same-name server must independently read its own snapshots, never
+ // borrow the former client's still-fresh count/time cache.
+ delete(m.pbsBackupCacheTime, instance)
+ if m.pbsClients == nil {
+ m.pbsClients = make(map[string]*pbs.Client)
+ }
+ if client == nil {
+ delete(m.pbsClients, instance)
+ } else {
+ m.pbsClients[instance] = client
+ }
+}
+
+// Hold only across local publication, never a provider read. WithoutCancel
+// retains the owner value, so an expired child cannot erase the runtime fence.
+func (m *Monitor) beginPBSBackupPublication(ctx context.Context, instance string) (func(), bool) {
+ if ctx.Err() != nil {
+ return nil, false
+ }
+ owner, _ := ctx.Value(pbsBackupPollContextKey{}).(*pbsBackupPoll)
+ if owner == nil {
+ return func() {}, true // focused collectors retain caller cancellation
+ }
+ m.pbsBackupPublicationMu.RLock()
+ m.mu.RLock()
+ current := ctx.Err() == nil && owner.ctx.Err() == nil && m.pbsBackupPollers[instance] == owner && m.pbsClients[instance] == owner.client
+ m.mu.RUnlock()
+ if !current {
+ m.pbsBackupPublicationMu.RUnlock()
+ return nil, false
+ }
+ return m.pbsBackupPublicationMu.RUnlock, true
+}
diff --git a/internal/monitoring/pbs_datastore_discovery_test.go b/internal/monitoring/pbs_datastore_discovery_test.go
index f9207ea64d..617eb0a94e 100644
--- a/internal/monitoring/pbs_datastore_discovery_test.go
+++ b/internal/monitoring/pbs_datastore_discovery_test.go
@@ -159,7 +159,7 @@ func TestPBSDatastoreDiscoverySkippedScopeIsNotEmpty(t *testing.T) {
// live setter deliberately clears timestamps for a catch-up poll.
f.m.config.BackupPollingInterval = time.Hour
case "in flight":
- f.m.pbsBackupPollers["pbs-main"] = true
+ f.m.pbsBackupPollers["pbs-main"] = &pbsBackupPoll{}
case "cancelled cycle":
cancelled, cancel := context.WithCancel(ctx)
cancel()
diff --git a/internal/monitoring/pbs_protection_observation_test.go b/internal/monitoring/pbs_protection_observation_test.go
index 170afec284..9908aeab7e 100644
--- a/internal/monitoring/pbs_protection_observation_test.go
+++ b/internal/monitoring/pbs_protection_observation_test.go
@@ -23,6 +23,12 @@ import (
"github.com/rcourtman/pulse-go-rewrite/pkg/pbs"
)
+// The ordinary poll must retain accepted scopes after a later child timeout,
+// without renewing unread history or confusing partial collection with safety.
+func TestPBSBudgetPublishesAcceptedListings(t *testing.T) {
+ exercisePBSBudgetAcceptedListings(t)
+}
+
func TestBuildPBSProtectionProviderObservationMapsEvidenceQuality(t *testing.T) {
t.Parallel()
@@ -392,6 +398,7 @@ func newPBSNamespacePollFixture(t *testing.T) *pbsNamespacePollFixture {
{Name: "pbs-main", Host: server.URL, MonitorDatastores: true, MonitorBackups: true},
{Name: "pbs-other", Host: server.URL, MonitorDatastores: true, MonitorBackups: true},
}}
+ m.pbsClients = map[string]*pbs.Client{"pbs-main": f.client, "pbs-other": f.client}
m.lastPBSBackupPoll = make(map[string]time.Time)
m.pollStatusMap = make(map[string]*pollStatus)
return f