Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Preserve accepted PBS listings through later collection expiry
pulse-triage[bot]623b5ce265f2
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex 990ce413eb..c07a065cfe 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -17,6 +17,32 @@## Purpose+### Accepted PBS inventory survives its collection budget, not its owner++The existing five-minute PBS backup budget bounds provider reads, not acceptance+of earlier successful namespace/snapshot listings. Expiry classifies remaining+scopes as unread without starting queued HTTP requests. Accepted snapshot rows+and their actual read times publish together; failed/queued reads preserve live+cache only, never renew durable point evidence. Incomplete enumeration remains+partial/unavailable and cannot reconcile durable absence or assert protection.+Explicit cancellation still withdraws the complete enumeration.++A background scan captures its runtime and client at admission. Runtime/client+replacement cancels that owner, invalidates its cadence claim, and fences local+publication; old completion cannot release a replacement worker. Same-name client+replacement discards old snapshot read times, requiring independent refresh.+The core cycle's own context still does not cancel detached backup work. A spent+budget never starts an already-expired backup-alert read. Ordinary cadence,+five workers, retention bounds, auth, guest operations and policy are unchanged.++The normal monitoring suite includes `pbs_backup_budget_test.go` through+`TestPBSBudgetPublishesAcceptedListings`, and `pbs_backup_lifecycle_test.go`.+They check VM/CT, datastore/namespace/queued-snapshot/live-task expiry, cold live+state, accepted emptiness, nonrenewal and ordinary recovery, plus actual async+admission, cancellation, runtime/client replacement and old-worker completion.+Source-bound desktop/narrow Backups proof consumes produced partial/recovery+frames. This is truthful collection, not original frozen-guest recovery.+### Docker image replacements invalidate older tag comparisonsSuccessful tag/platform results are cached for six hours. If an unseen localdiff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex 0ba8567256..85b762f480 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -21,6 +21,26 @@## Purpose+### PBS collection expiry preserves accepted scopes without inventing safety++PBS publishes namespace and snapshot listings accepted before a later child+budget expires. Unread datastores/namespaces and failed or queued snapshot groups+retain only existing raw history and cache times. Late responses are unread;+accepted empty scopes remove only their own raw inventory. Partial enumeration+does not reconcile durable absence, renew unread evidence, or claim protection.+Unknown live-writer visibility keeps incomplete artifacts non-successful, never+invents a completion. Explicit cancellation or an ended runtime/client binding+still prevents publication, even after dropping a spent child deadline.++`pbs_backup_budget_test.go` and the registered protection-observation entry point+exercise actual PBS HTTP collection through live state, mapper, durable ingest+and posture for VM/CT, warm/cold state, every expiry phase and ordinary recovery.+Lifecycle tests cover the background owner and same-name client replacement.+Desktop/narrow browser proof consumes those emitted inventory/posture frames+through existing Backups polling and disclosures; dates, partial protection and+independent unread records must remain legible. No new API, auth, policy,+retention, guest operation, release or installed-recovery claim is implied.+### Selected PBS datastore remains inspectable without hoverThe existing PBS row disclosure is named by backup reader and datastore, ratherdiff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.goindex f5c7b7346a..52735ddb5d 100644--- a/internal/monitoring/monitor.go+++ b/internal/monitoring/monitor.go@@ -1309,6 +1309,7 @@ type Monitor struct {pveGuestPublicationMu sync.RWMutex // Orders guest/backup publication with connection/runtime replacement and retirement.pveGuestPolls map[string]*pveGuestPoll // Guarded by m.mu; canceled cycles retain ownership until completion unless the binding ends.pbsClients map[string]*pbs.Client+ pbsBackupPublicationMu sync.RWMutex // Orders PBS backup publication with runtime/client replacement.pmgClients map[string]*pmg.ClientavailabilityStatuses map[string]AvailabilityProbeStatusavailabilityByLocation map[string]map[string]AvailabilityProbeStatus@@ -1380,7 +1381,7 @@ type Monitor struct {backupAlertEvalMu sync.Mutex // Coalesce global rollup reads and order failure/recovery publication.persistence *config.ConfigPersistence // Add persistence for saving updated configsinstanceTokens APITokenInventory // Instance inventory holding agent tokens bound to this tenant org; see APITokenInventories- pbsBackupPollers map[string]bool // Track PBS backup polling goroutines per instance+ pbsBackupPollers map[string]*pbsBackupPoll // Runtime/client-owned PBS backup scans; guarded by mupbsBackupCacheTime map[string]map[pbsBackupGroupKey]time.Time // Track when each PBS backup group was last fetchedruntimePollingMu sync.RWMutex // Guards runtimePollingOverride; polling goroutines read it every cycleruntimePollingOverride runtimePollingOverrides // Runtime polling-cadence overrides pushed by the settings API (#1619)@@ -1511,6 +1512,8 @@ type Monitor struct {func (m *Monitor) setRuntimeContext(ctx context.Context, hub *websocket.Hub) {m.pveGuestPublicationMu.Lock()defer m.pveGuestPublicationMu.Unlock()+ m.pbsBackupPublicationMu.Lock()+ defer m.pbsBackupPublicationMu.Unlock()m.mu.Lock()defer m.mu.Unlock()for name := range m.pveGuestPolls {@@ -1525,6 +1528,9 @@ func (m *Monitor) setRuntimeContext(ctx context.Context, hub *websocket.Hub) {for name := range m.pveBackupPolls {m.cancelPVEBackupPollLocked(name)}+ for name := range m.pbsBackupPollers {+ m.cancelPBSBackupPollLocked(name)+ }m.runtimeCtx = ctxm.wsHub = hub}@@ -2015,7 +2021,7 @@ func New(cfg *config.Config) (*Monitor, error) {pveBackupTemplateSubjects: make(map[string]map[string]struct{}),backupPermissionWarnings: make(map[string]string),persistence: config.NewConfigPersistence(cfg.DataPath),- pbsBackupPollers: make(map[string]bool),+ pbsBackupPollers: make(map[string]*pbsBackupPoll),pbsBackupCacheTime: make(map[string]map[pbsBackupGroupKey]time.Time),nodeSnapshots: make(map[string]NodeMemorySnapshot),guestSnapshots: make(map[string]GuestMemorySnapshot),diff --git a/internal/monitoring/monitor_backups.go b/internal/monitoring/monitor_backups.goindex b11d70479d..72fa946c67 100644--- a/internal/monitoring/monitor_backups.go+++ b/internal/monitoring/monitor_backups.go@@ -1918,13 +1918,6 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien// Process each datastorefor _, ds := range datastores {- if ctx.Err() != nil {- log.Warn().- Str("instance", instanceName).- Msg("PBS backup polling cancelled before completion")- return- }-namespacePaths := namespacePathsForDatastore(ds)log.Info().@@ -1952,16 +1945,23 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, cliengroupsRequested := 0for _, namespace := range namespacePaths {+ namespaceKey := pbsBackupNamespaceKey{datastore: ds.Name, namespace: namespace}if ctx.Err() != nil {- log.Warn().- Str("instance", instanceName).- Msg("PBS backup polling cancelled mid-datastore")- return+ // Classify every unread namespace without starting an expired+ // request. Earlier accepted namespaces remain independently useful.+ if ctx.Err() != context.DeadlineExceeded {+ return+ }+ datastoreNamespaceErrors+++ transientNamespaces[namespaceKey] = struct{}{}+ continue}-groups, err := client.ListBackupGroups(ctx, ds.Name, namespace)- namespaceKey := pbsBackupNamespaceKey{datastore: ds.Name, namespace: namespace}queriedNamespaces[namespaceKey] = struct{}{}+ if ctx.Err() != nil {+ // A late nil-error response is not an accepted empty listing.+ err = ctx.Err()+ }if err != nil {datastoreNamespaceErrors++if !shouldReuseCachedPBSBackups(err) {@@ -2071,8 +2071,7 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien// Stage successful read times until their matching rows are published.// A later cancellation must not freshen the previously published rows// when this poll's refreshed payload never reaches state.- fetchedAt := time.Now()- for _, key := range fetched.refreshed {+ for key, fetchedAt := range fetched.refreshed {observedGroups[key] = struct{}{}refreshedGroups[key] = fetchedAt}@@ -2115,8 +2114,8 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien}}- if ctx.Err() != nil {- // An interrupted enumeration cannot authoritatively replace history.+ if ctx.Err() != nil && ctx.Err() != context.DeadlineExceeded {+ // Explicit cancellation still withdraws the enumeration.return}@@ -2145,6 +2144,57 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clienInt("count", len(allBackups)).Msg("PBS backups fetched")+ // A manifestless snapshot is incomplete, but it is not necessarily still+ // being written: failed backup and PBS-to-PBS sync tasks can leave that+ // shape behind. Correlate incomplete artifacts with PBS's live task view+ // before they feed guest "Backup Running" state. If task visibility is+ // denied or transiently unavailable, keep the previous conservative+ // inference rather than claiming that no write is active.+ if hasIncompletePBSBackup(allBackups) {+ // Write-activity evidence belongs to this poll, not to the snapshot+ // cache. A failed snapshot refresh may have reused an incomplete+ // artifact whose previous poll authoritatively found no writer. Clear+ // that old observation before asking for the current task view so a+ // task-query failure leaves visibility unknown instead of publishing a+ // stale terminal-failure conclusion.+ clearPBSWriteActivityObservation(allBackups)+ var runningDataTasks []pbs.RunningDataTask+ runningTasksErr := ctx.Err()+ if runningTasksErr == nil {+ runningDataTasks, runningTasksErr = client.ListRunningDataTasks(ctx)+ if ctx.Err() != nil {+ runningTasksErr = ctx.Err()+ }+ }+ if runningTasksErr != nil {+ log.Warn().Err(runningTasksErr).+ Str("instance", instanceName).+ Msg("Could not correlate incomplete PBS snapshots with running tasks")+ } else {+ applyPBSWriteActivity(allBackups, runningDataTasks)+ }+ }++ if ctx.Err() != nil && ctx.Err() != context.DeadlineExceeded {+ return+ }+ if ctx.Err() == context.DeadlineExceeded && datastoreErrors == 0 {+ // The final live-writer read can spend the remaining budget too.+ // Its unknown result cannot turn a partial collection into protection.+ datastoreErrors = 1+ }++ publicationCtx := ctx+ if ctx.Err() == context.DeadlineExceeded {+ // Drop only the child budget, not its runtime/client owner value.+ publicationCtx = context.WithoutCancel(ctx)+ }+ unlock, current := m.beginPBSBackupPublication(publicationCtx, instanceName)+ if !current {+ return+ }+ defer unlock()+// With no accepted group listing, a mixture of rejected and transiently// unread namespaces does not establish that every datastore was denied.// Partial successful collection still reports any actual access rejection.@@ -2186,36 +2236,6 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clienreturn}- // A manifestless snapshot is incomplete, but it is not necessarily still- // being written: failed backup and PBS-to-PBS sync tasks can leave that- // shape behind. Correlate incomplete artifacts with PBS's live task view- // before they feed guest "Backup Running" state. If task visibility is- // denied or transiently unavailable, keep the previous conservative- // inference rather than claiming that no write is active.- if hasIncompletePBSBackup(allBackups) {- // Write-activity evidence belongs to this poll, not to the snapshot- // cache. A failed snapshot refresh may have reused an incomplete- // artifact whose previous poll authoritatively found no writer. Clear- // that old observation before asking for the current task view so a- // task-query failure leaves visibility unknown instead of publishing a- // stale terminal-failure conclusion.- clearPBSWriteActivityObservation(allBackups)- runningDataTasks, runningTasksErr := client.ListRunningDataTasks(ctx)- if runningTasksErr != nil {- log.Warn().Err(runningTasksErr).- Str("instance", instanceName).- Msg("Could not correlate incomplete PBS snapshots with running tasks")- } else {- applyPBSWriteActivity(allBackups, runningDataTasks)- }- }-- if ctx.Err() != nil {- // Running-task correlation is the last HTTP read and can also be- // interrupted. Do not publish its incomplete payload or renew cache.- return- }-// Publication is the commit boundary: no cancellation return may split a// successful read time from its matching payload. Per-instance poller// admission serializes this publication with the next ordinary poll.@@ -2280,7 +2300,11 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien// Sync backup times to VMs/Containers and republish them to canonical resources.m.syncGuestBackupTimesAndResourceStore()- m.checkBackupAlerts(context.Background())+ if ctx.Err() == nil {+ // Collection expiry is not a failed alert-store read or permission+ // to clear an existing wanted alert. The next ordinary check remains.+ m.checkBackupAlerts(ctx)+ }// Immediately broadcast the updated state so frontend sees new backupsm.broadcastStateUpdate()@@ -2450,7 +2474,7 @@ type pbsBackupSnapshotFetchResult struct {backups []models.PBSBackuperrors intterminalErrors int- refreshed []pbsBackupGroupKey+ refreshed map[pbsBackupGroupKey]time.Time}func (m *Monitor) fetchPBSBackupSnapshots(ctx context.Context, client *pbs.Client, instanceName string, requests []pbsBackupFetchRequest) pbsBackupSnapshotFetchResult {@@ -2481,7 +2505,15 @@ func (m *Monitor) fetchPBSBackupSnapshots(ctx context.Context, client *pbs.ClienStr("id", req.group.BackupID).Msg("Refreshing PBS backup group")- snapshots, err := client.ListBackupSnapshots(ctx, req.datastore, req.namespace, req.group.BackupType, req.group.BackupID)+ // Drain queued work as unread after expiry, without provider I/O.+ var snapshots []pbs.BackupSnapshot+ err := ctx.Err()+ if err == nil {+ snapshots, err = client.ListBackupSnapshots(ctx, req.datastore, req.namespace, req.group.BackupType, req.group.BackupID)+ if ctx.Err() != nil {+ err = ctx.Err()+ }+ }if err != nil {log.Error().Err(err).@@ -2503,17 +2535,14 @@ func (m *Monitor) fetchPBSBackupSnapshots(ctx context.Context, client *pbs.Cliencontinue}+ fetchedAt := time.Now()backups := convertPBSSnapshots(instanceName, req.datastore, req.namespace, snapshots)- select {- case results <- pbsBackupSnapshotFetchResult{+ results <- pbsBackupSnapshotFetchResult{backups: backups,- refreshed: []pbsBackupGroupKey{{+ refreshed: map[pbsBackupGroupKey]time.Time{{datastore: req.datastore, namespace: req.namespace,backupType: req.group.BackupType, backupID: req.group.BackupID,- }},- }:- case <-ctx.Done():- return+ }: fetchedAt},}}}()@@ -2522,11 +2551,9 @@ func (m *Monitor) fetchPBSBackupSnapshots(ctx context.Context, client *pbs.Cliengo func() {defer close(jobs)for _, req := range requests {- select {- case jobs <- req:- case <-ctx.Done():- return- }+ // Every planned group receives a result, including cached rows for+ // unread groups; only successful, on-time reads are refreshed.+ jobs <- req}}()@@ -2535,12 +2562,14 @@ func (m *Monitor) fetchPBSBackupSnapshots(ctx context.Context, client *pbs.Clienclose(results)}()- var combined pbsBackupSnapshotFetchResult+ combined := pbsBackupSnapshotFetchResult{refreshed: make(map[pbsBackupGroupKey]time.Time)}for result := range results {combined.backups = appendPBSBackupsWithinLimit(combined.backups, result.backups)combined.errors += result.errorscombined.terminalErrors += result.terminalErrors- combined.refreshed = append(combined.refreshed, result.refreshed...)+ for key, fetchedAt := range result.refreshed {+ combined.refreshed[key] = fetchedAt+ }}return combineddiff --git a/internal/monitoring/monitor_client_init.go b/internal/monitoring/monitor_client_init.goindex 24902da8a8..407fb29183 100644--- a/internal/monitoring/monitor_client_init.go+++ b/internal/monitoring/monitor_client_init.go@@ -123,7 +123,7 @@ func (m *Monitor) initPBSClients(cfg *config.Config) {}, monErr)continue}- m.pbsClients[pbsInst.Name] = client+ m.replacePBSClient(pbsInst.Name, client)log.Info().Str("instance", pbsInst.Name).Msg("PBS client created; awaiting first connectivity poll")diff --git a/internal/monitoring/monitor_client_reconnect.go b/internal/monitoring/monitor_client_reconnect.goindex 08cb8bf0f0..73df2656a6 100644--- a/internal/monitoring/monitor_client_reconnect.go+++ b/internal/monitoring/monitor_client_reconnect.go@@ -186,9 +186,7 @@ func (m *Monitor) retryFailedConnections(ctx context.Context) {continue}- m.mu.Lock()- m.pbsClients[pbsInst.Name] = client- m.mu.Unlock()+ m.replacePBSClient(pbsInst.Name, client)log.Info().Str("instance", pbsInst.Name).diff --git a/internal/monitoring/monitor_pbs_pmg.go b/internal/monitoring/monitor_pbs_pmg.goindex 1488592cee..3e368705d8 100644--- a/internal/monitoring/monitor_pbs_pmg.go+++ b/internal/monitoring/monitor_pbs_pmg.go@@ -558,43 +558,37 @@ func (m *Monitor) pollPBSBackupsAsync(instanceName string, client *pbs.Client, i}datastoreSnapshot := append([]models.PBSDatastore(nil), datastores...)- m.mu.Lock()- if m.pbsBackupPollers[instanceName] {- m.mu.Unlock()- log.Debug().Str("instance", instanceName).Msg("PBS backup polling already in progress")- return+ for i := range datastoreSnapshot {+ datastoreSnapshot[i].Namespaces = append([]models.PBSNamespace(nil), datastores[i].Namespaces...)}- if m.pbsBackupPollers == nil {- m.pbsBackupPollers = make(map[string]bool)+ namespaceErrorSnapshot := make(map[string]error, len(namespaceErrors))+ for name, err := range namespaceErrors {+ namespaceErrorSnapshot[name] = err}- if m.lastPBSBackupPoll == nil {- m.lastPBSBackupPoll = make(map[string]time.Time)+ owner := m.beginPBSBackupPoll(instanceName, client, lastPoll, newLast)+ if owner == nil {+ log.Debug().Str("instance", instanceName).Msg("PBS backup polling already in progress")+ return}- m.pbsBackupPollers[instanceName] = true- m.lastPBSBackupPoll[instanceName] = newLast- m.mu.Unlock()go func() {defer recoverFromPanic(fmt.Sprintf("pollPBSBackups-%s", instanceName))- defer func() {- m.mu.Lock()- delete(m.pbsBackupPollers, instanceName)- m.lastPBSBackupPoll[instanceName] = time.Now()- m.mu.Unlock()- }()+ defer m.finishPBSBackupPoll(instanceName, owner)// The main polling loop cancels its per-cycle context after returning.// Backup work keeps the existing independent, runtime-scoped deadline.- parentCtx := m.getRuntimeContext()- if parentCtx == nil {- parentCtx = context.Background()- }+ parentCtx := context.WithValue(owner.ctx, pbsBackupPollContextKey{}, owner)backupCtx, cancel := context.WithTimeout(parentCtx, 5*time.Minute)defer cancel()if backupCtx.Err() != nil {return}if discoveryErr != nil {+ unlock, current := m.beginPBSBackupPublication(backupCtx, instanceName)+ if !current {+ return+ }+ defer unlock()// One unreadable discovery scope, not a guessed datastore count.// Only actual authentication/authorization status establishes denial.denied := 0@@ -610,7 +604,7 @@ func (m *Monitor) pollPBSBackupsAsync(instanceName string, client *pbs.Client, i} else {// The explicit empty list is authoritative and reconciles only this// connection; the ordinary poller also drops its live rows/cache.- m.pollPBSBackups(backupCtx, instanceName, client, datastoreSnapshot, namespaceErrors)+ m.pollPBSBackups(backupCtx, instanceName, client, datastoreSnapshot, namespaceErrorSnapshot)}log.Info().Str("instance", instanceName).Dur("duration", time.Since(now)).Msg("Completed background PBS backup polling")}()diff --git a/internal/monitoring/pbs_backup_budget_test.go b/internal/monitoring/pbs_backup_budget_test.gonew file mode 100644index 0000000000..4f00725810--- /dev/null+++ b/internal/monitoring/pbs_backup_budget_test.go@@ -0,0 +1,261 @@+package monitoring++import (+ "context"+ "crypto/sha256"+ "encoding/json"+ "fmt"+ "net/http"+ "net/http/httptest"+ "os"+ "reflect"+ "strconv"+ "strings"+ "sync/atomic"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/models"+ "github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"+ "github.com/rcourtman/pulse-go-rewrite/internal/recovery"+ "github.com/rcourtman/pulse-go-rewrite/pkg/pbs"+)++// These are ordinary inventory reads against loopback fixtures, not guest+// operations, installed backup jobs or a reproduction of a frozen workload.+type pbsBudgetFixture struct {+ *pbsPosturePollFixture+ phase, reads, lateReads atomic.Int32+ at atomic.Int64+ boundary string+ empty bool+ datastores []models.PBSDatastore+ blocked chan struct{}+}++func newPBSBudgetFixture(t *testing.T, kind, boundary string) *pbsBudgetFixture {+ t.Helper()+ count := 3+ if boundary == "snapshots" {+ count = 10+ } else if boundary == "tasks" {+ count = 1+ }+ f := &pbsBudgetFixture{pbsPosturePollFixture: newPBSPosturePollFixture(t, count, kind), boundary: boundary, blocked: make(chan struct{}, 32)}+ f.at.Store(f.backupAt)+ f.datastores = []models.PBSDatastore{{Name: "archive"}, {Name: "slow"}, {Name: "never"}}+ if boundary == "namespace" {+ f.datastores = []models.PBSDatastore{{Name: "archive", Namespaces: []models.PBSNamespace{{Path: ""}, {Path: "slow"}, {Path: "never"}}}}+ } else if boundary == "snapshots" || boundary == "tasks" {+ f.datastores = []models.PBSDatastore{{Name: "archive"}}+ }+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ f.reads.Add(1)+ stage := r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]+ id := r.URL.Query().Get("backup-id")+ scope := ""+ if boundary == "namespace" {+ scope = r.URL.Query().Get("ns")+ } else if strings.Contains(r.URL.Path, "/slow/") {+ scope = "slow"+ } else if strings.Contains(r.URL.Path, "/never/") {+ scope = "never"+ }+ if f.phase.Load() == 1 {+ if scope == "never" {+ f.lateReads.Add(1)+ }+ blocked := scope == "slow" || boundary == "snapshots" && stage == "snapshots" && id != "100" || boundary == "tasks" && stage == "tasks"+ if blocked {+ f.blocked <- struct{}{}+ <-r.Context().Done()+ return+ }+ }+ var data any = []any{}+ switch stage {+ case "groups":+ ids := []int{100}+ if boundary == "snapshots" {+ // More groups than workers: unread queued groups must retain+ // their cache without making a request after the deadline.+ ids = []int{100, 101, 102, 103, 104, 105, 106, 107, 108, 109}+ } else if scope == "slow" {+ ids = []int{101}+ } else if scope == "never" {+ ids = []int{102}+ }+ groups := []map[string]any{}+ for _, vmid := range ids {+ if f.empty && f.phase.Load() == 1 && vmid == 100 {+ continue+ }+ groups = append(groups, map[string]any{"backup-type": kind, "backup-id": strconv.Itoa(vmid), "last-backup": f.at.Load(), "backup-count": 1})+ }+ data = groups+ case "snapshots":+ row := map[string]any{"backup-type": kind, "backup-id": id, "backup-time": f.at.Load(), "size": 1024,+ "files": []string{"index.json.blob"}, "verification": map[string]any{"state": "ok"}, "comment": "guest-" + id}+ if boundary == "tasks" && f.phase.Load() == 1 {+ row["size"], row["files"] = 0, []string{"guest.conf.blob"}+ }+ data = []map[string]any{row}+ case "tasks":+ default:+ http.NotFound(w, r)+ return+ }+ _ = json.NewEncoder(w).Encode(map[string]any{"data": data})+ }))+ t.Cleanup(server.Close)+ var err error+ f.client, err = pbs.NewClient(pbs.ClientConfig{Host: server.URL, TokenName: "fixture@pbs!reader", TokenValue: "fixture-only"})+ if err != nil {+ t.Fatal(err)+ }+ return f+}++func (f *pbsBudgetFixture) collect(t *testing.T, ctx context.Context) {+ t.Helper()+ f.m.pollPBSBackups(ctx, "pbs-main", f.client, f.datastores, nil)+ f.wait(t)+}++func (f *pbsBudgetFixture) frame(t *testing.T) map[string]any {+ t.Helper()+ return map[string]any{+ "pbs": map[string]any{"data": f.m.state.GetSnapshot().PBSBackups},+ "pve": map[string]any{"data": f.m.state.GetSnapshot().PVEBackups},+ "postures": map[string]any{"data": f.postures(t), "policy": recovery.DefaultProtectionPosturePolicy.Payload()},+ "points": f.points(t, "pbs-main"),+ }+}++func exercisePBSBudgetAcceptedListings(t *testing.T) {+ for _, kind := range []string{"vm", "ct"} {+ for _, boundary := range []string{"datastore", "namespace", "snapshots", "tasks"} {+ for _, cold := range []bool{false, true} {+ t.Run(fmt.Sprintf("%s/%s/cold=%t", kind, boundary, cold), func(t *testing.T) {+ f := newPBSBudgetFixture(t, kind, boundary)+ f.collect(t, context.Background())+ oldRaw := f.m.state.GetSnapshot().PBSBackups+ oldPoints := f.points(t, "pbs-main")+ frames := map[string]any{"observed": f.frame(t)}+ f.m.mu.RLock()+ cache := make(map[pbsBackupGroupKey]time.Time)+ for key, at := range f.m.pbsBackupCacheTime["pbs-main"] {+ cache[key] = at+ }+ f.m.mu.RUnlock()+ if cold {+ f.m.state.UpdatePBSBackups("pbs-main", nil)+ }+ f.at.Add(15 * 60)+ f.phase.Store(1)+ startReads := f.reads.Load()+ ctx, cancel := context.WithTimeout(context.Background(), time.Second)+ defer cancel()+ f.collect(t, ctx)+ if ctx.Err() != context.DeadlineExceeded || f.lateReads.Load() != 0 {+ t.Fatalf("budget did not expire cleanly: err=%v queued late reads=%d", ctx.Err(), f.lateReads.Load())+ }+ if boundary == "snapshots" && f.reads.Load()-startReads > 2+pbsBackupSnapshotFetchWorkers {+ t.Error("queued snapshot work started after expiry")+ }+ raw := f.m.state.GetSnapshot().PBSBackups+ advanced := false+ for _, row := range raw {+ if row.VMID == "100" {+ advanced = row.BackupTime.Unix() == f.at.Load()+ if boundary == "tasks" && (!row.InProgress || row.WriteActivityObserved || row.WriteActive) {+ t.Error("unread live tasks invented terminal or successful write evidence")+ }+ } else {+ found := false+ for _, old := range oldRaw {+ found = found || reflect.DeepEqual(row, old)+ }+ if cold || !found {+ t.Error("unread raw history was invented or freshened")+ }+ }+ }+ if !advanced || !cold && len(raw) != len(oldRaw) || cold && len(raw) != 1 {+ t.Errorf("accepted snapshot discarded or unread membership changed: advanced=%t rows=%d prior=%d cold=%t", advanced, len(raw), len(oldRaw), cold)+ }+ for key, old := range cache {+ at := f.m.pbsBackupCacheTimeFor("pbs-main", key)+ if key.backupID == "100" {+ if !at.After(old) || !at.Before(time.Now()) {+ t.Error("accepted read time was not published with its snapshot")+ }+ } else if !at.Equal(old) {+ t.Error("unread cached snapshot's read time changed")+ }+ }+ points := f.points(t, "pbs-main")+ for _, old := range oldPoints {+ found := false+ for _, now := range points {+ found = found || reflect.DeepEqual(old, now)+ }+ if !found {+ t.Error("partial enumeration changed/deleted previously accepted durable evidence")+ }+ }+ for _, posture := range f.postures(t) {+ if posture.State == recovery.ProtectionStateProtected || len(posture.ProviderStates) != 1 || posture.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || posture.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown {+ t.Errorf("partial collection invented protection: %+v", posture)+ }+ }+ frames["partial"] = f.frame(t)+ f.phase.Store(2)+ f.collect(t, context.Background())+ for _, posture := range f.postures(t) {+ if posture.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryComplete || posture.State != recovery.ProtectionStateProtected {+ t.Errorf("ordinary complete poll failed to recover: %+v", posture)+ }+ }+ frames["recovered"] = f.frame(t)+ if !cold && (boundary == "datastore" || boundary == "tasks") {+ if path := os.Getenv("PULSE_PBS_BUDGET_FRAMES"); path != "" {+ data, err := json.Marshal(frames)+ if err != nil {+ t.Fatal(err)+ }+ if err := os.WriteFile(path+"-"+kind+"-"+boundary+".json", data, 0600); err != nil {+ t.Fatal(err)+ }+ t.Logf("PULSE_PBS_FRAME %s-%s %x %s", kind, boundary, sha256.Sum256(data), data)+ }+ }+ t.Logf("kind=%s boundary=%s cold=%t: accepted snapshot advanced; unread raw/cache/durable evidence retained; partial posture and ordinary recovery proved", kind, boundary, cold)+ })+ }+ }+ }+}++func TestPBSBudgetAcceptedEmptyDoesNotProveUnreadAbsence(t *testing.T) {+ for _, kind := range []string{"vm", "ct"} {+ t.Run(kind, func(t *testing.T) {+ f := newPBSBudgetFixture(t, kind, "datastore")+ f.collect(t, context.Background())+ points := f.points(t, "pbs-main")+ f.empty = true+ f.phase.Store(1)+ ctx, cancel := context.WithTimeout(context.Background(), time.Second)+ defer cancel()+ f.collect(t, ctx)+ if len(f.m.state.GetSnapshot().PBSBackups) != 2 || !reflect.DeepEqual(points, f.points(t, "pbs-main")) {+ t.Error("accepted empty scope retained raw rows or partial scope deleted durable evidence")+ }+ for _, row := range f.m.state.GetSnapshot().PBSBackups {+ if row.VMID == "100" {+ t.Error("accepted empty listing failed to remove its raw row")+ }+ }+ })+ }+}diff --git a/internal/monitoring/pbs_backup_lifecycle_test.go b/internal/monitoring/pbs_backup_lifecycle_test.gonew file mode 100644index 0000000000..66498873c0--- /dev/null+++ b/internal/monitoring/pbs_backup_lifecycle_test.go@@ -0,0 +1,134 @@+package monitoring++import (+ "context"+ "reflect"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/config"+ "github.com/rcourtman/pulse-go-rewrite/pkg/pbs"+)++func waitPBSBudget(t *testing.T, ch <-chan struct{}, phase string) {+ t.Helper()+ select {+ case <-ch:+ case <-time.After(10 * time.Second):+ t.Fatalf("PBS worker did not reach %s", phase)+ }+}++func TestPBSBudgetOwnerWithdrawsLatePublication(t *testing.T) {+ for _, transition := range []string{"runtime cancellation", "runtime replacement", "client replacement", "client retirement"} {+ t.Run(transition, func(t *testing.T) {+ f := newPBSBudgetFixture(t, "vm", "datastore")+ f.collect(t, context.Background())+ before, raw := f.points(t, "pbs-main"), f.m.state.GetSnapshot().PBSBackups+ f.m.replacePBSClient("pbs-main", f.client)+ parent, cancel := context.WithCancel(context.Background())+ defer cancel()+ f.m.setRuntimeContext(parent, nil)+ f.m.config = &config.Config{EnableBackupPolling: true, BackupPollingCycles: 1}+ cfg := &config.PBSInstance{Name: "pbs-main", MonitorBackups: true, MonitorDatastores: true}+ f.at.Add(15 * 60)+ f.phase.Store(1)+ f.m.pollPBSBackupsAsync("pbs-main", f.client, cfg, f.datastores, nil, nil)+ waitPBSBudget(t, f.blocked, "later datastore read")+ f.m.mu.RLock()+ owner := f.m.pbsBackupPollers["pbs-main"]+ f.m.mu.RUnlock()+ if owner == nil || owner.ctx.Err() != nil {+ t.Fatal("background work lost its live runtime owner")+ }+ // A second ordinary cycle is not a second worker, even if cadence+ // says it is due while the first HTTP request is still in flight.+ f.m.pollPBSBackupsAsync("pbs-main", f.client, cfg, f.datastores, nil, nil)+ f.m.mu.RLock()+ if f.m.pbsBackupPollers["pbs-main"] != owner {+ t.Error("ordinary cycle stacked a backup worker")+ }+ f.m.mu.RUnlock()+ switch transition {+ case "runtime cancellation":+ cancel()+ case "runtime replacement":+ f.m.setRuntimeContext(context.Background(), nil) // Old parent intentionally remains live.+ case "client replacement":+ f.m.replacePBSClient("pbs-main", &pbs.Client{})+ case "client retirement":+ f.m.replacePBSClient("pbs-main", nil)+ }+ waitPBSBudget(t, owner.done, "withdrawn worker completion")+ f.wait(t)+ if !reflect.DeepEqual(raw, f.m.state.GetSnapshot().PBSBackups) || !reflect.DeepEqual(before, f.points(t, "pbs-main")) {+ t.Error("withdrawn owner published its earlier/late rows")+ }+ // Dropping an exhausted child deadline must never erase the owner.+ ctx := context.WithValue(context.Background(), pbsBackupPollContextKey{}, owner)+ if unlock, ok := f.m.beginPBSBackupPublication(context.WithoutCancel(ctx), "pbs-main"); ok {+ unlock()+ t.Error("late owner was admitted after its binding ended")+ }+ f.m.mu.RLock()+ at := f.m.lastPBSBackupPoll["pbs-main"]+ f.m.mu.RUnlock()+ if !at.IsZero() {+ t.Error("withdrawn worker postponed its replacement's first scan")+ }+ if transition != "runtime cancellation" {+ f.phase.Store(2)+ f.m.replacePBSClient("pbs-main", f.client)+ f.m.pollPBSBackupsAsync("pbs-main", f.client, cfg, f.datastores, nil, nil)+ deadline := time.Now().Add(10 * time.Second)+ for {+ f.m.mu.RLock()+ active := f.m.pbsBackupPollers["pbs-main"]+ f.m.mu.RUnlock()+ if active == nil {+ break+ }+ if time.Now().After(deadline) {+ t.Fatal("replacement ordinary scan did not finish")+ }+ time.Sleep(time.Millisecond)+ }+ f.wait(t)+ for _, row := range f.m.state.GetSnapshot().PBSBackups {+ if row.BackupTime.Unix() != f.at.Load() {+ t.Error("ordinary replacement borrowed former client's cached snapshots")+ }+ }+ }+ })+ }+}++func TestPBSBudgetOldCompletionCannotReleaseNewClaim(t *testing.T) {+ m := &Monitor{}+ first, second := &pbs.Client{}, &pbs.Client{}+ m.replacePBSClient("reader", first)+ m.setRuntimeContext(context.Background(), nil)+ old := m.beginPBSBackupPoll("reader", first, time.Time{}, time.Now())+ if old == nil {+ t.Fatal("first claim was not admitted")+ }+ m.replacePBSClient("reader", second)+ next := time.Now()+ current := m.beginPBSBackupPoll("reader", second, time.Time{}, next)+ if current == nil {+ t.Fatal("new binding did not get an independent first claim")+ }+ m.finishPBSBackupPoll("reader", old)+ if m.pbsBackupPollers["reader"] != current || !m.lastPBSBackupPoll["reader"].Equal(next) {+ t.Error("old completion removed the new worker or changed its cadence")+ }+ ctx, cancel := context.WithDeadline(context.WithValue(current.ctx, pbsBackupPollContextKey{}, current), time.Now().Add(-time.Second))+ defer cancel()+ unlock, ok := m.beginPBSBackupPublication(context.WithoutCancel(ctx), "reader")+ if !ok {+ t.Fatal("spent child budget blocked live owner's local publication")+ }+ unlock()+ m.finishPBSBackupPoll("reader", current)+}diff --git a/internal/monitoring/pbs_backup_poll.go b/internal/monitoring/pbs_backup_poll.gonew file mode 100644index 0000000000..8faa85cfb2--- /dev/null+++ b/internal/monitoring/pbs_backup_poll.go@@ -0,0 +1,113 @@+package monitoring++import (+ "context"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/pkg/pbs"+)++// A collection budget belongs to a single runtime/client binding. Spending+// that budget may publish earlier accepted reads, but ending the binding may+// not. Cancellation retains the admission until the worker has really left.+type pbsBackupPoll struct {+ ctx context.Context+ cancel context.CancelFunc+ client *pbs.Client+ done chan struct{}+}++type pbsBackupPollContextKey struct{}++func (m *Monitor) beginPBSBackupPoll(instance string, client *pbs.Client, last, next time.Time) *pbsBackupPoll {+ m.pbsBackupPublicationMu.RLock()+ defer m.pbsBackupPublicationMu.RUnlock()+ m.mu.Lock()+ defer m.mu.Unlock()+ if client == nil || m.pbsClients[instance] != client || !m.lastPBSBackupPoll[instance].Equal(last) || m.pbsBackupPollers[instance] != nil {+ return nil+ }+ parent := m.runtimeCtx+ if parent == nil {+ parent = context.Background()+ }+ if parent.Err() != nil {+ return nil+ }+ ctx, cancel := context.WithCancel(parent)+ owner := &pbsBackupPoll{ctx: ctx, cancel: cancel, client: client, done: make(chan struct{})}+ if m.pbsBackupPollers == nil {+ m.pbsBackupPollers = make(map[string]*pbsBackupPoll)+ }+ if m.lastPBSBackupPoll == nil {+ m.lastPBSBackupPoll = make(map[string]time.Time)+ }+ m.pbsBackupPollers[instance] = owner+ m.lastPBSBackupPoll[instance] = next+ return owner+}++func (m *Monitor) finishPBSBackupPoll(instance string, owner *pbsBackupPoll) {+ m.mu.Lock()+ if m.pbsBackupPollers[instance] == owner {+ delete(m.pbsBackupPollers, instance)+ if owner.ctx.Err() == nil && m.pbsClients[instance] == owner.client {+ m.lastPBSBackupPoll[instance] = time.Now()+ } else {+ delete(m.lastPBSBackupPoll, instance)+ }+ }+ m.mu.Unlock()+ owner.cancel()+ close(owner.done)+}++// Called with the publication writer and m.mu held. An old worker cannot+// publish into, or release, a later binding's claim even if its read returns late.+func (m *Monitor) cancelPBSBackupPollLocked(instance string) {+ if owner := m.pbsBackupPollers[instance]; owner != nil {+ owner.cancel()+ delete(m.pbsBackupPollers, instance)+ }+ delete(m.lastPBSBackupPoll, instance)+}++func (m *Monitor) replacePBSClient(instance string, client *pbs.Client) {+ m.pbsBackupPublicationMu.Lock()+ defer m.pbsBackupPublicationMu.Unlock()+ m.mu.Lock()+ defer m.mu.Unlock()+ m.cancelPBSBackupPollLocked(instance)+ // A same-name server must independently read its own snapshots, never+ // borrow the former client's still-fresh count/time cache.+ delete(m.pbsBackupCacheTime, instance)+ if m.pbsClients == nil {+ m.pbsClients = make(map[string]*pbs.Client)+ }+ if client == nil {+ delete(m.pbsClients, instance)+ } else {+ m.pbsClients[instance] = client+ }+}++// Hold only across local publication, never a provider read. WithoutCancel+// retains the owner value, so an expired child cannot erase the runtime fence.+func (m *Monitor) beginPBSBackupPublication(ctx context.Context, instance string) (func(), bool) {+ if ctx.Err() != nil {+ return nil, false+ }+ owner, _ := ctx.Value(pbsBackupPollContextKey{}).(*pbsBackupPoll)+ if owner == nil {+ return func() {}, true // focused collectors retain caller cancellation+ }+ m.pbsBackupPublicationMu.RLock()+ m.mu.RLock()+ current := ctx.Err() == nil && owner.ctx.Err() == nil && m.pbsBackupPollers[instance] == owner && m.pbsClients[instance] == owner.client+ m.mu.RUnlock()+ if !current {+ m.pbsBackupPublicationMu.RUnlock()+ return nil, false+ }+ return m.pbsBackupPublicationMu.RUnlock, true+}diff --git a/internal/monitoring/pbs_datastore_discovery_test.go b/internal/monitoring/pbs_datastore_discovery_test.goindex f9207ea64d..617eb0a94e 100644--- a/internal/monitoring/pbs_datastore_discovery_test.go+++ b/internal/monitoring/pbs_datastore_discovery_test.go@@ -159,7 +159,7 @@ func TestPBSDatastoreDiscoverySkippedScopeIsNotEmpty(t *testing.T) {// live setter deliberately clears timestamps for a catch-up poll.f.m.config.BackupPollingInterval = time.Hourcase "in flight":- f.m.pbsBackupPollers["pbs-main"] = true+ f.m.pbsBackupPollers["pbs-main"] = &pbsBackupPoll{}case "cancelled cycle":cancelled, cancel := context.WithCancel(ctx)cancel()diff --git a/internal/monitoring/pbs_protection_observation_test.go b/internal/monitoring/pbs_protection_observation_test.goindex 170afec284..9908aeab7e 100644--- a/internal/monitoring/pbs_protection_observation_test.go+++ b/internal/monitoring/pbs_protection_observation_test.go@@ -23,6 +23,12 @@ import ("github.com/rcourtman/pulse-go-rewrite/pkg/pbs")+// The ordinary poll must retain accepted scopes after a later child timeout,+// without renewing unread history or confusing partial collection with safety.+func TestPBSBudgetPublishesAcceptedListings(t *testing.T) {+ exercisePBSBudgetAcceptedListings(t)+}+func TestBuildPBSProtectionProviderObservationMapsEvidenceQuality(t *testing.T) {t.Parallel()@@ -392,6 +398,7 @@ func newPBSNamespacePollFixture(t *testing.T) *pbsNamespacePollFixture {{Name: "pbs-main", Host: server.URL, MonitorDatastores: true, MonitorBackups: true},{Name: "pbs-other", Host: server.URL, MonitorDatastores: true, MonitorBackups: true},}}+ m.pbsClients = map[string]*pbs.Client{"pbs-main": f.client, "pbs-other": f.client}m.lastPBSBackupPoll = make(map[string]time.Time)m.pollStatusMap = make(map[string]*pollStatus)return f