Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep re-read backups protected when one storage is unreadable
rcourtmanc86ca8d308f1
diff --git a/docs/release-control/v6/internal/subsystems/frontend-primitives.md b/docs/release-control/v6/internal/subsystems/frontend-primitives.mdindex 764d9fc27b..f0c63d7a7c 100644--- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md+++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md@@ -10877,6 +10877,17 @@ misrepresented. `ProxmoxCoverageTable.test.tsx` checks narrow/full projectionand complete long location text; the parsed inventory regressions also retainreader identity and full repository context without adding reads or routes.+### Unread backup storage is named once on the Backups page++When the postures on the page carry unread provider scopes, the Backups page+shows one amber status line under the health strip that names them (up to three,+then a count) and says Pulse cannot confirm the backups stored there, with one action:+check that the storage is reachable, or check the API token's permissions when+every named read was rejected. It reuses the existing read-notice style, adds no+request, and the connection name appears only when the gap spans several. The+per-workload drawer keeps the server's explanation, which names the same scopes.+`proxmoxBackupUnreadSources.test.ts` and `ProxmoxBackupsTable.test.tsx` cover it.+### Backup rows stay on one line (1 November 2026)Coverage and By date rows keep one line at every container width, phonesdiff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.mdindex c0897f0364..1c84f7a8bb 100644--- a/docs/release-control/v6/internal/subsystems/monitoring.md+++ b/docs/release-control/v6/internal/subsystems/monitoring.md@@ -777,6 +777,19 @@ producer feeds the production Backups coverage surface in`scripts/check-pve-backup-reconciliation.cjs`. Retained history is not a freshprovider read, an available archive, guest thaw or filesystem-liveness proof.+A partial PVE or PBS observation also names what went unread, from the same+read-failure bookkeeping that drives carry-forward: PVE unread nodes, node-local+storages and shared storages no node read (with the node when exactly one node+attempted it, and whether authorization rejected it); PBS datastores whose+namespace discovery failed and namespaces whose groups or snapshots were not all+read. The names never decide which points this poll confirmed; that remains the+renewed point evidence described in the storage-recovery contract, so a guest+whose backup this scan re-read is protected with partial coverage while one whose+only backup sits on the unread storage is not.+`TestPVEUnreadStorageLeavesReadBackupsProtected` covers one enabled but+unreachable PBS storage beside readable archives, a pruned archive retained by+the partial poll, and the complete recovery.+### Skipped PVE backup storage is unread, not emptyA configured backup target that is not safe to query (including disableddiff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex d6248b729a..1597872b98 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -279,6 +279,36 @@ checks all non-success outcomes through the real point/observation builder.The ordinary PBS writer lifecycle checks the same explanation alongside liveand durable artifact outcomes and feeds desktop/narrow Backups pixel proof.+### One unread storage does not withdraw backups Pulse did read++A partial PVE or PBS collection still confirms the points it re-read. Collectors+renew a point's evidence only for rows a poll returned, stamped with that poll's+observation time, so a point whose evidence was ingested no earlier than the+latest observation for its provider scope supports protection under partial+history with Partial or Unknown access. A point retained from an unread scope+keeps its older evidence and may have been pruned since, so it never supports+protection; neither does store-backfilled migration evidence, nor anything+under denied, unavailable or unknown history. The posture is then Protected with+partial coverage, and the cross-scope rule matches the unread-source rule: a+supported current backup is not withdrawn because another source is partial.+PBS snapshot-cache reuse within its 10-minute TTL counts as a read of that+group, exactly as complete collections already treat it.++Provider observations also record which scopes went unread (PVE node or+storage, with its node when exactly one node attempted it; PBS datastore+namespace discovery or namespace), persisted in+`protection_provider_observations.unread_scopes_json`, capped at 32 and carried+on each provider state as `unreadScopes`. The list explains the gap only;+confirmation never depends on it. Explanations name up to three scopes, and the+Backups page shows one notice line for the unread scopes across its postures.+`TestPVEUnreadStorageLeavesReadBackupsProtected` drives the production shape+(one enabled but unreachable PBS storage beside readable archives), a pruned+archive retained by the partial poll, and the complete recovery.+`TestBuildProtectionPosturePartialCollectionConfirmsOnlyReReadPoints`,+`TestStoreProtectionPostureKeepsReReadPointsUnderPartialCollection` and+`TestPBSMissingInventoryArrayRetainsUnreadScope` keep retained, legacy and+migration points out of protection.+### Coverage dates are inspectable without hoverOpen Coverage evidence retains each point’s exact reported date as wrappingdiff --git a/frontend-modern/browser-verification/0aabb658cb3ebf23.json b/frontend-modern/browser-verification/0aabb658cb3ebf23.jsonnew file mode 100644index 0000000000..dc91ab2757--- /dev/null+++ b/frontend-modern/browser-verification/0aabb658cb3ebf23.json@@ -0,0 +1,44 @@+{+ "version": 1,+ "base_sha": "598994dfc20f06bc293f676bbba438ee8cfbf0ed",+ "verified_at": "2026-11-01T16:38:23Z",+ "result": "passed",+ "changed_paths": [+ "frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx",+ "frontend-modern/src/features/proxmox/proxmoxBackupUnreadSources.ts",+ "frontend-modern/src/types/recovery.ts",+ "frontend-modern/src/utils/protectionPostureResponse.ts"+ ],+ "content_sha256": {+ "frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx": "784647117e6dfd2b5d536cd9e6abc427ac7c2957b3c3b5f768705d371ab20fba",+ "frontend-modern/src/features/proxmox/proxmoxBackupUnreadSources.ts": "e138b5b1f86f4f110be1a411d20f4cf55bebd225621157ed02d58b98b12f68a4",+ "frontend-modern/src/types/recovery.ts": "d52030f31023f078d65e40f3ba2433ef2289d94242e3feeb1e776fa8bab8c47c",+ "frontend-modern/src/utils/protectionPostureResponse.ts": "00aa34f7e724bb2daa7398855a1a9d6c66f3894818ddda66edfce93c0ba34c9c"+ },+ "routes": [+ "/proxmox/backups/coverage"+ ],+ "viewports": [+ {+ "width": 1440,+ "height": 1000+ },+ {+ "width": 375,+ "height": 812+ }+ ],+ "states": [+ "frames exported from the real PVE poll, recovery ingest and posture reader, replayed into the production Backups table (static build of browser-tests/backups.html)",+ "one enabled but unreachable PBS storage (pbs-energy-offsite on node-a) beside readable archives: 2 protected, one amber notice naming the storage",+ "node-b archive pruned while the storage stays unreadable: 1 protected, 1 attention, notice and guest-102 drawer explanation name the storage",+ "storage readable again: notice gone, complete coverage",+ "light theme only; the fixture does not apply the class-based dark theme"+ ],+ "interactions": [+ "loaded Coverage in Chromium 141 at 1440x1000 and WebKit 26 at 375x812, reloading per phase",+ "checked the notice text and role=status, the health-strip counts, and absence after recovery",+ "expanded and collapsed guest-102 by keyboard and matched its posture explanation",+ "asserted no page errors and no horizontal overflow in every phase; viewed screenshots"+ ]+}diff --git a/frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx b/frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsxindex 1c8b473b81..ba9be5aa35 100644--- a/frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx+++ b/frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx@@ -80,6 +80,10 @@ import {} from './proxmoxBackupsTableShared';import { getProxmoxBackupSourcePresentation } from './proxmoxBackupSourcePresentation';import { ProxmoxBackupsCoverageStrip } from './ProxmoxBackupsCoverageStrip';+import {+ collectUnreadBackupSources,+ describeUnreadBackupSources,+} from './proxmoxBackupUnreadSources';import { ProxmoxBackupServersTable } from './ProxmoxBackupServersTable';import { ProxmoxCoverageTable } from './ProxmoxCoverageTable';import { resolveBackupLocationFilter } from './proxmoxBackupLocation';@@ -195,6 +199,13 @@ export const ProxmoxBackupsTable: Component<{const protectionPostures = useProtectionPostures(() =>(props.workloads ?? []).map((workload) => workload.id),);+ // One unreadable storage limits every guest's coverage on its connection;+ // say which one once, where the operator can act on it.+ const unreadBackupSourcesNotice = createMemo(() =>+ describeUnreadBackupSources(+ collectUnreadBackupSources(protectionPostures.postureByResourceID().values()),+ ),+ );// Saved views persist the current route query, so every filter and the active// backup workspace must share one URL owner. View changes also clear the@@ -957,6 +968,14 @@ export const ProxmoxBackupsTable: Component<{]}/>+ <Show when={unreadBackupSourcesNotice()}>+ {(notice) => (+ <div role="status" class={BACKUP_READ_NOTICE_CLASS} data-unread-backup-sources>+ {notice()}+ </div>+ )}+ </Show>+<Show when={view() === 'date' && recoveryModel().recoverableArtifacts.length > 0}><Showwhen={scopedRecoverableArtifacts().some(diff --git a/frontend-modern/src/features/proxmox/__tests__/ProxmoxBackupsTable.test.tsx b/frontend-modern/src/features/proxmox/__tests__/ProxmoxBackupsTable.test.tsxindex e16f57b95b..7a7dce17fc 100644--- a/frontend-modern/src/features/proxmox/__tests__/ProxmoxBackupsTable.test.tsx+++ b/frontend-modern/src/features/proxmox/__tests__/ProxmoxBackupsTable.test.tsx@@ -565,6 +565,39 @@ describe('ProxmoxBackupsTable', () => {expect(postureURL.searchParams.getAll('resourceId')).toEqual(['ct-112']);});+ it('names a backup storage Pulse cannot read once, above the coverage list', async () => {+ mockBackupAPIs('protected');+ const response = await apiFetchJSONMock();+ response.data[0].coverage = 'partial';+ response.data[0].providerStates = [+ {+ provider: 'proxmox-pve',+ source: 'pve-backup-enumeration',+ scope: 'homelab',+ jobState: 'warning',+ historyCompleteness: 'partial',+ permissions: 'unknown',+ evidenceIds: ['evidence-2'],+ unreadScopes: [{ kind: 'storage', node: 'delly', storage: 'pbs-energy-offsite' }],+ },+ ];+ apiFetchJSONMock.mockResolvedValue(response);++ renderInRouter(() => (+ <ProxmoxBackupsTable+ emptyIcon={<span />}+ workloads={[workloadResource]}+ servers={[pbsServerResource]}+ />+ ));++ const notice = await screen.findByText(+ 'Pulse cannot read pbs-energy-offsite on delly, so it cannot confirm the backups stored there. Check that the storage is reachable from Proxmox.',+ );+ expect(notice).toHaveAttribute('role', 'status');+ expect(document.querySelectorAll('[data-unread-backup-sources]')).toHaveLength(1);+ });+it('offers By date / Coverage views and no legacy sub-tab tree', async () => {mockBackupAPIs();diff --git a/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupUnreadSources.test.ts b/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupUnreadSources.test.tsnew file mode 100644index 0000000000..d0d11401b8--- /dev/null+++ b/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupUnreadSources.test.ts@@ -0,0 +1,80 @@+import { describe, expect, it } from 'vitest';+import type { ProtectionPosture, ProtectionProviderState } from '@/types/recovery';+import { protectionPosture } from '@/utils/__fixtures__/protectionPosture';+import {+ collectUnreadBackupSources,+ describeUnreadBackupSources,+} from '../proxmoxBackupUnreadSources';++const withProvider = (+ id: string,+ provider: Partial<ProtectionProviderState>,+): ProtectionPosture => {+ const posture = protectionPosture(id);+ posture.providerStates = [{ ...posture.providerStates[0], ...provider }];+ return posture;+};++describe('unread backup sources', () => {+ it('collapses the same unread storage across guests into one actionable line', () => {+ const partial: Partial<ProtectionProviderState> = {+ provider: 'proxmox-pve',+ scope: 'homelab',+ historyCompleteness: 'partial',+ unreadScopes: [{ kind: 'storage', node: 'delly', storage: 'pbs-energy-offsite' }],+ };+ const sources = collectUnreadBackupSources([+ withProvider('vm:1', partial),+ withProvider('vm:2', partial),+ ]);+ expect(sources).toEqual([{ label: 'pbs-energy-offsite on delly', accessDenied: false }]);+ expect(describeUnreadBackupSources(sources)).toBe(+ 'Pulse cannot read pbs-energy-offsite on delly, so it cannot confirm the backups stored there. Check that the storage is reachable from Proxmox.',+ );+ });++ it('names connections only when the gap spans several, and reports denied access', () => {+ const sources = collectUnreadBackupSources([+ withProvider('vm:1', {+ provider: 'proxmox-pve',+ scope: 'east',+ historyCompleteness: 'partial',+ unreadScopes: [{ kind: 'node', node: 'pve2', accessDenied: true }],+ }),+ withProvider('vm:2', {+ provider: 'proxmox-pbs',+ scope: 'west',+ historyCompleteness: 'partial',+ unreadScopes: [+ { kind: 'namespace', storage: 'archive', namespace: '', accessDenied: true },+ ],+ }),+ ]);+ expect(sources.map((source) => source.label)).toEqual([+ 'the root namespace of datastore archive (west)',+ 'the storage list on node pve2 (east)',+ ]);+ expect(describeUnreadBackupSources(sources)).toMatch(+ /^Pulse does not have permission to read .* and .*\. Check the permissions of the Pulse API token\.$/,+ );+ });++ it('shows nothing for complete history or when no scope is named', () => {+ expect(+ collectUnreadBackupSources([+ protectionPosture('vm:1'),+ withProvider('vm:2', {+ historyCompleteness: 'complete',+ unreadScopes: [{ kind: 'storage', storage: 'stale' }],+ }),+ withProvider('vm:3', { historyCompleteness: 'partial' }),+ ]),+ ).toEqual([]);+ expect(describeUnreadBackupSources([])).toBeNull();+ });++ it('counts sources past the first three', () => {+ const sources = ['a', 'b', 'c', 'd', 'e'].map((label) => ({ label, accessDenied: false }));+ expect(describeUnreadBackupSources(sources)).toMatch(/^Pulse cannot read a, b, c and 2 more,/);+ });+});diff --git a/frontend-modern/src/features/proxmox/proxmoxBackupUnreadSources.ts b/frontend-modern/src/features/proxmox/proxmoxBackupUnreadSources.tsnew file mode 100644index 0000000000..78f29c3840--- /dev/null+++ b/frontend-modern/src/features/proxmox/proxmoxBackupUnreadSources.ts@@ -0,0 +1,88 @@+import type { ProtectionPosture, ProtectionUnreadScope } from '@/types/recovery';++// The server names what each provider's latest backup read could not reach.+// Every guest on that connection carries the same list, so collapse it to one+// page-level line: the operator fixes the storage, not each guest.++const scopeLabel = (scope: ProtectionUnreadScope): string => {+ const node = scope.node?.trim() ?? '';+ const storage = scope.storage?.trim() ?? '';+ const namespace = scope.namespace?.trim() ?? '';+ switch (scope.kind) {+ case 'node':+ return node ? `the storage list on node ${node}` : '';+ case 'storage':+ if (!storage) return '';+ return node ? `${storage} on ${node}` : storage;+ case 'datastore':+ return storage ? `the namespace list of datastore ${storage}` : '';+ case 'namespace':+ if (!storage) return '';+ return namespace+ ? `namespace ${namespace} on datastore ${storage}`+ : `the root namespace of datastore ${storage}`;+ default:+ return '';+ }+};++export interface UnreadBackupSource {+ label: string;+ accessDenied: boolean;+}++export const collectUnreadBackupSources = (+ postures: Iterable<ProtectionPosture>,+): UnreadBackupSource[] => {+ const byKey = new Map<string, UnreadBackupSource & { connection: string; base: string }>();+ for (const posture of postures) {+ for (const provider of posture.providerStates) {+ if (provider.historyCompleteness === 'complete') continue;+ for (const scope of provider.unreadScopes ?? []) {+ const base = scopeLabel(scope);+ if (!base) continue;+ const connection = provider.scope?.trim() ?? '';+ const key = `${provider.provider}\u0000${connection}\u0000${base}`;+ const existing = byKey.get(key);+ byKey.set(key, {+ label: base,+ base,+ connection,+ accessDenied: Boolean(scope.accessDenied) || Boolean(existing?.accessDenied),+ });+ }+ }+ }+ const entries = [...byKey.values()];+ // Name the connection only when the gap spans more than one of them.+ const connections = new Set(entries.map((entry) => entry.connection));+ return entries+ .map((entry) => ({+ label:+ connections.size > 1 && entry.connection+ ? `${entry.base} (${entry.connection})`+ : entry.base,+ accessDenied: entry.accessDenied,+ }))+ .sort((a, b) => a.label.localeCompare(b.label));+};++const NAMED = 3;++export const describeUnreadBackupSources = (+ sources: readonly UnreadBackupSource[],+): string | null => {+ if (sources.length === 0) return null;+ const names = sources.slice(0, NAMED).map((source) => source.label);+ if (sources.length > NAMED) names.push(`${sources.length - NAMED} more`);+ const list =+ names.length === 1+ ? names[0]+ : `${names.slice(0, -1).join(', ')} and ${names[names.length - 1]}`;+ const denied = sources.every((source) => source.accessDenied);+ const verb = denied ? 'Pulse does not have permission to read' : 'Pulse cannot read';+ const action = denied+ ? 'Check the permissions of the Pulse API token.'+ : 'Check that the storage is reachable from Proxmox.';+ return `${verb} ${list}, so it cannot confirm the backups stored there. ${action}`;+};diff --git a/frontend-modern/src/hooks/__tests__/useProtectionPostures.test.ts b/frontend-modern/src/hooks/__tests__/useProtectionPostures.test.tsindex 3854187077..147e8bbed4 100644--- a/frontend-modern/src/hooks/__tests__/useProtectionPostures.test.ts+++ b/frontend-modern/src/hooks/__tests__/useProtectionPostures.test.ts@@ -179,6 +179,36 @@ describe('useProtectionPostures transport', () => {},);+ it('keeps the unread provider scopes the server names and rejects a malformed one', async () => {+ const posture = protectionPosture('vm:1');+ posture.coverage = 'partial';+ posture.providerStates[0].historyCompleteness = 'partial';+ posture.providerStates[0].unreadScopes = [+ { kind: 'storage', node: 'delly', storage: 'pbs-energy-offsite' },+ ];+ apiFetchJSONMock.mockResolvedValue({ data: [posture], policy: protectionPolicy });+ let dispose = () => {};+ const hook = createRoot((rootDispose) => {+ dispose = rootDispose;+ return useProtectionPostures(() => ['vm:1']);+ });+ try {+ await vi.waitFor(() => expect(hook.postures()).toHaveLength(1));+ expect(hook.postureByResourceID().get('vm:1')?.providerStates[0].unreadScopes).toEqual([+ { kind: 'storage', node: 'delly', storage: 'pbs-energy-offsite' },+ ]);++ const malformed = structuredClone(posture);+ malformed.providerStates[0].unreadScopes = [{ kind: 'disk', storage: 'x' }] as never;+ apiFetchJSONMock.mockResolvedValue({ data: [malformed], policy: protectionPolicy });+ await hook.refetch();+ expect(hook.response.error).toBeInstanceOf(ProtectionPostureFormatError);+ expect(hook.postures()).toEqual([]);+ } finally {+ dispose();+ }+ });+it('does not restore a failed cached assessment when Coverage remounts', async () => {apiFetchJSONMock.mockResolvedValue({data: [protectionPosture('vm:1')],diff --git a/frontend-modern/src/types/recovery.ts b/frontend-modern/src/types/recovery.tsindex b112d2e10f..c08a3b26c2 100644--- a/frontend-modern/src/types/recovery.ts+++ b/frontend-modern/src/types/recovery.ts@@ -90,6 +90,18 @@ export type ProtectionVerification = 'verified' | 'unverified' | 'stale' | 'unknexport type ProtectionCoverage = 'complete' | 'partial' | 'none' | 'unknown';export type ProtectionHistoryCompleteness = 'complete' | 'partial' | 'unavailable' | 'unknown';+export type ProtectionUnreadScopeKind = 'node' | 'storage' | 'datastore' | 'namespace';++// One part of a provider's backup history that its latest read could not+// reach: a PVE node or storage, or a PBS datastore or namespace.+export interface ProtectionUnreadScope {+ kind: ProtectionUnreadScopeKind;+ node?: string;+ storage?: string;+ namespace?: string;+ accessDenied?: boolean;+}+export interface ProtectionProviderState {provider: RecoveryPlatform;source: string;@@ -102,6 +114,7 @@ export interface ProtectionProviderState {lastVerifiedAt?: string | null;evidenceIds: string[];verificationExpected?: boolean;+ unreadScopes?: ProtectionUnreadScope[];}export interface ProtectionPosture {diff --git a/frontend-modern/src/utils/protectionPostureResponse.ts b/frontend-modern/src/utils/protectionPostureResponse.tsindex 375a9c947f..9b99586cf6 100644--- a/frontend-modern/src/utils/protectionPostureResponse.ts+++ b/frontend-modern/src/utils/protectionPostureResponse.ts@@ -29,6 +29,19 @@ const choice =(value) =>typeof value === 'string' && values.includes(value);const idList: Check = (value) => value === null || (Array.isArray(value) && value.every(text));+const optionalText: Check = (value) => value === undefined || typeof value === 'string';+const unreadScope: Check = (value) =>+ value !== null &&+ typeof value === 'object' &&+ !Array.isArray(value) &&+ choice('node', 'storage', 'datastore', 'namespace')((value as Record<string, unknown>).kind) &&+ ['node', 'storage', 'namespace'].every((key) =>+ optionalText((value as Record<string, unknown>)[key]),+ ) &&+ ((value as Record<string, unknown>).accessDenied === undefined ||+ boolean((value as Record<string, unknown>).accessDenied));+const unreadScopeList: Check = (value) =>+ value == null || (Array.isArray(value) && value.every(unreadScope));function record(value: unknown, shape: Readonly<Record<string, Check>>): Record<string, unknown> {if (@@ -62,6 +75,7 @@ const providerShape = {lastVerifiedAt: optionalTime,evidenceIds: idList,verificationExpected: (value: unknown) => value === undefined || boolean(value),+ unreadScopes: unreadScopeList,};const postureShape = {subjectResourceId: text,diff --git a/internal/monitoring/backup_unread_scope_posture_test.go b/internal/monitoring/backup_unread_scope_posture_test.gonew file mode 100644index 0000000000..00650e26a1--- /dev/null+++ b/internal/monitoring/backup_unread_scope_posture_test.go@@ -0,0 +1,151 @@+package monitoring++import (+ "context"+ "fmt"+ "reflect"+ "strings"+ "testing"+ "time"++ "github.com/rcourtman/pulse-go-rewrite/internal/recovery"+ "github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"+)++// One enabled but unreachable PBS storage on one node, beside ordinary+// readable archive storage on every node: the shape that turned every guest on+// a production connection to attention.+type pveDeadOffsiteStorageClient struct {+ *pveReconciliationClient+ offsiteErr error+}++const deadOffsiteStorage = "pbs-energy-offsite"++func (c *pveDeadOffsiteStorageClient) GetStorage(ctx context.Context, node string) ([]proxmox.Storage, error) {+ rows, err := c.pveReconciliationClient.GetStorage(ctx, node)+ if err == nil && node == "node-a" {+ rows = append(rows, proxmox.Storage{Storage: deadOffsiteStorage, Type: "pbs", Content: "backup", Shared: 1, Active: 1, Enabled: 1})+ }+ return rows, err+}++func (c *pveDeadOffsiteStorageClient) GetStorageContent(ctx context.Context, node, storage string) ([]proxmox.StorageContent, error) {+ if storage == deadOffsiteStorage {+ if c.offsiteErr != nil {+ return nil, c.offsiteErr+ }+ return nil, nil+ }+ return c.pveReconciliationClient.GetStorageContent(ctx, node, storage)+}++func TestPVEUnreadStorageLeavesReadBackupsProtected(t *testing.T) {+ f := newPVEReconciliationFixture(t)+ c := &pveDeadOffsiteStorageClient{+ pveReconciliationClient: f.client,+ offsiteErr: fmt.Errorf("pbs-energy-offsite: error fetching datastores - 127.0.0.1:18007: connection refused"),+ }+ poll := func() {+ f.m.pollStorageBackupsWithNodes(context.Background(), "cluster", c, f.nodes, f.online)+ f.wait(t)+ }+ wantUnread := []recovery.ProtectionUnreadScope{{+ Kind: recovery.ProtectionUnreadScopeStorage, Node: "node-a", Storage: deadOffsiteStorage,+ }}+ assertNamed := func(p recovery.ProtectionPosture) {+ t.Helper()+ if len(p.ProviderStates) != 1 ||+ p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial ||+ !reflect.DeepEqual(p.ProviderStates[0].UnreadScopes, wantUnread) {+ t.Errorf("provider state lost the unread storage: %+v", p.ProviderStates)+ }+ if !strings.Contains(p.Explanation, "could not read "+deadOffsiteStorage+" on node-a") {+ t.Errorf("explanation does not name the unread storage: %q", p.Explanation)+ }+ }++ // Fresh backups on storages that read fine are protected; the dead+ // storage limits coverage and is named, but proves nothing about them.+ poll()+ for _, p := range f.postures(t) {+ if p.State != recovery.ProtectionStateProtected || p.Coverage != recovery.ProtectionCoveragePartial ||+ p.Freshness != recovery.ProtectionFreshnessCurrent {+ t.Errorf("read backup was not protected beside one unread storage: %+v", p)+ }+ assertNamed(p)+ }++ // Node-b's archive is pruned while the offsite storage is still dead. The+ // partial poll cannot delete the stored point, but it did not re-read it,+ // so that retained point no longer supports protection.+ retained := f.points(t)+ f.client.at = f.client.at.Add(15 * time.Minute)+ f.client.emptyB = true+ poll()+ points := f.points(t)+ if len(points) != len(retained) {+ t.Fatalf("partial poll deleted history: points=%d want %d", len(points), len(retained))+ }+ for _, p := range f.postures(t) {+ switch p.SubjectResourceID {+ case f.ids[0]:+ if p.State != recovery.ProtectionStateProtected {+ t.Errorf("re-read backup lost protection: %+v", p)+ }+ case f.ids[1]:+ if p.State == recovery.ProtectionStateProtected || p.Coverage != recovery.ProtectionCoveragePartial {+ t.Errorf("retained point that the latest poll did not read claimed protection: %+v", p)+ }+ }+ assertNamed(p)+ }++ // Once the storage reads again the collection is complete and reconciles+ // the pruned archive away; the unread names disappear with the gap.+ c.offsiteErr = nil+ poll()+ for _, p := range f.postures(t) {+ if len(p.ProviderStates) > 0 && len(p.ProviderStates[0].UnreadScopes) != 0 {+ t.Errorf("complete collection kept unread scopes: %+v", p.ProviderStates)+ }+ if p.SubjectResourceID == f.ids[0] &&+ (p.State != recovery.ProtectionStateProtected || p.Coverage != recovery.ProtectionCoverageComplete) {+ t.Errorf("complete read did not restore full protection: %+v", p)+ }+ if p.SubjectResourceID == f.ids[1] && p.State == recovery.ProtectionStateProtected {+ t.Errorf("confirmed-absent archive still protected its guest: %+v", p)+ }+ }+}++func TestPVEUnreadProtectionScopesNameWhatWentUnread(t *testing.T) {+ got := pveUnreadProtectionScopes(pveUnreadReads{+ nodes: map[string]struct{}{"node-c": {}},+ local: map[pveLocalBackupStorage]struct{}{{node: "node-b", storage: "local"}: {}},+ shared: map[string]struct{}{+ "offsite": {},+ "nfs": {},+ },+ sharedReads: map[pveLocalBackupStorage]struct{}{+ {node: "node-a", storage: "offsite"}: {},+ {node: "node-a", storage: "nfs"}: {},+ {node: "node-b", storage: "nfs"}: {},+ },+ denied: map[pveLocalBackupStorage]struct{}{+ {node: "node-c"}: {},+ {node: "node-b", storage: "nfs"}: {},+ {node: "node-a", storage: "unknown"}: {},+ },+ })+ want := []recovery.ProtectionUnreadScope{+ {Kind: recovery.ProtectionUnreadScopeNode, Node: "node-c", AccessDenied: true},+ {Kind: recovery.ProtectionUnreadScopeStorage, Node: "node-b", Storage: "local"},+ // Several nodes failed the same shared storage: name the storage only.+ {Kind: recovery.ProtectionUnreadScopeStorage, Storage: "nfs", AccessDenied: true},+ {Kind: recovery.ProtectionUnreadScopeStorage, Node: "node-a", Storage: "offsite"},+ }+ if !reflect.DeepEqual(got, want) {+ t.Fatalf("unread scopes = %+v\nwant %+v", got, want)+ }+}diff --git a/internal/monitoring/monitor_backups.go b/internal/monitoring/monitor_backups.goindex 72fa946c67..ef92183529 100644--- a/internal/monitoring/monitor_backups.go+++ b/internal/monitoring/monitor_backups.go@@ -281,6 +281,10 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceNamesharedStorageSuccess := map[string]struct{}{}unqueryableStorages := map[string]struct{}{}skippedLocalStorages := map[pveLocalBackupStorage]struct{}{}+ // Where unread reads were attempted, so the provider observation can name+ // them. A denied entry with no storage is a rejected node storage list.+ sharedUnreadReads := map[pveLocalBackupStorage]struct{}{}+ deniedReads := map[pveLocalBackupStorage]struct{}{}readState := m.backupReadStateForInstance(instanceName)// Build guest lookup map to find actual node for each VMID@@ -376,6 +380,7 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceNameif isPVEBackupPermissionError(err) {hadPermissionError = truepermissionFailureCount+++ deniedReads[pveLocalBackupStorage{node: node.Node}] = struct{}{}m.recordPVEBackupAccessWarning(ctx, instanceName, fmt.Sprintf("/nodes/%s/storage", node.Node), err)}monErr := errors.NewMonitorError(errors.ErrorTypeAPI, "get_storage_for_backups", instanceName, err).WithNode(node.Node)@@ -413,6 +418,7 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName} else {unqueryableStorages[storage.Storage] = struct{}{}storagePreserveNeeded[storage.Storage] = struct{}{}+ sharedUnreadReads[pveLocalBackupStorage{node: node.Node, storage: storage.Storage}] = struct{}{}}continue}@@ -433,6 +439,7 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceNameif isPVEBackupPermissionError(err) {hadPermissionError = truepermissionFailureCount+++ deniedReads[pveLocalBackupStorage{node: node.Node, storage: storage.Storage}] = struct{}{}m.recordPVEBackupAccessWarning(ctx, instanceName, fmt.Sprintf("/nodes/%s/storage/%s/content", node.Node, storage.Storage), err)} else {log.Debug().Err(monErr).@@ -444,6 +451,7 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceNamefailedLocalStorages[pveLocalBackupStorage{node: storageNode, storage: storage.Storage}] = struct{}{}} else {storagePreserveNeeded[storage.Storage] = struct{}{}+ sharedUnreadReads[pveLocalBackupStorage{node: node.Node, storage: storage.Storage}] = struct{}{}}contentFailures++continue@@ -659,6 +667,14 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceNamedelete(confirmationPreserveNeeded, storageName)}+ unreadScopes := pveUnreadProtectionScopes(pveUnreadReads{+ nodes: unreadNodes,+ local: failedLocalStorages,+ shared: storagePreserveNeeded,+ sharedReads: sharedUnreadReads,+ denied: deniedReads,+ })+// Carry-forward serves the live inventory, not a new provider observation.// Only rows actually returned this poll may renew point-level evidence.observedBackups := allBackups@@ -694,6 +710,7 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceNameContentSuccesses: contentSuccess,ContentFailures: contentFailures,PermissionFailureCount: permissionFailureCount,+ UnreadScopes: unreadScopes,},protectionObservedAt,)@@ -1897,6 +1914,13 @@ type pbsBackupFetchRequest struct {}// pollPBSBackups fetches all backups from PBS datastores+// pbsReadDenied reports an authorization rejection. Other terminal errors,+// such as a namespace that no longer exists, are not missing permission.+func pbsReadDenied(err error) bool {+ status, ok := pbs.HTTPStatus(err)+ return ok && (status == 401 || status == 403)+}+func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, client *pbs.Client, datastores []models.PBSDatastore, namespaceErrors map[string]error) {log.Debug().Str("instance", instanceName).Msg("polling PBS backups")@@ -1910,6 +1934,17 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clienqueriedNamespaces := make(map[pbsBackupNamespaceKey]struct{})transientNamespaces := make(map[pbsBackupNamespaceKey]struct{})transientDiscovery := make(map[string]struct{})+ // Names what went unread for the provider observation; it never decides+ // which points this poll confirmed.+ var unreadScopes []recovery.ProtectionUnreadScope+ unreadNamespace := func(datastore, namespace string, denied bool) {+ unreadScopes = append(unreadScopes, recovery.ProtectionUnreadScope{+ Kind: recovery.ProtectionUnreadScopeNamespace,+ Storage: datastore,+ Namespace: namespace,+ AccessDenied: denied,+ })+ }datastoreCount := len(datastores) // Number of datastores to querydatastoreFetches := 0 // Number of successful datastore fetchesdatastoreErrors := 0 // Datastores with incomplete history@@ -1940,6 +1975,11 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien} else {transientDiscovery[ds.Name] = struct{}{}}+ unreadScopes = append(unreadScopes, recovery.ProtectionUnreadScope{+ Kind: recovery.ProtectionUnreadScopeDatastore,+ Storage: ds.Name,+ AccessDenied: pbsReadDenied(err),+ })}groupsReused := 0groupsRequested := 0@@ -1954,6 +1994,7 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien}datastoreNamespaceErrors++transientNamespaces[namespaceKey] = struct{}{}+ unreadNamespace(ds.Name, namespace, false)continue}groups, err := client.ListBackupGroups(ctx, ds.Name, namespace)@@ -1969,6 +2010,7 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien} else {transientNamespaces[namespaceKey] = struct{}{}}+ unreadNamespace(ds.Name, namespace, pbsReadDenied(err))log.Error().Err(err).Str("instance", instanceName).@@ -1994,6 +2036,7 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien// Skipped groups are not a complete inventory, even when// every request we did make succeeded.datastoreNamespaceErrors+++ unreadNamespace(ds.Name, namespace, false)break}@@ -2067,6 +2110,10 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clienallBackups = appendPBSBackupsWithinLimit(allBackups, fetched.backups)datastoreNamespaceErrors += fetched.errorsdatastoreTerminalNamespaceErrors += fetched.terminalErrors+ if fetched.errors > 0 {+ // Snapshot failures carry no status here; do not call them denials.+ unreadNamespace(ds.Name, namespace, false)+ }// Stage successful read times until their matching rows are published.// A later cancellation must not freshen the previously published rows@@ -2217,6 +2264,8 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clienErr(protectionObservationErr).Str("instance", instanceName).Msg("Failed to build PBS protection provider observation")+ } else {+ protectionObservation = protectionObservation.WithUnreadScopes(unreadScopes)}// Decide whether to keep existing backups when all queries faileddiff --git a/internal/monitoring/monitor_backups_dir_storage_test.go b/internal/monitoring/monitor_backups_dir_storage_test.goindex 1890ab2c05..c08f8c29c3 100644--- a/internal/monitoring/monitor_backups_dir_storage_test.go+++ b/internal/monitoring/monitor_backups_dir_storage_test.go@@ -401,10 +401,12 @@ func TestPVEStorageRunningArchiveOriginLifecycle(t *testing.T) {if !last.Equal(f.client.at) || running != (phase != "completed") {t.Fatalf("phase=%s guest completion/running=%s/%t", phase, last, running)}+ // Node-a's completed archive is re-read in every phase, so it keeps+ // the guest protected; node-b's unread disk only limits coverage.posture := postures[0]if posture.LastSuccessfulPointAt == nil || !posture.LastSuccessfulPointAt.Equal(f.client.at) ||- (phase != "partial-read" && posture.State != recovery.ProtectionStateProtected) ||- (phase == "partial-read" && posture.State == recovery.ProtectionStateProtected) {+ posture.State != recovery.ProtectionStateProtected ||+ (phase == "partial-read") != (posture.Coverage == recovery.ProtectionCoveragePartial) {t.Fatalf("phase=%s wrong scoped protection: %+v", phase, posture)}frames[phase] = map[string]any{"pve": map[string]any{"data": snap.PVEBackups}, "postures": map[string]any{"data": postures, "policy": recovery.DefaultProtectionPosturePolicy.Payload()}, "vms": snap.VMs, "containers": snap.Containers}diff --git a/internal/monitoring/monitor_pbs_coverage_test.go b/internal/monitoring/monitor_pbs_coverage_test.goindex 1511d0c23b..bb7006bf59 100644--- a/internal/monitoring/monitor_pbs_coverage_test.go+++ b/internal/monitoring/monitor_pbs_coverage_test.go@@ -821,9 +821,19 @@ func TestPBSNamespaceDiscoveryFirstPollDoesNotInventScope(t *testing.T) {if len(f.points(t, "pbs-main")) != 1 || f.childReads.Load() != 0 {t.Fatal("failed first discovery invented a child namespace or lost readable root history")}+ // The root snapshot was read; only undiscovered namespaces are+ // unknown, so coverage is partial and names the datastore.for _, p := range f.postures(t) {- if p.State == recovery.ProtectionStateProtected {- t.Errorf("incomplete first discovery claimed current protection: %+v", p)+ if len(p.ProviderStates) == 0 {+ if p.State == recovery.ProtectionStateProtected {+ t.Errorf("guest without evidence claimed protection: %+v", p)+ }+ continue+ }+ if p.State != recovery.ProtectionStateProtected || p.Coverage != recovery.ProtectionCoveragePartial ||+ len(p.ProviderStates[0].UnreadScopes) != 1 ||+ p.ProviderStates[0].UnreadScopes[0].Kind != recovery.ProtectionUnreadScopeDatastore {+ t.Errorf("incomplete first discovery lost the read root or its partial coverage: %+v", p)}}})diff --git a/internal/monitoring/pbs_backup_budget_test.go b/internal/monitoring/pbs_backup_budget_test.goindex c3a8a867ce..c4ff1c9b90 100644--- a/internal/monitoring/pbs_backup_budget_test.go+++ b/internal/monitoring/pbs_backup_budget_test.go@@ -223,8 +223,14 @@ func exercisePBSBudgetAcceptedListings(t *testing.T) {}}for _, posture := range f.postures(t) {- if posture.State == recovery.ProtectionStateProtected || len(posture.ProviderStates) != 1 || posture.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || posture.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown {- t.Errorf("partial collection invented protection: %+v", posture)+ if len(posture.ProviderStates) != 1 || posture.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || posture.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown || posture.Coverage != recovery.ProtectionCoveragePartial {+ t.Errorf("partial collection invented complete history: %+v", posture)+ }+ // Only guest 100's snapshot was accepted before the budget+ // expired; at the tasks boundary it is still being written.+ wantProtected := posture.SubjectResourceID == f.ids[0] && boundary != "tasks"+ if (posture.State == recovery.ProtectionStateProtected) != wantProtected {+ t.Errorf("partial collection protection does not follow the accepted read (protected=%t): %+v", wantProtected, posture)}}frames["partial"] = f.frame(t)diff --git a/internal/monitoring/pbs_inventory_envelope_test.go b/internal/monitoring/pbs_inventory_envelope_test.goindex 0316b15094..2c256fe372 100644--- a/internal/monitoring/pbs_inventory_envelope_test.go+++ b/internal/monitoring/pbs_inventory_envelope_test.go@@ -85,10 +85,16 @@ func TestPBSMissingInventoryArrayRetainsUnreadScope(t *testing.T) {}continue}- if p.State == recovery.ProtectionStateProtected || len(p.ProviderStates) != 1 ||+ if len(p.ProviderStates) != 1 ||p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial ||- p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown {- t.Errorf("missing inventory invented complete protection or denied access: %+v", p)+ p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown ||+ p.Coverage != recovery.ProtectionCoveragePartial {+ t.Errorf("missing inventory invented complete history or denied access: %+v", p)+ }+ // The retained tenant point was not re-read and may have+ // been pruned since; the root snapshot was read this poll.+ if wantProtected := p.SubjectResourceID == f.ids[0]; (p.State == recovery.ProtectionStateProtected) != wantProtected {+ t.Errorf("protection does not follow what this poll read (protected=%t): %+v", wantProtected, p)}}}@@ -169,7 +175,10 @@ func TestPBSMissingInventoryArrayConsumerLifecycle(t *testing.T) {}postures := f.postures(t)for _, p := range postures {- healthy := phase == "observed" || phase == "recovered"+ // Incomplete phases still re-read the root snapshot; the retained+ // tenant point does not count until a read confirms it again.+ incomplete := phase == "missing-namespaces" || phase == "null-groups" || phase == "null-snapshots"+ healthy := phase == "observed" || phase == "recovered" || (incomplete && p.SubjectResourceID == f.ids[0])if healthy != (p.State == recovery.ProtectionStateProtected) {t.Fatalf("phase %s posture=%+v", phase, p)}diff --git a/internal/monitoring/pbs_inventory_response_test.go b/internal/monitoring/pbs_inventory_response_test.goindex 1d29d37c3a..8cba8fdea7 100644--- a/internal/monitoring/pbs_inventory_response_test.go+++ b/internal/monitoring/pbs_inventory_response_test.go@@ -67,8 +67,12 @@ func TestPBSWholeInventoryResponseRetainsEvidence(t *testing.T) {}continue}- if p.State == recovery.ProtectionStateProtected || len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown {- t.Errorf("unread response claimed protection or denied access: %+v", p)+ if len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown || p.Coverage != recovery.ProtectionCoveragePartial {+ t.Errorf("unread response claimed complete history or denied access: %+v", p)+ }+ // Only the root snapshot was re-read this poll.+ if wantProtected := p.SubjectResourceID == f.ids[0]; (p.State == recovery.ProtectionStateProtected) != wantProtected {+ t.Errorf("unread response protection does not follow what was read (protected=%t): %+v", wantProtected, p)}}}@@ -125,7 +129,10 @@ func TestPBSWholeInventoryConsumerLifecycle(t *testing.T) {}for _, p := range f.postures(t) {if p.SubjectResourceID == f.ids[1] || p.SubjectResourceID == f.ids[0] {- if (phase == "observed" || phase == "recovered") != (p.State == recovery.ProtectionStateProtected) {+ // Unread phases still re-read the root snapshot.+ unread := phase == "unread-namespaces" || phase == "unread-groups" || phase == "unread-snapshots"+ healthy := phase == "observed" || phase == "recovered" || (unread && p.SubjectResourceID == f.ids[0])+ if healthy != (p.State == recovery.ProtectionStateProtected) {t.Fatalf("phase %s posture=%+v", phase, p)}}diff --git a/internal/monitoring/pbs_namespace_continuity_test.go b/internal/monitoring/pbs_namespace_continuity_test.goindex d3cdac5b5d..8af05dacc9 100644--- a/internal/monitoring/pbs_namespace_continuity_test.go+++ b/internal/monitoring/pbs_namespace_continuity_test.go@@ -167,9 +167,18 @@ func TestPBSNamespaceTransientGroupReadKeepsLiveInventory(t *testing.T) {}}}+ // The accepted sibling was re-read; the failed namespace's+ // retained point was not and cannot count.+ acceptedID := f.ids[0]+ if accepted == "101" {+ acceptedID = f.ids[1]+ }for _, p := range f.postures(t) {- if p.State == recovery.ProtectionStateProtected || len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown {- t.Errorf("incomplete history claimed current protection: %+v", p)+ if len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown || p.Coverage != recovery.ProtectionCoveragePartial {+ t.Errorf("incomplete history claimed complete coverage: %+v", p)+ }+ if (p.State == recovery.ProtectionStateProtected) != (p.SubjectResourceID == acceptedID) {+ t.Errorf("protection does not follow the re-read namespace: %+v", p)}}}@@ -224,9 +233,11 @@ func TestPBSNamespaceDiscoveryContinuityAndAccessWithdrawal(t *testing.T) {if f.tenantReads.Load() != childReads {t.Error("discovery failure guessed a hidden namespace read")}+ // The root namespace was still read; only the tenant is unread.for _, p := range f.postures(t) {- if p.State == recovery.ProtectionStateProtected {- t.Errorf("failed discovery claimed protection: %+v", p)+ if (p.State == recovery.ProtectionStateProtected) != (p.SubjectResourceID == f.ids[0]) ||+ (p.State == recovery.ProtectionStateProtected && p.Coverage != recovery.ProtectionCoveragePartial) {+ t.Errorf("failed discovery protection does not follow the read root: %+v", p)}}}@@ -369,7 +380,10 @@ func TestPBSNamespaceContinuityLifecycle(t *testing.T) {}postures := f.postures(t)for _, p := range postures {- healthy := phase == "observed" || phase == "recovered" || phase == "restored"+ // Tenant failures leave the root snapshot re-read each poll.+ tenantUnread := phase == "throttled" || phase == "timed-out" || phase == "denied"+ healthy := phase == "observed" || phase == "recovered" || phase == "restored" ||+ (tenantUnread && p.SubjectResourceID == f.ids[0])if healthy != (p.State == recovery.ProtectionStateProtected) {t.Fatalf("phase %s posture=%+v", phase, p)}diff --git a/internal/monitoring/pbs_poll_posture_test.go b/internal/monitoring/pbs_poll_posture_test.goindex d5eb3d2858..e432799791 100644--- a/internal/monitoring/pbs_poll_posture_test.go+++ b/internal/monitoring/pbs_poll_posture_test.go@@ -6,6 +6,7 @@ import ("fmt""net/http""net/http/httptest"+ "reflect""strconv""strings""sync/atomic"@@ -283,8 +284,14 @@ func TestPBSOrdinaryPollSnapshotFailuresDoNotClaimCompleteHistory(t *testing.T)t.Errorf("raw snapshot rows=%d, want %d", got, wantRaw)}for _, p := range f.postures(t) {- if p.State != tc.state || len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || p.ProviderStates[0].Permissions != tc.permissions || p.ProviderStates[0].JobState != recovery.OutcomeWarning {- t.Errorf("snapshot failure posture=%+v, want %s/partial/%s/warning", p, tc.state, tc.permissions)+ // Only guest 100's snapshot read fails; guest 101's snapshot+ // is re-read and stays protected under partial coverage.+ state := tc.state+ if p.SubjectResourceID == f.ids[1] {+ state = recovery.ProtectionStateProtected+ }+ if p.State != state || p.Coverage != recovery.ProtectionCoveragePartial || len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || p.ProviderStates[0].Permissions != tc.permissions || p.ProviderStates[0].JobState != recovery.OutcomeWarning {+ t.Errorf("snapshot failure posture=%+v, want %s/partial/%s/warning", p, state, tc.permissions)}}// No TTL manipulation on the next poll: a failed refresh must not@@ -309,8 +316,11 @@ func TestPBSOrdinaryPollPartialNamespaceRetainsUnseenHistory(t *testing.T) {if len(f.points(t, "pbs-main")) != 4 {t.Error("partial namespace poll deleted unseen historical points")}+ // Both guests' root snapshots were re-read; only the unreadable+ // namespace's retained points went unconfirmed, and it is named.for _, p := range f.postures(t) {- if p.State != recovery.ProtectionStateAttention || len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsPartial {+ if p.State != recovery.ProtectionStateProtected || p.Coverage != recovery.ProtectionCoveragePartial || len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsPartial ||+ !reflect.DeepEqual(p.ProviderStates[0].UnreadScopes, []recovery.ProtectionUnreadScope{{Kind: recovery.ProtectionUnreadScopeNamespace, Storage: "archive", Namespace: "unreadable", AccessDenied: true}}) {t.Errorf("partial namespace posture=%+v", p)}}diff --git a/internal/monitoring/pbs_protection_observation_test.go b/internal/monitoring/pbs_protection_observation_test.goindex fc6ab19347..be8ba6058b 100644--- a/internal/monitoring/pbs_protection_observation_test.go+++ b/internal/monitoring/pbs_protection_observation_test.go@@ -527,10 +527,15 @@ func TestPBSNamespaceDiscoveryFailureRetainsRecoveryHistory(t *testing.T) {permissions = operationaltrust.EvidencePermissionsPartial}for _, p := range f.postures(t) {- if p.State == recovery.ProtectionStateProtected || len(p.ProviderStates) != 1 ||+ if len(p.ProviderStates) != 1 || p.Coverage != recovery.ProtectionCoveragePartial ||p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial ||p.ProviderStates[0].Permissions != permissions || p.ProviderStates[0].JobState != recovery.OutcomeWarning {- t.Errorf("discovery failure presented as complete/healthy: %+v", p)+ t.Errorf("discovery failure presented as complete history: %+v", p)+ }+ // The fresh root archive was read; the child guest's+ // retained point was not.+ if (p.State == recovery.ProtectionStateProtected) != (p.SubjectResourceID == f.ids[0]) {+ t.Errorf("discovery failure protection does not follow the read root: %+v", p)}}if f.childReads.Load() != reads {diff --git a/internal/monitoring/pbs_retryable_read_test.go b/internal/monitoring/pbs_retryable_read_test.goindex 3807a1f2e1..8aa01fda5f 100644--- a/internal/monitoring/pbs_retryable_read_test.go+++ b/internal/monitoring/pbs_retryable_read_test.go@@ -54,10 +54,14 @@ func TestPBSRetryableReadLifecycle(t *testing.T) {}}for _, p := range f.postures(t) {- if p.State == recovery.ProtectionStateProtected || len(p.ProviderStates) != 1 ||+ if len(p.ProviderStates) != 1 || p.Coverage != recovery.ProtectionCoveragePartial ||p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown ||p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial {- t.Errorf("temporary failure invented protection or denied access: %+v", p)+ t.Errorf("temporary failure invented complete history or denied access: %+v", p)+ }+ // The readable root sibling advanced; the tenant did not.+ if (p.State == recovery.ProtectionStateProtected) != (p.SubjectResourceID == f.ids[0]) {+ t.Errorf("temporary failure protection does not follow the read sibling: %+v", p)}}}diff --git a/internal/monitoring/pbs_snapshot_identity_test.go b/internal/monitoring/pbs_snapshot_identity_test.goindex 33b5a13235..944ee8ef32 100644--- a/internal/monitoring/pbs_snapshot_identity_test.go+++ b/internal/monitoring/pbs_snapshot_identity_test.go@@ -85,8 +85,9 @@ func TestPBSConflictingSnapshotIdentityRetainsUnreadScope(t *testing.T) {}}for _, p := range f.postures(t) {- if p.State == recovery.ProtectionStateProtected {- t.Errorf("invalid group advertised protected: %+v", p)+ // The root sibling was read; the malformed tenant group was not.+ if (p.State == recovery.ProtectionStateProtected) != (p.SubjectResourceID == f.ids[0]) {+ t.Errorf("invalid group protection does not follow the read root: %+v", p)}for _, provider := range p.ProviderStates {if provider.HistoryCompleteness != recovery.ProtectionHistoryPartial || provider.Permissions != operationaltrust.EvidencePermissionsUnknown {diff --git a/internal/monitoring/pve_backup_reconciliation_test.go b/internal/monitoring/pve_backup_reconciliation_test.goindex c17764df0b..a60d61b82d 100644--- a/internal/monitoring/pve_backup_reconciliation_test.go+++ b/internal/monitoring/pve_backup_reconciliation_test.go@@ -146,8 +146,18 @@ func (f *pveReconciliationFixture) assertHistory(t *testing.T, history recovery.if len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != history || p.ProviderStates[0].Permissions != permissions {t.Errorf("posture lost source-bound history/access evidence: %+v", p)}- if history != recovery.ProtectionHistoryComplete && p.State == recovery.ProtectionStateProtected {- t.Errorf("incomplete collection claimed protection: %+v", p)+ if history == recovery.ProtectionHistoryComplete || p.State != recovery.ProtectionStateProtected {+ continue+ }+ // An incomplete poll still confirms the archives it re-read, so a guest+ // whose backup it read may be protected, but only with partial coverage.+ // In two-guest fixtures node-b's guest is the unread one: its retained+ // point was not re-read and may have been pruned, so it never counts.+ // Single-guest fixtures also keep a re-read archive on node-a.+ unreadGuest := len(f.ids) > 1 && p.SubjectResourceID == f.ids[1]+ if history != recovery.ProtectionHistoryPartial || unreadGuest ||+ p.Coverage != recovery.ProtectionCoveragePartial {+ t.Errorf("incomplete collection claimed protection from unread history: %+v", p)}}}diff --git a/internal/monitoring/pve_local_archive_identity_test.go b/internal/monitoring/pve_local_archive_identity_test.goindex eefa582539..04da0cdf2d 100644--- a/internal/monitoring/pve_local_archive_identity_test.go+++ b/internal/monitoring/pve_local_archive_identity_test.go@@ -269,11 +269,16 @@ func TestPVELocalArchiveIdentityConsumerFrames(t *testing.T) {t.Fatalf("phase=%s live=%d history=%d want%d", phase, len(rows.StorageBackups), len(points), want)}for _, p := range postures {- if (phase == "observed" || phase == "recovered" || phase == "one-removed") && p.State != recovery.ProtectionStateProtected {+ // In the partial phase node-a's same-path archive is still+ // re-read, so the guest stays protected with partial coverage.+ if (phase == "observed" || phase == "recovered" || phase == "one-removed" || phase == "partial") && p.State != recovery.ProtectionStateProtected {t.Fatalf("accepted completed archives lost posture: %+v", p)}- if (phase == "partial" || phase == "empty") && p.State == recovery.ProtectionStateProtected {- t.Fatal("unread or absent archives claimed protection")+ if (phase == "partial") != (p.Coverage == recovery.ProtectionCoveragePartial) {+ t.Fatalf("phase=%s coverage does not reflect the unread disk: %+v", phase, p)+ }+ if phase == "empty" && p.State == recovery.ProtectionStateProtected {+ t.Fatal("absent archives claimed protection")}}frames[phase] = map[string]any{"pve": map[string]any{"data": rows}, "postures": map[string]any{"data": postures, "policy": recovery.DefaultProtectionPosturePolicy.Payload()}, "points": points}diff --git a/internal/monitoring/pve_local_storage_retention_test.go b/internal/monitoring/pve_local_storage_retention_test.goindex 8487b9d156..917712add7 100644--- a/internal/monitoring/pve_local_storage_retention_test.go+++ b/internal/monitoring/pve_local_storage_retention_test.go@@ -217,12 +217,15 @@ func TestPVELocalStorageAcceptedReadLifecycle(t *testing.T) {t.Fatalf("phase %s live/recorded inventory = %d/%d, want %d", phase, len(rows.StorageBackups), len(points), want)}for _, p := range postures {- if phase == "observed" || phase == "recovered" {+ // Node-a's guest keeps a re-read backup through the partial phase;+ // node-b's retained one was not re-read.+ readA := phase == "partial" && p.SubjectResourceID == f.ids[0]+ if phase == "observed" || phase == "recovered" || readA {if p.State != recovery.ProtectionStateProtected {t.Fatalf("phase %s accepted completed backup is not protected: %+v", phase, p)}} else if p.State == recovery.ProtectionStateProtected {- t.Fatalf("phase %s claimed protection without complete current backup evidence", phase)+ t.Fatalf("phase %s claimed protection without a current backup read: %+v", phase, p)}}frames[phase] = map[string]any{"pve": map[string]any{"data": rows}, "postures": map[string]any{"data": postures, "policy": recovery.DefaultProtectionPosturePolicy.Payload()}, "points": points}diff --git a/internal/monitoring/pve_protection_observation.go b/internal/monitoring/pve_protection_observation.goindex 2f28eabf28..f6909bb816 100644--- a/internal/monitoring/pve_protection_observation.go+++ b/internal/monitoring/pve_protection_observation.go@@ -16,6 +16,61 @@ type pveProtectionCollectionStats struct {ContentSuccesses intContentFailures intPermissionFailureCount int+ UnreadScopes []recovery.ProtectionUnreadScope+}++// pveUnreadReads is the unread part of one PVE backup enumeration.+type pveUnreadReads struct {+ nodes map[string]struct{} // nodes whose storage list was not read+ local map[pveLocalBackupStorage]struct{} // node-local storages not read+ shared map[string]struct{} // shared storages no node read+ sharedReads map[pveLocalBackupStorage]struct{} // nodes a shared read failed on+ denied map[pveLocalBackupStorage]struct{} // reads rejected by authorization+}++// pveUnreadProtectionScopes names what a PVE enumeration could not read. A+// shared storage names its node only when exactly one node attempted it.+func pveUnreadProtectionScopes(unread pveUnreadReads) []recovery.ProtectionUnreadScope {+ var scopes []recovery.ProtectionUnreadScope+ for node := range unread.nodes {+ _, denied := unread.denied[pveLocalBackupStorage{node: node}]+ scopes = append(scopes, recovery.ProtectionUnreadScope{+ Kind: recovery.ProtectionUnreadScopeNode,+ Node: node,+ AccessDenied: denied,+ })+ }+ for local := range unread.local {+ _, denied := unread.denied[local]+ scopes = append(scopes, recovery.ProtectionUnreadScope{+ Kind: recovery.ProtectionUnreadScopeStorage,+ Node: local.node,+ Storage: local.storage,+ AccessDenied: denied,+ })+ }+ for storage := range unread.shared {+ scope := recovery.ProtectionUnreadScope{+ Kind: recovery.ProtectionUnreadScopeStorage,+ Storage: storage,+ }+ attempts := 0+ for read := range unread.sharedReads {+ if read.storage != storage {+ continue+ }+ attempts+++ scope.Node = read.node+ if _, denied := unread.denied[read]; denied {+ scope.AccessDenied = true+ }+ }+ if attempts != 1 {+ scope.Node = ""+ }+ scopes = append(scopes, scope)+ }+ return recovery.NormalizeProtectionUnreadScopes(scopes)}func buildPVEProtectionProviderObservation(@@ -89,7 +144,7 @@ func buildPVEProtectionProviderObservation(}}- return recovery.NewProtectionProviderObservation(+ observation, err := recovery.NewProtectionProviderObservation(recovery.ProviderProxmoxPVE,"pve-backup-enumeration",instanceName,@@ -101,6 +156,10 @@ func buildPVEProtectionProviderObservation(observedAt,reason,)+ if err != nil {+ return recovery.ProtectionProviderObservation{}, err+ }+ return observation.WithUnreadScopes(stats.UnreadScopes), nil}func isPVEBackupPermissionError(err error) bool {diff --git a/internal/recovery/model/posture.go b/internal/recovery/model/posture.goindex bbb6347db6..0c1ab056a4 100644--- a/internal/recovery/model/posture.go+++ b/internal/recovery/model/posture.go@@ -113,6 +113,105 @@ func (completeness ProtectionHistoryCompleteness) Valid() bool {}}+// ProtectionUnreadScopeKind names the level of provider history that a+// collection could not read.+type ProtectionUnreadScopeKind string++const (+ // A PVE node whose storage list could not be read.+ ProtectionUnreadScopeNode ProtectionUnreadScopeKind = "node"+ // A PVE storage whose content could not be read. Node is set when the+ // failed read was made through, or belongs to, one node.+ ProtectionUnreadScopeStorage ProtectionUnreadScopeKind = "storage"+ // A PBS datastore whose namespaces could not all be discovered.+ ProtectionUnreadScopeDatastore ProtectionUnreadScopeKind = "datastore"+ // A PBS namespace (Namespace "" is the root) whose groups or snapshots+ // could not all be read.+ ProtectionUnreadScopeNamespace ProtectionUnreadScopeKind = "namespace"+)++// MaxProtectionUnreadScopes bounds the unread scopes kept on one observation.+// The list explains a partial collection; it is not an inventory.+const MaxProtectionUnreadScopes = 32++// ProtectionUnreadScope names one part of a provider's backup history that its+// latest collection could not read. Recovery points stored from that part are+// retained evidence, not current confirmation.+type ProtectionUnreadScope struct {+ Kind ProtectionUnreadScopeKind `json:"kind"`+ Node string `json:"node,omitempty"`+ Storage string `json:"storage,omitempty"`+ Namespace string `json:"namespace,omitempty"`+ AccessDenied bool `json:"accessDenied,omitempty"`+}++func (scope ProtectionUnreadScope) normalize() (ProtectionUnreadScope, bool) {+ scope.Node = strings.TrimSpace(scope.Node)+ scope.Storage = strings.TrimSpace(scope.Storage)+ scope.Namespace = strings.TrimSpace(scope.Namespace)+ switch scope.Kind {+ case ProtectionUnreadScopeNode:+ scope.Storage, scope.Namespace = "", ""+ return scope, scope.Node != ""+ case ProtectionUnreadScopeStorage:+ scope.Namespace = ""+ return scope, scope.Storage != ""+ case ProtectionUnreadScopeDatastore:+ scope.Node, scope.Namespace = "", ""+ return scope, scope.Storage != ""+ case ProtectionUnreadScopeNamespace:+ scope.Node = ""+ return scope, scope.Storage != ""+ default:+ return scope, false+ }+}++func (scope ProtectionUnreadScope) sortKey() string {+ return strings.Join([]string{+ string(scope.Kind), scope.Storage, scope.Namespace, scope.Node,+ }, "\x00")+}++// NormalizeProtectionUnreadScopes drops invalid entries, merges duplicates+// (a denied read wins), sorts, and caps the list.+func NormalizeProtectionUnreadScopes(scopes []ProtectionUnreadScope) []ProtectionUnreadScope {+ if len(scopes) == 0 {+ return nil+ }+ byKey := make(map[string]ProtectionUnreadScope, len(scopes))+ for _, scope := range scopes {+ normalized, ok := scope.normalize()+ if !ok {+ continue+ }+ key := normalized.sortKey()+ if existing, seen := byKey[key]; seen {+ normalized.AccessDenied = normalized.AccessDenied || existing.AccessDenied+ }+ byKey[key] = normalized+ }+ if len(byKey) == 0 {+ return nil+ }+ out := make([]ProtectionUnreadScope, 0, len(byKey))+ for _, scope := range byKey {+ out = append(out, scope)+ }+ sort.Slice(out, func(i, j int) bool { return out[i].sortKey() < out[j].sortKey() })+ if len(out) > MaxProtectionUnreadScopes {+ out = out[:MaxProtectionUnreadScopes]+ }+ return out+}++func cloneProtectionUnreadScopes(scopes []ProtectionUnreadScope) []ProtectionUnreadScope {+ if len(scopes) == 0 {+ return nil+ }+ return append([]ProtectionUnreadScope(nil), scopes...)+}+type ProtectionProviderState struct {Provider Provider `json:"provider"`Source string `json:"source"`@@ -125,6 +224,9 @@ type ProtectionProviderState struct {LastVerifiedAt *time.Time `json:"lastVerifiedAt,omitempty"`EvidenceIDs []string `json:"evidenceIds"`VerificationExpected bool `json:"verificationExpected,omitempty"`+ // UnreadScopes names what the provider's latest collection could not+ // read. Only a partial or unavailable history carries it.+ UnreadScopes []ProtectionUnreadScope `json:"unreadScopes,omitempty"`}func (state ProtectionProviderState) Clone() ProtectionProviderState {@@ -133,6 +235,7 @@ func (state ProtectionProviderState) Clone() ProtectionProviderState {clone.LastSuccessAt = cloneTime(state.LastSuccessAt)clone.LastVerifiedAt = cloneTime(state.LastVerifiedAt)clone.EvidenceIDs = append([]string(nil), state.EvidenceIDs...)+ clone.UnreadScopes = cloneProtectionUnreadScopes(state.UnreadScopes)return clone}@@ -248,11 +351,28 @@ type ProtectionProviderObservation struct {ObservedAt time.Time `json:"observedAt"`IngestedAt time.Time `json:"ingestedAt"`Evidence operationaltrust.EvidenceEnvelope `json:"evidence"`+ // UnreadScopes names the parts of a partial or unavailable collection that+ // went unread. It explains the gap; point confirmation does not depend on it.+ UnreadScopes []ProtectionUnreadScope `json:"unreadScopes,omitempty"`}func (observation ProtectionProviderObservation) Clone() ProtectionProviderObservation {clone := observationclone.Evidence = observation.Evidence.Clone()+ clone.UnreadScopes = cloneProtectionUnreadScopes(observation.UnreadScopes)+ return clone+}++// WithUnreadScopes returns the observation carrying the normalized unread+// scopes. A complete collection read everything, so it carries none.+func (observation ProtectionProviderObservation) WithUnreadScopes(+ scopes []ProtectionUnreadScope,+) ProtectionProviderObservation {+ clone := observation.Clone()+ clone.UnreadScopes = nil+ if observation.HistoryCompleteness != ProtectionHistoryComplete {+ clone.UnreadScopes = NormalizeProtectionUnreadScopes(scopes)+ }return clone}@@ -343,20 +463,27 @@ type ProtectionPostureQuery struct {}type ProtectionProviderSummary struct {- Provider Provider- Source string- Scope string- JobState Outcome- HistoryCompleteness ProtectionHistoryCompleteness- Permissions operationaltrust.EvidencePermissions- VerificationExpected bool- LastAttemptAt *time.Time- LastSuccessAt *time.Time- LastVerifiedAt *time.Time- BackupPointCount int- SnapshotPointCount int- RepositoryResourceIDs []string- EvidenceIDs []string+ Provider Provider+ Source string+ Scope string+ JobState Outcome+ HistoryCompleteness ProtectionHistoryCompleteness+ Permissions operationaltrust.EvidencePermissions+ VerificationExpected bool+ LastAttemptAt *time.Time+ LastSuccessAt *time.Time+ LastVerifiedAt *time.Time+ // ConfirmedLastSuccessAt and ConfirmedLastVerifiedAt cover only the points+ // that the provider's latest collection re-read. A partial collection can+ // still confirm those points; retained points from its unread scopes may+ // have been pruned since and stay out of these fields.+ ConfirmedLastSuccessAt *time.Time+ ConfirmedLastVerifiedAt *time.Time+ UnreadScopes []ProtectionUnreadScope+ BackupPointCount int+ SnapshotPointCount int+ RepositoryResourceIDs []string+ EvidenceIDs []string}func (summary ProtectionProviderSummary) normalize() ProtectionProviderSummary {diff --git a/internal/recovery/posture.go b/internal/recovery/posture.goindex bd6559d181..87ec92913b 100644--- a/internal/recovery/posture.go+++ b/internal/recovery/posture.go@@ -310,19 +310,27 @@ func BuildProtectionPostureFromPointsAt(switch point.Kind {case KindBackup:agg.summary.BackupPointCount+++ observation, observed := latestObservations[key]+ confirmed := observed && pointConfirmedByObservation(point, observation)if point.Outcome == OutcomeSuccess && !eventAt.IsZero() {- if agg.summary.LastSuccessAt == nil || eventAt.After(*agg.summary.LastSuccessAt) {- value := eventAt.UTC()- agg.summary.LastSuccessAt = &value+ agg.summary.LastSuccessAt = latestProtectionTime(agg.summary.LastSuccessAt, &eventAt)+ if confirmed {+ agg.summary.ConfirmedLastSuccessAt = latestProtectionTime(+ agg.summary.ConfirmedLastSuccessAt,+ &eventAt,+ )}}if point.Outcome == OutcomeSuccess &&point.Verified != nil &&*point.Verified &&!eventAt.IsZero() {- if agg.summary.LastVerifiedAt == nil || eventAt.After(*agg.summary.LastVerifiedAt) {- value := eventAt.UTC()- agg.summary.LastVerifiedAt = &value+ agg.summary.LastVerifiedAt = latestProtectionTime(agg.summary.LastVerifiedAt, &eventAt)+ if confirmed {+ agg.summary.ConfirmedLastVerifiedAt = latestProtectionTime(+ agg.summary.ConfirmedLastVerifiedAt,+ &eventAt,+ )}}case KindSnapshot:@@ -351,6 +359,7 @@ func BuildProtectionPostureFromPointsAt(agg.summary.HistoryCompleteness = observation.HistoryCompletenessagg.summary.Permissions = observation.Permissionsagg.summary.VerificationExpected = observation.VerificationExpected+ agg.summary.UnreadScopes = cloneUnreadScopes(observation.UnreadScopes)agg.summary.EvidenceIDs = append(agg.summary.EvidenceIDs, observation.Evidence.ID)if !observation.ObservedAt.Before(agg.lastEventAt) &&observation.JobState != OutcomeUnknown {@@ -367,6 +376,31 @@ func BuildProtectionPostureFromPointsAt(return DeriveProtectionPostureAt(subjectResourceID, summaries, policy, now)}+// recoveryPointMigrationCollector marks evidence the store backfilled for+// legacy points. It was never produced by a provider read.+const recoveryPointMigrationCollector = "recovery-point-migration"++// pointConfirmedByObservation reports whether the provider collection behind+// observation re-read this point. Collectors renew a point's evidence only for+// rows a poll actually returned, stamped with that poll's observation time, so+// evidence ingested no earlier than the observation proves the point still+// existed on a scope that collection read. A point retained from an unread+// scope keeps its older ingestion time and may have been pruned since.+func pointConfirmedByObservation(+ point RecoveryPoint,+ observation ProtectionProviderObservation,+) bool {+ evidence := point.Evidence+ if evidence == nil || evidence.IngestedAt.IsZero() {+ return false+ }+ if strings.TrimSpace(evidence.Source.Collector) == recoveryPointMigrationCollector ||+ strings.TrimSpace(evidence.Source.Provider) != strings.TrimSpace(string(observation.Provider)) {+ return false+ }+ return !evidence.IngestedAt.Before(observation.ObservedAt)+}+func DeriveProtectionPostureAt(subjectResourceID string,summaries []ProtectionProviderSummary,@@ -404,6 +438,7 @@ func DeriveProtectionPostureAt(hasSupportedQualifyingSuccess := falsehasInvalidatingFailure := falsehasSnapshotsOnly := false+ var unreadScopes []ProtectionUnreadScopeverificationExpected := policy.RequireVerificationsupportedVerificationExpected := policy.RequireVerificationvar supportedLastSuccessAt *time.Time@@ -443,6 +478,10 @@ func DeriveProtectionPostureAt(EvidenceIDs: append([]string(nil), summary.EvidenceIDs...),VerificationExpected: summary.VerificationExpected,}+ if summary.HistoryCompleteness != ProtectionHistoryComplete {+ state.UnreadScopes = NormalizeProtectionUnreadScopes(summary.UnreadScopes)+ unreadScopes = append(unreadScopes, state.UnreadScopes...)+ }posture.ProviderStates = append(posture.ProviderStates, state)posture.RepositoryResourceIDs = append(posture.RepositoryResourceIDs,@@ -483,17 +522,15 @@ func DeriveProtectionPostureAt(if summary.LastSuccessAt != nil {hasQualifyingSuccess = true}- if summary.LastSuccessAt != nil &&- summary.HistoryCompleteness == ProtectionHistoryComplete &&- summary.Permissions == operationaltrust.EvidencePermissionsSufficient {+ if supportedSuccessAt, supportedVerifiedAt := supportedProtectionTimes(summary); supportedSuccessAt != nil {hasSupportedQualifyingSuccess = truesupportedLastSuccessAt = latestProtectionTime(supportedLastSuccessAt,- summary.LastSuccessAt,+ supportedSuccessAt,)supportedLastVerifiedAt = latestProtectionTime(supportedLastVerifiedAt,- summary.LastVerifiedAt,+ supportedVerifiedAt,)if summary.VerificationExpected {supportedVerificationExpected = true@@ -571,15 +608,23 @@ func DeriveProtectionPostureAt(case hasSupportedQualifyingSuccess &&posture.Freshness == ProtectionFreshnessCurrent &&!hasInvalidatingFailure &&- !hasPartial &&(!stateVerificationExpected || posture.Verification == ProtectionVerificationVerified):+ // A backup the latest collection read stays a backup when some other+ // storage could not be read; coverage carries that gap.posture.State = ProtectionStateProtected- if posture.Verification == ProtectionVerificationVerified {+ switch {+ case posture.Verification == ProtectionVerificationVerified:posture.Explanation = "A recent backup is available and it was verified recently."- } else {+ case hasPartial || hasUnknownBlock:+ posture.Explanation = "A recent backup is available."+ default:posture.Explanation = "A recent backup is available, and Pulse can see the full backup history."}- if hasUnknownBlock {+ if names := describeUnreadScopes(unreadScopes); names != "" {+ posture.Explanation += " Pulse could not read " + names + ", but that does not change this backup."+ } else if hasPartial {+ posture.Explanation += " Part of the backup history could not be read, but that does not change this backup."+ } else if hasUnknownBlock {posture.Explanation += " Another backup source could not be read, but that does not change this backup."}case hasSupportedQualifyingSuccess:@@ -608,6 +653,9 @@ func DeriveProtectionPostureAt(} else {posture.Explanation = "Pulse can see only part of the backup history, so it cannot confirm a backup."}+ if names := describeUnreadScopes(unreadScopes); names != "" {+ posture.Explanation += " Pulse could not read " + names + "."+ }case hasCompleteHistory:posture.State = ProtectionStateUnprotectedif hasSnapshotsOnly {@@ -623,6 +671,77 @@ func DeriveProtectionPostureAt(return posture}+// supportedProtectionTimes returns the success and verification times a+// provider summary can stand behind. Complete history with sufficient access+// supports every point. A partial collection supports only the points it+// re-read; denied or unavailable history supports none.+func supportedProtectionTimes(summary ProtectionProviderSummary) (*time.Time, *time.Time) {+ switch {+ case summary.LastSuccessAt != nil &&+ summary.HistoryCompleteness == ProtectionHistoryComplete &&+ summary.Permissions == operationaltrust.EvidencePermissionsSufficient:+ return summary.LastSuccessAt, summary.LastVerifiedAt+ case summary.ConfirmedLastSuccessAt != nil &&+ summary.HistoryCompleteness == ProtectionHistoryPartial &&+ (summary.Permissions == operationaltrust.EvidencePermissionsPartial ||+ summary.Permissions == operationaltrust.EvidencePermissionsUnknown):+ return summary.ConfirmedLastSuccessAt, summary.ConfirmedLastVerifiedAt+ default:+ return nil, nil+ }+}++// describeUnreadScopes names unread provider scopes for an explanation, e.g.+// "pbs-offsite on delly". It names at most three and counts the rest.+func describeUnreadScopes(scopes []ProtectionUnreadScope) string {+ scopes = NormalizeProtectionUnreadScopes(scopes)+ if len(scopes) == 0 {+ return ""+ }+ names := make([]string, 0, len(scopes))+ for _, scope := range scopes {+ names = append(names, DescribeProtectionUnreadScope(scope))+ }+ const named = 3+ if len(names) > named {+ rest := len(names) - named+ names = append(names[:named], fmt.Sprintf("%d more", rest))+ }+ if len(names) == 1 {+ return names[0]+ }+ return strings.Join(names[:len(names)-1], ", ") + " and " + names[len(names)-1]+}++// DescribeProtectionUnreadScope renders one unread scope in operator terms.+func DescribeProtectionUnreadScope(scope ProtectionUnreadScope) string {+ switch scope.Kind {+ case ProtectionUnreadScopeNode:+ return "the storage list on node " + scope.Node+ case ProtectionUnreadScopeStorage:+ if scope.Node != "" {+ return scope.Storage + " on " + scope.Node+ }+ return scope.Storage+ case ProtectionUnreadScopeDatastore:+ return "the namespace list of datastore " + scope.Storage+ case ProtectionUnreadScopeNamespace:+ if scope.Namespace == "" {+ return "the root namespace of datastore " + scope.Storage+ }+ return "namespace " + scope.Namespace + " on datastore " + scope.Storage+ default:+ return ""+ }+}++func cloneUnreadScopes(scopes []ProtectionUnreadScope) []ProtectionUnreadScope {+ if len(scopes) == 0 {+ return nil+ }+ return append([]ProtectionUnreadScope(nil), scopes...)+}+func recoveryPointObservedAt(point RecoveryPoint) time.Time {if point.CompletedAt != nil && !point.CompletedAt.IsZero() {return point.CompletedAt.UTC()diff --git a/internal/recovery/posture_test.go b/internal/recovery/posture_test.goindex f1eb046496..c4204acecd 100644--- a/internal/recovery/posture_test.go+++ b/internal/recovery/posture_test.go@@ -738,3 +738,116 @@ func TestProtectionExplanationDoesNotEraseUnsuccessfulBackupEvidence(t *testing.})}}++// A partial provider collection supports exactly the points it re-read. A+// retained point from an unread scope has no newer evidence and may have been+// pruned since, so it cannot stand behind a protected claim.+func TestBuildProtectionPosturePartialCollectionConfirmsOnlyReReadPoints(t *testing.T) {+ t.Parallel()+ now := time.Date(2026, 11, 1, 12, 0, 0, 0, time.UTC)+ observedAt := now.Add(-time.Minute)+ backupAt := now.Add(-3 * time.Hour)+ verified := true+ unread := []ProtectionUnreadScope{{Kind: ProtectionUnreadScopeStorage, Node: "delly", Storage: "pbs-energy-offsite"}}+ for _, provider := range []Provider{ProviderProxmoxPVE, ProviderProxmoxPBS} {+ for _, permissions := range []operationaltrust.EvidencePermissions{+ operationaltrust.EvidencePermissionsUnknown,+ operationaltrust.EvidencePermissionsPartial,+ } {+ t.Run(fmt.Sprintf("%s/%s", provider, permissions), func(t *testing.T) {+ observation, err := NewProtectionProviderObservation(provider, "inventory", "source-a", OutcomeWarning, ProtectionHistoryPartial, permissions, true, observedAt, observedAt, nil)+ if err != nil {+ t.Fatal(err)+ }+ observation = observation.WithUnreadScopes(unread)+ withEvidence := func(collector string, ingestedAt time.Time) RecoveryPoint {+ point := RecoveryPoint{ID: "backup", Provider: provider, ProviderScope: "source-a", Kind: KindBackup, Mode: ModeRemote, Outcome: OutcomeSuccess, CompletedAt: &backupAt, Verified: &verified, SubjectResourceID: "resource:vm-100"}+ evidence, err := NewRecoveryPointEvidence(point, collector, ingestedAt)+ if err != nil {+ t.Fatal(err)+ }+ point.Evidence = evidence+ return point+ }+ build := func(point RecoveryPoint) ProtectionPosture {+ return BuildProtectionPostureFromPointsAt("resource:vm-100", []RecoveryPoint{point}, []ProtectionProviderObservation{observation}, DefaultProtectionPosturePolicy, now)+ }++ got := build(withEvidence("inventory", observedAt))+ if got.State != ProtectionStateProtected || got.Coverage != ProtectionCoveragePartial || got.Verification != ProtectionVerificationVerified {+ t.Fatalf("re-read point = %s/%s/%s, want protected/partial/verified", got.State, got.Coverage, got.Verification)+ }+ if !strings.Contains(got.Explanation, "Pulse could not read pbs-energy-offsite on delly, but that does not change this backup.") {+ t.Errorf("explanation = %q", got.Explanation)+ }+ if len(got.ProviderStates) != 1 || !reflect.DeepEqual(got.ProviderStates[0].UnreadScopes, unread) {+ t.Errorf("provider state unread scopes = %+v", got.ProviderStates)+ }+ if err := got.Validate(); err != nil {+ t.Fatal(err)+ }++ for name, point := range map[string]RecoveryPoint{+ "retained": withEvidence("inventory", observedAt.Add(-time.Millisecond)),+ "migration": withEvidence(recoveryPointMigrationCollector, now),+ "legacy": func() RecoveryPoint {+ p := withEvidence("inventory", observedAt)+ p.Evidence = nil+ return p+ }(),+ } {+ got := build(point)+ if got.State != ProtectionStateAttention || got.Coverage != ProtectionCoveragePartial {+ t.Errorf("%s point = %s/%s, want attention/partial", name, got.State, got.Coverage)+ }+ if !strings.Contains(got.Explanation, "Pulse could not read pbs-energy-offsite on delly.") {+ t.Errorf("%s explanation does not name the unread storage: %q", name, got.Explanation)+ }+ }+ })+ }+ }+}++// Denied or unavailable history confirms nothing, even for a renewed point.+func TestProtectionPostureUnavailableHistoryConfirmsNothing(t *testing.T) {+ t.Parallel()+ now := time.Date(2026, 11, 1, 12, 0, 0, 0, time.UTC)+ recent := now.Add(-time.Hour)+ for _, history := range []ProtectionHistoryCompleteness{ProtectionHistoryUnavailable, ProtectionHistoryUnknown} {+ summary := protectionTestSummary(now, &recent)+ summary.HistoryCompleteness = history+ summary.Permissions = operationaltrust.EvidencePermissionsUnknown+ summary.ConfirmedLastSuccessAt = &recent+ summary.ConfirmedLastVerifiedAt = &recent+ if got := DeriveProtectionPostureAt("resource:vm-100", []ProtectionProviderSummary{summary}, DefaultProtectionPosturePolicy, now); got.State != ProtectionStateUnknown {+ t.Errorf("%s history = %s, want unknown", history, got.State)+ }+ }+ summary := protectionTestSummary(now, &recent)+ summary.HistoryCompleteness = ProtectionHistoryPartial+ summary.Permissions = operationaltrust.EvidencePermissionsDenied+ summary.ConfirmedLastSuccessAt = &recent+ if got := DeriveProtectionPostureAt("resource:vm-100", []ProtectionProviderSummary{summary}, DefaultProtectionPosturePolicy, now); got.State == ProtectionStateProtected {+ t.Errorf("denied access claimed protection: %+v", got)+ }+}++func TestDescribeUnreadScopesNamesAtMostThree(t *testing.T) {+ t.Parallel()+ scopes := []ProtectionUnreadScope{+ {Kind: ProtectionUnreadScopeNode, Node: "pve2"},+ {Kind: ProtectionUnreadScopeStorage, Storage: "nfs"},+ {Kind: ProtectionUnreadScopeNamespace, Storage: "archive"},+ {Kind: ProtectionUnreadScopeNamespace, Storage: "archive", Namespace: "tenant"},+ {Kind: ProtectionUnreadScopeDatastore, Storage: "archive"},+ {Kind: "bogus", Storage: "ignored"},+ }+ want := "the namespace list of datastore archive, the root namespace of datastore archive, namespace tenant on datastore archive and 2 more"+ if got := describeUnreadScopes(scopes); got != want {+ t.Fatalf("describeUnreadScopes() = %q\nwant %q", got, want)+ }+ if got := describeUnreadScopes(scopes[:2]); got != "the storage list on node pve2 and nfs" {+ t.Fatalf("two scopes = %q", got)+ }+}diff --git a/internal/recovery/store/store.go b/internal/recovery/store/store.goindex 199ce7a09b..84af4b63f8 100644--- a/internal/recovery/store/store.go+++ b/internal/recovery/store/store.go@@ -687,6 +687,9 @@ func (s *Store) initSchema() error {if err := s.ensureColumn("recovery_points", "evidence_json", "TEXT"); err != nil {return err}+ if err := s.ensureColumn("protection_provider_observations", "unread_scopes_json", "TEXT"); err != nil {+ return err+ }// Migration: normalized index columns for efficient filtering/searching and consistent UI display.for _, col := range []struct {diff --git a/internal/recovery/store/store_posture.go b/internal/recovery/store/store_posture.goindex 5c64af212b..ee109284c1 100644--- a/internal/recovery/store/store_posture.go+++ b/internal/recovery/store/store_posture.go@@ -231,8 +231,8 @@ func (s *Store) UpsertProtectionProviderObservations(id, provider, source, scope, job_state,history_completeness, permissions, verification_expected,observed_at_ms, ingested_at_ms, evidence_json,- created_at_ms, updated_at_ms- ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)+ unread_scopes_json, created_at_ms, updated_at_ms+ ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)ON CONFLICT(id) DO UPDATE SETprovider=excluded.provider,source=excluded.source,@@ -244,6 +244,7 @@ func (s *Store) UpsertProtectionProviderObservations(observed_at_ms=excluded.observed_at_ms,ingested_at_ms=excluded.ingested_at_ms,evidence_json=excluded.evidence_json,+ unread_scopes_json=excluded.unread_scopes_json,updated_at_ms=excluded.updated_at_ms`)if err != nil {@@ -264,6 +265,14 @@ func (s *Store) UpsertProtectionProviderObservations(if err != nil {return err}+ var unreadScopesJSON any+ if scopes := recovery.NormalizeProtectionUnreadScopes(observation.UnreadScopes); len(scopes) > 0 {+ encoded, err := json.Marshal(scopes)+ if err != nil {+ return err+ }+ unreadScopesJSON = string(encoded)+ }verificationExpected := 0if observation.VerificationExpected {verificationExpected = 1@@ -281,6 +290,7 @@ func (s *Store) UpsertProtectionProviderObservations(observation.ObservedAt.UTC().UnixMilli(),observation.IngestedAt.UTC().UnixMilli(),string(evidenceJSON),+ unreadScopesJSON,nowMs,nowMs,); err != nil {@@ -470,7 +480,8 @@ func (s *Store) listLatestProtectionProviderObservations(observation.scope, observation.job_state,observation.history_completeness, observation.permissions,observation.verification_expected, observation.observed_at_ms,- observation.ingested_at_ms, observation.evidence_json+ observation.ingested_at_ms, observation.evidence_json,+ observation.unread_scopes_jsonFROM scopesJOIN protection_provider_observations observationON observation.id = (@@ -496,6 +507,7 @@ func (s *Store) listLatestProtectionProviderObservations(verificationExpected intobservedAtMs, ingestedAtMs int64evidenceRaw string+ unreadScopesRaw sql.NullString)if err := rows.Scan(&observation.ID,@@ -509,6 +521,7 @@ func (s *Store) listLatestProtectionProviderObservations(&observedAtMs,&ingestedAtMs,&evidenceRaw,+ &unreadScopesRaw,); err != nil {return nil, err}@@ -537,6 +550,12 @@ func (s *Store) listLatestProtectionProviderObservations(if err := observation.Validate(); err != nil {continue}+ // Unread scopes only explain a gap. An unreadable list must not drop+ // the observation, whose completeness still qualifies its points.+ var unreadScopes []recovery.ProtectionUnreadScope+ if unmarshalJSON(unreadScopesRaw, &unreadScopes) == nil {+ observation = observation.WithUnreadScopes(unreadScopes)+ }out = append(out, observation)}return out, rows.Err()diff --git a/internal/recovery/store/store_posture_test.go b/internal/recovery/store/store_posture_test.goindex d1b950f165..4036dbd2a0 100644--- a/internal/recovery/store/store_posture_test.go+++ b/internal/recovery/store/store_posture_test.go@@ -792,3 +792,90 @@ func TestStoreProtectionPosturePublishesCanonicalLegacySubject(t *testing.T) {t.Fatalf("obsolete attention posture remains: %+v total=%d err=%v", attention, total, err)}}++// A partial collection persists which scope it could not read, and a point it+// re-read keeps its standing while one it only retained does not.+func TestStoreProtectionPostureKeepsReReadPointsUnderPartialCollection(t *testing.T) {+ t.Parallel()++ store, err := Open(filepath.Join(t.TempDir(), "recovery.db"))+ if err != nil {+ t.Fatalf("Open() error = %v", err)+ }+ defer store.Close()+ ctx := context.Background()++ now := time.Now().UTC().Truncate(time.Millisecond)+ backupTime := now.Add(-time.Hour)+ point := recovery.RecoveryPoint{+ ID: "pve-backup:cluster:archive:vm-100",+ Provider: recovery.ProviderProxmoxPVE,+ Kind: recovery.KindBackup,+ Mode: recovery.ModeLocal,+ Outcome: recovery.OutcomeSuccess,+ CompletedAt: &backupTime,+ SubjectResourceID: "resource:vm-100",+ ProviderScope: "cluster",+ }+ unread := []recovery.ProtectionUnreadScope{{+ Kind: recovery.ProtectionUnreadScopeStorage,+ Node: "delly",+ Storage: "pbs-energy-offsite",+ }}+ collect := func(at time.Time, reRead bool) {+ t.Helper()+ observation, err := recovery.NewProtectionProviderObservation(+ recovery.ProviderProxmoxPVE, "pve-backup-enumeration", "cluster",+ recovery.OutcomeWarning, recovery.ProtectionHistoryPartial,+ operationaltrust.EvidencePermissionsUnknown, false, at, at, nil,+ )+ if err != nil {+ t.Fatal(err)+ }+ if reRead {+ evidence, err := recovery.NewRecoveryPointEvidence(point, "pve-storage-backup-inventory", at)+ if err != nil {+ t.Fatal(err)+ }+ read := point+ read.Evidence = evidence+ if err := store.UpsertPoints(ctx, []recovery.RecoveryPoint{read}); err != nil {+ t.Fatal(err)+ }+ }+ if err := store.UpsertProtectionProviderObservations(+ ctx,+ []recovery.ProtectionProviderObservation{observation.WithUnreadScopes(unread)},+ ); err != nil {+ t.Fatal(err)+ }+ }+ posture := func() recovery.ProtectionPosture {+ t.Helper()+ got, _, err := store.ListProtectionPostures(ctx, recovery.ProtectionPostureQuery{+ SubjectResourceIDs: []string{"resource:vm-100"},+ })+ if err != nil || len(got) != 1 {+ t.Fatalf("ListProtectionPostures() = %d, %v", len(got), err)+ }+ return got[0]+ }++ collect(now, true)+ got := posture()+ if got.State != recovery.ProtectionStateProtected || got.Coverage != recovery.ProtectionCoveragePartial {+ t.Fatalf("re-read point under partial collection = %s/%s, want protected/partial: %+v", got.State, got.Coverage, got)+ }+ if len(got.ProviderStates) != 1 || len(got.ProviderStates[0].UnreadScopes) != 1 ||+ got.ProviderStates[0].UnreadScopes[0] != unread[0] {+ t.Fatalf("unread scopes did not survive persistence: %+v", got.ProviderStates)+ }+ if !strings.Contains(got.Explanation, "pbs-energy-offsite on delly") {+ t.Fatalf("explanation does not name the unread storage: %q", got.Explanation)+ }++ collect(now.Add(time.Minute), false)+ if got := posture(); got.State != recovery.ProtectionStateAttention {+ t.Fatalf("retained point the latest collection did not read = %s, want attention", got.State)+ }+}diff --git a/internal/recovery/types.go b/internal/recovery/types.goindex de689647c5..860ce652ce 100644--- a/internal/recovery/types.go+++ b/internal/recovery/types.go@@ -111,6 +111,18 @@ const ()type ProtectionProviderState = model.ProtectionProviderState+type ProtectionUnreadScope = model.ProtectionUnreadScope+type ProtectionUnreadScopeKind = model.ProtectionUnreadScopeKind++const (+ ProtectionUnreadScopeNode ProtectionUnreadScopeKind = model.ProtectionUnreadScopeNode+ ProtectionUnreadScopeStorage ProtectionUnreadScopeKind = model.ProtectionUnreadScopeStorage+ ProtectionUnreadScopeDatastore ProtectionUnreadScopeKind = model.ProtectionUnreadScopeDatastore+ ProtectionUnreadScopeNamespace ProtectionUnreadScopeKind = model.ProtectionUnreadScopeNamespace+)++var NormalizeProtectionUnreadScopes = model.NormalizeProtectionUnreadScopes+type ProtectionPosture = model.ProtectionPosturetype ProtectionProviderObservation = model.ProtectionProviderObservationtype ProtectionPosturePolicy = model.ProtectionPosturePolicy