Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Keep re-read backups protected when one storage is unreadable

rcourtmanc86ca8d308f1
diff --git a/docs/release-control/v6/internal/subsystems/frontend-primitives.md b/docs/release-control/v6/internal/subsystems/frontend-primitives.md
index 764d9fc27b..f0c63d7a7c 100644
--- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md
+++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md
@@ -10877,6 +10877,17 @@ misrepresented. `ProxmoxCoverageTable.test.tsx` checks narrow/full projection
and complete long location text; the parsed inventory regressions also retain
reader identity and full repository context without adding reads or routes.
+### Unread backup storage is named once on the Backups page
+
+When the postures on the page carry unread provider scopes, the Backups page
+shows one amber status line under the health strip that names them (up to three,
+then a count) and says Pulse cannot confirm the backups stored there, with one action:
+check that the storage is reachable, or check the API token's permissions when
+every named read was rejected. It reuses the existing read-notice style, adds no
+request, and the connection name appears only when the gap spans several. The
+per-workload drawer keeps the server's explanation, which names the same scopes.
+`proxmoxBackupUnreadSources.test.ts` and `ProxmoxBackupsTable.test.tsx` cover it.
+
### Backup rows stay on one line (1 November 2026)
Coverage and By date rows keep one line at every container width, phones
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index c0897f0364..1c84f7a8bb 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -777,6 +777,19 @@ producer feeds the production Backups coverage surface in
`scripts/check-pve-backup-reconciliation.cjs`. Retained history is not a fresh
provider read, an available archive, guest thaw or filesystem-liveness proof.
+A partial PVE or PBS observation also names what went unread, from the same
+read-failure bookkeeping that drives carry-forward: PVE unread nodes, node-local
+storages and shared storages no node read (with the node when exactly one node
+attempted it, and whether authorization rejected it); PBS datastores whose
+namespace discovery failed and namespaces whose groups or snapshots were not all
+read. The names never decide which points this poll confirmed; that remains the
+renewed point evidence described in the storage-recovery contract, so a guest
+whose backup this scan re-read is protected with partial coverage while one whose
+only backup sits on the unread storage is not.
+`TestPVEUnreadStorageLeavesReadBackupsProtected` covers one enabled but
+unreachable PBS storage beside readable archives, a pruned archive retained by
+the partial poll, and the complete recovery.
+
### Skipped PVE backup storage is unread, not empty
A configured backup target that is not safe to query (including disabled
diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md
index d6248b729a..1597872b98 100644
--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md
+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md
@@ -279,6 +279,36 @@ checks all non-success outcomes through the real point/observation builder.
The ordinary PBS writer lifecycle checks the same explanation alongside live
and durable artifact outcomes and feeds desktop/narrow Backups pixel proof.
+### One unread storage does not withdraw backups Pulse did read
+
+A partial PVE or PBS collection still confirms the points it re-read. Collectors
+renew a point's evidence only for rows a poll returned, stamped with that poll's
+observation time, so a point whose evidence was ingested no earlier than the
+latest observation for its provider scope supports protection under partial
+history with Partial or Unknown access. A point retained from an unread scope
+keeps its older evidence and may have been pruned since, so it never supports
+protection; neither does store-backfilled migration evidence, nor anything
+under denied, unavailable or unknown history. The posture is then Protected with
+partial coverage, and the cross-scope rule matches the unread-source rule: a
+supported current backup is not withdrawn because another source is partial.
+PBS snapshot-cache reuse within its 10-minute TTL counts as a read of that
+group, exactly as complete collections already treat it.
+
+Provider observations also record which scopes went unread (PVE node or
+storage, with its node when exactly one node attempted it; PBS datastore
+namespace discovery or namespace), persisted in
+`protection_provider_observations.unread_scopes_json`, capped at 32 and carried
+on each provider state as `unreadScopes`. The list explains the gap only;
+confirmation never depends on it. Explanations name up to three scopes, and the
+Backups page shows one notice line for the unread scopes across its postures.
+`TestPVEUnreadStorageLeavesReadBackupsProtected` drives the production shape
+(one enabled but unreachable PBS storage beside readable archives), a pruned
+archive retained by the partial poll, and the complete recovery.
+`TestBuildProtectionPosturePartialCollectionConfirmsOnlyReReadPoints`,
+`TestStoreProtectionPostureKeepsReReadPointsUnderPartialCollection` and
+`TestPBSMissingInventoryArrayRetainsUnreadScope` keep retained, legacy and
+migration points out of protection.
+
### Coverage dates are inspectable without hover
Open Coverage evidence retains each point’s exact reported date as wrapping
diff --git a/frontend-modern/browser-verification/0aabb658cb3ebf23.json b/frontend-modern/browser-verification/0aabb658cb3ebf23.json
new file mode 100644
index 0000000000..dc91ab2757
--- /dev/null
+++ b/frontend-modern/browser-verification/0aabb658cb3ebf23.json
@@ -0,0 +1,44 @@
+{
+ "version": 1,
+ "base_sha": "598994dfc20f06bc293f676bbba438ee8cfbf0ed",
+ "verified_at": "2026-11-01T16:38:23Z",
+ "result": "passed",
+ "changed_paths": [
+ "frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx",
+ "frontend-modern/src/features/proxmox/proxmoxBackupUnreadSources.ts",
+ "frontend-modern/src/types/recovery.ts",
+ "frontend-modern/src/utils/protectionPostureResponse.ts"
+ ],
+ "content_sha256": {
+ "frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx": "784647117e6dfd2b5d536cd9e6abc427ac7c2957b3c3b5f768705d371ab20fba",
+ "frontend-modern/src/features/proxmox/proxmoxBackupUnreadSources.ts": "e138b5b1f86f4f110be1a411d20f4cf55bebd225621157ed02d58b98b12f68a4",
+ "frontend-modern/src/types/recovery.ts": "d52030f31023f078d65e40f3ba2433ef2289d94242e3feeb1e776fa8bab8c47c",
+ "frontend-modern/src/utils/protectionPostureResponse.ts": "00aa34f7e724bb2daa7398855a1a9d6c66f3894818ddda66edfce93c0ba34c9c"
+ },
+ "routes": [
+ "/proxmox/backups/coverage"
+ ],
+ "viewports": [
+ {
+ "width": 1440,
+ "height": 1000
+ },
+ {
+ "width": 375,
+ "height": 812
+ }
+ ],
+ "states": [
+ "frames exported from the real PVE poll, recovery ingest and posture reader, replayed into the production Backups table (static build of browser-tests/backups.html)",
+ "one enabled but unreachable PBS storage (pbs-energy-offsite on node-a) beside readable archives: 2 protected, one amber notice naming the storage",
+ "node-b archive pruned while the storage stays unreadable: 1 protected, 1 attention, notice and guest-102 drawer explanation name the storage",
+ "storage readable again: notice gone, complete coverage",
+ "light theme only; the fixture does not apply the class-based dark theme"
+ ],
+ "interactions": [
+ "loaded Coverage in Chromium 141 at 1440x1000 and WebKit 26 at 375x812, reloading per phase",
+ "checked the notice text and role=status, the health-strip counts, and absence after recovery",
+ "expanded and collapsed guest-102 by keyboard and matched its posture explanation",
+ "asserted no page errors and no horizontal overflow in every phase; viewed screenshots"
+ ]
+}
diff --git a/frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx b/frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx
index 1c8b473b81..ba9be5aa35 100644
--- a/frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx
+++ b/frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx
@@ -80,6 +80,10 @@ import {
} from './proxmoxBackupsTableShared';
import { getProxmoxBackupSourcePresentation } from './proxmoxBackupSourcePresentation';
import { ProxmoxBackupsCoverageStrip } from './ProxmoxBackupsCoverageStrip';
+import {
+ collectUnreadBackupSources,
+ describeUnreadBackupSources,
+} from './proxmoxBackupUnreadSources';
import { ProxmoxBackupServersTable } from './ProxmoxBackupServersTable';
import { ProxmoxCoverageTable } from './ProxmoxCoverageTable';
import { resolveBackupLocationFilter } from './proxmoxBackupLocation';
@@ -195,6 +199,13 @@ export const ProxmoxBackupsTable: Component<{
const protectionPostures = useProtectionPostures(() =>
(props.workloads ?? []).map((workload) => workload.id),
);
+ // One unreadable storage limits every guest's coverage on its connection;
+ // say which one once, where the operator can act on it.
+ const unreadBackupSourcesNotice = createMemo(() =>
+ describeUnreadBackupSources(
+ collectUnreadBackupSources(protectionPostures.postureByResourceID().values()),
+ ),
+ );
// Saved views persist the current route query, so every filter and the active
// backup workspace must share one URL owner. View changes also clear the
@@ -957,6 +968,14 @@ export const ProxmoxBackupsTable: Component<{
]}
/>
+ <Show when={unreadBackupSourcesNotice()}>
+ {(notice) => (
+ <div role="status" class={BACKUP_READ_NOTICE_CLASS} data-unread-backup-sources>
+ {notice()}
+ </div>
+ )}
+ </Show>
+
<Show when={view() === 'date' && recoveryModel().recoverableArtifacts.length > 0}>
<Show
when={scopedRecoverableArtifacts().some(
diff --git a/frontend-modern/src/features/proxmox/__tests__/ProxmoxBackupsTable.test.tsx b/frontend-modern/src/features/proxmox/__tests__/ProxmoxBackupsTable.test.tsx
index e16f57b95b..7a7dce17fc 100644
--- a/frontend-modern/src/features/proxmox/__tests__/ProxmoxBackupsTable.test.tsx
+++ b/frontend-modern/src/features/proxmox/__tests__/ProxmoxBackupsTable.test.tsx
@@ -565,6 +565,39 @@ describe('ProxmoxBackupsTable', () => {
expect(postureURL.searchParams.getAll('resourceId')).toEqual(['ct-112']);
});
+ it('names a backup storage Pulse cannot read once, above the coverage list', async () => {
+ mockBackupAPIs('protected');
+ const response = await apiFetchJSONMock();
+ response.data[0].coverage = 'partial';
+ response.data[0].providerStates = [
+ {
+ provider: 'proxmox-pve',
+ source: 'pve-backup-enumeration',
+ scope: 'homelab',
+ jobState: 'warning',
+ historyCompleteness: 'partial',
+ permissions: 'unknown',
+ evidenceIds: ['evidence-2'],
+ unreadScopes: [{ kind: 'storage', node: 'delly', storage: 'pbs-energy-offsite' }],
+ },
+ ];
+ apiFetchJSONMock.mockResolvedValue(response);
+
+ renderInRouter(() => (
+ <ProxmoxBackupsTable
+ emptyIcon={<span />}
+ workloads={[workloadResource]}
+ servers={[pbsServerResource]}
+ />
+ ));
+
+ const notice = await screen.findByText(
+ 'Pulse cannot read pbs-energy-offsite on delly, so it cannot confirm the backups stored there. Check that the storage is reachable from Proxmox.',
+ );
+ expect(notice).toHaveAttribute('role', 'status');
+ expect(document.querySelectorAll('[data-unread-backup-sources]')).toHaveLength(1);
+ });
+
it('offers By date / Coverage views and no legacy sub-tab tree', async () => {
mockBackupAPIs();
diff --git a/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupUnreadSources.test.ts b/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupUnreadSources.test.ts
new file mode 100644
index 0000000000..d0d11401b8
--- /dev/null
+++ b/frontend-modern/src/features/proxmox/__tests__/proxmoxBackupUnreadSources.test.ts
@@ -0,0 +1,80 @@
+import { describe, expect, it } from 'vitest';
+import type { ProtectionPosture, ProtectionProviderState } from '@/types/recovery';
+import { protectionPosture } from '@/utils/__fixtures__/protectionPosture';
+import {
+ collectUnreadBackupSources,
+ describeUnreadBackupSources,
+} from '../proxmoxBackupUnreadSources';
+
+const withProvider = (
+ id: string,
+ provider: Partial<ProtectionProviderState>,
+): ProtectionPosture => {
+ const posture = protectionPosture(id);
+ posture.providerStates = [{ ...posture.providerStates[0], ...provider }];
+ return posture;
+};
+
+describe('unread backup sources', () => {
+ it('collapses the same unread storage across guests into one actionable line', () => {
+ const partial: Partial<ProtectionProviderState> = {
+ provider: 'proxmox-pve',
+ scope: 'homelab',
+ historyCompleteness: 'partial',
+ unreadScopes: [{ kind: 'storage', node: 'delly', storage: 'pbs-energy-offsite' }],
+ };
+ const sources = collectUnreadBackupSources([
+ withProvider('vm:1', partial),
+ withProvider('vm:2', partial),
+ ]);
+ expect(sources).toEqual([{ label: 'pbs-energy-offsite on delly', accessDenied: false }]);
+ expect(describeUnreadBackupSources(sources)).toBe(
+ 'Pulse cannot read pbs-energy-offsite on delly, so it cannot confirm the backups stored there. Check that the storage is reachable from Proxmox.',
+ );
+ });
+
+ it('names connections only when the gap spans several, and reports denied access', () => {
+ const sources = collectUnreadBackupSources([
+ withProvider('vm:1', {
+ provider: 'proxmox-pve',
+ scope: 'east',
+ historyCompleteness: 'partial',
+ unreadScopes: [{ kind: 'node', node: 'pve2', accessDenied: true }],
+ }),
+ withProvider('vm:2', {
+ provider: 'proxmox-pbs',
+ scope: 'west',
+ historyCompleteness: 'partial',
+ unreadScopes: [
+ { kind: 'namespace', storage: 'archive', namespace: '', accessDenied: true },
+ ],
+ }),
+ ]);
+ expect(sources.map((source) => source.label)).toEqual([
+ 'the root namespace of datastore archive (west)',
+ 'the storage list on node pve2 (east)',
+ ]);
+ expect(describeUnreadBackupSources(sources)).toMatch(
+ /^Pulse does not have permission to read .* and .*\. Check the permissions of the Pulse API token\.$/,
+ );
+ });
+
+ it('shows nothing for complete history or when no scope is named', () => {
+ expect(
+ collectUnreadBackupSources([
+ protectionPosture('vm:1'),
+ withProvider('vm:2', {
+ historyCompleteness: 'complete',
+ unreadScopes: [{ kind: 'storage', storage: 'stale' }],
+ }),
+ withProvider('vm:3', { historyCompleteness: 'partial' }),
+ ]),
+ ).toEqual([]);
+ expect(describeUnreadBackupSources([])).toBeNull();
+ });
+
+ it('counts sources past the first three', () => {
+ const sources = ['a', 'b', 'c', 'd', 'e'].map((label) => ({ label, accessDenied: false }));
+ expect(describeUnreadBackupSources(sources)).toMatch(/^Pulse cannot read a, b, c and 2 more,/);
+ });
+});
diff --git a/frontend-modern/src/features/proxmox/proxmoxBackupUnreadSources.ts b/frontend-modern/src/features/proxmox/proxmoxBackupUnreadSources.ts
new file mode 100644
index 0000000000..78f29c3840
--- /dev/null
+++ b/frontend-modern/src/features/proxmox/proxmoxBackupUnreadSources.ts
@@ -0,0 +1,88 @@
+import type { ProtectionPosture, ProtectionUnreadScope } from '@/types/recovery';
+
+// The server names what each provider's latest backup read could not reach.
+// Every guest on that connection carries the same list, so collapse it to one
+// page-level line: the operator fixes the storage, not each guest.
+
+const scopeLabel = (scope: ProtectionUnreadScope): string => {
+ const node = scope.node?.trim() ?? '';
+ const storage = scope.storage?.trim() ?? '';
+ const namespace = scope.namespace?.trim() ?? '';
+ switch (scope.kind) {
+ case 'node':
+ return node ? `the storage list on node ${node}` : '';
+ case 'storage':
+ if (!storage) return '';
+ return node ? `${storage} on ${node}` : storage;
+ case 'datastore':
+ return storage ? `the namespace list of datastore ${storage}` : '';
+ case 'namespace':
+ if (!storage) return '';
+ return namespace
+ ? `namespace ${namespace} on datastore ${storage}`
+ : `the root namespace of datastore ${storage}`;
+ default:
+ return '';
+ }
+};
+
+export interface UnreadBackupSource {
+ label: string;
+ accessDenied: boolean;
+}
+
+export const collectUnreadBackupSources = (
+ postures: Iterable<ProtectionPosture>,
+): UnreadBackupSource[] => {
+ const byKey = new Map<string, UnreadBackupSource & { connection: string; base: string }>();
+ for (const posture of postures) {
+ for (const provider of posture.providerStates) {
+ if (provider.historyCompleteness === 'complete') continue;
+ for (const scope of provider.unreadScopes ?? []) {
+ const base = scopeLabel(scope);
+ if (!base) continue;
+ const connection = provider.scope?.trim() ?? '';
+ const key = `${provider.provider}\u0000${connection}\u0000${base}`;
+ const existing = byKey.get(key);
+ byKey.set(key, {
+ label: base,
+ base,
+ connection,
+ accessDenied: Boolean(scope.accessDenied) || Boolean(existing?.accessDenied),
+ });
+ }
+ }
+ }
+ const entries = [...byKey.values()];
+ // Name the connection only when the gap spans more than one of them.
+ const connections = new Set(entries.map((entry) => entry.connection));
+ return entries
+ .map((entry) => ({
+ label:
+ connections.size > 1 && entry.connection
+ ? `${entry.base} (${entry.connection})`
+ : entry.base,
+ accessDenied: entry.accessDenied,
+ }))
+ .sort((a, b) => a.label.localeCompare(b.label));
+};
+
+const NAMED = 3;
+
+export const describeUnreadBackupSources = (
+ sources: readonly UnreadBackupSource[],
+): string | null => {
+ if (sources.length === 0) return null;
+ const names = sources.slice(0, NAMED).map((source) => source.label);
+ if (sources.length > NAMED) names.push(`${sources.length - NAMED} more`);
+ const list =
+ names.length === 1
+ ? names[0]
+ : `${names.slice(0, -1).join(', ')} and ${names[names.length - 1]}`;
+ const denied = sources.every((source) => source.accessDenied);
+ const verb = denied ? 'Pulse does not have permission to read' : 'Pulse cannot read';
+ const action = denied
+ ? 'Check the permissions of the Pulse API token.'
+ : 'Check that the storage is reachable from Proxmox.';
+ return `${verb} ${list}, so it cannot confirm the backups stored there. ${action}`;
+};
diff --git a/frontend-modern/src/hooks/__tests__/useProtectionPostures.test.ts b/frontend-modern/src/hooks/__tests__/useProtectionPostures.test.ts
index 3854187077..147e8bbed4 100644
--- a/frontend-modern/src/hooks/__tests__/useProtectionPostures.test.ts
+++ b/frontend-modern/src/hooks/__tests__/useProtectionPostures.test.ts
@@ -179,6 +179,36 @@ describe('useProtectionPostures transport', () => {
},
);
+ it('keeps the unread provider scopes the server names and rejects a malformed one', async () => {
+ const posture = protectionPosture('vm:1');
+ posture.coverage = 'partial';
+ posture.providerStates[0].historyCompleteness = 'partial';
+ posture.providerStates[0].unreadScopes = [
+ { kind: 'storage', node: 'delly', storage: 'pbs-energy-offsite' },
+ ];
+ apiFetchJSONMock.mockResolvedValue({ data: [posture], policy: protectionPolicy });
+ let dispose = () => {};
+ const hook = createRoot((rootDispose) => {
+ dispose = rootDispose;
+ return useProtectionPostures(() => ['vm:1']);
+ });
+ try {
+ await vi.waitFor(() => expect(hook.postures()).toHaveLength(1));
+ expect(hook.postureByResourceID().get('vm:1')?.providerStates[0].unreadScopes).toEqual([
+ { kind: 'storage', node: 'delly', storage: 'pbs-energy-offsite' },
+ ]);
+
+ const malformed = structuredClone(posture);
+ malformed.providerStates[0].unreadScopes = [{ kind: 'disk', storage: 'x' }] as never;
+ apiFetchJSONMock.mockResolvedValue({ data: [malformed], policy: protectionPolicy });
+ await hook.refetch();
+ expect(hook.response.error).toBeInstanceOf(ProtectionPostureFormatError);
+ expect(hook.postures()).toEqual([]);
+ } finally {
+ dispose();
+ }
+ });
+
it('does not restore a failed cached assessment when Coverage remounts', async () => {
apiFetchJSONMock.mockResolvedValue({
data: [protectionPosture('vm:1')],
diff --git a/frontend-modern/src/types/recovery.ts b/frontend-modern/src/types/recovery.ts
index b112d2e10f..c08a3b26c2 100644
--- a/frontend-modern/src/types/recovery.ts
+++ b/frontend-modern/src/types/recovery.ts
@@ -90,6 +90,18 @@ export type ProtectionVerification = 'verified' | 'unverified' | 'stale' | 'unkn
export type ProtectionCoverage = 'complete' | 'partial' | 'none' | 'unknown';
export type ProtectionHistoryCompleteness = 'complete' | 'partial' | 'unavailable' | 'unknown';
+export type ProtectionUnreadScopeKind = 'node' | 'storage' | 'datastore' | 'namespace';
+
+// One part of a provider's backup history that its latest read could not
+// reach: a PVE node or storage, or a PBS datastore or namespace.
+export interface ProtectionUnreadScope {
+ kind: ProtectionUnreadScopeKind;
+ node?: string;
+ storage?: string;
+ namespace?: string;
+ accessDenied?: boolean;
+}
+
export interface ProtectionProviderState {
provider: RecoveryPlatform;
source: string;
@@ -102,6 +114,7 @@ export interface ProtectionProviderState {
lastVerifiedAt?: string | null;
evidenceIds: string[];
verificationExpected?: boolean;
+ unreadScopes?: ProtectionUnreadScope[];
}
export interface ProtectionPosture {
diff --git a/frontend-modern/src/utils/protectionPostureResponse.ts b/frontend-modern/src/utils/protectionPostureResponse.ts
index 375a9c947f..9b99586cf6 100644
--- a/frontend-modern/src/utils/protectionPostureResponse.ts
+++ b/frontend-modern/src/utils/protectionPostureResponse.ts
@@ -29,6 +29,19 @@ const choice =
(value) =>
typeof value === 'string' && values.includes(value);
const idList: Check = (value) => value === null || (Array.isArray(value) && value.every(text));
+const optionalText: Check = (value) => value === undefined || typeof value === 'string';
+const unreadScope: Check = (value) =>
+ value !== null &&
+ typeof value === 'object' &&
+ !Array.isArray(value) &&
+ choice('node', 'storage', 'datastore', 'namespace')((value as Record<string, unknown>).kind) &&
+ ['node', 'storage', 'namespace'].every((key) =>
+ optionalText((value as Record<string, unknown>)[key]),
+ ) &&
+ ((value as Record<string, unknown>).accessDenied === undefined ||
+ boolean((value as Record<string, unknown>).accessDenied));
+const unreadScopeList: Check = (value) =>
+ value == null || (Array.isArray(value) && value.every(unreadScope));
function record(value: unknown, shape: Readonly<Record<string, Check>>): Record<string, unknown> {
if (
@@ -62,6 +75,7 @@ const providerShape = {
lastVerifiedAt: optionalTime,
evidenceIds: idList,
verificationExpected: (value: unknown) => value === undefined || boolean(value),
+ unreadScopes: unreadScopeList,
};
const postureShape = {
subjectResourceId: text,
diff --git a/internal/monitoring/backup_unread_scope_posture_test.go b/internal/monitoring/backup_unread_scope_posture_test.go
new file mode 100644
index 0000000000..00650e26a1
--- /dev/null
+++ b/internal/monitoring/backup_unread_scope_posture_test.go
@@ -0,0 +1,151 @@
+package monitoring
+
+import (
+ "context"
+ "fmt"
+ "reflect"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/rcourtman/pulse-go-rewrite/internal/recovery"
+ "github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
+)
+
+// One enabled but unreachable PBS storage on one node, beside ordinary
+// readable archive storage on every node: the shape that turned every guest on
+// a production connection to attention.
+type pveDeadOffsiteStorageClient struct {
+ *pveReconciliationClient
+ offsiteErr error
+}
+
+const deadOffsiteStorage = "pbs-energy-offsite"
+
+func (c *pveDeadOffsiteStorageClient) GetStorage(ctx context.Context, node string) ([]proxmox.Storage, error) {
+ rows, err := c.pveReconciliationClient.GetStorage(ctx, node)
+ if err == nil && node == "node-a" {
+ rows = append(rows, proxmox.Storage{Storage: deadOffsiteStorage, Type: "pbs", Content: "backup", Shared: 1, Active: 1, Enabled: 1})
+ }
+ return rows, err
+}
+
+func (c *pveDeadOffsiteStorageClient) GetStorageContent(ctx context.Context, node, storage string) ([]proxmox.StorageContent, error) {
+ if storage == deadOffsiteStorage {
+ if c.offsiteErr != nil {
+ return nil, c.offsiteErr
+ }
+ return nil, nil
+ }
+ return c.pveReconciliationClient.GetStorageContent(ctx, node, storage)
+}
+
+func TestPVEUnreadStorageLeavesReadBackupsProtected(t *testing.T) {
+ f := newPVEReconciliationFixture(t)
+ c := &pveDeadOffsiteStorageClient{
+ pveReconciliationClient: f.client,
+ offsiteErr: fmt.Errorf("pbs-energy-offsite: error fetching datastores - 127.0.0.1:18007: connection refused"),
+ }
+ poll := func() {
+ f.m.pollStorageBackupsWithNodes(context.Background(), "cluster", c, f.nodes, f.online)
+ f.wait(t)
+ }
+ wantUnread := []recovery.ProtectionUnreadScope{{
+ Kind: recovery.ProtectionUnreadScopeStorage, Node: "node-a", Storage: deadOffsiteStorage,
+ }}
+ assertNamed := func(p recovery.ProtectionPosture) {
+ t.Helper()
+ if len(p.ProviderStates) != 1 ||
+ p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial ||
+ !reflect.DeepEqual(p.ProviderStates[0].UnreadScopes, wantUnread) {
+ t.Errorf("provider state lost the unread storage: %+v", p.ProviderStates)
+ }
+ if !strings.Contains(p.Explanation, "could not read "+deadOffsiteStorage+" on node-a") {
+ t.Errorf("explanation does not name the unread storage: %q", p.Explanation)
+ }
+ }
+
+ // Fresh backups on storages that read fine are protected; the dead
+ // storage limits coverage and is named, but proves nothing about them.
+ poll()
+ for _, p := range f.postures(t) {
+ if p.State != recovery.ProtectionStateProtected || p.Coverage != recovery.ProtectionCoveragePartial ||
+ p.Freshness != recovery.ProtectionFreshnessCurrent {
+ t.Errorf("read backup was not protected beside one unread storage: %+v", p)
+ }
+ assertNamed(p)
+ }
+
+ // Node-b's archive is pruned while the offsite storage is still dead. The
+ // partial poll cannot delete the stored point, but it did not re-read it,
+ // so that retained point no longer supports protection.
+ retained := f.points(t)
+ f.client.at = f.client.at.Add(15 * time.Minute)
+ f.client.emptyB = true
+ poll()
+ points := f.points(t)
+ if len(points) != len(retained) {
+ t.Fatalf("partial poll deleted history: points=%d want %d", len(points), len(retained))
+ }
+ for _, p := range f.postures(t) {
+ switch p.SubjectResourceID {
+ case f.ids[0]:
+ if p.State != recovery.ProtectionStateProtected {
+ t.Errorf("re-read backup lost protection: %+v", p)
+ }
+ case f.ids[1]:
+ if p.State == recovery.ProtectionStateProtected || p.Coverage != recovery.ProtectionCoveragePartial {
+ t.Errorf("retained point that the latest poll did not read claimed protection: %+v", p)
+ }
+ }
+ assertNamed(p)
+ }
+
+ // Once the storage reads again the collection is complete and reconciles
+ // the pruned archive away; the unread names disappear with the gap.
+ c.offsiteErr = nil
+ poll()
+ for _, p := range f.postures(t) {
+ if len(p.ProviderStates) > 0 && len(p.ProviderStates[0].UnreadScopes) != 0 {
+ t.Errorf("complete collection kept unread scopes: %+v", p.ProviderStates)
+ }
+ if p.SubjectResourceID == f.ids[0] &&
+ (p.State != recovery.ProtectionStateProtected || p.Coverage != recovery.ProtectionCoverageComplete) {
+ t.Errorf("complete read did not restore full protection: %+v", p)
+ }
+ if p.SubjectResourceID == f.ids[1] && p.State == recovery.ProtectionStateProtected {
+ t.Errorf("confirmed-absent archive still protected its guest: %+v", p)
+ }
+ }
+}
+
+func TestPVEUnreadProtectionScopesNameWhatWentUnread(t *testing.T) {
+ got := pveUnreadProtectionScopes(pveUnreadReads{
+ nodes: map[string]struct{}{"node-c": {}},
+ local: map[pveLocalBackupStorage]struct{}{{node: "node-b", storage: "local"}: {}},
+ shared: map[string]struct{}{
+ "offsite": {},
+ "nfs": {},
+ },
+ sharedReads: map[pveLocalBackupStorage]struct{}{
+ {node: "node-a", storage: "offsite"}: {},
+ {node: "node-a", storage: "nfs"}: {},
+ {node: "node-b", storage: "nfs"}: {},
+ },
+ denied: map[pveLocalBackupStorage]struct{}{
+ {node: "node-c"}: {},
+ {node: "node-b", storage: "nfs"}: {},
+ {node: "node-a", storage: "unknown"}: {},
+ },
+ })
+ want := []recovery.ProtectionUnreadScope{
+ {Kind: recovery.ProtectionUnreadScopeNode, Node: "node-c", AccessDenied: true},
+ {Kind: recovery.ProtectionUnreadScopeStorage, Node: "node-b", Storage: "local"},
+ // Several nodes failed the same shared storage: name the storage only.
+ {Kind: recovery.ProtectionUnreadScopeStorage, Storage: "nfs", AccessDenied: true},
+ {Kind: recovery.ProtectionUnreadScopeStorage, Node: "node-a", Storage: "offsite"},
+ }
+ if !reflect.DeepEqual(got, want) {
+ t.Fatalf("unread scopes = %+v\nwant %+v", got, want)
+ }
+}
diff --git a/internal/monitoring/monitor_backups.go b/internal/monitoring/monitor_backups.go
index 72fa946c67..ef92183529 100644
--- a/internal/monitoring/monitor_backups.go
+++ b/internal/monitoring/monitor_backups.go
@@ -281,6 +281,10 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
sharedStorageSuccess := map[string]struct{}{}
unqueryableStorages := map[string]struct{}{}
skippedLocalStorages := map[pveLocalBackupStorage]struct{}{}
+ // Where unread reads were attempted, so the provider observation can name
+ // them. A denied entry with no storage is a rejected node storage list.
+ sharedUnreadReads := map[pveLocalBackupStorage]struct{}{}
+ deniedReads := map[pveLocalBackupStorage]struct{}{}
readState := m.backupReadStateForInstance(instanceName)
// Build guest lookup map to find actual node for each VMID
@@ -376,6 +380,7 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
if isPVEBackupPermissionError(err) {
hadPermissionError = true
permissionFailureCount++
+ deniedReads[pveLocalBackupStorage{node: node.Node}] = struct{}{}
m.recordPVEBackupAccessWarning(ctx, instanceName, fmt.Sprintf("/nodes/%s/storage", node.Node), err)
}
monErr := errors.NewMonitorError(errors.ErrorTypeAPI, "get_storage_for_backups", instanceName, err).WithNode(node.Node)
@@ -413,6 +418,7 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
} else {
unqueryableStorages[storage.Storage] = struct{}{}
storagePreserveNeeded[storage.Storage] = struct{}{}
+ sharedUnreadReads[pveLocalBackupStorage{node: node.Node, storage: storage.Storage}] = struct{}{}
}
continue
}
@@ -433,6 +439,7 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
if isPVEBackupPermissionError(err) {
hadPermissionError = true
permissionFailureCount++
+ deniedReads[pveLocalBackupStorage{node: node.Node, storage: storage.Storage}] = struct{}{}
m.recordPVEBackupAccessWarning(ctx, instanceName, fmt.Sprintf("/nodes/%s/storage/%s/content", node.Node, storage.Storage), err)
} else {
log.Debug().Err(monErr).
@@ -444,6 +451,7 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
failedLocalStorages[pveLocalBackupStorage{node: storageNode, storage: storage.Storage}] = struct{}{}
} else {
storagePreserveNeeded[storage.Storage] = struct{}{}
+ sharedUnreadReads[pveLocalBackupStorage{node: node.Node, storage: storage.Storage}] = struct{}{}
}
contentFailures++
continue
@@ -659,6 +667,14 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
delete(confirmationPreserveNeeded, storageName)
}
+ unreadScopes := pveUnreadProtectionScopes(pveUnreadReads{
+ nodes: unreadNodes,
+ local: failedLocalStorages,
+ shared: storagePreserveNeeded,
+ sharedReads: sharedUnreadReads,
+ denied: deniedReads,
+ })
+
// Carry-forward serves the live inventory, not a new provider observation.
// Only rows actually returned this poll may renew point-level evidence.
observedBackups := allBackups
@@ -694,6 +710,7 @@ func (m *Monitor) pollStorageBackupsWithNodes(ctx context.Context, instanceName
ContentSuccesses: contentSuccess,
ContentFailures: contentFailures,
PermissionFailureCount: permissionFailureCount,
+ UnreadScopes: unreadScopes,
},
protectionObservedAt,
)
@@ -1897,6 +1914,13 @@ type pbsBackupFetchRequest struct {
}
// pollPBSBackups fetches all backups from PBS datastores
+// pbsReadDenied reports an authorization rejection. Other terminal errors,
+// such as a namespace that no longer exists, are not missing permission.
+func pbsReadDenied(err error) bool {
+ status, ok := pbs.HTTPStatus(err)
+ return ok && (status == 401 || status == 403)
+}
+
func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, client *pbs.Client, datastores []models.PBSDatastore, namespaceErrors map[string]error) {
log.Debug().Str("instance", instanceName).Msg("polling PBS backups")
@@ -1910,6 +1934,17 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien
queriedNamespaces := make(map[pbsBackupNamespaceKey]struct{})
transientNamespaces := make(map[pbsBackupNamespaceKey]struct{})
transientDiscovery := make(map[string]struct{})
+ // Names what went unread for the provider observation; it never decides
+ // which points this poll confirmed.
+ var unreadScopes []recovery.ProtectionUnreadScope
+ unreadNamespace := func(datastore, namespace string, denied bool) {
+ unreadScopes = append(unreadScopes, recovery.ProtectionUnreadScope{
+ Kind: recovery.ProtectionUnreadScopeNamespace,
+ Storage: datastore,
+ Namespace: namespace,
+ AccessDenied: denied,
+ })
+ }
datastoreCount := len(datastores) // Number of datastores to query
datastoreFetches := 0 // Number of successful datastore fetches
datastoreErrors := 0 // Datastores with incomplete history
@@ -1940,6 +1975,11 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien
} else {
transientDiscovery[ds.Name] = struct{}{}
}
+ unreadScopes = append(unreadScopes, recovery.ProtectionUnreadScope{
+ Kind: recovery.ProtectionUnreadScopeDatastore,
+ Storage: ds.Name,
+ AccessDenied: pbsReadDenied(err),
+ })
}
groupsReused := 0
groupsRequested := 0
@@ -1954,6 +1994,7 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien
}
datastoreNamespaceErrors++
transientNamespaces[namespaceKey] = struct{}{}
+ unreadNamespace(ds.Name, namespace, false)
continue
}
groups, err := client.ListBackupGroups(ctx, ds.Name, namespace)
@@ -1969,6 +2010,7 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien
} else {
transientNamespaces[namespaceKey] = struct{}{}
}
+ unreadNamespace(ds.Name, namespace, pbsReadDenied(err))
log.Error().
Err(err).
Str("instance", instanceName).
@@ -1994,6 +2036,7 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien
// Skipped groups are not a complete inventory, even when
// every request we did make succeeded.
datastoreNamespaceErrors++
+ unreadNamespace(ds.Name, namespace, false)
break
}
@@ -2067,6 +2110,10 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien
allBackups = appendPBSBackupsWithinLimit(allBackups, fetched.backups)
datastoreNamespaceErrors += fetched.errors
datastoreTerminalNamespaceErrors += fetched.terminalErrors
+ if fetched.errors > 0 {
+ // Snapshot failures carry no status here; do not call them denials.
+ unreadNamespace(ds.Name, namespace, false)
+ }
// Stage successful read times until their matching rows are published.
// A later cancellation must not freshen the previously published rows
@@ -2217,6 +2264,8 @@ func (m *Monitor) pollPBSBackups(ctx context.Context, instanceName string, clien
Err(protectionObservationErr).
Str("instance", instanceName).
Msg("Failed to build PBS protection provider observation")
+ } else {
+ protectionObservation = protectionObservation.WithUnreadScopes(unreadScopes)
}
// Decide whether to keep existing backups when all queries failed
diff --git a/internal/monitoring/monitor_backups_dir_storage_test.go b/internal/monitoring/monitor_backups_dir_storage_test.go
index 1890ab2c05..c08f8c29c3 100644
--- a/internal/monitoring/monitor_backups_dir_storage_test.go
+++ b/internal/monitoring/monitor_backups_dir_storage_test.go
@@ -401,10 +401,12 @@ func TestPVEStorageRunningArchiveOriginLifecycle(t *testing.T) {
if !last.Equal(f.client.at) || running != (phase != "completed") {
t.Fatalf("phase=%s guest completion/running=%s/%t", phase, last, running)
}
+ // Node-a's completed archive is re-read in every phase, so it keeps
+ // the guest protected; node-b's unread disk only limits coverage.
posture := postures[0]
if posture.LastSuccessfulPointAt == nil || !posture.LastSuccessfulPointAt.Equal(f.client.at) ||
- (phase != "partial-read" && posture.State != recovery.ProtectionStateProtected) ||
- (phase == "partial-read" && posture.State == recovery.ProtectionStateProtected) {
+ posture.State != recovery.ProtectionStateProtected ||
+ (phase == "partial-read") != (posture.Coverage == recovery.ProtectionCoveragePartial) {
t.Fatalf("phase=%s wrong scoped protection: %+v", phase, posture)
}
frames[phase] = map[string]any{"pve": map[string]any{"data": snap.PVEBackups}, "postures": map[string]any{"data": postures, "policy": recovery.DefaultProtectionPosturePolicy.Payload()}, "vms": snap.VMs, "containers": snap.Containers}
diff --git a/internal/monitoring/monitor_pbs_coverage_test.go b/internal/monitoring/monitor_pbs_coverage_test.go
index 1511d0c23b..bb7006bf59 100644
--- a/internal/monitoring/monitor_pbs_coverage_test.go
+++ b/internal/monitoring/monitor_pbs_coverage_test.go
@@ -821,9 +821,19 @@ func TestPBSNamespaceDiscoveryFirstPollDoesNotInventScope(t *testing.T) {
if len(f.points(t, "pbs-main")) != 1 || f.childReads.Load() != 0 {
t.Fatal("failed first discovery invented a child namespace or lost readable root history")
}
+ // The root snapshot was read; only undiscovered namespaces are
+ // unknown, so coverage is partial and names the datastore.
for _, p := range f.postures(t) {
- if p.State == recovery.ProtectionStateProtected {
- t.Errorf("incomplete first discovery claimed current protection: %+v", p)
+ if len(p.ProviderStates) == 0 {
+ if p.State == recovery.ProtectionStateProtected {
+ t.Errorf("guest without evidence claimed protection: %+v", p)
+ }
+ continue
+ }
+ if p.State != recovery.ProtectionStateProtected || p.Coverage != recovery.ProtectionCoveragePartial ||
+ len(p.ProviderStates[0].UnreadScopes) != 1 ||
+ p.ProviderStates[0].UnreadScopes[0].Kind != recovery.ProtectionUnreadScopeDatastore {
+ t.Errorf("incomplete first discovery lost the read root or its partial coverage: %+v", p)
}
}
})
diff --git a/internal/monitoring/pbs_backup_budget_test.go b/internal/monitoring/pbs_backup_budget_test.go
index c3a8a867ce..c4ff1c9b90 100644
--- a/internal/monitoring/pbs_backup_budget_test.go
+++ b/internal/monitoring/pbs_backup_budget_test.go
@@ -223,8 +223,14 @@ func exercisePBSBudgetAcceptedListings(t *testing.T) {
}
}
for _, posture := range f.postures(t) {
- if posture.State == recovery.ProtectionStateProtected || len(posture.ProviderStates) != 1 || posture.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || posture.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown {
- t.Errorf("partial collection invented protection: %+v", posture)
+ if len(posture.ProviderStates) != 1 || posture.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || posture.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown || posture.Coverage != recovery.ProtectionCoveragePartial {
+ t.Errorf("partial collection invented complete history: %+v", posture)
+ }
+ // Only guest 100's snapshot was accepted before the budget
+ // expired; at the tasks boundary it is still being written.
+ wantProtected := posture.SubjectResourceID == f.ids[0] && boundary != "tasks"
+ if (posture.State == recovery.ProtectionStateProtected) != wantProtected {
+ t.Errorf("partial collection protection does not follow the accepted read (protected=%t): %+v", wantProtected, posture)
}
}
frames["partial"] = f.frame(t)
diff --git a/internal/monitoring/pbs_inventory_envelope_test.go b/internal/monitoring/pbs_inventory_envelope_test.go
index 0316b15094..2c256fe372 100644
--- a/internal/monitoring/pbs_inventory_envelope_test.go
+++ b/internal/monitoring/pbs_inventory_envelope_test.go
@@ -85,10 +85,16 @@ func TestPBSMissingInventoryArrayRetainsUnreadScope(t *testing.T) {
}
continue
}
- if p.State == recovery.ProtectionStateProtected || len(p.ProviderStates) != 1 ||
+ if len(p.ProviderStates) != 1 ||
p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial ||
- p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown {
- t.Errorf("missing inventory invented complete protection or denied access: %+v", p)
+ p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown ||
+ p.Coverage != recovery.ProtectionCoveragePartial {
+ t.Errorf("missing inventory invented complete history or denied access: %+v", p)
+ }
+ // The retained tenant point was not re-read and may have
+ // been pruned since; the root snapshot was read this poll.
+ if wantProtected := p.SubjectResourceID == f.ids[0]; (p.State == recovery.ProtectionStateProtected) != wantProtected {
+ t.Errorf("protection does not follow what this poll read (protected=%t): %+v", wantProtected, p)
}
}
}
@@ -169,7 +175,10 @@ func TestPBSMissingInventoryArrayConsumerLifecycle(t *testing.T) {
}
postures := f.postures(t)
for _, p := range postures {
- healthy := phase == "observed" || phase == "recovered"
+ // Incomplete phases still re-read the root snapshot; the retained
+ // tenant point does not count until a read confirms it again.
+ incomplete := phase == "missing-namespaces" || phase == "null-groups" || phase == "null-snapshots"
+ healthy := phase == "observed" || phase == "recovered" || (incomplete && p.SubjectResourceID == f.ids[0])
if healthy != (p.State == recovery.ProtectionStateProtected) {
t.Fatalf("phase %s posture=%+v", phase, p)
}
diff --git a/internal/monitoring/pbs_inventory_response_test.go b/internal/monitoring/pbs_inventory_response_test.go
index 1d29d37c3a..8cba8fdea7 100644
--- a/internal/monitoring/pbs_inventory_response_test.go
+++ b/internal/monitoring/pbs_inventory_response_test.go
@@ -67,8 +67,12 @@ func TestPBSWholeInventoryResponseRetainsEvidence(t *testing.T) {
}
continue
}
- if p.State == recovery.ProtectionStateProtected || len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown {
- t.Errorf("unread response claimed protection or denied access: %+v", p)
+ if len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown || p.Coverage != recovery.ProtectionCoveragePartial {
+ t.Errorf("unread response claimed complete history or denied access: %+v", p)
+ }
+ // Only the root snapshot was re-read this poll.
+ if wantProtected := p.SubjectResourceID == f.ids[0]; (p.State == recovery.ProtectionStateProtected) != wantProtected {
+ t.Errorf("unread response protection does not follow what was read (protected=%t): %+v", wantProtected, p)
}
}
}
@@ -125,7 +129,10 @@ func TestPBSWholeInventoryConsumerLifecycle(t *testing.T) {
}
for _, p := range f.postures(t) {
if p.SubjectResourceID == f.ids[1] || p.SubjectResourceID == f.ids[0] {
- if (phase == "observed" || phase == "recovered") != (p.State == recovery.ProtectionStateProtected) {
+ // Unread phases still re-read the root snapshot.
+ unread := phase == "unread-namespaces" || phase == "unread-groups" || phase == "unread-snapshots"
+ healthy := phase == "observed" || phase == "recovered" || (unread && p.SubjectResourceID == f.ids[0])
+ if healthy != (p.State == recovery.ProtectionStateProtected) {
t.Fatalf("phase %s posture=%+v", phase, p)
}
}
diff --git a/internal/monitoring/pbs_namespace_continuity_test.go b/internal/monitoring/pbs_namespace_continuity_test.go
index d3cdac5b5d..8af05dacc9 100644
--- a/internal/monitoring/pbs_namespace_continuity_test.go
+++ b/internal/monitoring/pbs_namespace_continuity_test.go
@@ -167,9 +167,18 @@ func TestPBSNamespaceTransientGroupReadKeepsLiveInventory(t *testing.T) {
}
}
}
+ // The accepted sibling was re-read; the failed namespace's
+ // retained point was not and cannot count.
+ acceptedID := f.ids[0]
+ if accepted == "101" {
+ acceptedID = f.ids[1]
+ }
for _, p := range f.postures(t) {
- if p.State == recovery.ProtectionStateProtected || len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown {
- t.Errorf("incomplete history claimed current protection: %+v", p)
+ if len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown || p.Coverage != recovery.ProtectionCoveragePartial {
+ t.Errorf("incomplete history claimed complete coverage: %+v", p)
+ }
+ if (p.State == recovery.ProtectionStateProtected) != (p.SubjectResourceID == acceptedID) {
+ t.Errorf("protection does not follow the re-read namespace: %+v", p)
}
}
}
@@ -224,9 +233,11 @@ func TestPBSNamespaceDiscoveryContinuityAndAccessWithdrawal(t *testing.T) {
if f.tenantReads.Load() != childReads {
t.Error("discovery failure guessed a hidden namespace read")
}
+ // The root namespace was still read; only the tenant is unread.
for _, p := range f.postures(t) {
- if p.State == recovery.ProtectionStateProtected {
- t.Errorf("failed discovery claimed protection: %+v", p)
+ if (p.State == recovery.ProtectionStateProtected) != (p.SubjectResourceID == f.ids[0]) ||
+ (p.State == recovery.ProtectionStateProtected && p.Coverage != recovery.ProtectionCoveragePartial) {
+ t.Errorf("failed discovery protection does not follow the read root: %+v", p)
}
}
}
@@ -369,7 +380,10 @@ func TestPBSNamespaceContinuityLifecycle(t *testing.T) {
}
postures := f.postures(t)
for _, p := range postures {
- healthy := phase == "observed" || phase == "recovered" || phase == "restored"
+ // Tenant failures leave the root snapshot re-read each poll.
+ tenantUnread := phase == "throttled" || phase == "timed-out" || phase == "denied"
+ healthy := phase == "observed" || phase == "recovered" || phase == "restored" ||
+ (tenantUnread && p.SubjectResourceID == f.ids[0])
if healthy != (p.State == recovery.ProtectionStateProtected) {
t.Fatalf("phase %s posture=%+v", phase, p)
}
diff --git a/internal/monitoring/pbs_poll_posture_test.go b/internal/monitoring/pbs_poll_posture_test.go
index d5eb3d2858..e432799791 100644
--- a/internal/monitoring/pbs_poll_posture_test.go
+++ b/internal/monitoring/pbs_poll_posture_test.go
@@ -6,6 +6,7 @@ import (
"fmt"
"net/http"
"net/http/httptest"
+ "reflect"
"strconv"
"strings"
"sync/atomic"
@@ -283,8 +284,14 @@ func TestPBSOrdinaryPollSnapshotFailuresDoNotClaimCompleteHistory(t *testing.T)
t.Errorf("raw snapshot rows=%d, want %d", got, wantRaw)
}
for _, p := range f.postures(t) {
- if p.State != tc.state || len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || p.ProviderStates[0].Permissions != tc.permissions || p.ProviderStates[0].JobState != recovery.OutcomeWarning {
- t.Errorf("snapshot failure posture=%+v, want %s/partial/%s/warning", p, tc.state, tc.permissions)
+ // Only guest 100's snapshot read fails; guest 101's snapshot
+ // is re-read and stays protected under partial coverage.
+ state := tc.state
+ if p.SubjectResourceID == f.ids[1] {
+ state = recovery.ProtectionStateProtected
+ }
+ if p.State != state || p.Coverage != recovery.ProtectionCoveragePartial || len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || p.ProviderStates[0].Permissions != tc.permissions || p.ProviderStates[0].JobState != recovery.OutcomeWarning {
+ t.Errorf("snapshot failure posture=%+v, want %s/partial/%s/warning", p, state, tc.permissions)
}
}
// No TTL manipulation on the next poll: a failed refresh must not
@@ -309,8 +316,11 @@ func TestPBSOrdinaryPollPartialNamespaceRetainsUnseenHistory(t *testing.T) {
if len(f.points(t, "pbs-main")) != 4 {
t.Error("partial namespace poll deleted unseen historical points")
}
+ // Both guests' root snapshots were re-read; only the unreadable
+ // namespace's retained points went unconfirmed, and it is named.
for _, p := range f.postures(t) {
- if p.State != recovery.ProtectionStateAttention || len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsPartial {
+ if p.State != recovery.ProtectionStateProtected || p.Coverage != recovery.ProtectionCoveragePartial || len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial || p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsPartial ||
+ !reflect.DeepEqual(p.ProviderStates[0].UnreadScopes, []recovery.ProtectionUnreadScope{{Kind: recovery.ProtectionUnreadScopeNamespace, Storage: "archive", Namespace: "unreadable", AccessDenied: true}}) {
t.Errorf("partial namespace posture=%+v", p)
}
}
diff --git a/internal/monitoring/pbs_protection_observation_test.go b/internal/monitoring/pbs_protection_observation_test.go
index fc6ab19347..be8ba6058b 100644
--- a/internal/monitoring/pbs_protection_observation_test.go
+++ b/internal/monitoring/pbs_protection_observation_test.go
@@ -527,10 +527,15 @@ func TestPBSNamespaceDiscoveryFailureRetainsRecoveryHistory(t *testing.T) {
permissions = operationaltrust.EvidencePermissionsPartial
}
for _, p := range f.postures(t) {
- if p.State == recovery.ProtectionStateProtected || len(p.ProviderStates) != 1 ||
+ if len(p.ProviderStates) != 1 || p.Coverage != recovery.ProtectionCoveragePartial ||
p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial ||
p.ProviderStates[0].Permissions != permissions || p.ProviderStates[0].JobState != recovery.OutcomeWarning {
- t.Errorf("discovery failure presented as complete/healthy: %+v", p)
+ t.Errorf("discovery failure presented as complete history: %+v", p)
+ }
+ // The fresh root archive was read; the child guest's
+ // retained point was not.
+ if (p.State == recovery.ProtectionStateProtected) != (p.SubjectResourceID == f.ids[0]) {
+ t.Errorf("discovery failure protection does not follow the read root: %+v", p)
}
}
if f.childReads.Load() != reads {
diff --git a/internal/monitoring/pbs_retryable_read_test.go b/internal/monitoring/pbs_retryable_read_test.go
index 3807a1f2e1..8aa01fda5f 100644
--- a/internal/monitoring/pbs_retryable_read_test.go
+++ b/internal/monitoring/pbs_retryable_read_test.go
@@ -54,10 +54,14 @@ func TestPBSRetryableReadLifecycle(t *testing.T) {
}
}
for _, p := range f.postures(t) {
- if p.State == recovery.ProtectionStateProtected || len(p.ProviderStates) != 1 ||
+ if len(p.ProviderStates) != 1 || p.Coverage != recovery.ProtectionCoveragePartial ||
p.ProviderStates[0].Permissions != operationaltrust.EvidencePermissionsUnknown ||
p.ProviderStates[0].HistoryCompleteness != recovery.ProtectionHistoryPartial {
- t.Errorf("temporary failure invented protection or denied access: %+v", p)
+ t.Errorf("temporary failure invented complete history or denied access: %+v", p)
+ }
+ // The readable root sibling advanced; the tenant did not.
+ if (p.State == recovery.ProtectionStateProtected) != (p.SubjectResourceID == f.ids[0]) {
+ t.Errorf("temporary failure protection does not follow the read sibling: %+v", p)
}
}
}
diff --git a/internal/monitoring/pbs_snapshot_identity_test.go b/internal/monitoring/pbs_snapshot_identity_test.go
index 33b5a13235..944ee8ef32 100644
--- a/internal/monitoring/pbs_snapshot_identity_test.go
+++ b/internal/monitoring/pbs_snapshot_identity_test.go
@@ -85,8 +85,9 @@ func TestPBSConflictingSnapshotIdentityRetainsUnreadScope(t *testing.T) {
}
}
for _, p := range f.postures(t) {
- if p.State == recovery.ProtectionStateProtected {
- t.Errorf("invalid group advertised protected: %+v", p)
+ // The root sibling was read; the malformed tenant group was not.
+ if (p.State == recovery.ProtectionStateProtected) != (p.SubjectResourceID == f.ids[0]) {
+ t.Errorf("invalid group protection does not follow the read root: %+v", p)
}
for _, provider := range p.ProviderStates {
if provider.HistoryCompleteness != recovery.ProtectionHistoryPartial || provider.Permissions != operationaltrust.EvidencePermissionsUnknown {
diff --git a/internal/monitoring/pve_backup_reconciliation_test.go b/internal/monitoring/pve_backup_reconciliation_test.go
index c17764df0b..a60d61b82d 100644
--- a/internal/monitoring/pve_backup_reconciliation_test.go
+++ b/internal/monitoring/pve_backup_reconciliation_test.go
@@ -146,8 +146,18 @@ func (f *pveReconciliationFixture) assertHistory(t *testing.T, history recovery.
if len(p.ProviderStates) != 1 || p.ProviderStates[0].HistoryCompleteness != history || p.ProviderStates[0].Permissions != permissions {
t.Errorf("posture lost source-bound history/access evidence: %+v", p)
}
- if history != recovery.ProtectionHistoryComplete && p.State == recovery.ProtectionStateProtected {
- t.Errorf("incomplete collection claimed protection: %+v", p)
+ if history == recovery.ProtectionHistoryComplete || p.State != recovery.ProtectionStateProtected {
+ continue
+ }
+ // An incomplete poll still confirms the archives it re-read, so a guest
+ // whose backup it read may be protected, but only with partial coverage.
+ // In two-guest fixtures node-b's guest is the unread one: its retained
+ // point was not re-read and may have been pruned, so it never counts.
+ // Single-guest fixtures also keep a re-read archive on node-a.
+ unreadGuest := len(f.ids) > 1 && p.SubjectResourceID == f.ids[1]
+ if history != recovery.ProtectionHistoryPartial || unreadGuest ||
+ p.Coverage != recovery.ProtectionCoveragePartial {
+ t.Errorf("incomplete collection claimed protection from unread history: %+v", p)
}
}
}
diff --git a/internal/monitoring/pve_local_archive_identity_test.go b/internal/monitoring/pve_local_archive_identity_test.go
index eefa582539..04da0cdf2d 100644
--- a/internal/monitoring/pve_local_archive_identity_test.go
+++ b/internal/monitoring/pve_local_archive_identity_test.go
@@ -269,11 +269,16 @@ func TestPVELocalArchiveIdentityConsumerFrames(t *testing.T) {
t.Fatalf("phase=%s live=%d history=%d want%d", phase, len(rows.StorageBackups), len(points), want)
}
for _, p := range postures {
- if (phase == "observed" || phase == "recovered" || phase == "one-removed") && p.State != recovery.ProtectionStateProtected {
+ // In the partial phase node-a's same-path archive is still
+ // re-read, so the guest stays protected with partial coverage.
+ if (phase == "observed" || phase == "recovered" || phase == "one-removed" || phase == "partial") && p.State != recovery.ProtectionStateProtected {
t.Fatalf("accepted completed archives lost posture: %+v", p)
}
- if (phase == "partial" || phase == "empty") && p.State == recovery.ProtectionStateProtected {
- t.Fatal("unread or absent archives claimed protection")
+ if (phase == "partial") != (p.Coverage == recovery.ProtectionCoveragePartial) {
+ t.Fatalf("phase=%s coverage does not reflect the unread disk: %+v", phase, p)
+ }
+ if phase == "empty" && p.State == recovery.ProtectionStateProtected {
+ t.Fatal("absent archives claimed protection")
}
}
frames[phase] = map[string]any{"pve": map[string]any{"data": rows}, "postures": map[string]any{"data": postures, "policy": recovery.DefaultProtectionPosturePolicy.Payload()}, "points": points}
diff --git a/internal/monitoring/pve_local_storage_retention_test.go b/internal/monitoring/pve_local_storage_retention_test.go
index 8487b9d156..917712add7 100644
--- a/internal/monitoring/pve_local_storage_retention_test.go
+++ b/internal/monitoring/pve_local_storage_retention_test.go
@@ -217,12 +217,15 @@ func TestPVELocalStorageAcceptedReadLifecycle(t *testing.T) {
t.Fatalf("phase %s live/recorded inventory = %d/%d, want %d", phase, len(rows.StorageBackups), len(points), want)
}
for _, p := range postures {
- if phase == "observed" || phase == "recovered" {
+ // Node-a's guest keeps a re-read backup through the partial phase;
+ // node-b's retained one was not re-read.
+ readA := phase == "partial" && p.SubjectResourceID == f.ids[0]
+ if phase == "observed" || phase == "recovered" || readA {
if p.State != recovery.ProtectionStateProtected {
t.Fatalf("phase %s accepted completed backup is not protected: %+v", phase, p)
}
} else if p.State == recovery.ProtectionStateProtected {
- t.Fatalf("phase %s claimed protection without complete current backup evidence", phase)
+ t.Fatalf("phase %s claimed protection without a current backup read: %+v", phase, p)
}
}
frames[phase] = map[string]any{"pve": map[string]any{"data": rows}, "postures": map[string]any{"data": postures, "policy": recovery.DefaultProtectionPosturePolicy.Payload()}, "points": points}
diff --git a/internal/monitoring/pve_protection_observation.go b/internal/monitoring/pve_protection_observation.go
index 2f28eabf28..f6909bb816 100644
--- a/internal/monitoring/pve_protection_observation.go
+++ b/internal/monitoring/pve_protection_observation.go
@@ -16,6 +16,61 @@ type pveProtectionCollectionStats struct {
ContentSuccesses int
ContentFailures int
PermissionFailureCount int
+ UnreadScopes []recovery.ProtectionUnreadScope
+}
+
+// pveUnreadReads is the unread part of one PVE backup enumeration.
+type pveUnreadReads struct {
+ nodes map[string]struct{} // nodes whose storage list was not read
+ local map[pveLocalBackupStorage]struct{} // node-local storages not read
+ shared map[string]struct{} // shared storages no node read
+ sharedReads map[pveLocalBackupStorage]struct{} // nodes a shared read failed on
+ denied map[pveLocalBackupStorage]struct{} // reads rejected by authorization
+}
+
+// pveUnreadProtectionScopes names what a PVE enumeration could not read. A
+// shared storage names its node only when exactly one node attempted it.
+func pveUnreadProtectionScopes(unread pveUnreadReads) []recovery.ProtectionUnreadScope {
+ var scopes []recovery.ProtectionUnreadScope
+ for node := range unread.nodes {
+ _, denied := unread.denied[pveLocalBackupStorage{node: node}]
+ scopes = append(scopes, recovery.ProtectionUnreadScope{
+ Kind: recovery.ProtectionUnreadScopeNode,
+ Node: node,
+ AccessDenied: denied,
+ })
+ }
+ for local := range unread.local {
+ _, denied := unread.denied[local]
+ scopes = append(scopes, recovery.ProtectionUnreadScope{
+ Kind: recovery.ProtectionUnreadScopeStorage,
+ Node: local.node,
+ Storage: local.storage,
+ AccessDenied: denied,
+ })
+ }
+ for storage := range unread.shared {
+ scope := recovery.ProtectionUnreadScope{
+ Kind: recovery.ProtectionUnreadScopeStorage,
+ Storage: storage,
+ }
+ attempts := 0
+ for read := range unread.sharedReads {
+ if read.storage != storage {
+ continue
+ }
+ attempts++
+ scope.Node = read.node
+ if _, denied := unread.denied[read]; denied {
+ scope.AccessDenied = true
+ }
+ }
+ if attempts != 1 {
+ scope.Node = ""
+ }
+ scopes = append(scopes, scope)
+ }
+ return recovery.NormalizeProtectionUnreadScopes(scopes)
}
func buildPVEProtectionProviderObservation(
@@ -89,7 +144,7 @@ func buildPVEProtectionProviderObservation(
}
}
- return recovery.NewProtectionProviderObservation(
+ observation, err := recovery.NewProtectionProviderObservation(
recovery.ProviderProxmoxPVE,
"pve-backup-enumeration",
instanceName,
@@ -101,6 +156,10 @@ func buildPVEProtectionProviderObservation(
observedAt,
reason,
)
+ if err != nil {
+ return recovery.ProtectionProviderObservation{}, err
+ }
+ return observation.WithUnreadScopes(stats.UnreadScopes), nil
}
func isPVEBackupPermissionError(err error) bool {
diff --git a/internal/recovery/model/posture.go b/internal/recovery/model/posture.go
index bbb6347db6..0c1ab056a4 100644
--- a/internal/recovery/model/posture.go
+++ b/internal/recovery/model/posture.go
@@ -113,6 +113,105 @@ func (completeness ProtectionHistoryCompleteness) Valid() bool {
}
}
+// ProtectionUnreadScopeKind names the level of provider history that a
+// collection could not read.
+type ProtectionUnreadScopeKind string
+
+const (
+ // A PVE node whose storage list could not be read.
+ ProtectionUnreadScopeNode ProtectionUnreadScopeKind = "node"
+ // A PVE storage whose content could not be read. Node is set when the
+ // failed read was made through, or belongs to, one node.
+ ProtectionUnreadScopeStorage ProtectionUnreadScopeKind = "storage"
+ // A PBS datastore whose namespaces could not all be discovered.
+ ProtectionUnreadScopeDatastore ProtectionUnreadScopeKind = "datastore"
+ // A PBS namespace (Namespace "" is the root) whose groups or snapshots
+ // could not all be read.
+ ProtectionUnreadScopeNamespace ProtectionUnreadScopeKind = "namespace"
+)
+
+// MaxProtectionUnreadScopes bounds the unread scopes kept on one observation.
+// The list explains a partial collection; it is not an inventory.
+const MaxProtectionUnreadScopes = 32
+
+// ProtectionUnreadScope names one part of a provider's backup history that its
+// latest collection could not read. Recovery points stored from that part are
+// retained evidence, not current confirmation.
+type ProtectionUnreadScope struct {
+ Kind ProtectionUnreadScopeKind `json:"kind"`
+ Node string `json:"node,omitempty"`
+ Storage string `json:"storage,omitempty"`
+ Namespace string `json:"namespace,omitempty"`
+ AccessDenied bool `json:"accessDenied,omitempty"`
+}
+
+func (scope ProtectionUnreadScope) normalize() (ProtectionUnreadScope, bool) {
+ scope.Node = strings.TrimSpace(scope.Node)
+ scope.Storage = strings.TrimSpace(scope.Storage)
+ scope.Namespace = strings.TrimSpace(scope.Namespace)
+ switch scope.Kind {
+ case ProtectionUnreadScopeNode:
+ scope.Storage, scope.Namespace = "", ""
+ return scope, scope.Node != ""
+ case ProtectionUnreadScopeStorage:
+ scope.Namespace = ""
+ return scope, scope.Storage != ""
+ case ProtectionUnreadScopeDatastore:
+ scope.Node, scope.Namespace = "", ""
+ return scope, scope.Storage != ""
+ case ProtectionUnreadScopeNamespace:
+ scope.Node = ""
+ return scope, scope.Storage != ""
+ default:
+ return scope, false
+ }
+}
+
+func (scope ProtectionUnreadScope) sortKey() string {
+ return strings.Join([]string{
+ string(scope.Kind), scope.Storage, scope.Namespace, scope.Node,
+ }, "\x00")
+}
+
+// NormalizeProtectionUnreadScopes drops invalid entries, merges duplicates
+// (a denied read wins), sorts, and caps the list.
+func NormalizeProtectionUnreadScopes(scopes []ProtectionUnreadScope) []ProtectionUnreadScope {
+ if len(scopes) == 0 {
+ return nil
+ }
+ byKey := make(map[string]ProtectionUnreadScope, len(scopes))
+ for _, scope := range scopes {
+ normalized, ok := scope.normalize()
+ if !ok {
+ continue
+ }
+ key := normalized.sortKey()
+ if existing, seen := byKey[key]; seen {
+ normalized.AccessDenied = normalized.AccessDenied || existing.AccessDenied
+ }
+ byKey[key] = normalized
+ }
+ if len(byKey) == 0 {
+ return nil
+ }
+ out := make([]ProtectionUnreadScope, 0, len(byKey))
+ for _, scope := range byKey {
+ out = append(out, scope)
+ }
+ sort.Slice(out, func(i, j int) bool { return out[i].sortKey() < out[j].sortKey() })
+ if len(out) > MaxProtectionUnreadScopes {
+ out = out[:MaxProtectionUnreadScopes]
+ }
+ return out
+}
+
+func cloneProtectionUnreadScopes(scopes []ProtectionUnreadScope) []ProtectionUnreadScope {
+ if len(scopes) == 0 {
+ return nil
+ }
+ return append([]ProtectionUnreadScope(nil), scopes...)
+}
+
type ProtectionProviderState struct {
Provider Provider `json:"provider"`
Source string `json:"source"`
@@ -125,6 +224,9 @@ type ProtectionProviderState struct {
LastVerifiedAt *time.Time `json:"lastVerifiedAt,omitempty"`
EvidenceIDs []string `json:"evidenceIds"`
VerificationExpected bool `json:"verificationExpected,omitempty"`
+ // UnreadScopes names what the provider's latest collection could not
+ // read. Only a partial or unavailable history carries it.
+ UnreadScopes []ProtectionUnreadScope `json:"unreadScopes,omitempty"`
}
func (state ProtectionProviderState) Clone() ProtectionProviderState {
@@ -133,6 +235,7 @@ func (state ProtectionProviderState) Clone() ProtectionProviderState {
clone.LastSuccessAt = cloneTime(state.LastSuccessAt)
clone.LastVerifiedAt = cloneTime(state.LastVerifiedAt)
clone.EvidenceIDs = append([]string(nil), state.EvidenceIDs...)
+ clone.UnreadScopes = cloneProtectionUnreadScopes(state.UnreadScopes)
return clone
}
@@ -248,11 +351,28 @@ type ProtectionProviderObservation struct {
ObservedAt time.Time `json:"observedAt"`
IngestedAt time.Time `json:"ingestedAt"`
Evidence operationaltrust.EvidenceEnvelope `json:"evidence"`
+ // UnreadScopes names the parts of a partial or unavailable collection that
+ // went unread. It explains the gap; point confirmation does not depend on it.
+ UnreadScopes []ProtectionUnreadScope `json:"unreadScopes,omitempty"`
}
func (observation ProtectionProviderObservation) Clone() ProtectionProviderObservation {
clone := observation
clone.Evidence = observation.Evidence.Clone()
+ clone.UnreadScopes = cloneProtectionUnreadScopes(observation.UnreadScopes)
+ return clone
+}
+
+// WithUnreadScopes returns the observation carrying the normalized unread
+// scopes. A complete collection read everything, so it carries none.
+func (observation ProtectionProviderObservation) WithUnreadScopes(
+ scopes []ProtectionUnreadScope,
+) ProtectionProviderObservation {
+ clone := observation.Clone()
+ clone.UnreadScopes = nil
+ if observation.HistoryCompleteness != ProtectionHistoryComplete {
+ clone.UnreadScopes = NormalizeProtectionUnreadScopes(scopes)
+ }
return clone
}
@@ -343,20 +463,27 @@ type ProtectionPostureQuery struct {
}
type ProtectionProviderSummary struct {
- Provider Provider
- Source string
- Scope string
- JobState Outcome
- HistoryCompleteness ProtectionHistoryCompleteness
- Permissions operationaltrust.EvidencePermissions
- VerificationExpected bool
- LastAttemptAt *time.Time
- LastSuccessAt *time.Time
- LastVerifiedAt *time.Time
- BackupPointCount int
- SnapshotPointCount int
- RepositoryResourceIDs []string
- EvidenceIDs []string
+ Provider Provider
+ Source string
+ Scope string
+ JobState Outcome
+ HistoryCompleteness ProtectionHistoryCompleteness
+ Permissions operationaltrust.EvidencePermissions
+ VerificationExpected bool
+ LastAttemptAt *time.Time
+ LastSuccessAt *time.Time
+ LastVerifiedAt *time.Time
+ // ConfirmedLastSuccessAt and ConfirmedLastVerifiedAt cover only the points
+ // that the provider's latest collection re-read. A partial collection can
+ // still confirm those points; retained points from its unread scopes may
+ // have been pruned since and stay out of these fields.
+ ConfirmedLastSuccessAt *time.Time
+ ConfirmedLastVerifiedAt *time.Time
+ UnreadScopes []ProtectionUnreadScope
+ BackupPointCount int
+ SnapshotPointCount int
+ RepositoryResourceIDs []string
+ EvidenceIDs []string
}
func (summary ProtectionProviderSummary) normalize() ProtectionProviderSummary {
diff --git a/internal/recovery/posture.go b/internal/recovery/posture.go
index bd6559d181..87ec92913b 100644
--- a/internal/recovery/posture.go
+++ b/internal/recovery/posture.go
@@ -310,19 +310,27 @@ func BuildProtectionPostureFromPointsAt(
switch point.Kind {
case KindBackup:
agg.summary.BackupPointCount++
+ observation, observed := latestObservations[key]
+ confirmed := observed && pointConfirmedByObservation(point, observation)
if point.Outcome == OutcomeSuccess && !eventAt.IsZero() {
- if agg.summary.LastSuccessAt == nil || eventAt.After(*agg.summary.LastSuccessAt) {
- value := eventAt.UTC()
- agg.summary.LastSuccessAt = &value
+ agg.summary.LastSuccessAt = latestProtectionTime(agg.summary.LastSuccessAt, &eventAt)
+ if confirmed {
+ agg.summary.ConfirmedLastSuccessAt = latestProtectionTime(
+ agg.summary.ConfirmedLastSuccessAt,
+ &eventAt,
+ )
}
}
if point.Outcome == OutcomeSuccess &&
point.Verified != nil &&
*point.Verified &&
!eventAt.IsZero() {
- if agg.summary.LastVerifiedAt == nil || eventAt.After(*agg.summary.LastVerifiedAt) {
- value := eventAt.UTC()
- agg.summary.LastVerifiedAt = &value
+ agg.summary.LastVerifiedAt = latestProtectionTime(agg.summary.LastVerifiedAt, &eventAt)
+ if confirmed {
+ agg.summary.ConfirmedLastVerifiedAt = latestProtectionTime(
+ agg.summary.ConfirmedLastVerifiedAt,
+ &eventAt,
+ )
}
}
case KindSnapshot:
@@ -351,6 +359,7 @@ func BuildProtectionPostureFromPointsAt(
agg.summary.HistoryCompleteness = observation.HistoryCompleteness
agg.summary.Permissions = observation.Permissions
agg.summary.VerificationExpected = observation.VerificationExpected
+ agg.summary.UnreadScopes = cloneUnreadScopes(observation.UnreadScopes)
agg.summary.EvidenceIDs = append(agg.summary.EvidenceIDs, observation.Evidence.ID)
if !observation.ObservedAt.Before(agg.lastEventAt) &&
observation.JobState != OutcomeUnknown {
@@ -367,6 +376,31 @@ func BuildProtectionPostureFromPointsAt(
return DeriveProtectionPostureAt(subjectResourceID, summaries, policy, now)
}
+// recoveryPointMigrationCollector marks evidence the store backfilled for
+// legacy points. It was never produced by a provider read.
+const recoveryPointMigrationCollector = "recovery-point-migration"
+
+// pointConfirmedByObservation reports whether the provider collection behind
+// observation re-read this point. Collectors renew a point's evidence only for
+// rows a poll actually returned, stamped with that poll's observation time, so
+// evidence ingested no earlier than the observation proves the point still
+// existed on a scope that collection read. A point retained from an unread
+// scope keeps its older ingestion time and may have been pruned since.
+func pointConfirmedByObservation(
+ point RecoveryPoint,
+ observation ProtectionProviderObservation,
+) bool {
+ evidence := point.Evidence
+ if evidence == nil || evidence.IngestedAt.IsZero() {
+ return false
+ }
+ if strings.TrimSpace(evidence.Source.Collector) == recoveryPointMigrationCollector ||
+ strings.TrimSpace(evidence.Source.Provider) != strings.TrimSpace(string(observation.Provider)) {
+ return false
+ }
+ return !evidence.IngestedAt.Before(observation.ObservedAt)
+}
+
func DeriveProtectionPostureAt(
subjectResourceID string,
summaries []ProtectionProviderSummary,
@@ -404,6 +438,7 @@ func DeriveProtectionPostureAt(
hasSupportedQualifyingSuccess := false
hasInvalidatingFailure := false
hasSnapshotsOnly := false
+ var unreadScopes []ProtectionUnreadScope
verificationExpected := policy.RequireVerification
supportedVerificationExpected := policy.RequireVerification
var supportedLastSuccessAt *time.Time
@@ -443,6 +478,10 @@ func DeriveProtectionPostureAt(
EvidenceIDs: append([]string(nil), summary.EvidenceIDs...),
VerificationExpected: summary.VerificationExpected,
}
+ if summary.HistoryCompleteness != ProtectionHistoryComplete {
+ state.UnreadScopes = NormalizeProtectionUnreadScopes(summary.UnreadScopes)
+ unreadScopes = append(unreadScopes, state.UnreadScopes...)
+ }
posture.ProviderStates = append(posture.ProviderStates, state)
posture.RepositoryResourceIDs = append(
posture.RepositoryResourceIDs,
@@ -483,17 +522,15 @@ func DeriveProtectionPostureAt(
if summary.LastSuccessAt != nil {
hasQualifyingSuccess = true
}
- if summary.LastSuccessAt != nil &&
- summary.HistoryCompleteness == ProtectionHistoryComplete &&
- summary.Permissions == operationaltrust.EvidencePermissionsSufficient {
+ if supportedSuccessAt, supportedVerifiedAt := supportedProtectionTimes(summary); supportedSuccessAt != nil {
hasSupportedQualifyingSuccess = true
supportedLastSuccessAt = latestProtectionTime(
supportedLastSuccessAt,
- summary.LastSuccessAt,
+ supportedSuccessAt,
)
supportedLastVerifiedAt = latestProtectionTime(
supportedLastVerifiedAt,
- summary.LastVerifiedAt,
+ supportedVerifiedAt,
)
if summary.VerificationExpected {
supportedVerificationExpected = true
@@ -571,15 +608,23 @@ func DeriveProtectionPostureAt(
case hasSupportedQualifyingSuccess &&
posture.Freshness == ProtectionFreshnessCurrent &&
!hasInvalidatingFailure &&
- !hasPartial &&
(!stateVerificationExpected || posture.Verification == ProtectionVerificationVerified):
+ // A backup the latest collection read stays a backup when some other
+ // storage could not be read; coverage carries that gap.
posture.State = ProtectionStateProtected
- if posture.Verification == ProtectionVerificationVerified {
+ switch {
+ case posture.Verification == ProtectionVerificationVerified:
posture.Explanation = "A recent backup is available and it was verified recently."
- } else {
+ case hasPartial || hasUnknownBlock:
+ posture.Explanation = "A recent backup is available."
+ default:
posture.Explanation = "A recent backup is available, and Pulse can see the full backup history."
}
- if hasUnknownBlock {
+ if names := describeUnreadScopes(unreadScopes); names != "" {
+ posture.Explanation += " Pulse could not read " + names + ", but that does not change this backup."
+ } else if hasPartial {
+ posture.Explanation += " Part of the backup history could not be read, but that does not change this backup."
+ } else if hasUnknownBlock {
posture.Explanation += " Another backup source could not be read, but that does not change this backup."
}
case hasSupportedQualifyingSuccess:
@@ -608,6 +653,9 @@ func DeriveProtectionPostureAt(
} else {
posture.Explanation = "Pulse can see only part of the backup history, so it cannot confirm a backup."
}
+ if names := describeUnreadScopes(unreadScopes); names != "" {
+ posture.Explanation += " Pulse could not read " + names + "."
+ }
case hasCompleteHistory:
posture.State = ProtectionStateUnprotected
if hasSnapshotsOnly {
@@ -623,6 +671,77 @@ func DeriveProtectionPostureAt(
return posture
}
+// supportedProtectionTimes returns the success and verification times a
+// provider summary can stand behind. Complete history with sufficient access
+// supports every point. A partial collection supports only the points it
+// re-read; denied or unavailable history supports none.
+func supportedProtectionTimes(summary ProtectionProviderSummary) (*time.Time, *time.Time) {
+ switch {
+ case summary.LastSuccessAt != nil &&
+ summary.HistoryCompleteness == ProtectionHistoryComplete &&
+ summary.Permissions == operationaltrust.EvidencePermissionsSufficient:
+ return summary.LastSuccessAt, summary.LastVerifiedAt
+ case summary.ConfirmedLastSuccessAt != nil &&
+ summary.HistoryCompleteness == ProtectionHistoryPartial &&
+ (summary.Permissions == operationaltrust.EvidencePermissionsPartial ||
+ summary.Permissions == operationaltrust.EvidencePermissionsUnknown):
+ return summary.ConfirmedLastSuccessAt, summary.ConfirmedLastVerifiedAt
+ default:
+ return nil, nil
+ }
+}
+
+// describeUnreadScopes names unread provider scopes for an explanation, e.g.
+// "pbs-offsite on delly". It names at most three and counts the rest.
+func describeUnreadScopes(scopes []ProtectionUnreadScope) string {
+ scopes = NormalizeProtectionUnreadScopes(scopes)
+ if len(scopes) == 0 {
+ return ""
+ }
+ names := make([]string, 0, len(scopes))
+ for _, scope := range scopes {
+ names = append(names, DescribeProtectionUnreadScope(scope))
+ }
+ const named = 3
+ if len(names) > named {
+ rest := len(names) - named
+ names = append(names[:named], fmt.Sprintf("%d more", rest))
+ }
+ if len(names) == 1 {
+ return names[0]
+ }
+ return strings.Join(names[:len(names)-1], ", ") + " and " + names[len(names)-1]
+}
+
+// DescribeProtectionUnreadScope renders one unread scope in operator terms.
+func DescribeProtectionUnreadScope(scope ProtectionUnreadScope) string {
+ switch scope.Kind {
+ case ProtectionUnreadScopeNode:
+ return "the storage list on node " + scope.Node
+ case ProtectionUnreadScopeStorage:
+ if scope.Node != "" {
+ return scope.Storage + " on " + scope.Node
+ }
+ return scope.Storage
+ case ProtectionUnreadScopeDatastore:
+ return "the namespace list of datastore " + scope.Storage
+ case ProtectionUnreadScopeNamespace:
+ if scope.Namespace == "" {
+ return "the root namespace of datastore " + scope.Storage
+ }
+ return "namespace " + scope.Namespace + " on datastore " + scope.Storage
+ default:
+ return ""
+ }
+}
+
+func cloneUnreadScopes(scopes []ProtectionUnreadScope) []ProtectionUnreadScope {
+ if len(scopes) == 0 {
+ return nil
+ }
+ return append([]ProtectionUnreadScope(nil), scopes...)
+}
+
func recoveryPointObservedAt(point RecoveryPoint) time.Time {
if point.CompletedAt != nil && !point.CompletedAt.IsZero() {
return point.CompletedAt.UTC()
diff --git a/internal/recovery/posture_test.go b/internal/recovery/posture_test.go
index f1eb046496..c4204acecd 100644
--- a/internal/recovery/posture_test.go
+++ b/internal/recovery/posture_test.go
@@ -738,3 +738,116 @@ func TestProtectionExplanationDoesNotEraseUnsuccessfulBackupEvidence(t *testing.
})
}
}
+
+// A partial provider collection supports exactly the points it re-read. A
+// retained point from an unread scope has no newer evidence and may have been
+// pruned since, so it cannot stand behind a protected claim.
+func TestBuildProtectionPosturePartialCollectionConfirmsOnlyReReadPoints(t *testing.T) {
+ t.Parallel()
+ now := time.Date(2026, 11, 1, 12, 0, 0, 0, time.UTC)
+ observedAt := now.Add(-time.Minute)
+ backupAt := now.Add(-3 * time.Hour)
+ verified := true
+ unread := []ProtectionUnreadScope{{Kind: ProtectionUnreadScopeStorage, Node: "delly", Storage: "pbs-energy-offsite"}}
+ for _, provider := range []Provider{ProviderProxmoxPVE, ProviderProxmoxPBS} {
+ for _, permissions := range []operationaltrust.EvidencePermissions{
+ operationaltrust.EvidencePermissionsUnknown,
+ operationaltrust.EvidencePermissionsPartial,
+ } {
+ t.Run(fmt.Sprintf("%s/%s", provider, permissions), func(t *testing.T) {
+ observation, err := NewProtectionProviderObservation(provider, "inventory", "source-a", OutcomeWarning, ProtectionHistoryPartial, permissions, true, observedAt, observedAt, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ observation = observation.WithUnreadScopes(unread)
+ withEvidence := func(collector string, ingestedAt time.Time) RecoveryPoint {
+ point := RecoveryPoint{ID: "backup", Provider: provider, ProviderScope: "source-a", Kind: KindBackup, Mode: ModeRemote, Outcome: OutcomeSuccess, CompletedAt: &backupAt, Verified: &verified, SubjectResourceID: "resource:vm-100"}
+ evidence, err := NewRecoveryPointEvidence(point, collector, ingestedAt)
+ if err != nil {
+ t.Fatal(err)
+ }
+ point.Evidence = evidence
+ return point
+ }
+ build := func(point RecoveryPoint) ProtectionPosture {
+ return BuildProtectionPostureFromPointsAt("resource:vm-100", []RecoveryPoint{point}, []ProtectionProviderObservation{observation}, DefaultProtectionPosturePolicy, now)
+ }
+
+ got := build(withEvidence("inventory", observedAt))
+ if got.State != ProtectionStateProtected || got.Coverage != ProtectionCoveragePartial || got.Verification != ProtectionVerificationVerified {
+ t.Fatalf("re-read point = %s/%s/%s, want protected/partial/verified", got.State, got.Coverage, got.Verification)
+ }
+ if !strings.Contains(got.Explanation, "Pulse could not read pbs-energy-offsite on delly, but that does not change this backup.") {
+ t.Errorf("explanation = %q", got.Explanation)
+ }
+ if len(got.ProviderStates) != 1 || !reflect.DeepEqual(got.ProviderStates[0].UnreadScopes, unread) {
+ t.Errorf("provider state unread scopes = %+v", got.ProviderStates)
+ }
+ if err := got.Validate(); err != nil {
+ t.Fatal(err)
+ }
+
+ for name, point := range map[string]RecoveryPoint{
+ "retained": withEvidence("inventory", observedAt.Add(-time.Millisecond)),
+ "migration": withEvidence(recoveryPointMigrationCollector, now),
+ "legacy": func() RecoveryPoint {
+ p := withEvidence("inventory", observedAt)
+ p.Evidence = nil
+ return p
+ }(),
+ } {
+ got := build(point)
+ if got.State != ProtectionStateAttention || got.Coverage != ProtectionCoveragePartial {
+ t.Errorf("%s point = %s/%s, want attention/partial", name, got.State, got.Coverage)
+ }
+ if !strings.Contains(got.Explanation, "Pulse could not read pbs-energy-offsite on delly.") {
+ t.Errorf("%s explanation does not name the unread storage: %q", name, got.Explanation)
+ }
+ }
+ })
+ }
+ }
+}
+
+// Denied or unavailable history confirms nothing, even for a renewed point.
+func TestProtectionPostureUnavailableHistoryConfirmsNothing(t *testing.T) {
+ t.Parallel()
+ now := time.Date(2026, 11, 1, 12, 0, 0, 0, time.UTC)
+ recent := now.Add(-time.Hour)
+ for _, history := range []ProtectionHistoryCompleteness{ProtectionHistoryUnavailable, ProtectionHistoryUnknown} {
+ summary := protectionTestSummary(now, &recent)
+ summary.HistoryCompleteness = history
+ summary.Permissions = operationaltrust.EvidencePermissionsUnknown
+ summary.ConfirmedLastSuccessAt = &recent
+ summary.ConfirmedLastVerifiedAt = &recent
+ if got := DeriveProtectionPostureAt("resource:vm-100", []ProtectionProviderSummary{summary}, DefaultProtectionPosturePolicy, now); got.State != ProtectionStateUnknown {
+ t.Errorf("%s history = %s, want unknown", history, got.State)
+ }
+ }
+ summary := protectionTestSummary(now, &recent)
+ summary.HistoryCompleteness = ProtectionHistoryPartial
+ summary.Permissions = operationaltrust.EvidencePermissionsDenied
+ summary.ConfirmedLastSuccessAt = &recent
+ if got := DeriveProtectionPostureAt("resource:vm-100", []ProtectionProviderSummary{summary}, DefaultProtectionPosturePolicy, now); got.State == ProtectionStateProtected {
+ t.Errorf("denied access claimed protection: %+v", got)
+ }
+}
+
+func TestDescribeUnreadScopesNamesAtMostThree(t *testing.T) {
+ t.Parallel()
+ scopes := []ProtectionUnreadScope{
+ {Kind: ProtectionUnreadScopeNode, Node: "pve2"},
+ {Kind: ProtectionUnreadScopeStorage, Storage: "nfs"},
+ {Kind: ProtectionUnreadScopeNamespace, Storage: "archive"},
+ {Kind: ProtectionUnreadScopeNamespace, Storage: "archive", Namespace: "tenant"},
+ {Kind: ProtectionUnreadScopeDatastore, Storage: "archive"},
+ {Kind: "bogus", Storage: "ignored"},
+ }
+ want := "the namespace list of datastore archive, the root namespace of datastore archive, namespace tenant on datastore archive and 2 more"
+ if got := describeUnreadScopes(scopes); got != want {
+ t.Fatalf("describeUnreadScopes() = %q\nwant %q", got, want)
+ }
+ if got := describeUnreadScopes(scopes[:2]); got != "the storage list on node pve2 and nfs" {
+ t.Fatalf("two scopes = %q", got)
+ }
+}
diff --git a/internal/recovery/store/store.go b/internal/recovery/store/store.go
index 199ce7a09b..84af4b63f8 100644
--- a/internal/recovery/store/store.go
+++ b/internal/recovery/store/store.go
@@ -687,6 +687,9 @@ func (s *Store) initSchema() error {
if err := s.ensureColumn("recovery_points", "evidence_json", "TEXT"); err != nil {
return err
}
+ if err := s.ensureColumn("protection_provider_observations", "unread_scopes_json", "TEXT"); err != nil {
+ return err
+ }
// Migration: normalized index columns for efficient filtering/searching and consistent UI display.
for _, col := range []struct {
diff --git a/internal/recovery/store/store_posture.go b/internal/recovery/store/store_posture.go
index 5c64af212b..ee109284c1 100644
--- a/internal/recovery/store/store_posture.go
+++ b/internal/recovery/store/store_posture.go
@@ -231,8 +231,8 @@ func (s *Store) UpsertProtectionProviderObservations(
id, provider, source, scope, job_state,
history_completeness, permissions, verification_expected,
observed_at_ms, ingested_at_ms, evidence_json,
- created_at_ms, updated_at_ms
- ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+ unread_scopes_json, created_at_ms, updated_at_ms
+ ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(id) DO UPDATE SET
provider=excluded.provider,
source=excluded.source,
@@ -244,6 +244,7 @@ func (s *Store) UpsertProtectionProviderObservations(
observed_at_ms=excluded.observed_at_ms,
ingested_at_ms=excluded.ingested_at_ms,
evidence_json=excluded.evidence_json,
+ unread_scopes_json=excluded.unread_scopes_json,
updated_at_ms=excluded.updated_at_ms
`)
if err != nil {
@@ -264,6 +265,14 @@ func (s *Store) UpsertProtectionProviderObservations(
if err != nil {
return err
}
+ var unreadScopesJSON any
+ if scopes := recovery.NormalizeProtectionUnreadScopes(observation.UnreadScopes); len(scopes) > 0 {
+ encoded, err := json.Marshal(scopes)
+ if err != nil {
+ return err
+ }
+ unreadScopesJSON = string(encoded)
+ }
verificationExpected := 0
if observation.VerificationExpected {
verificationExpected = 1
@@ -281,6 +290,7 @@ func (s *Store) UpsertProtectionProviderObservations(
observation.ObservedAt.UTC().UnixMilli(),
observation.IngestedAt.UTC().UnixMilli(),
string(evidenceJSON),
+ unreadScopesJSON,
nowMs,
nowMs,
); err != nil {
@@ -470,7 +480,8 @@ func (s *Store) listLatestProtectionProviderObservations(
observation.scope, observation.job_state,
observation.history_completeness, observation.permissions,
observation.verification_expected, observation.observed_at_ms,
- observation.ingested_at_ms, observation.evidence_json
+ observation.ingested_at_ms, observation.evidence_json,
+ observation.unread_scopes_json
FROM scopes
JOIN protection_provider_observations observation
ON observation.id = (
@@ -496,6 +507,7 @@ func (s *Store) listLatestProtectionProviderObservations(
verificationExpected int
observedAtMs, ingestedAtMs int64
evidenceRaw string
+ unreadScopesRaw sql.NullString
)
if err := rows.Scan(
&observation.ID,
@@ -509,6 +521,7 @@ func (s *Store) listLatestProtectionProviderObservations(
&observedAtMs,
&ingestedAtMs,
&evidenceRaw,
+ &unreadScopesRaw,
); err != nil {
return nil, err
}
@@ -537,6 +550,12 @@ func (s *Store) listLatestProtectionProviderObservations(
if err := observation.Validate(); err != nil {
continue
}
+ // Unread scopes only explain a gap. An unreadable list must not drop
+ // the observation, whose completeness still qualifies its points.
+ var unreadScopes []recovery.ProtectionUnreadScope
+ if unmarshalJSON(unreadScopesRaw, &unreadScopes) == nil {
+ observation = observation.WithUnreadScopes(unreadScopes)
+ }
out = append(out, observation)
}
return out, rows.Err()
diff --git a/internal/recovery/store/store_posture_test.go b/internal/recovery/store/store_posture_test.go
index d1b950f165..4036dbd2a0 100644
--- a/internal/recovery/store/store_posture_test.go
+++ b/internal/recovery/store/store_posture_test.go
@@ -792,3 +792,90 @@ func TestStoreProtectionPosturePublishesCanonicalLegacySubject(t *testing.T) {
t.Fatalf("obsolete attention posture remains: %+v total=%d err=%v", attention, total, err)
}
}
+
+// A partial collection persists which scope it could not read, and a point it
+// re-read keeps its standing while one it only retained does not.
+func TestStoreProtectionPostureKeepsReReadPointsUnderPartialCollection(t *testing.T) {
+ t.Parallel()
+
+ store, err := Open(filepath.Join(t.TempDir(), "recovery.db"))
+ if err != nil {
+ t.Fatalf("Open() error = %v", err)
+ }
+ defer store.Close()
+ ctx := context.Background()
+
+ now := time.Now().UTC().Truncate(time.Millisecond)
+ backupTime := now.Add(-time.Hour)
+ point := recovery.RecoveryPoint{
+ ID: "pve-backup:cluster:archive:vm-100",
+ Provider: recovery.ProviderProxmoxPVE,
+ Kind: recovery.KindBackup,
+ Mode: recovery.ModeLocal,
+ Outcome: recovery.OutcomeSuccess,
+ CompletedAt: &backupTime,
+ SubjectResourceID: "resource:vm-100",
+ ProviderScope: "cluster",
+ }
+ unread := []recovery.ProtectionUnreadScope{{
+ Kind: recovery.ProtectionUnreadScopeStorage,
+ Node: "delly",
+ Storage: "pbs-energy-offsite",
+ }}
+ collect := func(at time.Time, reRead bool) {
+ t.Helper()
+ observation, err := recovery.NewProtectionProviderObservation(
+ recovery.ProviderProxmoxPVE, "pve-backup-enumeration", "cluster",
+ recovery.OutcomeWarning, recovery.ProtectionHistoryPartial,
+ operationaltrust.EvidencePermissionsUnknown, false, at, at, nil,
+ )
+ if err != nil {
+ t.Fatal(err)
+ }
+ if reRead {
+ evidence, err := recovery.NewRecoveryPointEvidence(point, "pve-storage-backup-inventory", at)
+ if err != nil {
+ t.Fatal(err)
+ }
+ read := point
+ read.Evidence = evidence
+ if err := store.UpsertPoints(ctx, []recovery.RecoveryPoint{read}); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if err := store.UpsertProtectionProviderObservations(
+ ctx,
+ []recovery.ProtectionProviderObservation{observation.WithUnreadScopes(unread)},
+ ); err != nil {
+ t.Fatal(err)
+ }
+ }
+ posture := func() recovery.ProtectionPosture {
+ t.Helper()
+ got, _, err := store.ListProtectionPostures(ctx, recovery.ProtectionPostureQuery{
+ SubjectResourceIDs: []string{"resource:vm-100"},
+ })
+ if err != nil || len(got) != 1 {
+ t.Fatalf("ListProtectionPostures() = %d, %v", len(got), err)
+ }
+ return got[0]
+ }
+
+ collect(now, true)
+ got := posture()
+ if got.State != recovery.ProtectionStateProtected || got.Coverage != recovery.ProtectionCoveragePartial {
+ t.Fatalf("re-read point under partial collection = %s/%s, want protected/partial: %+v", got.State, got.Coverage, got)
+ }
+ if len(got.ProviderStates) != 1 || len(got.ProviderStates[0].UnreadScopes) != 1 ||
+ got.ProviderStates[0].UnreadScopes[0] != unread[0] {
+ t.Fatalf("unread scopes did not survive persistence: %+v", got.ProviderStates)
+ }
+ if !strings.Contains(got.Explanation, "pbs-energy-offsite on delly") {
+ t.Fatalf("explanation does not name the unread storage: %q", got.Explanation)
+ }
+
+ collect(now.Add(time.Minute), false)
+ if got := posture(); got.State != recovery.ProtectionStateAttention {
+ t.Fatalf("retained point the latest collection did not read = %s, want attention", got.State)
+ }
+}
diff --git a/internal/recovery/types.go b/internal/recovery/types.go
index de689647c5..860ce652ce 100644
--- a/internal/recovery/types.go
+++ b/internal/recovery/types.go
@@ -111,6 +111,18 @@ const (
)
type ProtectionProviderState = model.ProtectionProviderState
+type ProtectionUnreadScope = model.ProtectionUnreadScope
+type ProtectionUnreadScopeKind = model.ProtectionUnreadScopeKind
+
+const (
+ ProtectionUnreadScopeNode ProtectionUnreadScopeKind = model.ProtectionUnreadScopeNode
+ ProtectionUnreadScopeStorage ProtectionUnreadScopeKind = model.ProtectionUnreadScopeStorage
+ ProtectionUnreadScopeDatastore ProtectionUnreadScopeKind = model.ProtectionUnreadScopeDatastore
+ ProtectionUnreadScopeNamespace ProtectionUnreadScopeKind = model.ProtectionUnreadScopeNamespace
+)
+
+var NormalizeProtectionUnreadScopes = model.NormalizeProtectionUnreadScopes
+
type ProtectionPosture = model.ProtectionPosture
type ProtectionProviderObservation = model.ProtectionProviderObservation
type ProtectionPosturePolicy = model.ProtectionPosturePolicy