Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Keep unobserved backup task durations unknown
pulse-triage[bot]6441932f3532
diff --git a/docs/release-control/v6/internal/subsystems/frontend-primitives.md b/docs/release-control/v6/internal/subsystems/frontend-primitives.mdindex 4455c8acf6..e680239989 100644--- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md+++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md@@ -20,6 +20,26 @@## Purpose+### Coverage duration requires observed task timing++The existing Coverage task disclosure derives a duration only from positive,+strict RFC3339 endpoints at or before the inventory observation time, with the+end after the start and no reported running status. Future/invalid/nonpositive+or reversed endpoints, or a running task carrying an end time, show **Unknown**+and an explanation, not a numeric completed duration or **Not reported**.+An absent end remains **Not reported**; a valid positive subsecond interval may+round to **0s**. Valid offsets are compared as instants. Task-reported status,+errors, identity, latest-task ordering, protection and restore evidence remain+independent and unchanged. Ordinary polling can withdraw and restore duration+within the same inspected task without replacing the panel or taking focus.++Normal-suite `taskDurationEvidence.test.ts` consumes parsed raw inventory and+covers adverse endpoints/status plus valid, missing, offset and zero controls.+`backupTaskEvidence.test.tsx` exercises those states through the actual inventory+consumer and ordinary polling. Desktop/narrow browser interactions and inspected+pixels cover the changed producer-to-disclosure path. No new requests, provider+operation, collection policy or native guest recovery is implied.+### Replication sync chronology stays unknown when unreadableThe existing Replication table and disclosure validate sync instants beforediff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.mdindex 5c21facf86..16885690ae 100644--- a/docs/release-control/v6/internal/subsystems/storage-recovery.md+++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md@@ -21,6 +21,26 @@## Purpose+### Coverage duration requires observed task timing++The existing Coverage task disclosure derives a duration only from positive,+strict RFC3339 endpoints at or before the inventory observation time, with the+end after the start and no reported running status. Future/invalid/nonpositive+or reversed endpoints, or a running task carrying an end time, show **Unknown**+and an explanation, not a numeric completed duration or **Not reported**.+An absent end remains **Not reported**; a valid positive subsecond interval may+round to **0s**. Valid offsets are compared as instants. Task-reported status,+errors, identity, latest-task ordering, protection and restore evidence remain+independent and unchanged. Ordinary polling can withdraw and restore duration+within the same inspected task without replacing the panel or taking focus.++Normal-suite `taskDurationEvidence.test.ts` consumes parsed raw inventory and+covers adverse endpoints/status plus valid, missing, offset and zero controls.+`backupTaskEvidence.test.tsx` exercises those states through the actual inventory+consumer and ordinary polling. Desktop/narrow browser interactions and inspected+pixels cover the changed producer-to-disclosure path. No new requests, provider+operation, collection policy or native guest recovery is implied.+### Withdrawn backup evidence retains keyboard positionWhen polling removes a focused restore-point disclosure/detail, filtered Coveragediff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.mdindex 9820a1ec25..343bf1d8f7 100644--- a/docs/release-control/v6/internal/subsystems/unified-resources.md+++ b/docs/release-control/v6/internal/subsystems/unified-resources.md@@ -23,6 +23,26 @@ and sort the complete canonical change table while startup and ingestion wait.## Purpose+### Coverage duration requires observed task timing++The existing Coverage task disclosure derives a duration only from positive,+strict RFC3339 endpoints at or before the inventory observation time, with the+end after the start and no reported running status. Future/invalid/nonpositive+or reversed endpoints, or a running task carrying an end time, show **Unknown**+and an explanation, not a numeric completed duration or **Not reported**.+An absent end remains **Not reported**; a valid positive subsecond interval may+round to **0s**. Valid offsets are compared as instants. Task-reported status,+errors, identity, latest-task ordering, protection and restore evidence remain+independent and unchanged. Ordinary polling can withdraw and restore duration+within the same inspected task without replacing the panel or taking focus.++Normal-suite `taskDurationEvidence.test.ts` consumes parsed raw inventory and+covers adverse endpoints/status plus valid, missing, offset and zero controls.+`backupTaskEvidence.test.tsx` exercises those states through the actual inventory+consumer and ordinary polling. Desktop/narrow browser interactions and inspected+pixels cover the changed producer-to-disclosure path. No new requests, provider+operation, collection policy or native guest recovery is implied.+### Withdrawn backup evidence retains keyboard positionWhen polling removes a focused restore-point disclosure/detail, filtered Coveragediff --git a/frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx b/frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsxindex 169f80ca75..da47861607 100644--- a/frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx+++ b/frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx@@ -248,11 +248,14 @@ const buildCoverageTaskSection = (task: CoverageTask, label: string): DetailSect),makeDetailRow('Duration',- task.durationSeconds === undefined- ? 'Not reported'- : task.durationSeconds === 0- ? '0s'- : formatDurationFromSeconds(task.durationSeconds),+ task.durationUnknown+ ? 'Unknown'+ : task.durationSeconds === undefined+ ? 'Not reported'+ : task.durationSeconds === 0+ ? '0s'+ : formatDurationFromSeconds(task.durationSeconds),+ { tone: task.durationUnknown ? 'warning' : undefined },),makeDetailRow('Error',@@ -272,6 +275,11 @@ const buildCoverageTaskSection = (task: CoverageTask, label: string): DetailSect]),footerContent: (<>+ <Show when={task.durationUnknown}>+ <p class="whitespace-normal text-[11px] leading-snug text-muted wrap-anywhere">+ The reported timestamps or running status do not establish a completed duration.+ </p>+ </Show><p class="whitespace-normal text-[11px] leading-snug text-muted wrap-anywhere">Task connection and node are reported by the task, not the workload’s current placement.</p>diff --git a/frontend-modern/src/features/proxmox/__tests__/ProxmoxCoverageTable.test.tsx b/frontend-modern/src/features/proxmox/__tests__/ProxmoxCoverageTable.test.tsxindex dc5db20c84..b766763c68 100644--- a/frontend-modern/src/features/proxmox/__tests__/ProxmoxCoverageTable.test.tsx+++ b/frontend-modern/src/features/proxmox/__tests__/ProxmoxCoverageTable.test.tsx@@ -1136,3 +1136,48 @@ describe('indeterminate task order in the canonical Coverage consumer', () => {},);});++describe('unknown duration in possible-latest task details', () => {+ it.each([320, 1200])('preserves both statuses and only the valid duration at %ipx', (width) => {+ const model = buildProxmoxBackupRecoveryModel({+ workloads: [identityGuest('east-100', 'east', 'pve-1')],+ pbsBackups: [],+ archives: [],+ snapshots: [],+ tasks: [+ identityTask({ id: 'future-end', endTime: '2026-10-04T12:05:00Z' }),+ identityTask({ id: 'reported-failure', status: 'failed', error: 'Retain this error' }),+ ],+ nowMs: BACKUP_IDENTITY_NOW,+ });+ renderInRouter(() => (+ <ProxmoxCoverageTable+ rows={model.coverageRows}+ hasAnyRows+ emptyIcon={<span />}+ emptyTitle=""+ emptyDescription=""+ sortKey={() => 'posture'}+ sortDirection={() => 'asc'}+ onSort={() => {}}+ expandedKey={model.coverageRows[0].key}+ onToggleExpand={() => {}}+ layoutWidth={() => width}+ showTaskColumn+ />+ ));+ const taskCard = (id: string) => screen.getByText(id, { exact: true }).closest('tbody')!;+ const duration = (card: HTMLElement) =>+ within(card).getByText('Duration', { exact: true, selector: 'td' }).closest('tr')!;+ expect(duration(taskCard('future-end'))).toHaveTextContent('Unknown');+ expect(taskCard('future-end')).toHaveTextContent('OK');+ expect(taskCard('future-end')).toHaveTextContent(+ 'The reported timestamps or running status do not establish a completed duration.',+ );+ expect(duration(taskCard('reported-failure'))).toHaveTextContent('5m 0s');+ expect(taskCard('reported-failure')).toHaveTextContent('Failed');+ expect(taskCard('reported-failure')).toHaveTextContent('Retain this error');+ expect(screen.getByTestId('proxmox-coverage-detail')).toHaveTextContent('Task order');+ expect(model.coverageRows[0].posture).toBe('unknown');+ });+});diff --git a/frontend-modern/src/features/proxmox/__tests__/backupTaskEvidence.test.tsx b/frontend-modern/src/features/proxmox/__tests__/backupTaskEvidence.test.tsxindex 409efa7837..1061557a87 100644--- a/frontend-modern/src/features/proxmox/__tests__/backupTaskEvidence.test.tsx+++ b/frontend-modern/src/features/proxmox/__tests__/backupTaskEvidence.test.tsx@@ -120,7 +120,10 @@ describe('latest backup task evidence through the inventory consumer', () => {mount();const section = taskSection(await open());expect(value(section, 'Started')).toHaveTextContent('Unknown');- expect(value(section, 'Duration')).toHaveTextContent('Not reported');+ expect(value(section, 'Duration')).toHaveTextContent('Unknown');+ expect(section).toHaveTextContent(+ 'The reported timestamps or running status do not establish a completed duration.',+ );expect(value(section, 'Error')).toHaveTextContent('Not reported');expect(section).not.toHaveTextContent('Invalid Date');expect(section).not.toHaveTextContent('also-invalid');@@ -128,7 +131,11 @@ describe('latest backup task evidence through the inventory consumer', () => {it.each([{ status: 'running', endTime: undefined, duration: 'Not reported' },- { status: 'OK', endTime: '2026-10-04T10:59:00Z', duration: 'Not reported' },+ { status: 'OK', endTime: '2026-10-04T10:59:00Z', duration: 'Unknown' },+ { status: 'OK', endTime: '2026-10-04T12:00:01Z', duration: 'Unknown' },+ { status: 'running', endTime: '2026-10-04T11:05:00Z', duration: 'Unknown' },+ { status: 'OK', startTime: '2027-01-01T00:00:00Z', endTime: '2027-01-01T00:05:00Z', duration: 'Unknown' },+ { status: 'OK', endTime: undefined, duration: 'Not reported' },{ status: 'OK', endTime: '2026-10-04T11:00:00.100Z', duration: '0s' },])('does not invent completion or protection for $status / $duration', async (extra) => {tasks = [identityTask(extra)];@@ -138,11 +145,41 @@ describe('latest backup task evidence through the inventory consumer', () => {extra.status === 'running' ? 'Running' : 'OK',);expect(value(section, 'Duration')).toHaveTextContent(extra.duration);+ const timingNotice = within(section).queryByText(+ 'The reported timestamps or running status do not establish a completed duration.',+ );+ if (extra.duration === 'Unknown') expect(timingNotice).toBeInTheDocument();+ else expect(timingNotice).toBeNull();expect(within(section).queryByText('Error', { exact: true, selector: 'td' })).toBeNull();expect(screen.getByRole('button', { name: '2 unknown' })).toBeInTheDocument();expect(document.body).not.toHaveTextContent('A current verified backup is available.');});+ it('withdraws and restores duration in the same open task without reclassifying its status', async () => {+ tasks = [identityTask()];+ mount();+ const panel = await open();+ const toggle = screen.getByRole('button', {+ name: 'Collapse details for same-name (VM 100) · PVE connection: east',+ });+ toggle.focus();+ expect(value(taskSection(panel), 'Duration')).toHaveTextContent('5m 0s');+ tasks = [identityTask({ endTime: '2026-10-04T12:05:00Z' })];+ await vi.advanceTimersByTimeAsync(30_000);+ await waitFor(() => expect(value(taskSection(panel), 'Duration')).toHaveTextContent('Unknown'));+ expect(value(taskSection(panel), 'Status')).toHaveTextContent('OK');+ expect(document.activeElement).toBe(toggle);+ expect(screen.getByTestId('proxmox-coverage-detail')).toBe(panel);+ tasks = [identityTask({ endTime: '2026-10-04T11:01:05Z' })];+ await vi.advanceTimersByTimeAsync(30_000);+ await waitFor(() => expect(value(taskSection(panel), 'Duration')).toHaveTextContent('1m 5s'));+ expect(taskSection(panel)).not.toHaveTextContent('do not establish a completed duration');+ expect(document.activeElement).toBe(toggle);+ expect(screen.getByTestId('proxmox-coverage-detail')).toBe(panel);+ expect(screen.getByRole('button', { name: '2 unknown' })).toBeInTheDocument();+ expect(api.fetch).toHaveBeenCalledTimes(6);+ });+it('updates failure, recovery and task removal in the same open panel on ordinary polling', async () => {mount();const panel = await open();diff --git a/frontend-modern/src/features/proxmox/__tests__/taskDurationEvidence.test.ts b/frontend-modern/src/features/proxmox/__tests__/taskDurationEvidence.test.tsnew file mode 100644index 0000000000..254ad93c43--- /dev/null+++ b/frontend-modern/src/features/proxmox/__tests__/taskDurationEvidence.test.ts@@ -0,0 +1,83 @@+import { describe, expect, it } from 'vitest';+import type { BackupTask } from '@/types/api';+import { parsePVEBackupInventory } from '../proxmoxBackupInventory';+import { buildProxmoxBackupRecoveryModel } from '../proxmoxBackupRecoveryModel';+import { BACKUP_IDENTITY_NOW, identityGuest, identityTask } from '../__fixtures__/backupIdentity';++function observe(extra: Partial<BackupTask>, nowMs = BACKUP_IDENTITY_NOW) {+ const pve = parsePVEBackupInventory({+ data: { backupTasks: [identityTask(extra)], storageBackups: [], guestSnapshots: [] },+ });+ return buildProxmoxBackupRecoveryModel({+ workloads: [identityGuest('east-100', 'east', 'pve-1')],+ pbsBackups: [],+ archives: pve.storageBackups,+ snapshots: pve.guestSnapshots,+ tasks: pve.backupTasks,+ nowMs,+ });+}++const unknownCases: { name: string; task: Partial<BackupTask> }[] = [+ { name: 'future end', task: { endTime: '2026-10-04T12:00:01Z' } },+ { name: 'future end across offset', task: { endTime: '2026-10-04T14:00:01+02:00' } },+ {+ name: 'both endpoints in future',+ task: { startTime: '2027-01-01T00:00:00Z', endTime: '2027-01-01T00:05:00Z' },+ },+ { name: 'running with reported end', task: { status: 'running' } },+ { name: 'normalised running status', task: { status: ' RUNNING ' } },+ { name: 'epoch start', task: { startTime: '1970-01-01T00:00:00Z' } },+ {+ name: 'negative instants',+ task: { startTime: '1969-01-01T00:00:00Z', endTime: '1969-01-01T00:05:00Z' },+ },+ { name: 'missing start with reported end', task: { startTime: '' } },+ { name: 'impossible end', task: { endTime: '2026-09-31T11:05:00Z' } },+ { name: 'timezone-less end', task: { endTime: '2026-10-04T11:05:00' } },+ { name: 'reversed interval', task: { endTime: '2026-10-04T10:59:59Z' } },+ { name: 'equal endpoints', task: { endTime: '2026-10-04T11:00:00Z' } },+];++describe('Coverage task duration needs observed, consistent endpoints', () => {+ it.each(unknownCases)('keeps $name unknown without changing provider evidence', ({ task }) => {+ const model = observe({ ...task, error: 'retained provider error' });+ const row = model.coverageRows[0];+ expect(row.latestTask?.durationSeconds).toBeUndefined();+ expect(row.latestTask?.durationUnknown).toBe(true);+ expect(row.latestTask?.status).toBe(task.status ?? 'OK');+ expect(row.latestTask?.error).toBe('retained provider error');+ expect(row.latestTask?.id).toBe('task-100');+ expect(row.workload.resourceId).toBe('east-100');+ expect(row.posture).toBe('unknown');+ expect(model.coverageSummary.recoverableArtifacts).toBe(0);+ });++ it.each([undefined, ''])('keeps an unreported end distinct from bad evidence: %s', (endTime) => {+ const task = observe({ endTime }).coverageRows[0].latestTask!;+ expect(task.durationSeconds).toBeUndefined();+ expect(task.durationUnknown).not.toBe(true);+ });++ it.each([+ { name: 'ordinary OK', task: {}, seconds: 300 },+ { name: 'reported failure', task: { status: 'failed' }, seconds: 300 },+ { name: 'success alias', task: { status: 'success' }, seconds: 300 },+ { name: 'completed alias', task: { status: 'completed' }, seconds: 300 },+ { name: 'end at observation', task: { endTime: '2026-10-04T12:00:00Z' }, seconds: 3600 },+ {+ name: 'offsets describe instants',+ task: { startTime: '2026-10-04T13:00:00+02:00', endTime: '2026-10-04T08:05:00-03:00' },+ seconds: 300,+ },+ { name: 'observed subsecond rounds to zero', task: { endTime: '2026-10-04T11:00:00.1Z' }, seconds: 0 },+ ])('retains $name', ({ task: input, seconds }) => {+ const task = observe(input).coverageRows[0].latestTask!;+ expect(task.durationSeconds).toBe(seconds);+ expect(task.durationUnknown).not.toBe(true);+ });++ it('does not fabricate elapsed time without a finite observation clock', () => {+ expect(observe({}, Number.NaN).coverageRows[0].latestTask).toMatchObject({ durationUnknown: true });+ });+});diff --git a/frontend-modern/src/features/proxmox/proxmoxBackupRecoveryModel.ts b/frontend-modern/src/features/proxmox/proxmoxBackupRecoveryModel.tsindex 01582358b4..eeaf1badae 100644--- a/frontend-modern/src/features/proxmox/proxmoxBackupRecoveryModel.ts+++ b/frontend-modern/src/features/proxmox/proxmoxBackupRecoveryModel.ts@@ -63,6 +63,8 @@ export interface CoverageTask {startedAt: string;startedMs?: number;durationSeconds?: number;+ /** Reported timing/status cannot establish a completed duration; not missing data or zero. */+ durationUnknown?: boolean;error?: string;/** Task-reported origin, independent of the workload's current placement. */instance?: string;@@ -542,11 +544,27 @@ function taskLabel(task: BackupTask): string {return task.status || 'Unknown';}-function taskDurationSeconds(task: BackupTask): number | undefined {+function taskDuration(+ task: BackupTask,+ nowMs: number,+): Pick<CoverageTask, 'durationSeconds' | 'durationUnknown'> {+ if (!task.endTime) return {};const start = parseBackupTimestampMs(task.startTime);const end = parseBackupTimestampMs(task.endTime);- if (start === undefined || end === undefined || end <= start) return undefined;- return Math.round((end - start) / 1000);+ // Parseable endpoints alone do not establish observed elapsed time. Apply+ // the same positive/past-instant rule as task ordering, and never use a+ // contradictory end time to make a running task look completed.+ if (+ normalizeKey(task.status) === 'running' ||+ start === undefined ||+ end === undefined ||+ getRecoveryAgeBand(start, nowMs) === 'unknown' ||+ getRecoveryAgeBand(end, nowMs) === 'unknown' ||+ end <= start+ ) {+ return { durationUnknown: true };+ }+ return { durationSeconds: Math.round((end - start) / 1000) };}function selectLatestTask(@@ -759,7 +777,7 @@ export function buildProxmoxBackupRecoveryModel(startedAt: task.startTime,startedMs:getRecoveryAgeBand(parsedStart, input.nowMs) === 'unknown' ? undefined : parsedStart,- durationSeconds: taskDurationSeconds(task),+ ...taskDuration(task, input.nowMs),error: task.error,instance: task.instance?.trim() || undefined,node: task.node?.trim() || undefined,