Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Explain backup task failures and uncertain ordering in Recovery Help
pulse-triage[bot]352c3dd7959d
diff --git a/docs/PBS.md b/docs/PBS.mdindex 358752580f..5a7f9b3951 100644--- a/docs/PBS.md+++ b/docs/PBS.md@@ -379,6 +379,11 @@ For local PVE `vzdump` archives rather than PBS records, use the[PVE Coverage comparison](RECOVERY.md#local-pve-backups-are-newer-than-coverage).It does not require adding a PBS connection.+For a recent backup beside a failed or unknown **Task**, compare+[task results and backup age](RECOVERY.md#backup-task-results-and-backup-age)+separately. A task result, artifact verification and workload posture answer+different questions; use existing records rather than running another backup.+The **Backup health** strip and the backup list answer different questions.If they disagree, use **PBS's own backup and verification records** for thesame guest while checking Pulse's assessment; do not assume either a faileddiff --git a/docs/RECOVERY.md b/docs/RECOVERY.mdindex b3575d3892..98ca99467a 100644--- a/docs/RECOVERY.md+++ b/docs/RECOVERY.md@@ -202,6 +202,49 @@ new backup or changing stored history. For local `vzdump` archives, use the[PVE comparison below](#local-pve-backups-are-newer-than-coverage); PBS is notrequired.+### Backup task results and backup age++A recent backup and a failed task can both be genuine. In **Proxmox → Backups+→ Coverage**, read these columns separately:++| Column | What to compare |+|---|---|+| **Last backup** (**Backup** or **Age** at narrower widths) | The latest completed backup visible for the selected backup location, not the result of the latest task. A guest snapshot does not fill this column. |+| **Task** | A collected task result for this workload, where the evidence establishes which task started last. It does not verify an artifact or replace its timestamp. |+| **Posture** | Pulse's assessment of subject-linked protection evidence, not a synonym for either task success or backup age. A location filter does not recalculate this assessment. |++Expand the workload, including on a phone, to inspect **Latest task**: **Status**,+**Started**, **Duration**, **Error** and **Task ID**, where reported. Keep the+workload's type, ID, node and **PVE connection** separate from the provider+reader's **Scope**. Match the existing native task and backup record to that+same owning installation and guest, not just a repeated name or VMID.++- **Failed** does not erase an earlier completed backup. Inspect the task's+ error and the existing artifact independently; a recent backup age does not+ cancel the failure either.+- **Running** is not a new completed recovery point. **OK** is a task result,+ not verification, a tested restore or confirmation that the guest thawed.+- If **Task order** says **Latest task: Unknown**, missing, future or tied+ start times prevent Pulse from ordering several possible latest tasks.+ Inspect each **Possible latest task** section. Its number is not a time+ order: do not choose an OK task over a failed one to declare recovery.+- **None** or a dash means no matching recent task is shown, not that no task+ ran in the provider. An **Unknown** status is not OK, and **Not reported**+ under Error is not proof that there was no error. If your version has no+ task details, use the existing native task record rather than assuming success.++Use the records already available. Do not run another backup, verification,+restore or guest-agent probe, restart monitoring, clear History or change+retention merely to reconcile these readings. Keep full task IDs, private+paths and unredacted errors private; a public report needs only the affected+view, running Pulse version, task status/time and a locally reviewed error+summary with consistent aliases.++If a guest is unresponsive or freeze/thaw is uncertain, stop this display+comparison and follow [Backup safety](VM_DISK_MONITORING.md#backup-safety).+Neither an OK task nor a corrected badge confirms independent thaw, fresh+successful writes to every covered filesystem and workload liveness.+### Local PVE backups are newer than CoverageA recent archive in PVE and an old or **Unprotected** Coverage row arediff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.mdindex 8610f2adf7..93cd586bed 100644--- a/docs/TROUBLESHOOTING.md+++ b/docs/TROUBLESHOOTING.md@@ -761,6 +761,11 @@ row, use the [PVE Coverage comparison](RECOVERY.md#local-pve-backups-are-newer-tCompare the owning installation and archive without adding PBS, re-enrollingnodes or changing stored history.+For a recent backup beside a failed or unknown **Task**, use+[task results and backup age](RECOVERY.md#backup-task-results-and-backup-age).+Expand the workload to inspect the task result and any task-order uncertainty;+a recent artifact does not cancel a task failure.+#### Replication jobs are Pending, stale or missing**Proxmox → Replication** monitors storage replication between PVE nodes,diff --git a/frontend-modern/public/docs/PBS.md b/frontend-modern/public/docs/PBS.mdindex 358752580f..5a7f9b3951 100644--- a/frontend-modern/public/docs/PBS.md+++ b/frontend-modern/public/docs/PBS.md@@ -379,6 +379,11 @@ For local PVE `vzdump` archives rather than PBS records, use the[PVE Coverage comparison](RECOVERY.md#local-pve-backups-are-newer-than-coverage).It does not require adding a PBS connection.+For a recent backup beside a failed or unknown **Task**, compare+[task results and backup age](RECOVERY.md#backup-task-results-and-backup-age)+separately. A task result, artifact verification and workload posture answer+different questions; use existing records rather than running another backup.+The **Backup health** strip and the backup list answer different questions.If they disagree, use **PBS's own backup and verification records** for thesame guest while checking Pulse's assessment; do not assume either a faileddiff --git a/frontend-modern/public/docs/RECOVERY.md b/frontend-modern/public/docs/RECOVERY.mdindex b3575d3892..98ca99467a 100644--- a/frontend-modern/public/docs/RECOVERY.md+++ b/frontend-modern/public/docs/RECOVERY.md@@ -202,6 +202,49 @@ new backup or changing stored history. For local `vzdump` archives, use the[PVE comparison below](#local-pve-backups-are-newer-than-coverage); PBS is notrequired.+### Backup task results and backup age++A recent backup and a failed task can both be genuine. In **Proxmox → Backups+→ Coverage**, read these columns separately:++| Column | What to compare |+|---|---|+| **Last backup** (**Backup** or **Age** at narrower widths) | The latest completed backup visible for the selected backup location, not the result of the latest task. A guest snapshot does not fill this column. |+| **Task** | A collected task result for this workload, where the evidence establishes which task started last. It does not verify an artifact or replace its timestamp. |+| **Posture** | Pulse's assessment of subject-linked protection evidence, not a synonym for either task success or backup age. A location filter does not recalculate this assessment. |++Expand the workload, including on a phone, to inspect **Latest task**: **Status**,+**Started**, **Duration**, **Error** and **Task ID**, where reported. Keep the+workload's type, ID, node and **PVE connection** separate from the provider+reader's **Scope**. Match the existing native task and backup record to that+same owning installation and guest, not just a repeated name or VMID.++- **Failed** does not erase an earlier completed backup. Inspect the task's+ error and the existing artifact independently; a recent backup age does not+ cancel the failure either.+- **Running** is not a new completed recovery point. **OK** is a task result,+ not verification, a tested restore or confirmation that the guest thawed.+- If **Task order** says **Latest task: Unknown**, missing, future or tied+ start times prevent Pulse from ordering several possible latest tasks.+ Inspect each **Possible latest task** section. Its number is not a time+ order: do not choose an OK task over a failed one to declare recovery.+- **None** or a dash means no matching recent task is shown, not that no task+ ran in the provider. An **Unknown** status is not OK, and **Not reported**+ under Error is not proof that there was no error. If your version has no+ task details, use the existing native task record rather than assuming success.++Use the records already available. Do not run another backup, verification,+restore or guest-agent probe, restart monitoring, clear History or change+retention merely to reconcile these readings. Keep full task IDs, private+paths and unredacted errors private; a public report needs only the affected+view, running Pulse version, task status/time and a locally reviewed error+summary with consistent aliases.++If a guest is unresponsive or freeze/thaw is uncertain, stop this display+comparison and follow [Backup safety](VM_DISK_MONITORING.md#backup-safety).+Neither an OK task nor a corrected badge confirms independent thaw, fresh+successful writes to every covered filesystem and workload liveness.+### Local PVE backups are newer than CoverageA recent archive in PVE and an old or **Unprotected** Coverage row arediff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.mdindex 8610f2adf7..93cd586bed 100644--- a/frontend-modern/public/docs/TROUBLESHOOTING.md+++ b/frontend-modern/public/docs/TROUBLESHOOTING.md@@ -761,6 +761,11 @@ row, use the [PVE Coverage comparison](RECOVERY.md#local-pve-backups-are-newer-tCompare the owning installation and archive without adding PBS, re-enrollingnodes or changing stored history.+For a recent backup beside a failed or unknown **Task**, use+[task results and backup age](RECOVERY.md#backup-task-results-and-backup-age).+Expand the workload to inspect the task result and any task-order uncertainty;+a recent artifact does not cancel a task failure.+#### Replication jobs are Pending, stale or missing**Proxmox → Replication** monitors storage replication between PVE nodes,diff --git a/frontend-modern/src/features/docs/__tests__/backupTaskEvidenceGuide.test.ts b/frontend-modern/src/features/docs/__tests__/backupTaskEvidenceGuide.test.tsnew file mode 100644index 0000000000..023a7a95aa--- /dev/null+++ b/frontend-modern/src/features/docs/__tests__/backupTaskEvidenceGuide.test.ts@@ -0,0 +1,99 @@+import { readFileSync } from 'node:fs';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { describe, expect, it } from 'vitest';+import { COVERAGE_COLUMNS } from '@/features/proxmox/proxmoxBackupsTablePresentation';+import { createGuideFixture } from './guideFixture';++const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../../..');+const read = (file: string) => readFileSync(path.join(root, file), 'utf8');+const guide = createGuideFixture((name) =>+ name === 'RECOVERY' && process.env.PULSE_RECOVERY_GUIDE_TEST_INPUT+ ? readFileSync(process.env.PULSE_RECOVERY_GUIDE_TEST_INPUT, 'utf8')+ : read(`frontend-modern/public/docs/${name}.md`),+);+const fragment = '#backup-task-results-and-backup-age';+const section = () => {+ const heading = guide('RECOVERY').querySelector(fragment);+ expect(heading?.textContent).toBe('Backup task results and backup age');+ const result = document.createElement('section');+ for (let next = heading!.nextElementSibling; next; next = next.nextElementSibling) {+ if (/^H[1-3]$/.test(next.tagName)) break;+ result.appendChild(next.cloneNode(true));+ }+ return result;+};+const prose = () => section().textContent!.replace(/\s+/g, ' ');++describe('backup task evidence guidance', () => {+ it('keeps both discovery links usable and all three shipped mirrors equal', () => {+ for (const name of ['PBS', 'TROUBLESHOOTING']) {+ const link = guide(name).querySelector(`a[href="/docs/RECOVERY${fragment}"]`);+ expect(link?.textContent).toBe('task results and backup age');+ }+ for (const name of ['RECOVERY', 'PBS', 'TROUBLESHOOTING']) {+ expect(read(`frontend-modern/public/docs/${name}.md`)).toBe(read(`docs/${name}.md`));+ }+ });++ it('separates the actual Coverage columns and preserves failure beside a recent backup', () => {+ const rows = [...section().querySelectorAll('tbody tr')];+ expect(rows).toHaveLength(3);+ for (const id of ['latest', 'task', 'posture']) {+ const column = COVERAGE_COLUMNS.find((value) => value.id === id)!;+ expect(rows.some((row) => row.firstElementChild?.textContent?.includes(column.label))).toBe(+ true,+ );+ }+ const text = prose();+ for (const distinction of [+ 'A recent backup and a failed task can both be genuine',+ 'not the result of the latest task',+ 'A guest snapshot does not fill this column',+ 'A location filter does not recalculate this assessment',+ 'Failed does not erase an earlier completed backup',+ 'a recent backup age does not cancel the failure either',+ 'Running is not a new completed recovery point',+ 'OK is a task result, not verification',+ ]) {+ expect(text).toContain(distinction);+ }+ });++ it('explains uncertain ordering without turning absent evidence into a successful task', () => {+ const text = prose();+ for (const distinction of [+ 'Latest task: Unknown',+ 'missing, future or tied start times',+ 'Possible latest task',+ 'Its number is not a time order',+ 'do not choose an OK task over a failed one',+ 'None or a dash means no matching recent task is shown',+ 'not that no task ran in the provider',+ 'An Unknown status is not OK',+ 'Not reported under Error is not proof that there was no error',+ 'Status, Started, Duration, Error and Task ID',+ 'same owning installation and guest',+ ]) {+ expect(text).toContain(distinction);+ }+ });++ it('keeps comparison passive, private and distinct from the guest-safety procedure', () => {+ const text = prose();+ for (const boundary of [+ 'Use the records already available',+ 'Do not run another backup, verification, restore or guest-agent probe',+ 'restart monitoring, clear History or change retention',+ 'Keep full task IDs, private paths and unredacted errors private',+ 'stop this display comparison',+ 'independent thaw, fresh successful writes to every covered filesystem and workload liveness',+ ]) {+ expect(text).toContain(boundary);+ }+ expect(section().querySelector('pre')).toBeNull();+ const safety = '#backup-safety';+ expect(section().querySelector(`a[href="/docs/VM_DISK_MONITORING${safety}"]`)).not.toBeNull();+ expect(guide('VM_DISK_MONITORING').querySelector(safety)).not.toBeNull();+ });+});