Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
fix([issue-12340]): preserve healthy backups alongside unreadable snapshots (#12347)
[._.]/7f35a4eaf21f
diff --git a/docs/BACKUP.md b/docs/BACKUP.mdindex 5354bcfcd2..9e2f7f0776 100644--- a/docs/BACKUP.md+++ b/docs/BACKUP.md@@ -568,6 +568,7 @@ Left unbounded, `snapshots/<hostname>/` grows forever — every prior full DB du- **`retentionCount` absent or explicitly `null` both mean unlimited** — no pruning runs. This is why an install that predates this setting keeps every snapshot until the operator saves a choice: it has no stored value, and absence resolves to unlimited, not to the new-install default.- **Pruning runs only after a run reaches a completed snapshot with a usable database dump** (`pg_dump` succeeded, or was explicitly skipped by the file-backend escape hatch). A degraded run whose database dump failed keeps older snapshots and reports zero pruned snapshots; retention resumes after a later successful dump. A run that fails before completion never prunes, and a prune failure is logged without failing an otherwise-successful backup.- **Pruning is scoped to the CURRENT machine's namespace** (`snapshots/<hostname>/`) and only ever deletes snapshots whose `snapshotState()` reports neither `incomplete` nor `failed`. It never touches another machine's namespace in a shared destination, the legacy pre-namespace root, or an in-progress/failed snapshot — those stay until the operator deletes them explicitly. It also skips a snapshot that a restore or download is still reading (see [Snapshot source lifetime](#how-restore-works)); the skipped snapshot is not counted as pruned and becomes a candidate again on a later run.+- **Unreadable or corrupt manifests are protected in addition to the retention count.** They never displace readable completed snapshots, even when an incorrect clock gave a protected snapshot a future-dated ID. Snapshot IDs determine age; manifest timestamps do not change that ordering. Legacy completed snapshots with no manifest remain eligible for retention as before.- **`DELETE /api/backup/snapshots/:snapshotId?source=<source>`** (`backup.deleteSnapshot()`) permanently removes exactly the selected source/ID pair, immediately — there is no undo and it is never automatic. It shares `resolveSnapshotPath()`'s path-traversal and symlink guards with restore/download, and refuses a snapshot that is still being written (`SNAPSHOT_INCOMPLETE`) or that a restore or download is still reading (`SNAPSHOT_IN_USE`); unlike restore, a `.failed` snapshot **is** deletable. The Backup settings tab's snapshot history exposes this as a per-row delete action behind a confirmation dialog.## Scheduling & statusdiff --git a/server/services/backup.js b/server/services/backup.jsindex 55f37aadcf..731b588e16 100644--- a/server/services/backup.js+++ b/server/services/backup.js@@ -1098,7 +1098,8 @@ export async function listSnapshots(destPath) {: source === MACHINE_HOST ? `${source} (current machine)` : source,selectionKey: `${source}/${id}`,currentMachine: source === MACHINE_HOST,- createdAt: manifest?.generatedAt ?? null,+ createdAt: typeof manifest?.generatedAt === 'string' && !Number.isNaN(Date.parse(manifest.generatedAt))+ ? manifest.generatedAt : null,fileCount: manifest?.fileCount ?? 0,incomplete,failed,@@ -1118,8 +1119,8 @@ export async function listSnapshots(destPath) {/*** Delete the oldest COMPLETED snapshots in the CURRENT machine's source* namespace beyond `retentionCount`. Called only from `runBackup()` after a- * run reaches a completed snapshot (rsync + a pg_dump attempt, even a- * degraded one) — a run that fails earlier never reaches this call, so an+ * run reaches a completed snapshot with a successful or explicitly skipped+ * database dump — a run that fails earlier never reaches this call, so an* in-progress or `.failed` snapshot is never a candidate; both are re-checked* here anyway via `snapshotState` in case an unrelated run left one behind.* Only ever walks `snapshots/<MACHINE_HOST>/` — never another machine's@@ -1158,15 +1159,15 @@ async function pruneOldSnapshots(destPath, retentionCount) {descriptors.sort((a, b) => b.id.localeCompare(a.id)|| (b.createdAt ?? '').localeCompare(a.createdAt ?? ''));- // Unreadable-manifest snapshots still count toward retention but are never deleted.- const toDelete = [];- descriptors.slice(retentionCount).forEach(d => {+ // Keep protected snapshots in addition to the readable retention budget,+ // even when a bad clock placed a protected snapshot ahead of healthy ones.+ const toDelete = descriptors.filter(d => {if (d.protectedFromPrune) {console.warn(`⚠️ Backup retention: kept snapshot ${d.id} (manifest unreadable)`);- } else {- toDelete.push(d);+ return false;}- });+ return true;+ }).slice(retentionCount);let pruned = 0;for (const { snapshotDir, id } of toDelete) {// A snapshot a restore or download is reading stays; a later run prunes itdiff --git a/server/services/backup.test.js b/server/services/backup.test.jsindex 6f356eb719..20af526503 100644--- a/server/services/backup.test.js+++ b/server/services/backup.test.js@@ -522,6 +522,24 @@ describe('listSnapshots', () => {readdirSpy.mockRestore();});+ it('lists healthy snapshots alongside malformed or absent manifest timestamps', async () => {+ const generatedAt = [{ malformed: true }, ['2026-01-01'], 'not-a-date', null, '2026-01-05T00:00:00Z'];+ const ids = generatedAt.map((_, index) => `2026-01-0${index + 1}T00-00-00`);+ vi.spyOn(fs, 'readdir').mockImplementation(async path =>+ String(path) === joinPath('/dest', 'snapshots')+ ? [dirent(machineHost, true)]+ : ids.map(id => dirent(id, true)));+ vi.spyOn(fs, 'readFile').mockImplementation(async path =>+ JSON.stringify({ generatedAt: generatedAt[ids.indexOf(basename(dirname(String(path))))], fileCount: 0 }));++ const result = await listSnapshots('/dest');++ expect(result.map(({ id, createdAt }) => ({ id, createdAt }))).toEqual([+ { id: ids[4], createdAt: generatedAt[4] },+ ...ids.slice(0, 4).reverse().map(id => ({ id, createdAt: null })),+ ]);+ });+it('discovers legacy and machine snapshots with collision-safe identities', async () => {const destRoot = await fs.mkdtemp(joinPath(tmpdir(), 'portos-backup-sources-'));const snapshotId = '2026-06-08T15-18-34';@@ -4952,10 +4970,11 @@ describe('runBackup lifecycle', () => {await expect(fsp.stat(old)).rejects.toThrow();});- it('keeps the newest snapshot and prunes the oldest when the newest manifest is corrupt, and never deletes an unreadable-manifest snapshot (#11859)', async () => {+ it('retains the readable snapshot budget in addition to corrupt manifests (#12340)', async () => {const fsp = await actualFs();- const oldest = await seedSnapshot(machineHost, '2020-01-01T00-00-00', { generatedAt: '2020-01-01T00:00:00.000Z' });- const corrupt = await seedSnapshot(machineHost, '2020-01-02T00-00-00');+ const oldest = await seedSnapshot(machineHost, '2020-01-01T00-00-00', { generatedAt: { malformed: true } });+ const readable = await seedSnapshot(machineHost, '2020-01-02T00-00-00', { generatedAt: '2099-01-01T00:00:00.000Z' });+ const corrupt = await seedSnapshot(machineHost, '2020-01-03T00-00-00');await fsp.writeFile(joinPath(corrupt, 'manifest.json'), '{truncated');const unreadable = await seedSnapshot(machineHost, '2019-12-31T00-00-00');await fsp.writeFile(joinPath(unreadable, 'manifest.json'), '{truncated');@@ -4966,14 +4985,43 @@ describe('runBackup lifecycle', () => {const pending = runBackup(destRoot, { emit: vi.fn() }, { retentionCount: 2 });await waitFor(() => spawn.mock.calls.length === 1, 'rsync spawn');rsync.emit('close', 0);- await expect(pending).resolves.toMatchObject({ status: 'ok', prunedSnapshots: 1 });+ const result = await pending;+ expect(result).toMatchObject({ status: 'ok', prunedSnapshots: 1 });await expect(fsp.stat(oldest)).rejects.toThrow();+ await expect(fsp.stat(readable)).resolves.toBeDefined();+ await expect(fsp.stat(joinPath(destRoot, 'snapshots', machineHost, result.snapshotId))).resolves.toBeDefined();await expect(fsp.stat(corrupt)).resolves.toBeDefined();await expect(fsp.stat(unreadable)).resolves.toBeDefined();expect(warn).toHaveBeenCalledWith('⚠️ Backup retention: kept snapshot 2019-12-31T00-00-00 (manifest unreadable)');});+ it('keeps a completed backup after clock correction when a future snapshot manifest cannot be read (#12340)', async () => {+ const fsp = await actualFs();+ const future = await seedSnapshot(machineHost, '2099-01-01T00-00-00', { generatedAt: '2099-01-01T00:00:00.000Z' });+ const manifestPath = joinPath(future, 'manifest.json');+ fs.readFile.mockImplementation((path, ...args) => path === manifestPath+ ? Promise.reject(Object.assign(new Error('unreadable'), { code: 'EACCES' }))+ : fsp.readFile(path, ...args));+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});+ vi.useFakeTimers({ toFake: ['Date'] });+ vi.setSystemTime(new Date('2026-01-02T03:04:05Z'));+ try {+ const rsync = fakeProc();+ spawn.mockReturnValue(rsync);+ const pending = runBackup(destRoot, null, { retentionCount: 1 });+ await waitFor(() => spawn.mock.calls.length === 1, 'rsync spawn');+ rsync.emit('close', 0);+ const result = await pending;+ expect(result).toMatchObject({ status: 'ok', prunedSnapshots: 0 });+ await expect(fsp.stat(joinPath(destRoot, 'snapshots', machineHost, result.snapshotId))).resolves.toBeDefined();+ await expect(fsp.stat(future)).resolves.toBeDefined();+ expect(warn).toHaveBeenCalledWith('⚠️ Backup retention: kept snapshot 2099-01-01T00-00-00 (manifest unreadable)');+ } finally {+ vi.useRealTimers();+ }+ });+it('does not prune anything when retentionCount is null (unlimited — the legacy/unset default)', async () => {const fsp = await actualFs();const old1 = await seedSnapshot(machineHost, '2020-01-01T00-00-00', { generatedAt: '2020-01-01T00:00:00.000Z' });