Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix([issue-12340]): preserve healthy backups alongside unreadable snapshots (#12347)

[._.]/7f35a4eaf21f
diff --git a/docs/BACKUP.md b/docs/BACKUP.md
index 5354bcfcd2..9e2f7f0776 100644
--- a/docs/BACKUP.md
+++ b/docs/BACKUP.md
@@ -568,6 +568,7 @@ Left unbounded, `snapshots/<hostname>/` grows forever — every prior full DB du
- **`retentionCount` absent or explicitly `null` both mean unlimited** — no pruning runs. This is why an install that predates this setting keeps every snapshot until the operator saves a choice: it has no stored value, and absence resolves to unlimited, not to the new-install default.
- **Pruning runs only after a run reaches a completed snapshot with a usable database dump** (`pg_dump` succeeded, or was explicitly skipped by the file-backend escape hatch). A degraded run whose database dump failed keeps older snapshots and reports zero pruned snapshots; retention resumes after a later successful dump. A run that fails before completion never prunes, and a prune failure is logged without failing an otherwise-successful backup.
- **Pruning is scoped to the CURRENT machine's namespace** (`snapshots/<hostname>/`) and only ever deletes snapshots whose `snapshotState()` reports neither `incomplete` nor `failed`. It never touches another machine's namespace in a shared destination, the legacy pre-namespace root, or an in-progress/failed snapshot — those stay until the operator deletes them explicitly. It also skips a snapshot that a restore or download is still reading (see [Snapshot source lifetime](#how-restore-works)); the skipped snapshot is not counted as pruned and becomes a candidate again on a later run.
+- **Unreadable or corrupt manifests are protected in addition to the retention count.** They never displace readable completed snapshots, even when an incorrect clock gave a protected snapshot a future-dated ID. Snapshot IDs determine age; manifest timestamps do not change that ordering. Legacy completed snapshots with no manifest remain eligible for retention as before.
- **`DELETE /api/backup/snapshots/:snapshotId?source=<source>`** (`backup.deleteSnapshot()`) permanently removes exactly the selected source/ID pair, immediately — there is no undo and it is never automatic. It shares `resolveSnapshotPath()`'s path-traversal and symlink guards with restore/download, and refuses a snapshot that is still being written (`SNAPSHOT_INCOMPLETE`) or that a restore or download is still reading (`SNAPSHOT_IN_USE`); unlike restore, a `.failed` snapshot **is** deletable. The Backup settings tab's snapshot history exposes this as a per-row delete action behind a confirmation dialog.
## Scheduling & status
diff --git a/server/services/backup.js b/server/services/backup.js
index 55f37aadcf..731b588e16 100644
--- a/server/services/backup.js
+++ b/server/services/backup.js
@@ -1098,7 +1098,8 @@ export async function listSnapshots(destPath) {
: source === MACHINE_HOST ? `${source} (current machine)` : source,
selectionKey: `${source}/${id}`,
currentMachine: source === MACHINE_HOST,
- createdAt: manifest?.generatedAt ?? null,
+ createdAt: typeof manifest?.generatedAt === 'string' && !Number.isNaN(Date.parse(manifest.generatedAt))
+ ? manifest.generatedAt : null,
fileCount: manifest?.fileCount ?? 0,
incomplete,
failed,
@@ -1118,8 +1119,8 @@ export async function listSnapshots(destPath) {
/**
* Delete the oldest COMPLETED snapshots in the CURRENT machine's source
* namespace beyond `retentionCount`. Called only from `runBackup()` after a
- * run reaches a completed snapshot (rsync + a pg_dump attempt, even a
- * degraded one) — a run that fails earlier never reaches this call, so an
+ * run reaches a completed snapshot with a successful or explicitly skipped
+ * database dump — a run that fails earlier never reaches this call, so an
* in-progress or `.failed` snapshot is never a candidate; both are re-checked
* here anyway via `snapshotState` in case an unrelated run left one behind.
* Only ever walks `snapshots/<MACHINE_HOST>/` — never another machine's
@@ -1158,15 +1159,15 @@ async function pruneOldSnapshots(destPath, retentionCount) {
descriptors.sort((a, b) => b.id.localeCompare(a.id)
|| (b.createdAt ?? '').localeCompare(a.createdAt ?? ''));
- // Unreadable-manifest snapshots still count toward retention but are never deleted.
- const toDelete = [];
- descriptors.slice(retentionCount).forEach(d => {
+ // Keep protected snapshots in addition to the readable retention budget,
+ // even when a bad clock placed a protected snapshot ahead of healthy ones.
+ const toDelete = descriptors.filter(d => {
if (d.protectedFromPrune) {
console.warn(`⚠️ Backup retention: kept snapshot ${d.id} (manifest unreadable)`);
- } else {
- toDelete.push(d);
+ return false;
}
- });
+ return true;
+ }).slice(retentionCount);
let pruned = 0;
for (const { snapshotDir, id } of toDelete) {
// A snapshot a restore or download is reading stays; a later run prunes it
diff --git a/server/services/backup.test.js b/server/services/backup.test.js
index 6f356eb719..20af526503 100644
--- a/server/services/backup.test.js
+++ b/server/services/backup.test.js
@@ -522,6 +522,24 @@ describe('listSnapshots', () => {
readdirSpy.mockRestore();
});
+ it('lists healthy snapshots alongside malformed or absent manifest timestamps', async () => {
+ const generatedAt = [{ malformed: true }, ['2026-01-01'], 'not-a-date', null, '2026-01-05T00:00:00Z'];
+ const ids = generatedAt.map((_, index) => `2026-01-0${index + 1}T00-00-00`);
+ vi.spyOn(fs, 'readdir').mockImplementation(async path =>
+ String(path) === joinPath('/dest', 'snapshots')
+ ? [dirent(machineHost, true)]
+ : ids.map(id => dirent(id, true)));
+ vi.spyOn(fs, 'readFile').mockImplementation(async path =>
+ JSON.stringify({ generatedAt: generatedAt[ids.indexOf(basename(dirname(String(path))))], fileCount: 0 }));
+
+ const result = await listSnapshots('/dest');
+
+ expect(result.map(({ id, createdAt }) => ({ id, createdAt }))).toEqual([
+ { id: ids[4], createdAt: generatedAt[4] },
+ ...ids.slice(0, 4).reverse().map(id => ({ id, createdAt: null })),
+ ]);
+ });
+
it('discovers legacy and machine snapshots with collision-safe identities', async () => {
const destRoot = await fs.mkdtemp(joinPath(tmpdir(), 'portos-backup-sources-'));
const snapshotId = '2026-06-08T15-18-34';
@@ -4952,10 +4970,11 @@ describe('runBackup lifecycle', () => {
await expect(fsp.stat(old)).rejects.toThrow();
});
- it('keeps the newest snapshot and prunes the oldest when the newest manifest is corrupt, and never deletes an unreadable-manifest snapshot (#11859)', async () => {
+ it('retains the readable snapshot budget in addition to corrupt manifests (#12340)', async () => {
const fsp = await actualFs();
- const oldest = await seedSnapshot(machineHost, '2020-01-01T00-00-00', { generatedAt: '2020-01-01T00:00:00.000Z' });
- const corrupt = await seedSnapshot(machineHost, '2020-01-02T00-00-00');
+ const oldest = await seedSnapshot(machineHost, '2020-01-01T00-00-00', { generatedAt: { malformed: true } });
+ const readable = await seedSnapshot(machineHost, '2020-01-02T00-00-00', { generatedAt: '2099-01-01T00:00:00.000Z' });
+ const corrupt = await seedSnapshot(machineHost, '2020-01-03T00-00-00');
await fsp.writeFile(joinPath(corrupt, 'manifest.json'), '{truncated');
const unreadable = await seedSnapshot(machineHost, '2019-12-31T00-00-00');
await fsp.writeFile(joinPath(unreadable, 'manifest.json'), '{truncated');
@@ -4966,14 +4985,43 @@ describe('runBackup lifecycle', () => {
const pending = runBackup(destRoot, { emit: vi.fn() }, { retentionCount: 2 });
await waitFor(() => spawn.mock.calls.length === 1, 'rsync spawn');
rsync.emit('close', 0);
- await expect(pending).resolves.toMatchObject({ status: 'ok', prunedSnapshots: 1 });
+ const result = await pending;
+ expect(result).toMatchObject({ status: 'ok', prunedSnapshots: 1 });
await expect(fsp.stat(oldest)).rejects.toThrow();
+ await expect(fsp.stat(readable)).resolves.toBeDefined();
+ await expect(fsp.stat(joinPath(destRoot, 'snapshots', machineHost, result.snapshotId))).resolves.toBeDefined();
await expect(fsp.stat(corrupt)).resolves.toBeDefined();
await expect(fsp.stat(unreadable)).resolves.toBeDefined();
expect(warn).toHaveBeenCalledWith('⚠️ Backup retention: kept snapshot 2019-12-31T00-00-00 (manifest unreadable)');
});
+ it('keeps a completed backup after clock correction when a future snapshot manifest cannot be read (#12340)', async () => {
+ const fsp = await actualFs();
+ const future = await seedSnapshot(machineHost, '2099-01-01T00-00-00', { generatedAt: '2099-01-01T00:00:00.000Z' });
+ const manifestPath = joinPath(future, 'manifest.json');
+ fs.readFile.mockImplementation((path, ...args) => path === manifestPath
+ ? Promise.reject(Object.assign(new Error('unreadable'), { code: 'EACCES' }))
+ : fsp.readFile(path, ...args));
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ vi.useFakeTimers({ toFake: ['Date'] });
+ vi.setSystemTime(new Date('2026-01-02T03:04:05Z'));
+ try {
+ const rsync = fakeProc();
+ spawn.mockReturnValue(rsync);
+ const pending = runBackup(destRoot, null, { retentionCount: 1 });
+ await waitFor(() => spawn.mock.calls.length === 1, 'rsync spawn');
+ rsync.emit('close', 0);
+ const result = await pending;
+ expect(result).toMatchObject({ status: 'ok', prunedSnapshots: 0 });
+ await expect(fsp.stat(joinPath(destRoot, 'snapshots', machineHost, result.snapshotId))).resolves.toBeDefined();
+ await expect(fsp.stat(future)).resolves.toBeDefined();
+ expect(warn).toHaveBeenCalledWith('⚠️ Backup retention: kept snapshot 2099-01-01T00-00-00 (manifest unreadable)');
+ } finally {
+ vi.useRealTimers();
+ }
+ });
+
it('does not prune anything when retentionCount is null (unlimited — the legacy/unset default)', async () => {
const fsp = await actualFs();
const old1 = await seedSnapshot(machineHost, '2020-01-01T00-00-00', { generatedAt: '2020-01-01T00:00:00.000Z' });