Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: keep snapshots alive while restore and download consumers read them (#10898)

Adam Eivyfb86f08c69d5
diff --git a/docs/BACKUP.md b/docs/BACKUP.md
index 29c3b5f483..5931213d92 100644
--- a/docs/BACKUP.md
+++ b/docs/BACKUP.md
@@ -487,6 +487,8 @@ Key behaviors, accurate to the code:
Restore is two independent operations — restoring files and restoring the DB are separate decisions. Both are **dry-run by default** and validate `snapshotId` and an optional source namespace against path traversal before touching anything. Explicit source selections also reject symbolic-link aliases for the snapshots root, source namespace, or selected snapshot before archive or restore reads begin.
+**Snapshot source lifetime (#10898).** A consumer that reads snapshot bytes holds a read lease on that snapshot for as long as it reads: a file restore or preview from before its restorability and integrity preflight until rsync's child closes (and, for execution, caches are reconciled); a database restore from its restorability check through dump admission and spooling; a download until the `tar` child closes — including after the client disconnects, since `tar` may still be reading. The lease is released on every exit path: success, refusal, failure, timeout and abort. Deletion needs the snapshot exclusively: explicit deletion of a snapshot in use is refused with `409 SNAPSHOT_IN_USE`, automatic retention skips it (logged, and pruned by a later run), and a reader that arrives while a deletion is in progress is refused with `409 SNAPSHOT_DELETING`. Deletion takes ownership *before* its final existence and in-progress checks, so a deletion that was admitted first cannot race a new reader. Ownership is keyed by the snapshot's canonical path, so aliases — an omitted versus an explicit current-machine source, a symlinked destination, a different letter case on a case-insensitive volume — share one owner. The lease coordinates this server's own requests and scheduled jobs; it is not a lock between machines that share a destination.
+
A backup run that fails after creating its snapshot directory records a durable `.failed` marker before releasing its `.in-progress` guards. Failed snapshots are never eligible for file or database restore (`SNAPSHOT_FAILED`); they remain downloadable so their partial files can be inspected or recovered manually. If PortOS cannot write the failed marker, it keeps the existing incomplete markers instead, which also block restore. Snapshots created by older PortOS versions without a manifest or failure marker retain their legacy behavior because an unmarked historical failure cannot be distinguished reliably from a genuine pre-manifest snapshot.
### Files — `restoreSnapshot()`
@@ -497,6 +499,8 @@ Snapshots from PortOS versions that predate `manifest.json` remain restorable as
After the preflight, rsync copies `<snapshot>/data/` back to `./data/`. Restore always passes `--checksum`, so rsync compares file contents even when the live file has the same size and modification time as the snapshot; equal-content files remain skippable, while differing bytes appear in previews and are restored. This applies to dry-run and live restores, including selective subdirectory restores and legacy snapshots without a manifest. `dryRun: true` (the default) reports what would change without writing; an optional `subdirFilter` limits the restore to one subdirectory.
+**Every nonzero rsync exit fails a restore, including 24.** A backup tolerates exit 24 ("some source files vanished") because it reads the live data tree, where a file deleted mid-scan is normal. A restore reads a snapshot that must not change under it, so a vanished source file means live data may now mix restored and older files: preview and execution both reject it. An execution failure is reported as a partial restore — the error says some files may already have been overwritten because file restore is not transactional — and caches (settings, Brain, media index) are reconciled before the error is returned, exactly as for any other failed transfer.
+
#### Database authority is not restored
`data/database-authority.json` is backed up as cutover evidence but is never installed by a file restore. It records which database backend a completed cutover retired on the machine that ran it, so restoring it from another machine's snapshot (or from an older snapshot after a reverse cutover) would make PortOS refuse its own healthy database with `DATABASE_RETIRED_BACKEND`, including after a restart. Every file restore therefore excludes it, in preview and execution alike, and the integrity preflight ignores it too. The destination's existing file stays byte-for-byte as it was: an absent record stays absent and a damaged one stays fail-closed. The same preservation covers the other machine-local authority and recovery records — `peer-execution/`, `peer-execution-catalog.json`, `peer-execution-grants.json`, `peer-execution-authority.json`, `peer-execution-recovery.jsonl`, `workflow-maintenance/` (the graceful-maintenance journal and abandoned-run receipts) and `backup-admission/` (which snapshots also skip) — so a snapshot's copy is evidence only (see [STORAGE.md](./STORAGE.md)). Other records in the snapshot restore normally. Selecting any of these paths, including `database-authority.json` itself, is refused with `BACKUP_RESTORE_MACHINE_LOCAL` before anything is transferred. This does not touch an active maintenance journal (`database-maintenance/`): its recovery semantics are unchanged, and another machine's cutover endpoints are never imported. To change which backend is authoritative, run a database cutover.
@@ -554,8 +558,8 @@ Left unbounded, `snapshots/<hostname>/` grows forever — every prior full DB du
- **New installs default to 30 completed snapshots per source.** The default ships as `backup.retentionCount: 30` in `data.reference/settings.json`, copied only into an install that has no `data/settings.json` yet — see the comment on `resolveRetentionCount()` for why this, not a migration, is what keeps an existing install from silently losing its archive. Operators choose 1–365, or Unlimited, from the Backup settings tab.
- **`retentionCount` absent or explicitly `null` both mean unlimited** — no pruning runs. This is why an install that predates this setting keeps every snapshot until the operator saves a choice: it has no stored value, and absence resolves to unlimited, not to the new-install default.
- **Pruning runs only after a run reaches a completed snapshot with a usable database dump** (`pg_dump` succeeded, or was explicitly skipped by the file-backend escape hatch). A degraded run whose database dump failed keeps older snapshots and reports zero pruned snapshots; retention resumes after a later successful dump. A run that fails before completion never prunes, and a prune failure is logged without failing an otherwise-successful backup.
-- **Pruning is scoped to the CURRENT machine's namespace** (`snapshots/<hostname>/`) and only ever deletes snapshots whose `snapshotState()` reports neither `incomplete` nor `failed`. It never touches another machine's namespace in a shared destination, the legacy pre-namespace root, or an in-progress/failed snapshot — those stay until the operator deletes them explicitly.
-- **`DELETE /api/backup/snapshots/:snapshotId?source=<source>`** (`backup.deleteSnapshot()`) permanently removes exactly the selected source/ID pair, immediately — there is no undo and it is never automatic. It shares `resolveSnapshotPath()`'s path-traversal and symlink guards with restore/download, and refuses a snapshot that is still being written (`SNAPSHOT_INCOMPLETE`); unlike restore, a `.failed` snapshot **is** deletable. The Backup settings tab's snapshot history exposes this as a per-row delete action behind a confirmation dialog.
+- **Pruning is scoped to the CURRENT machine's namespace** (`snapshots/<hostname>/`) and only ever deletes snapshots whose `snapshotState()` reports neither `incomplete` nor `failed`. It never touches another machine's namespace in a shared destination, the legacy pre-namespace root, or an in-progress/failed snapshot — those stay until the operator deletes them explicitly. It also skips a snapshot that a restore or download is still reading (see [Snapshot source lifetime](#how-restore-works)); the skipped snapshot is not counted as pruned and becomes a candidate again on a later run.
+- **`DELETE /api/backup/snapshots/:snapshotId?source=<source>`** (`backup.deleteSnapshot()`) permanently removes exactly the selected source/ID pair, immediately — there is no undo and it is never automatic. It shares `resolveSnapshotPath()`'s path-traversal and symlink guards with restore/download, and refuses a snapshot that is still being written (`SNAPSHOT_INCOMPLETE`) or that a restore or download is still reading (`SNAPSHOT_IN_USE`); unlike restore, a `.failed` snapshot **is** deletable. The Backup settings tab's snapshot history exposes this as a per-row delete action behind a confirmation dialog.
## Scheduling & status
diff --git a/server/routes/backup.test.js b/server/routes/backup.test.js
index e250cef2a1..718218c2af 100644
--- a/server/routes/backup.test.js
+++ b/server/routes/backup.test.js
@@ -297,6 +297,19 @@ describe('backup routes', () => {
expect(backup.deleteSnapshot).not.toHaveBeenCalled();
});
+ it('surfaces the structured conflict when a restore or download is reading the snapshot (#10898)', async () => {
+ getSettings.mockResolvedValue({ backup: { destPath: '/dest' } });
+ backup.deleteSnapshot.mockRejectedValueOnce(new ServerError(
+ 'Snapshot is in use by a restore or download: snap-1. Retry when it finishes.',
+ { status: 409, code: 'SNAPSHOT_IN_USE', context: { snapshotId: 'snap-1' } },
+ ));
+
+ const res = await request(buildApp()).delete('/api/backup/snapshots/snap-1');
+
+ expect(res.status).toBe(409);
+ expect(res.body).toMatchObject({ code: 'SNAPSHOT_IN_USE', context: { snapshotId: 'snap-1' } });
+ });
+
it('returns 400 when the backup destination is not configured', async () => {
getSettings.mockResolvedValue({ backup: {} });
const res = await request(buildApp()).delete('/api/backup/snapshots/snap-1');
@@ -338,6 +351,21 @@ describe('backup routes', () => {
);
});
+ it('refuses a restore of a snapshot whose deletion is in progress with a structured conflict (#10898)', async () => {
+ getSettings.mockResolvedValue({ backup: { destPath: '/dest' } });
+ backup.restoreSnapshot.mockRejectedValueOnce(new ServerError(
+ 'Snapshot is being deleted: snap-1',
+ { status: 409, code: 'SNAPSHOT_DELETING', context: { snapshotId: 'snap-1' } },
+ ));
+
+ const res = await request(buildApp())
+ .post('/api/backup/restore')
+ .send({ snapshotId: 'snap-1', dryRun: false });
+
+ expect(res.status).toBe(409);
+ expect(res.body).toMatchObject({ code: 'SNAPSHOT_DELETING', context: { snapshotId: 'snap-1' } });
+ });
+
it('forwards source with file restore preview and execution requests', async () => {
getSettings.mockResolvedValue({ backup: { destPath: '/dest' } });
backup.restoreSnapshot.mockResolvedValue({ changedFiles: [] });
diff --git a/server/services/backup.js b/server/services/backup.js
index 109e9cd892..84570b8775 100644
--- a/server/services/backup.js
+++ b/server/services/backup.js
@@ -9,7 +9,7 @@
import { dashboardEvents } from './dashboardEvents.js';
import { spawn } from '../lib/childProcess.js';
import { killWithEscalation } from '../lib/killWithEscalation.js';
-import { access, lstat, mkdir, mkdtemp, readdir, rm, stat, unlink, writeFile } from 'fs/promises';
+import { access, lstat, mkdir, mkdtemp, readdir, realpath, rm, stat, unlink, writeFile } from 'fs/promises';
import { randomUUID } from 'node:crypto';
import { PassThrough } from 'node:stream';
import { hostname, tmpdir } from 'os';
@@ -140,6 +140,75 @@ async function assertSnapshotRestorable(snapshotDir, snapshotId, currentSource)
}
}
+// Snapshot source lifetime (#10898). Verifying snapshot bytes proves nothing if
+// the snapshot can be deleted while a consumer is still reading it, so every
+// consumer holds a READ lease from before its preflight until its last read
+// settles (file restore/preview through rsync's close and reconciliation,
+// database restore through dump admission, a download through tar's close),
+// and every deletion — explicit or retention — needs the snapshot EXCLUSIVELY.
+// Readers share; a deletion refuses while any reader holds it, and a reader
+// refuses while a deletion holds it. Keyed by the canonical path (realpath
+// folds the omitted-vs-explicit current source, destination symlinks and, on a
+// case-insensitive volume, letter case into one owner); a path that does not
+// exist keeps its resolved form, which every alias of it shares too. This
+// coordinates this server's own callers — it is not a cross-machine lock.
+const snapshotLeases = new Map();
+
+const SNAPSHOT_LEASE_REFUSALS = Object.freeze({
+ SNAPSHOT_DELETING: (snapshotId) => `Snapshot is being deleted: ${snapshotId}`,
+ SNAPSHOT_IN_USE: (snapshotId) => `Snapshot is in use by a restore or download: ${snapshotId}. Retry when it finishes.`,
+});
+
+/**
+ * Claim `snapshotDir` for `mode` ('read' | 'delete'). Resolves `{ release }`,
+ * or `{ refusal }` naming the conflicting owner; never throws. The check and
+ * the claim run in one synchronous step after the key is resolved, so two
+ * claimants cannot both win.
+ */
+async function claimSnapshotLease(snapshotDir, mode) {
+ // snapshotDir comes from resolveSnapshotPath, so it is already resolved.
+ const key = await realpath(snapshotDir).catch(() => snapshotDir);
+ const lease = snapshotLeases.get(key) ?? { readers: 0, deleting: false };
+ if (lease.deleting) return { refusal: 'SNAPSHOT_DELETING' };
+ if (mode === 'delete' && lease.readers > 0) return { refusal: 'SNAPSHOT_IN_USE' };
+ if (mode === 'delete') lease.deleting = true;
+ else lease.readers += 1;
+ snapshotLeases.set(key, lease);
+ let held = true;
+ return {
+ release: () => {
+ if (!held) return;
+ held = false;
+ if (mode === 'delete') lease.deleting = false;
+ else lease.readers -= 1;
+ if (!lease.deleting && lease.readers === 0) snapshotLeases.delete(key);
+ },
+ };
+}
+
+/** Claim a snapshot lease or throw the structured 409 naming the conflict. */
+async function acquireSnapshotLease(snapshotDir, snapshotId, mode) {
+ const { release, refusal } = await claimSnapshotLease(snapshotDir, mode);
+ if (refusal) {
+ throw new ServerError(SNAPSHOT_LEASE_REFUSALS[refusal](snapshotId), {
+ status: 409,
+ code: refusal,
+ context: { snapshotId },
+ });
+ }
+ return release;
+}
+
+/** Hold a read lease on `snapshotDir` for exactly as long as `operation` runs. */
+async function withSnapshotRead(snapshotDir, snapshotId, operation) {
+ const release = await acquireSnapshotLease(snapshotDir, snapshotId, 'read');
+ try {
+ return await operation();
+ } finally {
+ release();
+ }
+}
+
// Serialize state read-merge-write so two saveState() calls (e.g. a run
// completing while the scheduler stamps a status) can't each read the same
// pre-image and clobber the other's fields. Single tail per shared state file.
@@ -418,7 +487,8 @@ function watchBackupProcess(proc, { label, progressPath = null, idleTimeoutMs }
/**
* Run rsync from srcDir to destDir with optional flags.
- * Resolves with array of changed file lines. Rejects on non-zero exit (except 24).
+ * Resolves with array of changed file lines. Rejects on non-zero exit (24 only
+ * when the caller opts in — see `allowVanishedSources`).
*/
export function resolveRsyncBinary(env = process.env) {
const override = typeof env.PORTOS_RSYNC === 'string' ? env.PORTOS_RSYNC.trim() : '';
@@ -428,7 +498,13 @@ export function resolveRsyncBinary(env = process.env) {
return override || 'rsync';
}
-function runRsync(srcDir, destDir, flags = [], { idleTimeoutMs } = {}) {
+// `allowVanishedSources` is the per-operation exit policy (#10898). A backup
+// reads the LIVE data tree, where a file deleted mid-scan (exit 24) is normal
+// and the snapshot is still a usable point-in-time copy. A restore reads a
+// snapshot that must not change under it: exit 24 means part of its source
+// disappeared, so live data may now mix restored and old files. Restore keeps
+// the strict default and treats 24 like every other nonzero exit.
+function runRsync(srcDir, destDir, flags = [], { idleTimeoutMs, allowVanishedSources = false } = {}) {
return new Promise((resolve, reject) => {
// `--itemize-changes` emits only after each file finishes. `--progress` is
// also supported by macOS's bundled rsync 2.6.9 and emits within a large
@@ -467,12 +543,13 @@ function runRsync(srcDir, destDir, flags = [], { idleTimeoutMs } = {}) {
reject(timeoutError);
return;
}
- // Exit code 24 = some files vanished mid-transfer (normal for active system)
- if (code === 0 || code === 24) {
+ // Exit code 24 = some source files vanished mid-transfer.
+ if (code === 0 || (code === 24 && allowVanishedSources)) {
stdoutReader.flush();
resolve(changed);
} else {
- reject(new Error(`rsync exited with code ${code}: ${stderr.trim()}`));
+ const detail = code === 24 ? ' (source files vanished during transfer)' : '';
+ reject(new Error(`rsync exited with code ${code}${detail}: ${stderr.trim()}`));
}
});
@@ -643,7 +720,7 @@ export async function runBackup(destPath, io = null, { excludePaths = [], disabl
let pgResult;
try {
const excludeFlags = effectiveExcludes.flatMap(p => ['--exclude', p]);
- changedFiles = await runRsync(PATHS.data, dataDestDir, excludeFlags);
+ changedFiles = await runRsync(PATHS.data, dataDestDir, excludeFlags, { allowVanishedSources: true });
console.log(`💾 Backup rsync complete: ${changedFiles.length} files changed (exit 0)`);
// A configured-but-failed dump degrades the backup and alerts the user.
@@ -1054,14 +1131,25 @@ async function pruneOldSnapshots(destPath, retentionCount) {
});
const toDelete = descriptors.slice(retentionCount);
+ let pruned = 0;
for (const { snapshotDir, id } of toDelete) {
- await rm(snapshotDir, { recursive: true, force: true }).catch(err =>
- console.error(`❌ Backup retention: failed to remove snapshot ${id}: ${err.message}`));
+ // A snapshot a restore or download is reading stays; a later run prunes it
+ // once the reader finishes (#10898).
+ const { release, refusal } = await claimSnapshotLease(snapshotDir, 'delete');
+ if (refusal) {
+ console.warn(`⚠️ Backup retention: skipped snapshot ${id} (${refusal})`);
+ continue;
+ }
+ await rm(snapshotDir, { recursive: true, force: true }).then(
+ () => { pruned += 1; },
+ err => console.error(`❌ Backup retention: failed to remove snapshot ${id}: ${err.message}`),
+ );
+ release();
}
- if (toDelete.length) {
- console.log(`💾 Backup retention: pruned ${toDelete.length} snapshot(s), keeping ${retentionCount}`);
+ if (pruned) {
+ console.log(`💾 Backup retention: pruned ${pruned} snapshot(s), keeping ${retentionCount}`);
}
- return { pruned: toDelete.length };
+ return { pruned };
}
function resolveSnapshotPath(destPath, snapshotId, source) {
@@ -1144,7 +1232,8 @@ async function assertExplicitSnapshotSourceSafe({
* (`.in-progress`), matching the guard that blocks restore and download,
* because the partial directory may still be owned by an in-flight
* `runBackup()`. A `.failed` snapshot IS deletable — unlike restore, deletion
- * has no reason to require a successful backup.
+ * has no reason to require a successful backup. A snapshot a restore or
+ * download is still reading is refused with `SNAPSHOT_IN_USE`.
* @param {string} destPath - Path to external drive backup root
* @param {string} snapshotId - Snapshot ID to delete
* @param {{ source?: string }} [options]
@@ -1154,18 +1243,26 @@ export async function deleteSnapshot(destPath, snapshotId, { source } = {}) {
const resolved = resolveSnapshotPath(destPath, snapshotId, source);
const { snapshotDir, currentSource, resolvedSource } = resolved;
await assertExplicitSnapshotSourceSafe(resolved, snapshotId);
- const info = await stat(snapshotDir).catch(() => null);
- if (!info?.isDirectory?.()) {
- throw new ServerError(`Snapshot not found: ${snapshotId}`, { status: 404, code: 'NOT_FOUND' });
- }
- const { incomplete } = await snapshotState(snapshotDir, snapshotId, currentSource);
- if (incomplete) {
- throw new ServerError(`Snapshot is still being written: ${snapshotId}`, {
- status: 409,
- code: 'SNAPSHOT_INCOMPLETE',
- });
+ // Own the snapshot BEFORE the final existence and in-progress checks: an
+ // active reader refuses this deletion (SNAPSHOT_IN_USE), and once admitted no
+ // new reader can start until the directory is gone (#10898).
+ const release = await acquireSnapshotLease(snapshotDir, snapshotId, 'delete');
+ try {
+ const info = await stat(snapshotDir).catch(() => null);
+ if (!info?.isDirectory?.()) {
+ throw new ServerError(`Snapshot not found: ${snapshotId}`, { status: 404, code: 'NOT_FOUND' });
+ }
+ const { incomplete } = await snapshotState(snapshotDir, snapshotId, currentSource);
+ if (incomplete) {
+ throw new ServerError(`Snapshot is still being written: ${snapshotId}`, {
+ status: 409,
+ code: 'SNAPSHOT_INCOMPLETE',
+ });
+ }
+ await rm(snapshotDir, { recursive: true, force: true });
+ } finally {
+ release();
}
- await rm(snapshotDir, { recursive: true, force: true });
dashboardEvents.emit('backup:changed');
console.log(`💾 Backup snapshot deleted: ${resolvedSource}/${snapshotId}`);
return { deleted: true, snapshotId, source: resolvedSource };
@@ -1181,18 +1278,31 @@ export async function openSnapshotStream(destPath, snapshotId, { source } = {})
const resolved = resolveSnapshotPath(destPath, snapshotId, source);
const { snapshotsRoot, snapshotDir, currentSource } = resolved;
await assertExplicitSnapshotSourceSafe(resolved, snapshotId);
- const info = await stat(snapshotDir).catch(() => null);
- if (!info?.isDirectory?.()) {
- throw new ServerError(`Snapshot not found: ${snapshotId}`, { status: 404, code: 'NOT_FOUND' });
+ // The read lease outlives this call: tar reads the snapshot until its child
+ // closes, however the download ends (#10898).
+ const releaseRead = await acquireSnapshotLease(snapshotDir, snapshotId, 'read');
+ let proc;
+ try {
+ const info = await stat(snapshotDir).catch(() => null);
+ if (!info?.isDirectory?.()) {
+ throw new ServerError(`Snapshot not found: ${snapshotId}`, { status: 404, code: 'NOT_FOUND' });
+ }
+ await assertSnapshotComplete(snapshotDir, snapshotId, currentSource);
+
+ // tar's stderr is a pipe (spawn's default) and MUST be drained: left unread
+ // it fills its ~64KB buffer on a tree that warns a lot — files changing under
+ // the archiver, unreadable modes — and tar then blocks on the write forever,
+ // hanging the download with the process still alive. Keep the tail so a
+ // non-zero exit can say why rather than just reporting the code.
+ proc = spawn('tar', ['-czf', '-', '-C', snapshotsRoot, snapshotId], { shell: false });
+ } catch (err) {
+ releaseRead();
+ throw err;
}
- await assertSnapshotComplete(snapshotDir, snapshotId, currentSource);
-
- // tar's stderr is a pipe (spawn's default) and MUST be drained: left unread
- // it fills its ~64KB buffer on a tree that warns a lot — files changing under
- // the archiver, unreadable modes — and tar then blocks on the write forever,
- // hanging the download with the process still alive. Keep the tail so a
- // non-zero exit can say why rather than just reporting the code.
- const proc = spawn('tar', ['-czf', '-', '-C', snapshotsRoot, snapshotId], { shell: false });
+ proc.on('close', releaseRead);
+ // A child that never spawned has no pid and nothing reading the snapshot.
+ // Any other error (a failed kill) leaves the child alive until `close`.
+ proc.on('error', () => { if (proc.pid === undefined) releaseRead(); });
const archive = new PassThrough();
let stderrTail = '';
proc.stderr?.on('data', (chunk) => { stderrTail = (stderrTail + chunk).slice(-500); });
@@ -1503,10 +1613,7 @@ function restoreIdleTimeoutMs(largestFileBytes = 0) {
*/
export async function restoreSnapshot(destPath, snapshotId, { dryRun = true, subdirFilter = null, source } = {}) {
const resolved = resolveSnapshotPath(destPath, snapshotId, source);
- const { snapshotDir, currentSource } = resolved;
await assertExplicitSnapshotSourceSafe(resolved, snapshotId);
- await assertSnapshotRestorable(snapshotDir, snapshotId, currentSource);
- const srcDir = join(snapshotDir, 'data');
// Defense-in-depth for non-route callers (the route already validates via
// subdirFilterSchema). subdirFilter is interpolated into an rsync include arg,
@@ -1525,6 +1632,18 @@ export async function restoreSnapshot(destPath, snapshotId, { dryRun = true, sub
);
}
+ // Hold the source from before the restorability and integrity preflight
+ // until rsync has closed (and, for execution, caches are reconciled), so
+ // neither explicit deletion nor retention can remove verified bytes while
+ // rsync still reads them (#10898).
+ return withSnapshotRead(resolved.snapshotDir, snapshotId,
+ () => restoreHeldSnapshot(resolved, snapshotId, { dryRun, subdirFilter }));
+}
+
+async function restoreHeldSnapshot({ snapshotDir, currentSource }, snapshotId, { dryRun, subdirFilter }) {
+ await assertSnapshotRestorable(snapshotDir, snapshotId, currentSource);
+ const srcDir = join(snapshotDir, 'data');
+
// Run the preflight independently for preview and execution. A preview is an
// aid to confirmation, not an integrity lease: snapshot bytes may change
// between requests, especially on removable or network-backed destinations.
@@ -1726,8 +1845,16 @@ const DUMP_UNREADABLE = Object.freeze({ status: 'failed', reason: 'dump_unreadab
*/
export async function restorePostgres(destPath, snapshotId, { dryRun = true, source } = {}) {
const resolved = resolveSnapshotPath(destPath, snapshotId, source);
- const { snapshotDir, currentSource } = resolved;
await assertExplicitSnapshotSourceSafe(resolved, snapshotId);
+ // Held for the whole call, which spans every read of snapshot bytes: the
+ // restorability check, the manifest and the dump admission/spool read. (The
+ // replay itself reads only the private spool copy.) Neither deletion nor
+ // retention can remove the dump while it is being admitted (#10898).
+ return withSnapshotRead(resolved.snapshotDir, snapshotId,
+ () => restoreHeldDatabase(resolved, snapshotId, { dryRun }));
+}
+
+async function restoreHeldDatabase({ snapshotDir, currentSource }, snapshotId, { dryRun }) {
await assertSnapshotRestorable(snapshotDir, snapshotId, currentSource);
// A committed restore awaiting repair fences the database; neither a preview
// nor another replay may start until it is resolved (#9725).
diff --git a/server/services/backup.test.js b/server/services/backup.test.js
index b5691c6381..f1f7e615f6 100644
--- a/server/services/backup.test.js
+++ b/server/services/backup.test.js
@@ -845,6 +845,40 @@ describe('openSnapshotStream', () => {
vi.useRealTimers();
await expect(failed).resolves.toMatchObject({ message: 'Snapshot download aborted: snap-1' });
+ // The killed child closes, ending its hold on the snapshot (#10898).
+ proc.emit('close', null, 'SIGKILL');
+ });
+
+ it('holds the snapshot against deletion until tar closes, even after the download aborts (#10898)', async () => {
+ const proc = readySnapshot();
+ const stream = await openSnapshotStream('/dest', 'snap-1');
+ stream.on('error', () => {});
+
+ await expect(deleteSnapshot('/dest', 'snap-1', { source: machineHost }))
+ .rejects.toMatchObject({ status: 409, code: 'SNAPSHOT_IN_USE' });
+ // The response is gone but tar may still be reading the snapshot.
+ stream.abort();
+ await expect(deleteSnapshot('/dest', 'snap-1'))
+ .rejects.toMatchObject({ status: 409, code: 'SNAPSHOT_IN_USE' });
+
+ proc.exitCode = 143;
+ proc.emit('close', null, 'SIGTERM');
+ // Past the lease, deletion reaches its own existence check.
+ fs.stat.mockResolvedValue({ isDirectory: () => false });
+ await expect(deleteSnapshot('/dest', 'snap-1'))
+ .rejects.toMatchObject({ status: 404, code: 'NOT_FOUND' });
+ });
+
+ it('releases the snapshot when tar cannot be spawned (#10898)', async () => {
+ const proc = readySnapshot();
+ const stream = await openSnapshotStream('/dest', 'snap-1');
+ const failed = errored(stream);
+ proc.emit('error', Object.assign(new Error('spawn tar ENOENT'), { code: 'ENOENT' }));
+ await expect(failed).resolves.toMatchObject({ code: 'ENOENT' });
+
+ fs.stat.mockResolvedValue({ isDirectory: () => false });
+ await expect(deleteSnapshot('/dest', 'snap-1'))
+ .rejects.toMatchObject({ status: 404, code: 'NOT_FOUND' });
});
it('does not escalate once tar has already exited', async () => {
@@ -860,6 +894,7 @@ describe('openSnapshotStream', () => {
await vi.advanceTimersByTimeAsync(10_000);
expect(proc.kill).not.toHaveBeenCalled();
vi.useRealTimers();
+ proc.emit('close', 0);
});
});
@@ -1315,6 +1350,28 @@ describe('restorePostgres', () => {
expect(runDbMigrations).not.toHaveBeenCalled();
});
+ it('holds the snapshot against deletion while its dump is being admitted, then releases it (#10898)', async () => {
+ const snapshotId = '2026-10-09T00-00-00';
+ // Same module instance as restorePostgres, whose lease table is module state.
+ const { deleteSnapshot } = await import('./backup.js');
+ vi.spyOn(fs, 'stat').mockResolvedValue({ size: 4096, isFile: () => true, isDirectory: () => false });
+ mockLegacyDumpRead();
+ let releaseAdmission;
+ getServerMajorVersion.mockImplementationOnce(() => new Promise(resolveMajor => { releaseAdmission = () => resolveMajor(16); }));
+
+ const restoring = restorePostgres('/dest', snapshotId, { dryRun: true });
+ await vi.waitFor(() => expect(releaseAdmission).toBeTypeOf('function'));
+ // An explicit current-machine source names the same snapshot as the default.
+ await expect(deleteSnapshot('/dest', snapshotId, { source: machineHost }))
+ .rejects.toMatchObject({ status: 409, code: 'SNAPSHOT_IN_USE' });
+
+ releaseAdmission();
+ await expect(restoring).resolves.toMatchObject({ status: 'ok', dryRun: true });
+ // Past the lease, deletion reaches its own existence check.
+ await expect(deleteSnapshot('/dest', snapshotId))
+ .rejects.toMatchObject({ status: 404, code: 'NOT_FOUND' });
+ });
+
it('refuses unexpected schema objects before entering maintenance or spawning replay', async () => {
vi.spyOn(fs, 'stat').mockResolvedValue({ size: 4096, isFile: () => true });
mockLegacyDumpRead();
@@ -2355,6 +2412,94 @@ describe('restoreSnapshot manifest verification', () => {
}
});
+ // #10898: a verified source is worth nothing if it can be deleted while rsync
+ // still reads it. The child is paused after spawn so each interleaving is
+ // deterministic; verification, ownership and deletion run for real.
+ describe('snapshot source lifetime (#10898)', () => {
+ const relativePath = 'restore-integrity/example.json';
+
+ beforeEach(async () => {
+ await writeManifest({ [relativePath]: await writeSnapshotFile(relativePath, 'trusted backup') });
+ });
+
+ async function startRestore(options) {
+ const proc = fakeProc();
+ spawn.mockReturnValueOnce(proc);
+ const pending = restoreSnapshot(tmpRoot, 'snap-1', options);
+ await vi.waitFor(() => expect(spawn).toHaveBeenCalled());
+ return { proc, pending };
+ }
+
+ it('refuses deletion through a source alias while a manifested restore reads the snapshot, until rsync closes', async () => {
+ // A symlinked destination plus an explicit current-machine source is a
+ // second spelling of the same snapshot; it must meet the same owner.
+ const aliasRoot = await realFs.mkdtemp(joinPath(tmpdir(), 'portos-restore-alias-'));
+ const aliasDest = joinPath(aliasRoot, 'dest');
+ await realFs.symlink(tmpRoot, aliasDest, 'dir');
+ try {
+ const { proc, pending } = await startRestore({ dryRun: false, subdirFilter: 'restore-integrity' });
+
+ await expect(deleteSnapshot(aliasDest, 'snap-1', { source: machineHost }))
+ .rejects.toMatchObject({ status: 409, code: 'SNAPSHOT_IN_USE', context: { snapshotId: 'snap-1' } });
+ await expect(realFs.stat(snapshotDir)).resolves.toBeDefined();
+
+ proc.emit('close', 0);
+ await expect(pending).resolves.toMatchObject({ dryRun: false, verification: { status: 'verified' } });
+ await deleteSnapshot(aliasDest, 'snap-1', { source: machineHost });
+ await expect(realFs.stat(snapshotDir)).rejects.toMatchObject({ code: 'ENOENT' });
+ } finally {
+ await realFs.rm(aliasRoot, { recursive: true, force: true });
+ }
+ });
+
+ it('refuses a new reader while an admitted deletion is still running', async () => {
+ let openGate;
+ const gate = new Promise(resolveGate => { openGate = resolveGate; });
+ // deleteSnapshot's first stat is its existence check, made under the lease.
+ fs.stat.mockImplementationOnce(async (path) => { await gate; return realFs.stat(path); });
+ const deleting = deleteSnapshot(tmpRoot, 'snap-1');
+ await vi.waitFor(() => expect(fs.stat).toHaveBeenCalled());
+
+ await expect(restoreSnapshot(tmpRoot, 'snap-1', { dryRun: true }))
+ .rejects.toMatchObject({ status: 409, code: 'SNAPSHOT_DELETING' });
+ await expect(openSnapshotStream(tmpRoot, 'snap-1', { source: machineHost }))
+ .rejects.toMatchObject({ status: 409, code: 'SNAPSHOT_DELETING' });
+ expect(spawn).not.toHaveBeenCalled();
+
+ openGate();
+ await expect(deleting).resolves.toMatchObject({ deleted: true });
+ await expect(realFs.stat(snapshotDir)).rejects.toMatchObject({ code: 'ENOENT' });
+ });
+
+ it('rejects a live restore whose source vanished (rsync exit 24) as partial, reconciles caches, and releases the source', async () => {
+ const { proc, pending } = await startRestore({ dryRun: false, subdirFilter: 'restore-integrity' });
+ reloadSettings.mockClear();
+ proc.stderr.emit('data', Buffer.from('file has vanished: "restore-integrity/example.json"'));
+ proc.emit('close', 24);
+
+ await expect(pending).rejects.toThrow(/rsync exited with code 24 \(source files vanished during transfer\).*not transactional/s);
+ expect(reloadSettings).toHaveBeenCalledTimes(1);
+ await expect(deleteSnapshot(tmpRoot, 'snap-1')).resolves.toMatchObject({ deleted: true });
+ });
+
+ it('rejects a preview whose source vanished (rsync exit 24) rather than reporting a clean plan', async () => {
+ const { proc, pending } = await startRestore({ dryRun: true });
+ proc.emit('close', 24);
+
+ await expect(pending).rejects.toThrow(/rsync exited with code 24/);
+ await expect(deleteSnapshot(tmpRoot, 'snap-1')).resolves.toMatchObject({ deleted: true });
+ });
+
+ it('releases the source when the integrity preflight refuses the restore', async () => {
+ await writeSnapshotFile('restore-integrity/unrecorded.json', 'added after backup');
+
+ await expect(restoreSnapshot(tmpRoot, 'snap-1', { dryRun: false }))
+ .rejects.toMatchObject({ code: 'BACKUP_FILE_INTEGRITY_FAILED' });
+ expect(spawn).not.toHaveBeenCalled();
+ await expect(deleteSnapshot(tmpRoot, 'snap-1')).resolves.toMatchObject({ deleted: true });
+ });
+ });
+
// #10064: database-authority.json is machine-local admission state. A snapshot
// keeps it as recovery evidence, but a restore must never install it: another
// machine's (or an older cutover's) record would fence a healthy local backend
@@ -4478,6 +4623,43 @@ describe('runBackup lifecycle', () => {
expect(warn).toHaveBeenCalledWith(expect.stringMatching(/^⚠️ Backup retention skipped: DB dump dump_error — keeping older snapshots$/));
});
+ it('skips a snapshot a restore is reading and still prunes the rest (#10898)', async () => {
+ const fsp = await actualFs();
+ const { restoreSnapshot } = await import('./backup.js');
+ const reading = await seedSnapshot(machineHost, '2020-01-01T00-00-00');
+ await fsp.writeFile(joinPath(reading, 'manifest.json'), JSON.stringify({ generatedAt: '2020-01-01T00:00:00.000Z', fileCount: 0, files: {} }));
+ const idle = await seedSnapshot(machineHost, '2020-01-02T00-00-00', { generatedAt: '2020-01-02T00:00:00.000Z' });
+
+ const restoreRsync = fakeProc();
+ const backupRsync = fakeProc();
+ spawn.mockReturnValueOnce(restoreRsync).mockReturnValueOnce(backupRsync);
+ const restoring = restoreSnapshot(destRoot, '2020-01-01T00-00-00', { dryRun: true });
+ await waitFor(() => spawn.mock.calls.length === 1, 'restore rsync spawn');
+
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const pending = runBackup(destRoot, { emit: vi.fn() }, { retentionCount: 1 });
+ await waitFor(() => spawn.mock.calls.length === 2, 'backup rsync spawn');
+ backupRsync.emit('close', 0);
+
+ await expect(pending).resolves.toMatchObject({ status: 'ok', prunedSnapshots: 1 });
+ await expect(fsp.stat(idle)).rejects.toThrow();
+ await expect(fsp.stat(reading)).resolves.toBeDefined();
+ expect(warn).toHaveBeenCalledWith('⚠️ Backup retention: skipped snapshot 2020-01-01T00-00-00 (SNAPSHOT_IN_USE)');
+
+ restoreRsync.emit('close', 0);
+ await expect(restoring).resolves.toMatchObject({ dryRun: true, verification: { status: 'verified' } });
+ });
+
+ it('keeps tolerating live source files that vanish mid-backup (rsync exit 24)', async () => {
+ const rsync = fakeProc();
+ spawn.mockReturnValue(rsync);
+ const pending = runBackup(destRoot, { emit: vi.fn() });
+ await waitFor(() => spawn.mock.calls.length === 1, 'rsync spawn');
+ rsync.emit('close', 24);
+
+ await expect(pending).resolves.toMatchObject({ status: 'ok' });
+ });
+
it('still prunes when the explicit file-backend dump is skipped', async () => {
const fsp = await actualFs();
const old = await seedSnapshot(machineHost, '2020-01-01T00-00-00', { generatedAt: '2020-01-01T00:00:00.000Z' });