Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: refuse unsafe writer inventories before database cutover (#12062)

Adam Eivya2c005ddb6c8
diff --git a/docs/STORAGE.md b/docs/STORAGE.md
index 27dbc97dd8..dcb3c44285 100644
--- a/docs/STORAGE.md
+++ b/docs/STORAGE.md
@@ -209,9 +209,9 @@ There is deliberately no PID-death, elapsed-time, or force recovery. A launch in
For local diagnostics at any stage, run `node scripts/database-maintenance.mjs status`. From `exporting` onward it adds `transfer: { dump: "absent" | "recorded", import: "pending" | "committed" }`; malformed transfer records fail closed. An owned operation adds a bounded `coordinator` result: `unregistered` means no worker reservation; `awaiting-exit` means no supervisor completion proof (either running or interrupted); `exited` includes the supervisor exit code. Neither is a transfer-success verdict. Malformed ownership, binding, or exit records fail closed. Only an **unowned accepted** operation can use `node scripts/database-maintenance.mjs cancel <operation-id>` after checking saved source configuration. For an owned operation whose worker exited (`coordinator.state: "exited"`), `node scripts/database-maintenance.mjs recover <operation-id>` relaunches the worker for the SAME recorded operation and prints `{ recovery: "launched" | "running" }`; there is no launch of a new operation, force, skip or reverse command. `awaiting-exit` means no recovery is possible yet: wait for the running worker, or — if it was interrupted before its supervisor recorded an exit — keep the fence and inspect it by hand. Run `recover` from a shell without `PGHOST`/`PGPORT` overrides; the worker refuses when the saved configuration does not resolve to the recorded target. Preserve the journal, worker records, `transfer-*.json`, the source database and the `portos-maintenance-<operation-id>.sql` dump; do not delete markers, edit or replace the dump, or call `scripts/db.sh migrate` (still refused). Stage-specific meaning: `accepted` — nothing stopped; `quiescing` — producers may be stopped, no dump yet; `exporting` + `dump: absent` — the next attempt re-exports from the quiesced source; `exporting`/`importing` + `dump: recorded` — the source copy is fixed and only it may be imported; `importing` + `import: pending` — the target transaction did not commit, retry imports the same dump; `import: committed` — mode commit and restart verification remain; `committing` — saved mode is being set to the target; `verifying` — saved mode names the target, waiting for a restarted server's own proof; `verified` — only release remains. After release `status` reports `{ stage: "idle", lastCutover: { id, source, target } }`. If a successor refuses because an earlier coordinator's process is still running, inspect this install's process table (`ps -A -o pid,pgid,command`) and stop that leftover dump/import yourself before the next recovery.
-`data/database-writers/<launch-id>/` is machine-local, file-primary **detached launch evidence**. `spawnDetached` publishes and fsyncs a reservation before its first asynchronous setup step, then checks admission again. A launch that races fence acceptance is therefore either refused or already represented in the inventory. It checks admission again after setup; a launch refused before any launcher started records `abandoned.json`, the only proof that no child exists. On POSIX the detached outer `sh` is a session and process-group leader, so its PID (`launcher.json`) names the group every supervisor/job descendant inherits; `launch.json` adds the job PID and launch time. Reservations contain only a random launch ID, timestamp, control directory, and process-group mode. Commands, arguments, environment variables, credentials, and application records are excluded. A partial/unreadable record fails inventory closed. On completion, POSIX keeps the identity and retires the entry only after two consecutive process-table snapshots show the job gone and every recorded group empty (a group ID cannot be reused while it has members); a surviving descendant or slow supervisor leaves the record for maintenance reconciliation. Windows has no group proof, so its completion still compacts into the identity-free `unreconciled-exits.json` marker, published from fsynced bytes with its parent directory synced before deletion. Ordinary completed jobs do not accumulate one permanent directory per launch.
+`data/database-writers/<launch-id>/` is machine-local, file-primary **detached launch evidence**. `spawnDetached` publishes and fsyncs a reservation before its first asynchronous setup step, then checks admission again. A launch that races fence acceptance is therefore either refused or already represented in the inventory. It checks admission again after setup; a launch refused before any launcher started (including a POSIX launcher spawn failure with no PID) records `abandoned.json`, the only proof that no child exists. On POSIX the detached outer `sh` is a session and process-group leader, so its PID (`launcher.json`) names the group every supervisor/job descendant inherits; `launch.json` adds the job PID and launch time. Reservations contain only a random launch ID, timestamp, control directory, and process-group mode. Commands, arguments, environment variables, credentials, and application records are excluded. A partial/unreadable record fails inventory closed. Regular dotfiles in the registry root are ignored; other unexpected entries, including hidden directories or symlinks, refuse with the entry name. On completion, POSIX keeps the identity and retires the entry only after two consecutive process-table snapshots show the job gone and every recorded group empty (a group ID cannot be reused while it has members); a surviving descendant or slow supervisor leaves the record for maintenance reconciliation. Windows has no group proof, so its completion still compacts into the identity-free `unreconciled-exits.json` marker, published from fsynced bytes with its parent directory synced before deletion. Ordinary completed jobs do not accumulate one permanent directory per launch.
-Run `node scripts/database-maintenance.mjs writers` for bounded counts at any stage (`unresolved`, `launching`, `launched`, `exited`, `abandoned`); it always reports `quiescenceVerified: false` and exposes no control paths or process identities. A malformed inventory refuses the command. A `{ state: 'unresolved', kind: 'retired-exits' }` entry represents legacy or Windows compacted completion history. **An inventory read never proves descendant quiescence.** Only the internal `reconcileDetachedWriters(id, token)` stage (`server/services/databaseMaintenanceQuiescence.js`) can, and only for the owning coordinator worker after `stopOwnedDatabaseProducers` recorded producer shutdown at `quiescing`. It classifies each record against a POSIX process-table snapshot: a job still alive with a verified identity (its PGID is recorded and it started inside the launch window) has its recorded groups sent SIGTERM, then SIGKILL after a grace period; a group seen empty is never signalled again. It refuses, leaving the fence and every record in place, when an admitted launch has not recorded a process identity, when a record is legacy/compacted, lacks a launcher group, or is a process-group job whose own PID was never recorded, when a live PID does not match its recorded identity (possible reuse), when an exited job still has surviving group members, when groups do not terminate, when a detached supervisor of this install's data directory is running without a record (for example a launch from code that predates the registry), or on Windows for anything but abandoned launches. Two consecutive quiescent snapshots are required before it archives every record to `data/database-maintenance/reconciled-writers/<launch-id>/` and returns `quiescenceVerified: true, transferReady: false`. It never infers absence from a PID probe or elapsed time and has no force/skip option. It is internal: `quiesceDatabaseWriters(id, token)` runs producer shutdown, the predecessor-coordinator proof, then reconciliation inside the entered transfer worker, which repeats it on every attempt. Limits: a descendant that deliberately leaves its session and process group (`setsid`) is not observable by group, and orphans of a pre-registry launch whose supervisor already exited cannot be identified. Records are local recovery evidence included in backups, never a federated application collection or reference seed; do not delete them to unblock maintenance. Damaged ownership or inventory makes the transfer worker exit with code 1 and a redacted refusal. Its stdout/stderr, owner binding, one-use entry marker, group record and supervisor receipt remain in the reserved directory.
+Run `node scripts/database-maintenance.mjs writers` for bounded counts at any stage (`unresolved`, `launching`, `launched`, `exited`, `abandoned`) and `unresolvedIds` identifying retained launch reservations; it always reports `quiescenceVerified: false` and exposes no control paths or process identities. A malformed inventory refuses the command. A `{ state: 'unresolved', kind: 'retired-exits' }` entry represents legacy or Windows compacted completion history. **An inventory read never proves descendant quiescence.** Only the internal `reconcileDetachedWriters(id, token)` stage (`server/services/databaseMaintenanceQuiescence.js`) can, and only for the owning coordinator worker after `stopOwnedDatabaseProducers` recorded producer shutdown at `quiescing`. It classifies each record against a POSIX process-table snapshot: a job still alive with a verified identity (its PGID is recorded and it started inside the launch window) has its recorded groups sent SIGTERM, then SIGKILL after a grace period; a group seen empty is never signalled again. It refuses, leaving the fence and every record in place, when an admitted launch has not recorded a process identity, when a record is legacy/compacted, lacks a launcher group, or is a process-group job whose own PID was never recorded, when a live PID does not match its recorded identity (possible reuse), when an exited job still has surviving group members, when groups do not terminate, when a detached supervisor of this install's data directory is running without a record (for example a launch from code that predates the registry), or on Windows for anything but abandoned launches. Two consecutive quiescent snapshots are required before it archives every record to `data/database-maintenance/reconciled-writers/<launch-id>/` and returns `quiescenceVerified: true, transferReady: false`. It never infers absence from a PID probe or elapsed time and has no force/skip option. It is internal: `quiesceDatabaseWriters(id, token)` runs producer shutdown, the predecessor-coordinator proof, then reconciliation inside the entered transfer worker, which repeats it on every attempt. Limits: a descendant that deliberately leaves its session and process group (`setsid`) is not observable by group, and orphans of a pre-registry launch whose supervisor already exited cannot be identified. Records are local recovery evidence included in backups, never a federated application collection or reference seed; do not delete them to unblock maintenance. Damaged ownership or inventory makes the transfer worker exit with code 1 and a redacted refusal. Its stdout/stderr, owner binding, one-use entry marker, group record and supervisor receipt remain in the reserved directory.
A cancelled unowned accepted operation moves to `data/database-maintenance-cancelled/<operation-id>/` as local recovery evidence. These records remain filesystem-backed and included in backups; they are not a growing application collection. Restoring an active journal deliberately restores its fence: do not delete it merely to make startup succeed. A released operation moves to `data/database-maintenance-completed/<operation-id>/` the same way. `data/database-authority.json` is `file-primary`, machine-local state read before PostgreSQL: one versioned record naming the last released cutover's operation id and its retired source and released target endpoints (no password or records). Absent means no cutover has completed; malformed fails closed. It is never federated, has no reference seed, and needs no migration. Snapshots still contain it as recovery evidence, but `restoreSnapshot` excludes it from every file restore (full or scoped, preview and execution alike): the record describes a cutover on the machine that made it, so installing another machine's — or an older snapshot's, after a reverse cutover — would fence a healthy local backend as retired. The destination's file is left byte-for-byte as it was (absent stays absent, damaged stays damaged), and an explicit `database-authority.json` selection is refused with `BACKUP_RESTORE_MACHINE_LOCAL`. Ordinary records in the same snapshot restore normally. See [database maintenance admission](BACKUP.md#database-maintenance-admission) for the operator contract and current limitations.
@@ -452,7 +452,7 @@ backend names through the ordinary instance authentication gate. It checks the
saved direction against the running pool and requires complete, trusted, idle
work state. A successful response is `{ source, target, advisory: true, accepted: false }`: it creates no operation, reserves no maintenance window, and does
not promise that a later request is safe. Cutover acceptance repeats
-these checks under its final admission protocol. Missing/unreadable work state,
+these checks under its final admission protocol. Both inspect the detached writer inventory against the process table and refuse pending, ambiguous, or orphaned writers before producer shutdown. Acceptance repeats the inventory check after draining publications, before creating the fence, and under the fence before taking coordinator ownership. An unsafe final check cancels the unowned operation when the saved source still matches. Missing/unreadable work state,
configuration drift, or an existing maintenance fence refuses the check.
The internal `stopOwnedDatabaseProducers` coordinator stage retains the original
diff --git a/scripts/database-maintenance.mjs b/scripts/database-maintenance.mjs
index 3adcbb37ed..8992e2b9cb 100644
--- a/scripts/database-maintenance.mjs
+++ b/scripts/database-maintenance.mjs
@@ -34,8 +34,12 @@ async function main() {
const [command, ...args] = process.argv.slice(2);
if (command === 'writers' && args.length === 0) {
const counts = { unresolved: 0, launching: 0, launched: 0, exited: 0, abandoned: 0 };
- for (const writer of createDatabaseWriterRegistry().read()) counts[writer.state] += 1;
- return { ...counts, quiescenceVerified: false };
+ const unresolvedIds = [];
+ for (const writer of createDatabaseWriterRegistry().read()) {
+ counts[writer.state] += 1;
+ if (writer.state === 'unresolved' && writer.id) unresolvedIds.push(writer.id);
+ }
+ return { ...counts, unresolvedIds, quiescenceVerified: false };
}
if (command === 'status' && args.length === 0) {
const record = journal.read();
diff --git a/server/lib/databaseWriterRegistry.js b/server/lib/databaseWriterRegistry.js
index 41bc9b0f51..320c72ff61 100644
--- a/server/lib/databaseWriterRegistry.js
+++ b/server/lib/databaseWriterRegistry.js
@@ -24,7 +24,7 @@ const completionSchema = z.object({
}).strict();
const retirementSchema = z.object({ version: z.literal(1) }).strict();
const RETIRED_EXITS = 'unreconciled-exits.json';
-const refused = () => new Error('Detached writer inventory is incomplete or unreadable; maintenance must remain fenced.');
+const refused = name => new Error(`Detached writer inventory is incomplete or unreadable${name === undefined ? '' : ` at entry ${JSON.stringify(name)}`}; maintenance must remain fenced.`);
function syncDirectory(directory) {
if (process.platform === 'win32') return;
@@ -148,9 +148,15 @@ export function createDatabaseWriterRegistry(dataDir = PATHS.data) {
}
const retired = names.includes(RETIRED_EXITS);
if (retired) readRecord(directory, RETIRED_EXITS, retirementSchema);
- const writers = names.filter(name => name !== RETIRED_EXITS).sort().map(name => {
- try { return readEntry(name); } catch { throw refused(); }
- });
+ const writers = [];
+ for (const name of names.filter(name => name !== RETIRED_EXITS).sort()) {
+ try {
+ // Finder/editor regular dotfiles cannot contain a launch reservation.
+ // Hidden directories and symlinks still fail closed.
+ if (name.startsWith('.') && lstatSync(join(directory, name)).isFile()) continue;
+ writers.push(readEntry(name));
+ } catch { throw refused(name); }
+ }
// Keep this in the inventory itself so a consumer cannot accidentally read
// only current launch rows and mistake a compacted history for an empty set.
if (retired) writers.push({ state: 'unresolved', kind: 'retired-exits' });
diff --git a/server/lib/databaseWriterRegistry.test.js b/server/lib/databaseWriterRegistry.test.js
index 8015bf458a..1ef333448d 100644
--- a/server/lib/databaseWriterRegistry.test.js
+++ b/server/lib/databaseWriterRegistry.test.js
@@ -3,6 +3,8 @@ import { mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, unlinkSync,
import { join } from 'node:path';
import { tmpdir } from 'node:os';
import { fileURLToPath } from 'node:url';
+import { EventEmitter } from 'node:events';
+import * as childProcess from './childProcess.js';
import { spawn, spawnSync } from 'node:child_process';
import { classifyWriterQuiescence, createDatabaseWriterRegistry } from './databaseWriterRegistry.js';
import { createDatabaseMaintenanceJournal } from './databaseMaintenanceJournal.js';
@@ -39,6 +41,7 @@ beforeEach(() => {
});
afterEach(async () => {
context.setupBarrier = null;
+ vi.restoreAllMocks();
for (const { handle, done } of handles) {
if (handle.exitCode === null && handle.signalCode === null) handle.kill('SIGKILL');
await done;
@@ -90,6 +93,33 @@ describe('durable detached launch admission', () => {
expect(readdirSync(join(context.data, 'database-writers'))).toEqual([]);
});
+ it.skipIf(process.platform === 'win32')('records a launcher spawn refusal as abandoned', async () => {
+ const error = Object.assign(new Error('spawn sh EAGAIN'), { code: 'EAGAIN' });
+ vi.spyOn(childProcess, 'spawn').mockImplementationOnce(() => {
+ const child = new EventEmitter();
+ child.unref = () => {};
+ process.nextTick(() => child.emit('error', error));
+ return child;
+ });
+ const handle = await spawnDetached(process.execPath, ['-e', 'process.exit(0)'], {
+ controlDir: join(root, 'failed-control'), pollMs: 1,
+ });
+ expect((await observe(handle)).error).toBe(error);
+ expect(registry.read()).toEqual([expect.objectContaining({ state: 'abandoned' })]);
+ });
+
+ it('ignores only regular dotfiles and identifies unsafe inventory entries', () => {
+ const directory = join(context.data, 'database-writers');
+ mkdirSync(directory, { recursive: true });
+ writeFileSync(join(directory, '.DS_Store'), 'finder');
+ expect(registry.read()).toEqual([]);
+ writeFileSync(join(directory, 'unexpected.txt'), 'unknown');
+ expect(() => registry.read()).toThrow(/unexpected.txt/);
+ unlinkSync(join(directory, 'unexpected.txt'));
+ mkdirSync(join(directory, '.hidden'));
+ expect(() => registry.read()).toThrow(/\.hidden/);
+ });
+
it('fails closed on legacy compacted exit history and refuses to extend it when unreadable', () => {
mkdirSync(join(context.data, 'database-writers'), { recursive: true });
writeFileSync(join(context.data, 'database-writers', 'unreconciled-exits.json'), JSON.stringify({ version: 1 }));
@@ -112,13 +142,13 @@ describe('durable detached launch admission', () => {
});
it('reports bounded operator diagnostics without publishing local identities or claiming quiescence', () => {
- registry.reserve(join(root, 'private-control'));
+ const writer = registry.reserve(join(root, 'private-control'));
const cli = new URL('../../scripts/database-maintenance.mjs', import.meta.url);
const result = spawnSync(process.execPath, [fileURLToPath(cli), 'writers'], {
encoding: 'utf8', env: { ...process.env, PORTOS_DATA_ROOT: root }, timeout: 10000,
});
expect(result.status).toBe(0);
- expect(JSON.parse(result.stdout)).toEqual({ unresolved: 1, launching: 0, launched: 0, exited: 0, abandoned: 0, quiescenceVerified: false });
+ expect(JSON.parse(result.stdout)).toEqual({ unresolved: 1, launching: 0, launched: 0, exited: 0, abandoned: 0, unresolvedIds: [writer.id], quiescenceVerified: false });
expect(result.stdout).not.toContain(root);
});
diff --git a/server/lib/detachedSpawn.js b/server/lib/detachedSpawn.js
index 45d486546d..1796bdc591 100644
--- a/server/lib/detachedSpawn.js
+++ b/server/lib/detachedSpawn.js
@@ -807,6 +807,10 @@ async function spawnDetachedWithReservation(bin, args, {
}
}
+ // No POSIX launcher PID means no process was started. A PID-bearing
+ // launcher remains evidence even if it later errors or its publication fails.
+ if (!isWindows && launchError && !launcher?.pid) markAbandoned(writer);
+
if (isWindows && launchError) {
// Persist cancellation before returning or scheduling cleanup. A supervisor
// still cold-starting must not launch an orphan after this failed handoff.
diff --git a/server/routes/databasePreflight.test.js b/server/routes/databasePreflight.test.js
index a617451af7..6fe4f3a92f 100644
--- a/server/routes/databasePreflight.test.js
+++ b/server/routes/databasePreflight.test.js
@@ -26,6 +26,7 @@ vi.mock('../lib/childProcess.js', async importOriginal => ({
// The detached cutover worker is launched only after acceptance; never here.
vi.mock('../lib/detachedSpawn.js', async importOriginal => ({
...await importOriginal(),
+ snapshotProcesses: vi.fn(),
spawnDatabaseMaintenanceWorker: vi.fn(async () => ({ on() {} })),
}));
@@ -33,12 +34,13 @@ import databaseRoutes from './database.js';
import { PATHS } from '../lib/paths.js';
import { POOL_CONFIG, query } from '../lib/db.js';
import { execFile, spawn } from '../lib/childProcess.js';
+import { createDatabaseWriterRegistry } from '../lib/databaseWriterRegistry.js';
import { createDatabaseMaintenanceJournal } from '../lib/databaseMaintenanceJournal.js';
import { acquireBackupSnapshotCut, withBackupAssetPublication } from '../lib/backupSnapshotBoundary.js';
import { cleanupTempDataRoots } from '../lib/mockPathsDataRoot.js';
import { getSystemActivity } from '../services/activeProcessing.js';
import { countActiveCosAgents, getPersistentMindImageWorkGuard } from '../services/updatePreflight.js';
-import { spawnDatabaseMaintenanceWorker } from '../lib/detachedSpawn.js';
+import { snapshotProcesses, spawnDatabaseMaintenanceWorker } from '../lib/detachedSpawn.js';
import { hostControlRouteGate } from '../services/authGate.js';
const idle = () => ({
@@ -56,6 +58,7 @@ const accept = (body = { source: 'native', target: 'docker' }) => request(app).p
beforeEach(() => {
vi.clearAllMocks();
+ snapshotProcesses.mockReset().mockResolvedValue([]);
for (const [key, value] of Object.entries({ PGHOST: 'localhost', PGPORT: '5432', PORTOS_NATIVE_PGPORT: '5432', PGPORT_DOCKER: '5561', PGUSER: 'example_role', PGDATABASE: 'example_test', PGPASSWORD: 'example-only' })) vi.stubEnv(key, value);
Object.assign(POOL_CONFIG, { host: 'localhost', port: 5432, user: 'example_role', database: 'example_test' });
copyEcosystemConfig(PATHS.installRoot);
@@ -68,6 +71,7 @@ afterEach(() => {
vi.unstubAllEnvs();
rmSync(join(PATHS.data, 'database-maintenance'), { recursive: true, force: true });
rmSync(join(PATHS.data, 'database-maintenance-cancelled'), { recursive: true, force: true });
+ rmSync(join(PATHS.data, 'database-writers'), { recursive: true, force: true });
expect(execFile).not.toHaveBeenCalled();
expect(spawn).not.toHaveBeenCalled();
expect(query).not.toHaveBeenCalled();
@@ -87,6 +91,41 @@ describe('database maintenance preflight HTTP contract', () => {
expect(readFileSync(join(PATHS.installRoot, '.env'), 'utf8')).toBe(`PGMODE=${source}\n`);
});
+ it.each(['pending', 'ambiguous', 'orphaned'])('refuses a %s writer before advice or acceptance closes admission', async verdict => {
+ const writer = createDatabaseWriterRegistry().reserve(join(PATHS.data, 'example-control'));
+ if (verdict === 'ambiguous') writer.launched(4242);
+ if (verdict === 'orphaned') {
+ writer.launcher(4242);
+ snapshotProcesses.mockResolvedValue([{ pid: 4243, pgid: 4242, startedAt: Date.now() }]);
+ }
+ for (const call of [preflight, accept]) {
+ const result = await call();
+ expect(result.status).toBe(409);
+ expect(result.body.code).toBe('DATABASE_PREFLIGHT_BUSY');
+ expect(createDatabaseMaintenanceJournal().read()).toBeNull();
+ expect(spawnDatabaseMaintenanceWorker).not.toHaveBeenCalled();
+ }
+ });
+
+ it.skipIf(process.platform === 'win32')('sanitizes an unavailable process snapshot without fencing the server', async () => {
+ snapshotProcesses.mockRejectedValue(new Error('example private process command'));
+ const result = await accept();
+ expect(result.body.code).toBe('DATABASE_PREFLIGHT_UNTRUSTED');
+ expect(result.body.error).not.toContain('example private');
+ expect(createDatabaseMaintenanceJournal().read()).toBeNull();
+ expect(spawnDatabaseMaintenanceWorker).not.toHaveBeenCalled();
+ });
+
+ it('refuses an unreadable writer inventory with bounded diagnostics', async () => {
+ mkdirSync(join(PATHS.data, 'database-writers'), { recursive: true });
+ writeFileSync(join(PATHS.data, 'database-writers', 'example-private-file'), 'unknown');
+ const result = await accept();
+ expect(result.body.code).toBe('DATABASE_PREFLIGHT_UNTRUSTED');
+ expect(result.body.error).not.toContain('example-private-file');
+ expect(createDatabaseMaintenanceJournal().read()).toBeNull();
+ expect(spawnDatabaseMaintenanceWorker).not.toHaveBeenCalled();
+ });
+
it.each([
{}, { source: 'native', target: 'native' }, { source: 'native', target: 'other' },
{ source: 'native', target: 'docker', force: true },
@@ -223,6 +262,45 @@ describe('database cutover acceptance HTTP contract', () => {
expect((await accept({ source, target })).status).toBe(503);
});
+ it('accepts never-started writers and harmless registry dotfiles', async () => {
+ createDatabaseWriterRegistry().reserve(join(PATHS.data, 'example-abandoned')).abandoned();
+ writeFileSync(join(PATHS.data, 'database-writers', '.DS_Store'), 'finder');
+ expect((await accept()).status).toBe(202);
+ await vi.waitFor(() => expect(spawnDatabaseMaintenanceWorker).toHaveBeenCalledTimes(1));
+ });
+
+ it.skipIf(process.platform === 'win32')('allows a verified running writer for the worker to terminate after acceptance', async () => {
+ const writer = createDatabaseWriterRegistry().reserve(join(PATHS.data, 'example-running'));
+ writer.launcher(4242);
+ writer.launched(4243);
+ snapshotProcesses.mockResolvedValue([{ pid: 4243, pgid: 4242, startedAt: Date.now() }]);
+ expect((await accept()).status).toBe(202);
+ await vi.waitFor(() => expect(spawnDatabaseMaintenanceWorker).toHaveBeenCalledTimes(1));
+ });
+
+ it.skipIf(process.platform === 'win32')('rechecks writers after draining and releases the cut without publishing an unsafe fence', async () => {
+ const registry = createDatabaseWriterRegistry();
+ snapshotProcesses.mockResolvedValueOnce([]).mockImplementationOnce(async () => {
+ registry.reserve(join(PATHS.data, 'example-late-control'));
+ return [];
+ });
+ expect((await accept()).body.code).toBe('DATABASE_PREFLIGHT_BUSY');
+ expect(createDatabaseMaintenanceJournal().read()).toBeNull();
+ expect(spawnDatabaseMaintenanceWorker).not.toHaveBeenCalled();
+ (await acquireBackupSnapshotCut())();
+ });
+
+ it.skipIf(process.platform === 'win32')('cancels before worker launch when a writer becomes unprovable under the fence', async () => {
+ const writer = createDatabaseWriterRegistry().reserve(join(PATHS.data, 'example-racing-control'));
+ writer.launcher(4242);
+ snapshotProcesses.mockResolvedValueOnce([]).mockResolvedValueOnce([]).mockResolvedValueOnce([
+ { pid: 4243, pgid: 4242, startedAt: Date.now() },
+ ]);
+ expect((await accept()).body.code).toBe('DATABASE_PREFLIGHT_BUSY');
+ expect(createDatabaseMaintenanceJournal().read()).toBeNull();
+ expect(spawnDatabaseMaintenanceWorker).not.toHaveBeenCalled();
+ });
+
it('refuses stale and reversed requests without publishing an operation', async () => {
expect((await accept({ source: 'docker', target: 'native' })).body.code).toBe('DATABASE_PREFLIGHT_STALE');
POOL_CONFIG.port = 6000;
diff --git a/server/services/databasePreflight.js b/server/services/databasePreflight.js
index cf9606115f..48ac7e166a 100644
--- a/server/services/databasePreflight.js
+++ b/server/services/databasePreflight.js
@@ -9,6 +9,7 @@ import { acquireBackupSnapshotCut } from '../lib/backupSnapshotBoundary.js';
import { assertDatabaseAdmission, createDatabaseMaintenanceJournal, databaseMaintenanceEndpointSchema } from '../lib/databaseMaintenanceJournal.js';
import { databaseMaintenancePreflightSchema } from '../lib/validation.js';
import { ServerError } from '../lib/errorHandler.js';
+import { classifyWriterQuiescence, createDatabaseWriterRegistry } from '../lib/databaseWriterRegistry.js';
import { summarizeSystemActivity } from '../lib/systemIdle.js';
const require = createRequire(import.meta.url);
@@ -75,6 +76,20 @@ async function assertTrustedIdle() {
if (!summarizeSystemActivity(activity.data).idle || activeAgents > 0 || !mindGuard.safe) throw busy();
}
+async function assertReconcilableWriters() {
+ let rows, processes;
+ try {
+ const { snapshotProcesses } = await import('../lib/detachedSpawn.js');
+ processes = process.platform === 'win32' ? [] : await snapshotProcesses();
+ // Read after the asynchronous snapshot so newly admitted reservations are
+ // included. Acceptance repeats this under the fence before worker launch.
+ rows = createDatabaseWriterRegistry().read();
+ } catch { throw unavailable(); }
+ if (rows.some(row => process.platform === 'win32'
+ ? row.state !== 'abandoned'
+ : ['pending', 'ambiguous', 'orphaned'].includes(classifyWriterQuiescence(row, processes).verdict))) throw busy();
+}
+
/** Non-mutating advice only; no operation token, reservation, or writer drain. */
export async function preflightDatabaseMaintenance(input) {
const direction = databaseMaintenancePreflightSchema.parse(input);
@@ -82,6 +97,7 @@ export async function preflightDatabaseMaintenance(input) {
const initial = readConfiguration();
validateDirection(direction, initial);
await assertTrustedIdle();
+ await assertReconcilableWriters();
// No await between the final fence/configuration checks and the response.
// This narrows the observation race, but is deliberately NOT acceptance.
assertDatabaseAdmission();
@@ -119,14 +135,18 @@ export async function acceptDatabaseMaintenance(input) {
});
let operation;
try {
- operation = journal.begin({ source: configuration.source, target: configuration.target });
- } catch {
- throw refuse('DATABASE_MAINTENANCE', 'Another database maintenance operation is active or requires recovery.');
+ await assertReconcilableWriters();
+ try {
+ operation = journal.begin({ source: configuration.source, target: configuration.target });
+ } catch {
+ throw refuse('DATABASE_MAINTENANCE', 'Another database maintenance operation is active or requires recovery.');
+ }
} finally {
releaseSnapshotCut();
}
try {
await assertTrustedIdle();
+ await assertReconcilableWriters();
const final = readConfiguration();
validateDirection(direction, final);
if (final.revision !== configuration.revision) throw stale();