Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: resume proven initialized retirement journals automatically

[._.]/ Adam Eivy14e971a4afa6
diff --git a/docs/features/branch-reconciliation.md b/docs/features/branch-reconciliation.md
index 2778166dd6..9122f52716 100644
--- a/docs/features/branch-reconciliation.md
+++ b/docs/features/branch-reconciliation.md
@@ -217,6 +217,57 @@ A move-based restore retry refuses when both originals are already restored and
its working-move proof is present, preserving that evidence for this separate
finalization path instead of replacing it with no-op observations.
+### Automatic continuation of initialized retirement journals
+
+CoS discovers existing private initialized-retirement journals at startup evaluation
+and every 30 minutes, independently of branch-reconcile scheduling or free agent
+slots. It acts only while the daemon is running, unpaused, and CoS autonomy is
+`execute`. Discovery never starts a retirement or grants authority from a journal
+phase alone. The bounded drain handles at most two transactions per pass, rotates
+past held work, and refreshes evidence on subsequent passes. It creates no agent,
+worktree, or additional persistent queue.
+
+Automatic continuation requires a valid original transaction, a bound preservation
+manifest, the original exact reservation, and fresh service/native proofs. The
+local tip must equal the merged PR head; the complete scoped comparison must be
+`integrated` with no differences. The native classifier permits only the archived
+residual-compression removal and root-provenance addition described below. Every
+other source field must still match the preserved baseline. A successful review
+is passed to the same purpose-specific operation with its exact digest; the
+operation recomputes the proof before moving anything. After its internal scan,
+the native helper waits for an exact, fresh server acknowledgement before rename;
+pause, a lost executor, invalid replies, and parent exit grant nothing. The branch
+receiver guard runs after preservation and fsyncs deletion intent before invoking
+Git. Before lengthy branch verification, `branch-retirement-prepared` durably
+binds the new exact reservation without granting deletion. Restart obtains a fresh
+branch review under that same cut. The daemon never accepts
+supersession. Branch retirement requires its own subsequent fresh review.
+
+A plain `quarantined` journal whose reservation release was interrupted can finish
+through this same proof without moving either original again. Durably confirmed
+branch deletion can replay its receipt without another Git deletion; it still
+requires archive, placement, absent-ref, ownership, and reservation checks. A
+completed receipt or completed restore with an already released reservation needs
+no further action and consumes no replay slot. A missing owner file inside an
+existing cut is uncertainty, not a released reservation.
+
+Unknown metadata, changed sources, superseded work, active owners, manual policy
+holds, unsupported pre-move/restoration phases, and uncertain deletion results
+remain held for explicit review. CoS does not restore originals automatically,
+transfer a cut, infer ownership from age/PIDs, or turn ref absence into a deletion
+receipt. Existing schedule status exposes `reconcileOutcomes[appId].initializedRetirements`
+with each transaction, branch, observed journal digest, phase, and reason. These
+observations are diagnostics, never cached approval. Invalid unbound journals are
+reported in the global reconcile outcome. The journals and archives remain the
+recovery evidence after restart. A crash between cut creation and publication of
+its preparation record remains an explicit reservation-mismatch hold; the daemon
+never adopts an unrecorded owner. A pause or interruption after deletion intent
+was fsynced remains uncertain even when no receiver result was observed.
+
+The narrow machine-verified continuation above does not require another runtime
+agent or human approval. The explicit operations below remain available for
+independent inspection, supersession decisions, and unsupported recovery cases.
+
### Completing an interrupted quarantine
When the working directory has already moved into quarantine, restoring it
@@ -280,6 +331,6 @@ age, timer or automatic retry clears this evidence. Reservation release atomical
moves the intact owner into a durable private release record, preserving evidence
across a crash without leaving an ownerless cut. Receipt replay keeps the original
archive identity and timestamp so a newer checkout obligation cannot be satisfied
-by replaying an older retirement. An interrupted transaction
-requires explicit status and recovery; ambiguous identities remain held. The
+by replaying an older retirement. Interrupted transactions outside the machine-verified continuation above
+require explicit status and recovery; ambiguous identities remain held. The
private journal and all copies remain available after success or recovery.
diff --git a/scripts/lib/initialized_checkout_archive.py b/scripts/lib/initialized_checkout_archive.py
index 1f37623a15..4a89e442b6 100644
--- a/scripts/lib/initialized_checkout_archive.py
+++ b/scripts/lib/initialized_checkout_archive.py
@@ -1350,6 +1350,29 @@ def verify_retired_quarantine(request):
return verified_quarantine_resumption(request, completed=True, retired=True)
+def await_mutation_permit(request):
+ """Automatic callers recheck the live daemon gate after native verification.
+
+ The backend lock stays held while waiting. EOF, missing/invalid replies and
+ parent death grant nothing; explicit operations retain their old protocol.
+ """
+ token = request.get('mutationPermit')
+ if token is None:
+ return
+ if not isinstance(token, str) or not re.fullmatch(r'[a-f0-9-]{36}', token):
+ fail('mutation-permit-invalid')
+ print(json.dumps({'mutationReady': token, 'part': 'administration'}), flush=True)
+ reply = sys.stdin.readline(4097)
+ try:
+ accepted = json.loads(reply)
+ except (ValueError, TypeError):
+ fail('mutation-permit-missing')
+ if (len(reply) > 4096 or not reply.endswith('\n') or not isinstance(accepted, dict)
+ or set(accepted) != {'mutationPermit', 'allow'}
+ or accepted['mutationPermit'] != token or accepted['allow'] is not True):
+ fail('mutation-permit-invalid')
+
+
def resume_quarantine(request):
verified = verify_quarantine_resumption(request)
if verified['placements']['administration']['state'] == 'quarantine':
@@ -1360,6 +1383,9 @@ def resume_quarantine(request):
source, destination = Path(facts['privateGitDir']), root / 'quarantine/administration'
safe_path(str(source))
safe_path(str(destination.parent))
+ if identity(source) != facts['identity']['administration'] or os.path.lexists(destination):
+ fail('move-lease-or-destination-changed')
+ await_mutation_permit(request)
if identity(source) != facts['identity']['administration'] or os.path.lexists(destination):
fail('move-lease-or-destination-changed')
exclusive_rename(source, destination)
diff --git a/scripts/lib/initialized_checkout_archive_test.py b/scripts/lib/initialized_checkout_archive_test.py
index 5454a36fec..413647c069 100644
--- a/scripts/lib/initialized_checkout_archive_test.py
+++ b/scripts/lib/initialized_checkout_archive_test.py
@@ -780,6 +780,43 @@ class NativeArchive(unittest.TestCase):
self.assertFalse(self.work.exists())
self.assertEqual((self.txn / 'manifest.json').read_bytes(), baseline)
+ def test_automatic_quarantine_requires_fresh_exact_permit_after_native_verification(self):
+ request = {**self.quarantine_resumption_fixture(), 'command': 'resume-quarantine',
+ 'mutationPermit': '11111111-1111-1111-1111-111111111111'}
+ saved = archive.load_transaction(request)[1]
+ source = Path(saved['facts']['privateGitDir'])
+ original = archive.identity(source)
+ destination = self.txn / 'quarantine/administration'
+ for reply in ('', '{}\n', json.dumps({'mutationPermit': request['mutationPermit'], 'allow': False}) + '\n',
+ json.dumps({'mutationPermit': request['mutationPermit'], 'allow': 1}) + '\n'):
+ with subprocess.Popen([sys.executable, str(Path(archive.__file__))], env=self.env,
+ stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ text=True) as child:
+ child.stdin.write(json.dumps(request) + '\n')
+ child.stdin.flush()
+ ready = json.loads(child.stdout.readline())
+ self.assertEqual(ready, {'mutationReady': request['mutationPermit'], 'part': 'administration'})
+ # This readiness record is emitted after full native verification,
+ # while the exact original and backend lock remain in place.
+ self.assertEqual(archive.identity(source), original)
+ self.assertFalse(destination.exists())
+ output, _ = child.communicate(reply, timeout=30)
+ self.assertNotEqual(child.returncode, 0)
+ self.assertFalse(json.loads(output)['ok'])
+ self.assertEqual(archive.identity(source), original)
+ self.assertFalse(destination.exists())
+ with subprocess.Popen([sys.executable, str(Path(archive.__file__))], env=self.env,
+ stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ text=True) as child:
+ child.stdin.write(json.dumps(request) + '\n')
+ child.stdin.flush()
+ self.assertEqual(json.loads(child.stdout.readline())['mutationReady'], request['mutationPermit'])
+ output, _ = child.communicate(json.dumps({'mutationPermit': request['mutationPermit'], 'allow': True}) + '\n', timeout=30)
+ self.assertEqual(child.returncode, 0, output)
+ self.assertTrue(json.loads(output)['moved'])
+ self.assertEqual(archive.identity(destination), original)
+ self.assertFalse(source.exists())
+
def test_quarantine_resumption_recovers_postmove_evidence_interruption_without_replaying_rename(self):
request = self.quarantine_resumption_fixture()
write = archive.private_json
diff --git a/server/routes/cosScheduleRoutes.js b/server/routes/cosScheduleRoutes.js
index 87a2efae9d..030c7beb66 100644
--- a/server/routes/cosScheduleRoutes.js
+++ b/server/routes/cosScheduleRoutes.js
@@ -314,7 +314,8 @@ router.post('/schedule/branch-reconcile/checkouts/retire', asyncHandler(async (r
res.json(result);
}));
-// Explicit operator capability; automatic cleanup retains its initialized-module guard.
+// Initial retirement remains explicit. Only existing proof-bound journals can be
+// continued by the deterministic daemon; ordinary cleanup keeps its module guard.
router.post('/schedule/branch-reconcile/initialized-checkouts/review', asyncHandler(async (req, res) => {
const { appId, prUrl, ...release } = validateRequest(heldCheckoutReviewBodySchema.strict(), req.body);
const { reviewInitializedCheckout } = await import('../services/initializedCheckoutRecovery.js');
diff --git a/server/services/cos.js b/server/services/cos.js
index 0a2b986e2c..966464172a 100644
--- a/server/services/cos.js
+++ b/server/services/cos.js
@@ -562,6 +562,20 @@ async function bootDaemon(state) {
// set to `execute` and the daemon is not paused.
// The handoff follow-up poll (prWatcher.followUpHandoffReceipt) shares the
// gate and is cancelled with the daemon.
+ // Existing retirement journals must recover even when branch-reconcile is
+ // disabled or every agent slot is occupied. The service reuses its exact cut;
+ // it never goes through normal agent/worktree admission.
+ const { INITIALIZED_RECOVERY_INTERVAL_MS, sweepInitializedCheckoutRecoveries } = await import('./initializedCheckoutRecoverySweep.js');
+ scheduleEvent({
+ id: 'cos-initialized-retirement-recovery',
+ type: 'interval',
+ intervalMs: INITIALIZED_RECOVERY_INTERVAL_MS,
+ handler: async () => {
+ await sweepInitializedCheckoutRecoveries(pendingMergeSweepBlocked)
+ .catch(error => console.warn(`⚠️ Initialized retirement recovery deferred: ${error.message}`));
+ },
+ metadata: { description: 'Resume verified initialized retirement journals' },
+ });
const { PENDING_MERGE_SWEEP_INTERVAL_MS, setHandoffFollowUpGate } = await import('./prWatcher.js');
setHandoffFollowUpGate(pendingMergeSweepBlocked);
scheduleEvent({
@@ -661,6 +675,7 @@ async function teardownDaemon() {
cancelEvent('cos-rehabilitation-check');
cancelEvent('cos-improvement-check');
cancelEvent('cos-pending-merge-sweep');
+ cancelEvent('cos-initialized-retirement-recovery');
(await import('./prWatcher.js')).cancelHandoffFollowUps();
await unregisterJobSchedules();
diff --git a/server/services/cos.startStop.test.js b/server/services/cos.startStop.test.js
index 1de12667d2..5bc76054e3 100644
--- a/server/services/cos.startStop.test.js
+++ b/server/services/cos.startStop.test.js
@@ -13,12 +13,22 @@
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
const scheduled = new Set();
+const handlers = new Map();
vi.mock('./eventScheduler.js', async (importOriginal) => ({
...(await importOriginal()),
- schedule: vi.fn(({ id }) => { scheduled.add(id); return id; }),
+ schedule: vi.fn(({ id, handler }) => { scheduled.add(id); handlers.set(id, handler); return id; }),
cancel: vi.fn((id) => scheduled.delete(id)),
}));
+const recoveryDrain = vi.hoisted(() => ({ calls: [] }));
+vi.mock('./initializedCheckoutRecoverySweep.js', () => ({
+ INITIALIZED_RECOVERY_INTERVAL_MS: 30 * 60_000,
+ sweepInitializedCheckoutRecoveries: async blocked => {
+ const reason = await blocked();
+ if (!reason) recoveryDrain.calls.push('replay');
+ return { skipped: reason };
+ },
+}));
let persisted = { running: false, paused: false, agents: {}, config: { healthCheckIntervalMs: 900000 } };
vi.mock('./cosState.js', async (importOriginal) => ({
...(await importOriginal()),
@@ -153,6 +163,25 @@ describe('CoS daemon start/stop lifecycle (#11860)', () => {
expect(handoffFollowUp.gate).toBe(null);
});
+ it('arms journal recovery independently of agent slots and branch tasks, and enforces live autonomy on every timer', async () => {
+ recoveryDrain.calls.length = 0;
+ persisted = { ...persisted, config: { ...persisted.config, maxConcurrentAgents: 0, domainAutonomy: { cos: 'execute' } } };
+ const starting = start();
+ await waitFor(() => typeof releaseRecovery === 'function');
+ releaseRecovery(); await starting;
+ expect(cosEventIds()).toContain('cos-initialized-retirement-recovery');
+ const tick = handlers.get('cos-initialized-retirement-recovery');
+ await tick();
+ expect(recoveryDrain.calls).toEqual(['replay']);
+ persisted.paused = true; await tick();
+ persisted.paused = false;
+ for (const mode of ['off', 'dry-run']) { persisted.config.domainAutonomy.cos = mode; await tick(); }
+ expect(recoveryDrain.calls).toEqual(['replay']);
+ await stop(); await tick();
+ expect(recoveryDrain.calls).toEqual(['replay']);
+ expect(cosEventIds()).not.toContain('cos-initialized-retirement-recovery');
+ });
+
it('a boot step that throws rolls the daemon back so the next start() succeeds', async () => {
boot.cleanupOrphanedAgents = vi.fn(async () => { throw new Error('agent state unreadable'); });
await expect(start()).rejects.toThrow('agent state unreadable');
diff --git a/server/services/cosTaskGenerator.js b/server/services/cosTaskGenerator.js
index 9d1822a8d5..13b9565594 100644
--- a/server/services/cosTaskGenerator.js
+++ b/server/services/cosTaskGenerator.js
@@ -882,6 +882,14 @@ export async function evaluateTasks(options) {
// meters agent RUNS — a deterministic merge spawns nothing and consumes none
// of it — while the mode is the user's "may PortOS act on its own" switch.
if (!paused && getDomainMode(state.config, 'cos') === 'execute') {
+ const { sweepInitializedCheckoutRecoveries } = await import('./initializedCheckoutRecoverySweep.js');
+ // This drain has its own cadence/single-flight guard and rechecks autonomy
+ // between stages, including after native archive verification.
+ void sweepInitializedCheckoutRecoveries(async () => {
+ if (!isDaemonRunning()) return 'daemon-stopped';
+ const current = await loadState();
+ return current.paused ? 'daemon-paused' : getDomainMode(current.config, 'cos') !== 'execute' ? 'autonomy-not-execute' : null;
+ }).catch(error => console.warn(`⚠️ Initialized retirement recovery deferred: ${error.message}`));
const prWatcher = await import('./prWatcher.js');
const sweep = await prWatcher.sweepPendingMergePrs();
if (sweep.merged || sweep.escalated || sweep.timedOut) {
diff --git a/server/services/initializedCheckoutNativePermit.test.js b/server/services/initializedCheckoutNativePermit.test.js
new file mode 100644
index 0000000000..d44c5ee941
--- /dev/null
+++ b/server/services/initializedCheckoutNativePermit.test.js
@@ -0,0 +1,70 @@
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+import { EventEmitter } from 'node:events';
+import { PassThrough } from 'node:stream';
+import { pinPlatform } from '../lib/testHelper.js';
+vi.mock('../lib/childProcess.js', () => ({ spawn: vi.fn() }));
+vi.mock('./cosClaimOwnershipState.js', () => ({ withClaimOwnershipLock: work => work() }));
+import { spawn } from '../lib/childProcess.js';
+import { _runInitializedCheckoutNative } from './initializedCheckoutRecovery.js';
+
+let restore, child, sent;
+beforeEach(() => {
+ restore = pinPlatform('darwin');
+ sent = [];
+ child = Object.assign(new EventEmitter(), { stdout: new PassThrough(), stderr: new PassThrough(), stdin: new PassThrough(),
+ kill: vi.fn(() => { queueMicrotask(() => child.emit('close', 1)); }) });
+ child.stdin.on('data', value => sent.push(JSON.parse(value.toString())));
+ spawn.mockReturnValue(child);
+});
+afterEach(() => { restore(); vi.clearAllMocks(); });
+
+const request = { command: 'resume-quarantine', transactionDir: '/fixture/transaction' };
+const ready = () => child.stdout.write(`${JSON.stringify({ mutationReady: sent[0].mutationPermit, part: 'administration' })}\n`);
+
+describe('native mutation permission transport', () => {
+ it('keeps stdin open until post-verification readiness and obtains a fresh gate decision before acknowledging', async () => {
+ const gate = vi.fn(async () => {});
+ const result = _runInitializedCheckoutNative(request, { beforeMutation: gate });
+ expect(sent[0]).toMatchObject({ ...request, mutationPermit: expect.any(String) });
+ expect(child.stdin.writableEnded).toBe(false);
+ expect(gate).not.toHaveBeenCalled();
+ ready();
+ await vi.waitFor(() => expect(sent).toHaveLength(2));
+ expect(sent[1]).toEqual({ mutationPermit: sent[0].mutationPermit, allow: true });
+ child.stdout.write('{"ok":true,"moved":true}\n'); child.emit('close', 0);
+ expect(await result).toEqual({ ok: true, moved: true });
+ expect(gate).toHaveBeenCalledTimes(1);
+ });
+ it('sends no permit when the daemon pauses during native verification, and waits for child exit', async () => {
+ const gate = vi.fn(async () => { throw new Error('daemon-paused'); });
+ const result = _runInitializedCheckoutNative(request, { beforeMutation: gate });
+ const rejected = expect(result).rejects.toThrow('daemon-paused');
+ ready();
+ await rejected;
+ expect(sent).toHaveLength(1);
+ expect(child.kill).toHaveBeenCalledWith('SIGTERM');
+ });
+ it('refuses a mismatched readiness token without calling the gate or acknowledging', async () => {
+ const gate = vi.fn();
+ const result = _runInitializedCheckoutNative(request, { beforeMutation: gate });
+ const rejected = expect(result).rejects.toThrow('retirement-native-permit-invalid');
+ child.stdout.write('{"mutationReady":"foreign","part":"administration"}\n');
+ await rejected;
+ expect(gate).not.toHaveBeenCalled(); expect(sent).toHaveLength(1);
+ });
+ it('does not acknowledge after child exit while the gate is still resolving', async () => {
+ let finish;
+ const result = _runInitializedCheckoutNative(request, { beforeMutation: () => new Promise(resolve => { finish = resolve; }) });
+ const rejected = expect(result).rejects.toThrow();
+ ready(); await vi.waitFor(() => expect(finish).toBeTypeOf('function'));
+ child.emit('close', 1); await rejected;
+ finish(); await Promise.resolve(); await Promise.resolve();
+ expect(sent).toHaveLength(1);
+ });
+ it('preserves the explicit/manual single-request protocol without a mutation permit', async () => {
+ const result = _runInitializedCheckoutNative(request);
+ expect(sent).toEqual([request]); expect(child.stdin.writableEnded).toBe(true);
+ child.stdout.write('{"ok":true,"moved":true}\n'); child.emit('close', 0);
+ expect(await result).toMatchObject({ ok: true, moved: true });
+ });
+});
diff --git a/server/services/initializedCheckoutRecovery.js b/server/services/initializedCheckoutRecovery.js
index 501374c290..019169bc2c 100644
--- a/server/services/initializedCheckoutRecovery.js
+++ b/server/services/initializedCheckoutRecovery.js
@@ -5,8 +5,8 @@
*/
import { spawn } from '../lib/childProcess.js';
import { createHash, randomUUID } from 'node:crypto';
-import { link, lstat, mkdir, open, readFile, readdir } from 'node:fs/promises';
-import { join } from 'node:path';
+import { link, lstat, mkdir, open, opendir, readFile, readdir } from 'node:fs/promises';
+import { isAbsolute, join } from 'node:path';
import { type as hostOsType } from 'node:os';
import { fileURLToPath } from 'node:url';
import { PATHS } from '../lib/paths.js';
@@ -17,13 +17,16 @@ import { withClaimOwnershipLock } from './cosClaimOwnershipState.js';
const helper = fileURLToPath(new URL('../../scripts/lib/initialized_checkout_archive.py', import.meta.url));
const hash = value => createHash('sha256').update(JSON.stringify(value)).digest('hex');
const fail = reason => { throw new Error(reason); };
-const sameRelease = (a, b) => ['path', 'head', 'branch', 'locked', 'lockReason'].every(key => (a[key] ?? null) === (b[key] ?? null));
+const sameRelease = (a = {}, b = {}) => ['path', 'head', 'branch', 'locked', 'lockReason'].every(key => (a[key] ?? null) === (b[key] ?? null));
const transactionRoot = () => join(PATHS.cos, 'initialized-checkout-retirements');
/** Native helper holds its own flock until the actual child exits. */
-function runInitializedCheckoutNative(input, { keepOpen = false } = {}) {
+export function _runInitializedCheckoutNative(input, { keepOpen = false, beforeMutation } = {}) {
if (process.platform !== 'darwin') return Promise.reject(new Error('retirement-platform-unsupported'));
return new Promise((resolve, reject) => {
+ const permit = beforeMutation ? randomUUID() : null;
+ if (permit && (keepOpen || input.command !== 'resume-quarantine')) return reject(new Error('retirement-native-permit-command-invalid'));
+ let permissionRequested = false;
const child = spawn('/usr/bin/python3', [helper], { stdio: ['pipe', 'pipe', 'pipe'], env: {
PATH: process.env.PATH, HOME: process.env.HOME, TMPDIR: process.env.TMPDIR,
PYTHONDONTWRITEBYTECODE: '1',
@@ -56,6 +59,21 @@ function runInitializedCheckoutNative(input, { keepOpen = false } = {}) {
child.stdout.on('data', data => {
output += data;
if (output.length > 4 * 1024 * 1024) return stop(new Error('retirement-native-output-limit'));
+ if (permit && !permissionRequested && output.includes('\n')) {
+ const newline = output.indexOf('\n');
+ let ready;
+ try { ready = JSON.parse(output.slice(0, newline)); } catch { return stop(new Error('retirement-native-permit-invalid')); }
+ // A refused native verification may finish without requesting permission.
+ if (ready.ok === false) return;
+ if (ready.mutationReady !== permit || ready.part !== 'administration'
+ || Object.keys(ready).length !== 2) return stop(new Error('retirement-native-permit-invalid'));
+ permissionRequested = true;
+ output = output.slice(newline + 1);
+ Promise.resolve().then(beforeMutation).then(() => {
+ if (closed || failure) return;
+ child.stdin.end(`${JSON.stringify({ mutationPermit: permit, allow: true })}\n`);
+ }).catch(stop);
+ }
if (keepOpen && !acquired && output.includes('\n')) {
try {
const result = JSON.parse(output.slice(0, output.indexOf('\n')));
@@ -70,8 +88,8 @@ function runInitializedCheckoutNative(input, { keepOpen = false } = {}) {
// Metadata and native diagnostics stay out of public logs.
child.stderr.resume();
child.stdin.on('error', error => { failure ||= error; });
- child.stdin.write(`${JSON.stringify(input)}\n`);
- if (!keepOpen) child.stdin.end();
+ child.stdin.write(`${JSON.stringify(permit ? { ...input, mutationPermit: permit } : input)}\n`);
+ if (!keepOpen && !permit) child.stdin.end();
});
}
@@ -107,14 +125,27 @@ async function appendStep(directory, state, phase, patch = {}) {
return next;
}
-async function readJournal(directory) {
+async function readJournal(directory, { bounded = false, budget } = {}) {
await privateDirectory(directory);
- const files = (await readdir(directory)).filter(name => /^\d{6}\.json$/.test(name)).sort();
- let previous = null;
+ const names = [];
+ if (bounded) {
+ const listing = await opendir(directory);
+ for await (const entry of listing) {
+ if (names.length >= 1024) fail('retirement-journal-budget-exceeded');
+ names.push(entry.name);
+ }
+ } else names.push(...await readdir(directory));
+ const files = names.filter(name => /^\d{6}\.json$/.test(name)).sort();
+ if (bounded && files.length > 128) fail('retirement-journal-budget-exceeded');
+ let previous = null, bytes = 0;
for (const [sequence, name] of files.entries()) {
const info = await lstat(join(directory, name));
if (!info.isFile() || info.isSymbolicLink() || (hostOsType() !== 'Windows_NT' && (info.mode & 0o777) !== 0o600)) fail('retirement-journal-unsafe');
+ bytes += info.size;
+ if (budget) { budget.remaining -= info.size; if (budget.remaining < 0) fail('retirement-discovery-budget-exceeded'); }
+ if (bounded && (info.size > 16 * 1024 * 1024 || bytes > 64 * 1024 * 1024)) fail('retirement-journal-budget-exceeded');
const row = JSON.parse(await readFile(join(directory, name), 'utf8'));
+ if (bounded && previous && ['id', 'appId', 'repoPath', 'release', 'review'].some(key => JSON.stringify(row[key]) !== JSON.stringify(previous[key]))) fail('retirement-journal-identity-changed');
if (row.version !== 1 || row.sequence !== sequence || row.previous !== (previous ? hash(previous) : null)) fail('retirement-journal-invalid');
previous = row;
}
@@ -135,7 +166,7 @@ async function defaultContext(appId, release) {
/** Factory allows workflow tests to replace native IO and forge/network boundaries. */
export function _createInitializedCheckoutRecovery(deps = {}) {
- const native = deps.native || runInitializedCheckoutNative;
+ const native = deps.native || _runInitializedCheckoutNative;
const root = deps.root || transactionRoot;
const context = deps.context || defaultContext;
const fenced = deps.fenced || withClaimOwnershipLock;
@@ -252,8 +283,8 @@ export function _createInitializedCheckoutRecovery(deps = {}) {
quarantineResumption: accepted.quarantineResumption };
};
const branchReview = async ({ directory, state, ctx }) => {
- const transitioned = state.phase === 'quarantined-with-metadata-transitions';
- if ((!transitioned && state.phase !== 'quarantined') || state.branchDeleted) fail('retirement-branch-phase-not-ready');
+ const transitioned = Boolean(state.quarantineResumptionReview);
+ if (!['quarantined', 'quarantined-with-metadata-transitions', 'branch-retirement-prepared'].includes(state.phase) || state.branchDeleted) fail('retirement-branch-phase-not-ready');
const policy = await ensureOwner(ctx, state.release, false);
const proof = await native(transitioned
? { command: 'verify-completed-quarantine', ...quarantineProofInput({ directory, state, ctx }) }
@@ -279,17 +310,24 @@ export function _createInitializedCheckoutRecovery(deps = {}) {
const executor = await native({ command: 'lock', transactionDir: location(input.transactionId) }, { keepOpen: true });
try {
const bound = await loadBoundTransaction(input);
- return await (await admission()).withCheckoutPublication(bound.ctx.repoPath, async () => {
+ const gate = await admission();
+ const work = async () => {
executor.assertHeld();
if (hash(await readJournal(bound.directory)) !== hash(bound.state)) fail('retirement-journal-changed');
return { ok: true, review: await branchReview(bound) };
- });
+ };
+ if (bound.state.phase === 'branch-retirement-prepared') {
+ const reservation = await gate.readCheckoutReservation(bound.ctx.repoPath, bound.state.id);
+ if (JSON.stringify(reservation) !== JSON.stringify(bound.state.reservation)) fail('retirement-reservation-changed');
+ return await gate.withCheckoutReservation(reservation, work);
+ }
+ return await gate.withCheckoutPublication(bound.ctx.repoPath, work);
} finally { await executor.release(); }
}).catch(error => ({ ok: false, reason: error.message }));
// A receiver success was fsynced before recording the schedule receipt. Replay
// only those completed phases: this never issues a Git mutation or recreates
// a ref, and still refuses a new branch/holder or changed archive.
- const finishBranchReceipt = async ({ directory, ctx, state }) => {
+ const finishBranchReceipt = async ({ directory, ctx, state }, beforeMutation) => {
const executor = await native({ command: 'lock', transactionDir: directory }, { keepOpen: true });
const gate = await admission();
try {
@@ -323,12 +361,14 @@ export function _createInitializedCheckoutRecovery(deps = {}) {
if (reservation && JSON.stringify(reservation) !== JSON.stringify(state.reservation)) fail('retirement-reservation-changed');
const finish = async () => {
await verify();
+ await beforeMutation();
if (state.phase === 'branch-retired') {
const record = deps.recordRetirement || (await import('./taskSchedule.js')).recordHeldCheckoutRetirement;
await record(state.appId, { path: state.release.path, branch: state.release.branch,
head: state.release.head, branchDeleted: true, archiveDir: directory });
state = await appendStep(directory, state, 'retired', { receiptRecorded: true });
}
+ await beforeMutation();
if (reservation) await gate.releaseCheckoutReservation(reservation);
return { ok: true, transactionId: state.id, branchDeleted: true, receiptRecorded: true, archiveDir: directory };
};
@@ -336,11 +376,11 @@ export function _createInitializedCheckoutRecovery(deps = {}) {
: await gate.withCheckoutPublication(ctx.repoPath, finish);
} finally { await executor.release(); }
};
- const retireBranch = input => fenced(async () => {
+ const retireBranch = (input, { beforeMutation = async () => {} } = {}) => fenced(async () => {
const bound = await loadBoundTransaction(input);
const { directory, ctx } = bound;
let { state } = bound;
- if (['branch-retired', 'retired'].includes(state.phase) && state.branchDeleted === true) return finishBranchReceipt(bound);
+ if (['branch-retired', 'retired'].includes(state.phase) && state.branchDeleted === true) return finishBranchReceipt(bound, beforeMutation);
// An interrupted receiver may have completed without its journal result.
// It is an explicit uncertainty hold, never permission to recreate a ref.
if (state.phase === 'branch-retirement-intent') fail('retirement-branch-deletion-uncertain');
@@ -349,21 +389,54 @@ export function _createInitializedCheckoutRecovery(deps = {}) {
try {
if (hash(await readJournal(directory)) !== hash(state)) fail('retirement-journal-changed');
executor.assertHeld();
- const current = await branchReview(bound);
- assertReview(current, input.review);
- const reservation = await gate.reserveCheckoutRetirement(ctx.repoPath, state.id);
+ let reservation, current;
+ const prepared = state.phase === 'branch-retirement-prepared';
+ if (prepared) {
+ reservation = await gate.readCheckoutReservation(ctx.repoPath, state.id);
+ if (JSON.stringify(reservation) !== JSON.stringify(state.reservation)) fail('retirement-reservation-changed');
+ } else {
+ current = await branchReview(bound);
+ assertReview(current, input.review);
+ await beforeMutation();
+ reservation = await gate.reserveCheckoutRetirement(ctx.repoPath, state.id);
+ // Durable preparation binds this new cut before lengthy verification or
+ // branch archival. It grants no deletion and can be reviewed after restart.
+ state = await appendStep(directory, state, 'branch-retirement-prepared', { reservation });
+ }
return await gate.withCheckoutReservation(reservation, async () => {
- assertReview(await branchReview(bound), input.review);
+ const fresh = await branchReview({ ...bound, state });
+ if (prepared) assertReview(fresh, input.review);
+ else {
+ const evidence = value => { const { digest: _digest, journalDigest: _journal, ...rest } = value; return rest; };
+ // Only our just-published journal transition changed. Every substantive
+ // source/archive/forge/owner field must equal the accepted review.
+ if (JSON.stringify(evidence(fresh)) !== JSON.stringify(evidence(current))) fail('retirement-evidence-changed');
+ }
+ current = fresh;
executor.assertHeld();
- state = await appendStep(directory, state, 'branch-retirement-intent', { reservation, branchReview: current });
+ await beforeMutation();
const remove = deps.deleteRef || (await import('./worktreeManager.js')).deleteCompletedLocalBranch;
- const deleted = await remove(ctx.repoPath, state.release.branch, state.release.head);
+ let receiverAuthorized = false;
+ const deleted = await remove(ctx.repoPath, state.release.branch, state.release.head, { beforeMutation: async () => {
+ executor.assertHeld();
+ if (JSON.stringify(await ensureOwner(ctx, state.release, false)) !== JSON.stringify(current.policy)
+ || JSON.stringify(await forge(ctx, state.release, state.review.forge.prUrl)) !== JSON.stringify(current.forge)) fail('retirement-evidence-changed');
+ await beforeMutation();
+ executor.assertHeld();
+ state = await appendStep(directory, state, 'branch-retirement-intent', { reservation, branchReview: current });
+ receiverAuthorized = true;
+ await beforeMutation();
+ executor.assertHeld();
+ } });
if (!deleted) fail('retirement-branch-delete-refused-or-uncertain');
+ if (!receiverAuthorized) fail('retirement-branch-receiver-intent-missing');
state = await appendStep(directory, state, 'branch-retired', { branchDeleted: true, branchRetainedReason: null });
+ await beforeMutation();
const record = deps.recordRetirement || (await import('./taskSchedule.js')).recordHeldCheckoutRetirement;
await record(state.appId, { path: state.release.path, branch: state.release.branch,
head: state.release.head, branchDeleted: true, archiveDir: directory });
state = await appendStep(directory, state, 'retired', { receiptRecorded: true });
+ await beforeMutation();
await gate.releaseCheckoutReservation(reservation);
return { ok: true, transactionId: state.id, branchDeleted: true, receiptRecorded: true, archiveDir: directory };
});
@@ -397,6 +470,7 @@ export function _createInitializedCheckoutRecovery(deps = {}) {
...(unboundPreservation ? { recoveryRequired: true, recoveryAction: 'restore', reason: 'preservation-proof-unbound' } : {}),
...(state.phase === 'branch-retirement-intent' ? { branchDeletion: 'uncertain', branchRef, recoveryRequired: true } : {}),
...(state.phase === 'branch-retired' ? { receiptPending: true, recoveryRequired: true } : {}),
+ ...(state.phase === 'branch-retirement-prepared' ? { recoveryRequired: true, recoveryAction: 'review-branch' } : {}),
...(state.quarantineResumptionReview ? { quarantineResumption: state.quarantineResumptionReview.quarantineResumption,
quarantineResumptionComplete: state.phase === 'quarantined-with-metadata-transitions' } : {}),
...(state.metadataRestoreReview ? { metadataRecovery: state.metadataRestoreReview.metadataRecovery,
@@ -566,7 +640,7 @@ export function _createInitializedCheckoutRecovery(deps = {}) {
const quarantineReview = async (bound, reservation) => {
const { state, directory, ctx } = bound;
const allowed = ['quarantine-working-intent', 'quarantined-working', 'quarantine-administration-intent',
- 'quarantined-administration', 'quarantine-resumption-approved', 'quarantine-resumption-administration-intent',
+ 'quarantined-administration', 'quarantined', 'quarantine-resumption-approved', 'quarantine-resumption-administration-intent',
'quarantine-resumption-administration-complete', 'quarantined-with-metadata-transitions'];
if (!allowed.includes(state.phase) || state.branchDeleted || !state.proof?.manifestSha256) fail('quarantine-resumption-phase-not-ready');
if (JSON.stringify(reservation) !== JSON.stringify(state.reservation)) fail('quarantine-resumption-reservation-changed');
@@ -587,9 +661,11 @@ export function _createInitializedCheckoutRecovery(deps = {}) {
};
const reviewQuarantineResumption = input => withMetadataRestore(input,
async (bound, { reservation }) => ({ ok: true, review: await quarantineReview(bound, reservation) }));
- const resumeQuarantine = input => withMetadataRestore(input, async (bound, { executor, gate, reservation }) => {
+ const resumeQuarantine = (input, { beforeMutation: mutationGuard } = {}) => withMetadataRestore(input, async (bound, { executor, gate, reservation }) => {
+ const beforeMutation = mutationGuard || (async () => {});
const current = await quarantineReview(bound, reservation);
assertReview(current, input.review);
+ await beforeMutation();
const { directory } = bound;
let state = await appendStep(directory, bound.state, 'quarantine-resumption-approved',
{ quarantineResumptionReview: current, quarantineResumptionObservation: null });
@@ -598,13 +674,19 @@ export function _createInitializedCheckoutRecovery(deps = {}) {
const fresh = await quarantineSafety({ ...bound, state }, administrationQuarantined);
if (hash(fresh) !== current.safetyEvidenceDigest) fail('retirement-evidence-changed');
executor.assertHeld();
+ await beforeMutation();
};
await assertSafety(current.initialPlacements.administration === 'quarantine');
await native({ command: 'verify-quarantine-resumption', ...proof });
if (current.initialPlacements.administration === 'original') {
await assertSafety(false);
state = await appendStep(directory, state, 'quarantine-resumption-administration-intent');
- const moved = await native({ command: 'resume-quarantine', ...proof });
+ await beforeMutation();
+ const moved = await native({ command: 'resume-quarantine', ...proof }, mutationGuard ? { beforeMutation: async () => {
+ executor.assertHeld();
+ await assertSafety(false);
+ executor.assertHeld();
+ } } : undefined);
if (moved.moved !== true || !moved.postMoveObservation?.file
|| !/^[a-f0-9]{64}$/.test(moved.postMoveObservation.sha256 || '')
|| !/^[a-f0-9]{64}$/.test(moved.postMoveObservation.digest || '')) fail('quarantine-postmove-evidence-missing');
@@ -616,6 +698,7 @@ export function _createInitializedCheckoutRecovery(deps = {}) {
await assertSafety(true);
state = await appendStep(directory, state, 'quarantined-with-metadata-transitions',
{ branchDeleted: false, branchRetainedReason: 'quarantined-checkout-rollback-ref' });
+ await beforeMutation();
await gate.releaseCheckoutReservation(reservation);
return { ok: true, transactionId: state.id, phase: state.phase, originalsQuarantined: true,
branchDeleted: false, branchRetainedReason: state.branchRetainedReason, archiveDir: directory };
@@ -700,7 +783,140 @@ export function _createInitializedCheckoutRecovery(deps = {}) {
archivesRetained: true, metadataTransitionsPreserved: true, restoredFinalization: current.restoredFinalization,
archiveDir: directory };
}, true);
- return { review, retire, status, restore, verifyArchive, reviewBranch, retireBranch, reviewMetadataRestore, restoreMetadata,
+ const recoveryBinding = async (id, expectedDigest, budget) => {
+ const directory = location(id);
+ const state = await readJournal(directory, { bounded: true, budget });
+ const original = state.review;
+ const unsigned = { ...original }; delete unsigned.digest;
+ if (state.id !== id || typeof state.appId !== 'string' || !state.appId
+ || !isAbsolute(state.repoPath || '') || !isAbsolute(state.release?.path || '')
+ || typeof state.release?.branch !== 'string' || !state.release.branch
+ || !/^[a-f0-9]{40}(?:[a-f0-9]{24})?$/.test(state.release?.head || '')
+ || original?.mode !== 'initialized-private-modules' || original.digest !== hash(unsigned)
+ || !sameRelease(original.release, state.release)) fail('retirement-journal-binding-invalid');
+ const ctx = await context(state.appId);
+ if (ctx.repoPath !== state.repoPath) fail('retirement-transaction-app-mismatch');
+ if (expectedDigest && hash(state) !== expectedDigest) fail('retirement-journal-changed');
+ return { directory, state, ctx };
+ };
+ const completedWithoutCut = async ({ state, ctx }) => {
+ const complete = (state.phase === 'retired' && state.branchDeleted === true && state.receiptRecorded === true)
+ || ['restored', 'restored-with-metadata-transitions', 'restored-with-reviewed-finalization'].includes(state.phase);
+ if (!complete) return false;
+ const gate = await admission();
+ const cut = await gate.readCheckoutReservation(ctx.repoPath, state.id).catch(async error => {
+ if (error.code === 'CHECKOUT_RETIREMENT_RESERVED') {
+ // A durably completed historical transaction must not consume replay
+ // slots merely because a different transaction now owns this repo.
+ // Validate and re-read that owner; never enter or release its cut.
+ const { directory } = await gate._checkoutAdmission(ctx.repoPath);
+ const ownerPath = join(directory, 'cut', 'owner.json');
+ const info = await lstat(ownerPath);
+ if (!info.isFile() || info.isSymbolicLink() || info.size > 4096
+ || (hostOsType() !== 'Windows_NT' && (info.mode & 0o777) !== 0o600)) fail('retirement-reservation-owner-invalid');
+ const owner = JSON.parse(await readFile(ownerPath, 'utf8'));
+ if (owner.version !== 1 || !/^[a-f0-9-]{36}$/.test(owner.id || '') || owner.id === state.id
+ || owner.generation !== owner.id || owner.kind !== 'snapshot' || !Number.isInteger(owner.pid)
+ || typeof owner.startedAt !== 'string' || !Number.isFinite(Date.parse(owner.startedAt))) fail('retirement-reservation-owner-invalid');
+ const fresh = await gate.readCheckoutReservation(ctx.repoPath, owner.id);
+ if (JSON.stringify(fresh.owner) !== JSON.stringify(owner)) fail('retirement-reservation-changed');
+ return null;
+ }
+ if (error.code !== 'ENOENT') throw error;
+ // Missing owner.json inside an extant cut is uncertainty, not release.
+ const { directory } = await gate._checkoutAdmission(ctx.repoPath);
+ const exists = await lstat(join(directory, 'cut')).catch(readError => {
+ if (readError.code === 'ENOENT') return null;
+ throw readError;
+ });
+ if (exists) fail('retirement-reservation-owner-missing');
+ return null;
+ });
+ return cut === null;
+ };
+ // Discovery reads only journals; expensive native/forge checks happen on a
+ // bounded cadence in the daemon drain. A journal is a locator, never a grant.
+ const discover = async () => {
+ const rows = [];
+ const exists = await lstat(root()).catch(error => { if (error.code === 'ENOENT') return null; throw error; });
+ if (!exists) return { ok: true, transactions: [] };
+ await privateDirectory(root());
+ const entries = await opendir(root());
+ const budget = { remaining: 256 * 1024 * 1024 };
+ let count = 0;
+ for await (const entry of entries) {
+ if (++count > 1000) return { ok: false, reason: 'retirement-discovery-budget-exceeded', transactions: [] };
+ if (!/^[a-f0-9-]{36}$/.test(entry.name)) {
+ rows.push({ transactionId: null, reason: 'retirement-journal-name-invalid' });
+ continue;
+ }
+ const row = await recoveryBinding(entry.name, undefined, budget).then(async bound => {
+ const { state } = bound;
+ const settled = await completedWithoutCut(bound).catch(() => false);
+ return { transactionId: state.id, appId: state.appId, repoPath: state.repoPath,
+ phase: state.phase, journalDigest: hash(state), release: state.release, settled };
+ }).catch(error => ({ transactionId: entry.name, reason: error.message }));
+ if (row.reason === 'retirement-discovery-budget-exceeded') return { ok: false, reason: row.reason, transactions: [] };
+ rows.push(row);
+ }
+ return { ok: true, transactions: rows.sort((a, b) => String(a.transactionId).localeCompare(String(b.transactionId))) };
+ };
+ const automaticReview = value => {
+ if (!value || value.release?.head !== value.forge?.prHead) fail('automatic-retirement-exact-pr-head-required');
+ if (value.comparison?.disposition !== 'integrated' || value.comparison.difference !== '') fail('automatic-retirement-independent-review-required');
+ };
+ // Only the deterministic daemon calls this continuation. It never starts a
+ // transaction, restores originals, adopts another cut, or accepts supersession.
+ const recoverAutomatically = async (input, { blocked = async () => 'automatic-retirement-gate-required' } = {}) => {
+ const permitted = async () => { const reason = await blocked(); if (reason) fail(reason); };
+ const bound = await recoveryBinding(input.transactionId, input.journalDigest);
+ const { state, ctx } = bound;
+ if (state.appId !== input.appId) fail('retirement-transaction-app-mismatch');
+ await permitted();
+ if (await completedWithoutCut(bound)) return { ok: true, alreadyComplete: true, phase: state.phase, transactionId: state.id };
+ if (!/^[a-f0-9]{64}$/.test(state.proof?.manifestSha256 || '')) fail('preservation-proof-unbound');
+ if (state.phase === 'branch-retirement-intent') fail('retirement-branch-deletion-uncertain');
+ if (['branch-retired', 'retired'].includes(state.phase) && state.branchDeleted === true) {
+ // Confirmed deletion is already authorized and durable. A changed default
+ // branch cannot prevent recording its receipt; this path never deletes.
+ return retireBranch(input, { beforeMutation: permitted });
+ }
+ const finishBranch = async current => {
+ await permitted();
+ const reviewed = await reviewBranch(current);
+ if (!reviewed.ok) return reviewed;
+ automaticReview(reviewed.review);
+ await permitted();
+ return retireBranch({ ...current, review: { digest: reviewed.review.digest } }, { beforeMutation: permitted });
+ };
+ if (state.phase === 'branch-retirement-prepared') return finishBranch(input);
+ const resumable = ['quarantine-working-intent', 'quarantined-working', 'quarantine-administration-intent',
+ 'quarantined-administration', 'quarantined', 'quarantine-resumption-approved',
+ 'quarantine-resumption-administration-intent', 'quarantine-resumption-administration-complete',
+ 'quarantined-with-metadata-transitions'];
+ if (!resumable.includes(state.phase) || state.branchDeleted) fail('automatic-retirement-phase-requires-review');
+ const gate = await admission();
+ const reservation = await gate.readCheckoutReservation(ctx.repoPath, state.id).catch(error => {
+ if (error.code === 'ENOENT' && ['quarantined', 'quarantined-with-metadata-transitions'].includes(state.phase)) return null;
+ throw error;
+ });
+ let current = input;
+ if (reservation) {
+ if (JSON.stringify(reservation) !== JSON.stringify(state.reservation)) fail('retirement-reservation-changed');
+ const reviewed = await reviewQuarantineResumption(current);
+ if (!reviewed.ok) return reviewed;
+ automaticReview(reviewed.review);
+ await permitted();
+ const resumed = await resumeQuarantine({ ...current, review: { digest: reviewed.review.digest } }, { beforeMutation: permitted });
+ if (!resumed.ok) return resumed;
+ const fresh = await recoveryBinding(state.id);
+ current = { ...current, journalDigest: hash(fresh.state) };
+ }
+ return finishBranch(current);
+ };
+ const automatic = async (input, options) => recoverAutomatically(input, options)
+ .catch(error => ({ ok: false, reason: error.message, transactionId: input.transactionId, recoveryRequired: true }));
+ return { discover, recoverAutomatically: automatic, review, retire, status, restore, verifyArchive, reviewBranch, retireBranch, reviewMetadataRestore, restoreMetadata,
reviewRestoredFinalization, finalizeRestored, reviewQuarantineResumption, resumeQuarantine };
}
@@ -721,3 +937,6 @@ export const finalizeInitializedCheckoutRestored = recovery.finalizeRestored;
export const reviewInitializedCheckoutQuarantineResumption = recovery.reviewQuarantineResumption;
export const resumeInitializedCheckoutQuarantine = recovery.resumeQuarantine;
+
+export const discoverInitializedCheckoutRecoveries = recovery.discover;
+export const recoverInitializedCheckoutAutomatically = recovery.recoverAutomatically;
diff --git a/server/services/initializedCheckoutRecovery.test.js b/server/services/initializedCheckoutRecovery.test.js
index 1a8c9b3e1f..4527fafbf3 100644
--- a/server/services/initializedCheckoutRecovery.test.js
+++ b/server/services/initializedCheckoutRecovery.test.js
@@ -1,6 +1,6 @@
import { pinPlatform } from '../lib/testHelper.js';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
-import { mkdtemp, readFile, readdir, rm, writeFile } from 'node:fs/promises';
+import { mkdir, mkdtemp, open, readFile, readdir, rm, writeFile } from 'node:fs/promises';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
import { createHash } from 'node:crypto';
@@ -71,10 +71,11 @@ beforeEach(async () => {
fixture = { cut: null, facts, snapshot, headExists: true, placement: { working: 'original', administration: 'original' }, failPart: null,
metadataToken: 'f'.repeat(64), finalizationToken: '7'.repeat(64), quarantineToken: '6'.repeat(64) };
const admission = {
+ _checkoutAdmission: async () => ({ directory: join(root, 'admission') }),
withCheckoutPublication: async (_repo, work) => { if (fixture.cut) throw new Error('checkout-retirement-reserved'); return work(); },
reserveCheckoutRetirement: vi.fn(async (_repo, id) => {
if (fixture.cut) throw new Error('checkout-retirement-reserved');
- fixture.cut = { owner: { id }, directory: '/fixture/admission' }; return fixture.cut;
+ fixture.cut = { owner: { version: 1, id, generation: id, kind: 'snapshot', pid: process.pid, startedAt: new Date().toISOString() }, directory: '/fixture/admission' }; return fixture.cut;
}),
withCheckoutReservation: async (_cut, work) => work(),
readCheckoutReservation: async () => {
@@ -161,11 +162,11 @@ beforeEach(async () => {
ownership: vi.fn(async () => ({ ok: true, policy: null })),
forge: async () => ({ ...fixture.facts }),
comparison: async (_ctx, _release, _url, _snapshot, policy, forge) => ({
- forge, policy, comparison: { disposition: fixture.superseded ? 'review-supersession' : 'ancestor' },
+ release: _release, forge, policy, comparison: { disposition: fixture.superseded ? 'review-supersession' : 'integrated', difference: fixture.superseded ? 'review the changed work' : '' },
}),
listWorktrees: async () => fixture.newHolder ? [{ path: '/fixture/new', branch: release.branch }] : [],
execGit: vi.fn(async () => ({ exitCode: fixture.headExists ? 0 : 1, stdout: fixture.headExists ? release.head : '' })),
- deleteRef: vi.fn(async () => { fixture.headExists = false; return true; }),
+ deleteRef: vi.fn(async (_repo, _branch, _head, options) => { await options.beforeMutation(); fixture.headExists = false; return true; }),
recordRetirement: vi.fn(async () => {}),
};
fixture.deps = deps; fixture.service = _createInitializedCheckoutRecovery(deps);
@@ -618,7 +619,7 @@ describe('initialized checkout transaction', () => {
it('never promotes absent-ref quarantine deletion intent to success', async () => {
const input = await resumedQuarantine();
const prepared = await fixture.service.reviewBranch(input);
- fixture.deps.deleteRef.mockImplementationOnce(async () => { fixture.headExists = false; throw new Error('receiver-outcome-lost'); });
+ fixture.deps.deleteRef.mockImplementationOnce(async (_repo, _branch, _head, options) => { await options.beforeMutation(); fixture.headExists = false; throw new Error('receiver-outcome-lost'); });
await fixture.service.retireBranch({ ...input, review: { digest: prepared.review.digest } });
const status = await fixture.service.status(input);
expect(status.phase).toBe('branch-retirement-intent');
@@ -869,7 +870,7 @@ describe('initialized checkout transaction', () => {
const input = await quarantinedReview();
const result = await fixture.service.retireBranch(input);
expect(result).toMatchObject({ ok: true, branchDeleted: true, receiptRecorded: true });
- expect(fixture.deps.deleteRef).toHaveBeenCalledWith('/fixture/repo', release.branch, release.head);
+ expect(fixture.deps.deleteRef).toHaveBeenCalledWith('/fixture/repo', release.branch, release.head, { beforeMutation: expect.any(Function) });
expect(fixture.deps.recordRetirement).toHaveBeenCalledWith('app', { path: release.path, branch: release.branch,
head: release.head, branchDeleted: true, archiveDir: join(root, input.transactionId) });
expect(fixture.cut).toBeNull();
@@ -893,7 +894,7 @@ describe('initialized checkout transaction', () => {
});
it('retains the delete intent and reports uncertainty after deletion without a journal outcome', async () => {
const input = await quarantinedReview();
- fixture.deps.deleteRef.mockImplementation(async () => { fixture.headExists = false; throw new Error('receiver-result-lost'); });
+ fixture.deps.deleteRef.mockImplementation(async (_repo, _branch, _head, options) => { await options.beforeMutation(); fixture.headExists = false; throw new Error('receiver-result-lost'); });
expect(await fixture.service.retireBranch(input)).toMatchObject({ ok: false, reason: 'receiver-result-lost', recoveryRequired: true });
const restarted = _createInitializedCheckoutRecovery(fixture.deps);
const status = await restarted.status(input);
@@ -1069,3 +1070,252 @@ describe('initialized checkout transaction', () => {
await expect(fixture.service.status({ appId: 'app', transactionId: result.transactionId })).rejects.toThrow('retirement-journal-invalid');
});
});
+
+// These exercise persisted journal discovery through the actual service
+// continuation, not a synthetic phase-to-command mapping.
+describe('automatic initialized retirement continuation', () => {
+ const automaticInput = async () => {
+ const result = await fixture.service.discover();
+ expect(result.ok).toBe(true);
+ expect(result.transactions).toHaveLength(1);
+ const row = result.transactions[0];
+ expect(row.reason).toBeUndefined();
+ return { appId: row.appId, transactionId: row.transactionId, journalDigest: row.journalDigest };
+ };
+ const permitted = { blocked: async () => null };
+ const interrupted = async () => {
+ fixture.failPart = 'administration';
+ expect(await fixture.service.retire(await reviewed())).toMatchObject({ ok: false, recoveryRequired: true });
+ fixture.failPart = null;
+ fixture.service = _createInitializedCheckoutRecovery(fixture.deps);
+ return automaticInput();
+ };
+ it('discovers after restart and finishes archive-backed quarantine plus a separate fresh branch retirement', async () => {
+ const input = await interrupted();
+ const originalReservation = structuredClone(fixture.cut);
+ await expect(fixture.deps.admission.withCheckoutPublication('/fixture/repo', () => {}))
+ .rejects.toThrow('checkout-retirement-reserved');
+ const before = fixture.deps.native.mock.calls.length;
+ expect(await fixture.service.recoverAutomatically(input, permitted))
+ .toMatchObject({ ok: true, branchDeleted: true, receiptRecorded: true });
+ expect(fixture.deps.admission.releaseCheckoutReservation.mock.calls[0][0]).toEqual(originalReservation);
+ expect(fixture.placement).toEqual({ working: 'quarantined', administration: 'quarantined' });
+ expect(fixture.deps.native.mock.calls.slice(before).some(([arg]) => ['inspect', 'preserve', 'move'].includes(arg.command))).toBe(false);
+ expect(fixture.deps.deleteRef).toHaveBeenCalledTimes(1);
+ expect(fixture.deps.recordRetirement).toHaveBeenCalledTimes(1);
+ expect(fixture.cut).toBeNull();
+ });
+ it('finishes the plain quarantined journal left before releasing its exact reservation', async () => {
+ fixture.deps.admission.releaseCheckoutReservation.mockRejectedValueOnce(new Error('release-interrupted'));
+ const result = await fixture.service.retire(await reviewed());
+ expect(result).toMatchObject({ ok: false, reason: 'release-interrupted' });
+ expect((await fixture.service.status({ appId: 'app', transactionId: result.transactionId })).phase).toBe('quarantined');
+ const moves = fixture.deps.native.mock.calls.filter(([arg]) => arg.command === 'move').length;
+ expect(await fixture.service.recoverAutomatically(await automaticInput(), permitted)).toMatchObject({ ok: true, receiptRecorded: true });
+ expect(fixture.deps.native.mock.calls.filter(([arg]) => arg.command === 'move')).toHaveLength(moves);
+ expect(fixture.deps.native.mock.calls.some(([arg]) => arg.command === 'resume-quarantine')).toBe(false);
+ });
+ it.each(['superseded', 'different-pr-head', 'unknown-metadata', 'active-owner', 'new-holder', 'archive-changed', 'executor-busy', 'reservation-changed'])(
+ 'retains the original reservation and all sources when fresh evidence is %s', async cause => {
+ const input = await interrupted();
+ if (cause === 'superseded') fixture.superseded = true;
+ if (cause === 'different-pr-head') fixture.facts.prHead = '9'.repeat(40);
+ if (cause === 'unknown-metadata') fixture.forbiddenMetadata = true;
+ if (cause === 'active-owner') fixture.deps.ownership.mockResolvedValue({ ok: false, reason: 'active-owner' });
+ if (cause === 'new-holder') fixture.newHolder = true;
+ if (cause === 'archive-changed') fixture.failArchive = true;
+ if (cause === 'executor-busy') fixture.deps.native.mockImplementationOnce(async () => { throw new Error('retirement-executor-busy'); });
+ if (cause === 'reservation-changed') fixture.cut.owner.id = 'other-transaction';
+ expect(await fixture.service.recoverAutomatically(input, permitted)).toMatchObject({ ok: false });
+ expect(fixture.cut).not.toBeNull();
+ expect(fixture.placement.administration).toBe('original');
+ expect(fixture.deps.deleteRef).not.toHaveBeenCalled();
+ expect(fixture.deps.recordRetirement).not.toHaveBeenCalled();
+ });
+ it.each(['pause', 'default', 'owner'])('rechecks %s after preparing a review and before the remaining move', async changed => {
+ const input = await interrupted();
+ let gates = 0;
+ const blocked = async () => {
+ if (++gates !== 2) return null;
+ if (changed === 'pause') return 'daemon-paused';
+ if (changed === 'default') fixture.facts.defaultHead = '8'.repeat(40);
+ if (changed === 'owner') fixture.deps.ownership.mockResolvedValue({ ok: false, reason: 'active-owner' });
+ return null;
+ };
+ expect(await fixture.service.recoverAutomatically(input, { blocked })).toMatchObject({ ok: false });
+ expect(fixture.placement.administration).toBe('original');
+ expect(fixture.cut).not.toBeNull();
+ });
+ it.each(['remaining-rename', 'branch-delete'])('honors a pause arriving inside the second native review before %s', async stage => {
+ const input = stage === 'remaining-rename' ? await interrupted() : await (async () => {
+ await fixture.service.retire(await reviewed()); return automaticInput();
+ })();
+ let paused = false, checks = 0;
+ const native = fixture.deps.native.getMockImplementation();
+ fixture.deps.native.mockImplementation(async request => {
+ const result = await native(request);
+ if (stage === 'remaining-rename' && request.command === 'verify-quarantine-resumption') paused = true;
+ if (stage === 'branch-delete' && request.command === 'verify' && request.mode === 'restore' && ++checks === 3) paused = true;
+ return result;
+ });
+ expect(await fixture.service.recoverAutomatically(input, { blocked: async () => paused ? 'daemon-paused' : null }))
+ .toMatchObject({ ok: false, reason: 'daemon-paused' });
+ expect(fixture.deps.deleteRef).not.toHaveBeenCalled();
+ expect(fixture.cut).not.toBeNull();
+ if (stage === 'remaining-rename') expect(fixture.placement.administration).toBe('original');
+ });
+ it('restarts a durably prepared branch retirement under the exact original cut without creating another reservation', async () => {
+ await fixture.service.retire(await reviewed());
+ fixture.deps.deleteRef.mockRejectedValueOnce(new Error('branch-archive-interrupted'));
+ expect(await fixture.service.recoverAutomatically(await automaticInput(), permitted)).toMatchObject({ ok: false, reason: 'branch-archive-interrupted' });
+ const input = await automaticInput();
+ expect((await fixture.service.status(input)).phase).toBe('branch-retirement-prepared');
+ const reservation = structuredClone(fixture.cut);
+ const reservations = fixture.deps.admission.reserveCheckoutRetirement.mock.calls.length;
+ fixture.service = _createInitializedCheckoutRecovery(fixture.deps);
+ expect(await fixture.service.recoverAutomatically(input, permitted)).toMatchObject({ ok: true, receiptRecorded: true });
+ expect(fixture.deps.admission.reserveCheckoutRetirement).toHaveBeenCalledTimes(reservations);
+ expect(fixture.deps.admission.releaseCheckoutReservation.mock.calls.at(-1)[0]).toEqual(reservation);
+ });
+ it.each(['owner', 'archive', 'forge', 'reservation'])('holds prepared retirement after fresh %s evidence changes', async changed => {
+ await fixture.service.retire(await reviewed());
+ fixture.deps.deleteRef.mockRejectedValueOnce(new Error('branch-archive-interrupted'));
+ await fixture.service.recoverAutomatically(await automaticInput(), permitted);
+ const input = await automaticInput();
+ if (changed === 'owner') fixture.deps.ownership.mockResolvedValue({ ok: false, reason: 'active-owner' });
+ if (changed === 'archive') fixture.failArchive = true;
+ if (changed === 'forge') fixture.facts.prHead = '9'.repeat(40);
+ if (changed === 'reservation') fixture.cut.owner.id = 'foreign';
+ expect(await fixture.service.recoverAutomatically(input, permitted)).toMatchObject({ ok: false });
+ expect(fixture.deps.deleteRef).toHaveBeenCalledTimes(1);
+ expect(fixture.cut).not.toBeNull(); expect(fixture.headExists).toBe(true);
+ });
+ it('rechecks pause after durable intent publication and retains uncertainty without calling the receiver', async () => {
+ await fixture.service.retire(await reviewed());
+ let received = false;
+ fixture.deps.deleteRef.mockImplementation(async (_repo, _branch, _head, options) => {
+ await options.beforeMutation(); received = true; fixture.headExists = false; return true;
+ });
+ const input = await automaticInput();
+ const blocked = async () => {
+ const status = await fixture.service.status(input);
+ return status.phase === 'branch-retirement-intent' ? 'daemon-paused' : null;
+ };
+ expect(await fixture.service.recoverAutomatically(input, { blocked })).toMatchObject({ ok: false, reason: 'daemon-paused' });
+ expect(received).toBe(false); expect(fixture.headExists).toBe(true); expect(fixture.cut).not.toBeNull();
+ expect(await fixture.service.recoverAutomatically(await automaticInput(), permitted))
+ .toMatchObject({ ok: false, reason: 'retirement-branch-deletion-uncertain' });
+ });
+ it.each(['restored', 'restored-with-metadata-transitions', 'restored-with-reviewed-finalization'])(
+ 'treats completed %s with released reservation as settled, never as new retirement intent', async phase => {
+ const input = await interrupted();
+ if (phase === 'restored') {
+ expect(await fixture.service.restore(input)).toMatchObject({ ok: true });
+ } else {
+ const review = await fixture.service.reviewMetadataRestore(input);
+ fixture.delayedRestoreDrift = phase === 'restored-with-reviewed-finalization';
+ expect(await fixture.service.restoreMetadata({ ...input, review: { digest: review.review.digest } }))
+ .toMatchObject({ ok: !fixture.delayedRestoreDrift });
+ if (phase === 'restored-with-reviewed-finalization') {
+ const current = await automaticInput();
+ const final = await fixture.service.reviewRestoredFinalization(current);
+ expect(await fixture.service.finalizeRestored({ ...current, review: { digest: final.review.digest } })).toMatchObject({ ok: true });
+ }
+ }
+ const discovered = await fixture.service.discover();
+ expect(discovered.transactions[0]).toMatchObject({ phase, settled: true });
+ const commands = fixture.deps.native.mock.calls.length;
+ expect(await fixture.service.recoverAutomatically(await automaticInput(), permitted)).toMatchObject({ ok: true, alreadyComplete: true, phase });
+ expect(fixture.deps.native).toHaveBeenCalledTimes(commands);
+ expect(fixture.deps.deleteRef).not.toHaveBeenCalled();
+ });
+ it('skips completed historical journals beside a fresh foreign cut without starving its exact owner', async () => {
+ for (let i = 0; i < 3; i++) {
+ const retired = await fixture.service.retire(await reviewed());
+ expect(await fixture.service.restore({ appId: 'app', transactionId: retired.transactionId })).toMatchObject({ ok: true });
+ }
+ fixture.failPart = 'administration';
+ const pending = await fixture.service.retire(await reviewed());
+ fixture.failPart = null;
+ const originalCut = fixture.cut;
+ const owner = originalCut.owner;
+ await mkdir(join(root, 'admission/cut'), { recursive: true, mode: 0o700 });
+ await writeFile(join(root, 'admission/cut/owner.json'), JSON.stringify(owner), { mode: 0o600 });
+ fixture.deps.admission.readCheckoutReservation = async (_repo, id) => {
+ if (id !== pending.transactionId) throw Object.assign(new Error('checkout-retirement-reserved'), { code: 'CHECKOUT_RETIREMENT_RESERVED' });
+ return { ...originalCut, owner };
+ };
+ // Discovery proves terminal history is unrelated to the fresh owner. Its
+ // sweep slot must go to that owner, not any of the three completed rows.
+ const discovered = await fixture.service.discover();
+ expect(discovered.transactions.filter(row => row.settled)).toHaveLength(3);
+ const { _createInitializedRecoverySweep } = await import('./initializedCheckoutRecoverySweep.js');
+ const recover = vi.fn(async () => ({ ok: false, reason: 'fixture-current-owner-hold' }));
+ await _createInitializedRecoverySweep({ discover: () => fixture.service.discover(), recover })(async () => null);
+ expect(recover.mock.calls.map(([input]) => input.transactionId)).toEqual([pending.transactionId]);
+ expect(fixture.cut).toBe(originalCut);
+ expect(fixture.deps.deleteRef).not.toHaveBeenCalled();
+ // Unknown or same-owner metadata is never treated as proof of release.
+ await writeFile(join(root, 'admission/cut/owner.json'), '{}', { mode: 0o600 });
+ expect((await fixture.service.discover()).transactions.some(row => row.settled)).toBe(false);
+ });
+ it('replays a durably confirmed receipt without another deletion, but retains uncertain deletion', async () => {
+ await fixture.service.retire(await reviewed());
+ fixture.deps.recordRetirement.mockRejectedValueOnce(new Error('receipt-disk-full'));
+ expect(await fixture.service.recoverAutomatically(await automaticInput(), permitted)).toMatchObject({ ok: false });
+ fixture.superseded = true; // Receipt replay does not delete superseded work.
+ fixture.service = _createInitializedCheckoutRecovery(fixture.deps);
+ expect(await fixture.service.recoverAutomatically(await automaticInput(), permitted)).toMatchObject({ ok: true, receiptRecorded: true });
+ expect(fixture.deps.deleteRef).toHaveBeenCalledTimes(1);
+ expect(fixture.cut).toBeNull();
+ const calls = fixture.deps.native.mock.calls.length;
+ expect(await fixture.service.recoverAutomatically(await automaticInput(), permitted)).toMatchObject({ ok: true, alreadyComplete: true });
+ expect(fixture.deps.native).toHaveBeenCalledTimes(calls);
+ });
+ it('never interprets an absent ref as a completed deletion without the committed result', async () => {
+ await fixture.service.retire(await reviewed());
+ fixture.deps.deleteRef.mockImplementation(async (_repo, _branch, _head, options) => { await options.beforeMutation(); fixture.headExists = false; throw new Error('receiver-result-lost'); });
+ expect(await fixture.service.recoverAutomatically(await automaticInput(), permitted)).toMatchObject({ ok: false });
+ expect(await fixture.service.recoverAutomatically(await automaticInput(), permitted))
+ .toMatchObject({ ok: false, reason: 'retirement-branch-deletion-uncertain' });
+ expect(fixture.deps.deleteRef).toHaveBeenCalledTimes(1);
+ expect(fixture.deps.recordRetirement).not.toHaveBeenCalled();
+ });
+ it('reports unsupported pre-move phases and corrupt journals without creating a fresh transaction', async () => {
+ fixture.failPreserve = true;
+ await fixture.service.retire(await reviewed());
+ const input = await automaticInput();
+ expect(await fixture.service.recoverAutomatically(input, permitted)).toMatchObject({ ok: false, reason: 'preservation-proof-unbound' });
+ const first = join(root, input.transactionId, '000000.json');
+ await writeFile(first, '{', { mode: 0o600 });
+ const found = await fixture.service.discover();
+ expect(found.transactions[0]).toMatchObject({ transactionId: input.transactionId, reason: expect.any(String) });
+ expect(fixture.deps.native.mock.calls.some(([arg]) => arg.command === 'move')).toBe(false);
+ expect(fixture.cut).not.toBeNull();
+ });
+ it('does not call a terminal receipt settled when its owner file disappeared but the cut still exists', async () => {
+ await fixture.service.retire(await reviewed());
+ expect(await fixture.service.recoverAutomatically(await automaticInput(), permitted)).toMatchObject({ ok: true });
+ await mkdir(join(root, 'admission/cut'), { recursive: true, mode: 0o700 });
+ const found = await fixture.service.discover();
+ expect(found.transactions[0].settled).toBe(false);
+ expect(await fixture.service.recoverAutomatically(await automaticInput(), permitted))
+ .toMatchObject({ ok: false, reason: 'retirement-reservation-owner-missing' });
+ expect(fixture.deps.deleteRef).toHaveBeenCalledTimes(1);
+ });
+ it('holds an oversized journal before reading or dispatching its contents', async () => {
+ const input = await interrupted();
+ const file = await open(join(root, input.transactionId, '000000.json'), 'r+');
+ try { await file.truncate(16 * 1024 * 1024 + 1); } finally { await file.close(); }
+ const found = await fixture.service.discover();
+ expect(found.transactions[0]).toMatchObject({ transactionId: input.transactionId, reason: 'retirement-journal-budget-exceeded' });
+ expect(fixture.deps.deleteRef).not.toHaveBeenCalled();
+ });
+ it('refuses stale discovery digests and requires an explicit running gate', async () => {
+ const input = await interrupted();
+ expect(await fixture.service.recoverAutomatically(input)).toMatchObject({ ok: false, reason: 'automatic-retirement-gate-required' });
+ expect(await fixture.service.recoverAutomatically({ ...input, journalDigest: '0'.repeat(64) }, permitted))
+ .toMatchObject({ ok: false, reason: 'retirement-journal-changed' });
+ expect(fixture.deps.deleteRef).not.toHaveBeenCalled();
+ });
+});
diff --git a/server/services/initializedCheckoutRecoverySweep.js b/server/services/initializedCheckoutRecoverySweep.js
new file mode 100644
index 0000000000..e66b9438eb
--- /dev/null
+++ b/server/services/initializedCheckoutRecoverySweep.js
@@ -0,0 +1,77 @@
+/** Deterministic continuation of existing private retirement transactions.
+ * The journal stays authoritative; this bounded in-memory retry cache grants
+ * nothing and is rebuilt after restart. No task, agent, or checkout is created.
+ */
+export const INITIALIZED_RECOVERY_INTERVAL_MS = 30 * 60_000;
+const MAX_ATTEMPTS = 2;
+
+export function _createInitializedRecoverySweep({ discover, recover, now = Date.now, report = () => {} }) {
+ let running = null;
+ let nextScan = 0;
+ let cursor = '';
+ const attempted = new Map();
+ const sweep = async blocked => {
+ const reason = await blocked();
+ if (reason) return { skipped: reason, outcomes: [] };
+ if (now() < nextScan) return { skipped: 'recovery-cadence', outcomes: [] };
+ nextScan = now() + INITIALIZED_RECOVERY_INTERVAL_MS;
+ const found = await discover();
+ if (!found.ok) {
+ await report({ transactionId: null, reason: found.reason });
+ return { skipped: found.reason, outcomes: [] };
+ }
+ const outcomes = [];
+ const valid = [];
+ for (const row of found.transactions) {
+ if (row.reason) { await report(row); outcomes.push(row); }
+ else if (row.settled) {
+ const outcome = { transactionId: row.transactionId, appId: row.appId, branch: row.release?.branch,
+ phase: row.phase, journalDigest: row.journalDigest, ok: true, reason: 'retirement-recovery-completed' };
+ await report(outcome); outcomes.push(outcome);
+ } else valid.push(row);
+ }
+ const ids = new Set(valid.map(row => row.transactionId));
+ for (const key of attempted.keys()) if (!ids.has(key)) attempted.delete(key);
+ // Rotate bounded batches so an old held transaction cannot starve later
+ // journals. Every retry obtains fresh evidence; cached holds expire.
+ const sorted = [...valid.filter(row => row.transactionId > cursor), ...valid.filter(row => row.transactionId <= cursor)];
+ let count = 0;
+ for (const row of sorted) {
+ const prior = attempted.get(row.transactionId);
+ if (prior?.digest === row.journalDigest && now() < prior.retryAt) continue;
+ const stopped = await blocked();
+ if (stopped) return { skipped: stopped, outcomes };
+ if (++count > MAX_ATTEMPTS) break;
+ cursor = row.transactionId;
+ const result = await recover({ appId: row.appId, transactionId: row.transactionId,
+ journalDigest: row.journalDigest }, { blocked });
+ const outcome = { transactionId: row.transactionId, appId: row.appId,
+ phase: result.receiptRecorded ? 'retired' : result.phase || row.phase,
+ branch: row.release?.branch, journalDigest: row.journalDigest, ok: result.ok, reason: result.reason || (result.ok ? 'retirement-recovery-completed' : 'retirement-recovery-held') };
+ attempted.set(row.transactionId, { digest: row.journalDigest, retryAt: now() + INITIALIZED_RECOVERY_INTERVAL_MS });
+ await report(outcome); outcomes.push(outcome);
+ }
+ return { outcomes };
+ };
+ return blocked => {
+ if (running) return running;
+ running = sweep(blocked).finally(() => { running = null; });
+ return running;
+ };
+}
+
+const drain = _createInitializedRecoverySweep({
+ discover: async () => (await import('./initializedCheckoutRecovery.js')).discoverInitializedCheckoutRecoveries(),
+ recover: async (...args) => (await import('./initializedCheckoutRecovery.js')).recoverInitializedCheckoutAutomatically(...args),
+ report: async row => {
+ const { recordInitializedRetirementRecovery } = await import('./taskSchedule.js');
+ const { appId, ...observation } = row;
+ await recordInitializedRetirementRecovery(appId || null, observation);
+ // No paths, native diagnostics, patches, or raw metadata in daemon logs.
+ const reason = /^[a-z][a-z0-9-]+$/.test(row.reason) ? row.reason : 'retirement-recovery-held';
+ const message = `Initialized retirement ${row.transactionId || 'discovery'}: ${reason}`;
+ console.info(`🔁 ${message}`);
+ },
+});
+
+export const sweepInitializedCheckoutRecoveries = drain;
diff --git a/server/services/initializedCheckoutRecoverySweep.test.js b/server/services/initializedCheckoutRecoverySweep.test.js
new file mode 100644
index 0000000000..f5f436fad7
--- /dev/null
+++ b/server/services/initializedCheckoutRecoverySweep.test.js
@@ -0,0 +1,64 @@
+import { describe, expect, it, vi } from 'vitest';
+import { _createInitializedRecoverySweep, INITIALIZED_RECOVERY_INTERVAL_MS } from './initializedCheckoutRecoverySweep.js';
+
+const transaction = id => ({ transactionId: id, appId: 'app', journalDigest: `${id}-digest`,
+ phase: 'quarantined-working', release: { branch: `claim/${id}` } });
+
+describe('initialized retirement recovery drain', () => {
+ it('serializes overlapping triggers, bounds batches and rotates past held work with fresh retry evidence', async () => {
+ let time = 0, release;
+ const rows = ['a', 'b', 'c'].map(transaction);
+ const discover = vi.fn(async () => ({ ok: true, transactions: rows }));
+ const recover = vi.fn(async () => ({ ok: false, reason: 'automatic-retirement-independent-review-required' }));
+ recover.mockImplementationOnce(() => new Promise(resolve => { release = resolve; }));
+ const report = vi.fn();
+ const drain = _createInitializedRecoverySweep({ discover, recover, report, now: () => time });
+ const first = drain(async () => null);
+ expect(drain(async () => null)).toBe(first);
+ await vi.waitFor(() => expect(release).toBeTypeOf('function'));
+ release({ ok: false, reason: 'active-owner' });
+ await first;
+ expect(recover.mock.calls.map(([input]) => input.transactionId)).toEqual(['a', 'b']);
+ expect(report.mock.calls[0][0]).toMatchObject({ transactionId: 'a', branch: 'claim/a', reason: 'active-owner' });
+ expect(await drain(async () => null)).toMatchObject({ skipped: 'recovery-cadence' });
+ time += INITIALIZED_RECOVERY_INTERVAL_MS;
+ rows[0].journalDigest = 'new-journal';
+ await drain(async () => null);
+ expect(recover.mock.calls.slice(2).map(([input]) => input.transactionId)).toEqual(['c', 'a']);
+ expect(recover.mock.calls.at(-1)[0].journalDigest).toBe('new-journal');
+ expect(discover).toHaveBeenCalledTimes(2);
+ });
+ it('clears completed diagnostics without letting terminal journals consume replay slots', async () => {
+ const report = vi.fn();
+ const recover = vi.fn(async () => ({ ok: false, reason: 'active-owner' }));
+ const rows = ['a', 'b', 'c'].map(id => ({ ...transaction(id), settled: true, phase: 'restored-with-reviewed-finalization' }));
+ const drain = _createInitializedRecoverySweep({ discover: async () => ({ ok: true, transactions: [...rows, transaction('d')] }), recover, report });
+ await drain(async () => null);
+ expect(recover.mock.calls.map(([input]) => input.transactionId)).toEqual(['d']);
+ expect(report.mock.calls.slice(0, 3).every(([row]) => row.ok && row.reason === 'retirement-recovery-completed')).toBe(true);
+ });
+ it('does no discovery while stopped, paused, off or dry-run and checks the gate again before replay', async () => {
+ const discover = vi.fn(async () => ({ ok: true, transactions: [transaction('a')] }));
+ const recover = vi.fn();
+ const drain = _createInitializedRecoverySweep({ discover, recover });
+ for (const reason of ['daemon-stopped', 'daemon-paused', 'autonomy-off', 'autonomy-dry-run']) {
+ expect(await drain(async () => reason)).toMatchObject({ skipped: reason });
+ }
+ expect(discover).not.toHaveBeenCalled();
+ const gate = vi.fn().mockResolvedValueOnce(null).mockResolvedValue('daemon-paused');
+ expect(await drain(gate)).toMatchObject({ skipped: 'daemon-paused' });
+ expect(recover).not.toHaveBeenCalled();
+ });
+ it('surfaces invalid journals and refuses an incomplete discovery without invoking recovery', async () => {
+ const report = vi.fn();
+ const recover = vi.fn();
+ const discover = vi.fn(async () => ({ ok: true, transactions: [{ transactionId: 'invalid', reason: 'retirement-journal-invalid' }] }));
+ const drain = _createInitializedRecoverySweep({ discover, recover, report, now: () => 0 });
+ await drain(async () => null);
+ expect(report).toHaveBeenCalledWith({ transactionId: 'invalid', reason: 'retirement-journal-invalid' });
+ const capped = _createInitializedRecoverySweep({ discover: async () => ({ ok: false,
+ reason: 'retirement-discovery-budget-exceeded', transactions: [transaction('a')] }), recover, report });
+ expect(await capped(async () => null)).toMatchObject({ skipped: 'retirement-discovery-budget-exceeded' });
+ expect(recover).not.toHaveBeenCalled();
+ });
+});
diff --git a/server/services/taskSchedule.js b/server/services/taskSchedule.js
index ac42b424a6..5ccf9469b0 100644
--- a/server/services/taskSchedule.js
+++ b/server/services/taskSchedule.js
@@ -762,6 +762,24 @@ export async function recordBranchReconcileOutcome(appId, { status, reason, rema
});
}
+/** Observations are diagnostic only; journals and fresh service proofs own recovery. */
+export async function recordInitializedRetirementRecovery(appId, observation) {
+ return updateSchedule(async schedule => {
+ const record = ensureExecutionRecord(schedule, 'branch-reconcile', appId);
+ const previous = record.branchReconcileOutcome || {};
+ const rows = previous.initializedRetirements || [];
+ const index = rows.findIndex(row => row.transactionId === observation.transactionId);
+ if (index < 0 && rows.length >= 1000) throw new Error('retirement-observation-budget-exceeded');
+ const prior = index < 0 ? null : rows[index];
+ if (prior && Object.entries(observation).every(([key, value]) => prior[key] === value)) return { result: prior, changed: false };
+ const next = [...rows];
+ const row = { ...observation, observedAt: new Date().toISOString() };
+ if (index < 0) next.push(row); else next[index] = row;
+ record.branchReconcileOutcome = { ...previous, initializedRetirements: next };
+ return { result: row, changed: true };
+ });
+}
+
export async function getBranchReconcileOutcome(appId) {
const schedule = await loadSchedule();
return resolveExecutionRecord(schedule, 'branch-reconcile', appId)?.branchReconcileOutcome ?? null;
diff --git a/server/services/taskSchedule.test.js b/server/services/taskSchedule.test.js
index 121d86bd34..0351f609c7 100644
--- a/server/services/taskSchedule.test.js
+++ b/server/services/taskSchedule.test.js
@@ -143,6 +143,7 @@ import {
getPerpetualDrainState,
recordPerpetualDispatch,
recordBranchReconcileOutcome,
+ recordInitializedRetirementRecovery,
recordHeldCheckoutRetirement,
applyOnDemandRunResets,
queuePerpetualRefill,
@@ -3187,6 +3188,11 @@ describe('taskSchedule', () => {
writeFile.mockImplementation(async (_path, contents) => { saved = JSON.parse(contents); });
const remaining = [{ branch: 'fix/example', path: '/example/checkout', reason: 'worktree-locked' }];
await recordBranchReconcileOutcome('app-1', { status: 'running', reason: 'batch-pending', remaining, nextAction: 'Continue.' });
+ await recordInitializedRetirementRecovery('app-1', { transactionId: 'transaction-a', branch: 'claim/held',
+ phase: 'quarantined-working', journalDigest: 'digest-a', ok: false, reason: 'automatic-retirement-independent-review-required' });
+ expect((await getScheduleStatus()).tasks['branch-reconcile'].reconcileOutcomes['app-1'].initializedRetirements)
+ .toEqual([expect.objectContaining({ transactionId: 'transaction-a', branch: 'claim/held', reason: 'automatic-retirement-independent-review-required' })]);
+
const observedAt = saved.executions['task:branch-reconcile'].perApp['app-1'].branchReconcileOutcome.observedAt;
await recordBranchReconcileOutcome('app-1', { status: 'incomplete', reason: 'agent-failed', nextAction: 'Resolve the provider failure and run again.' });
expect(saved.executions['task:branch-reconcile'].perApp['app-1'].branchReconcileOutcome)
diff --git a/server/services/worktreeManager.js b/server/services/worktreeManager.js
index 100123a359..81bea75fbf 100644
--- a/server/services/worktreeManager.js
+++ b/server/services/worktreeManager.js
@@ -248,7 +248,8 @@ async function queueWorktreeCreate(repo, fn) {
* policy still apply. Share the creation lane so managed attachments cannot
* enter between the receiver's holder check and ref transaction.
*/
-export function deleteCompletedLocalBranch(repoPath, branch, expectedHead) {
+export function deleteCompletedLocalBranch(repoPath, branch, expectedHead, { beforeMutation = async () => {} } = {}) {
+ let callbackError;
return queueWorktreeCreate(repoPath, async () => {
if (PROTECTED_BRANCHES.includes(branch)) return false;
const { getDefaultBranch } = await import('./git.js');
@@ -261,11 +262,12 @@ export function deleteCompletedLocalBranch(repoPath, branch, expectedHead) {
if (!isAbsolute(common)) return false;
const preservation = await preserveBranchRetirement(repoPath, common, branch, expectedHead);
if (!await preservation.verify()) return false;
+ await Promise.resolve().then(beforeMutation).catch(error => { callbackError = error; throw error; });
const result = await execGit(['push',
'--receive-pack=git -c receive.denyCurrentBranch=refuse -c receive.denyDeleteCurrent=refuse -c receive.autogc=false receive-pack',
`--force-with-lease=${ref}:${expectedHead}`, '--', common, `:${ref}`], repoPath, { ignoreExitCode: true });
return result.exitCode === 0;
- }).catch(() => false);
+ }).catch(error => { if (error === callbackError) throw error; return false; });
}
/**
@@ -417,7 +419,7 @@ export async function forceRemoveWorktreeDir(repo, worktreePath, opts = {}) {
.catch(error => ({ removed: false, reason: error.message, warning: 'Checkout retained: repository reservation unavailable' }));
}
-async function forceRemoveWorktreeDirAdmitted(repo, worktreePath, { label, log = 'remove', subject = worktreePath, verifyRemoval } = {}) {
+async function forceRemoveWorktreeDirAdmitted(repo, worktreePath, { label, log = 'remove', subject = worktreePath, verifyRemoval, beforeMutation = async () => {} } = {}) {
const logAll = label && log === 'all';
let lastFailure = null;
let heldReason = null;
@@ -439,6 +441,7 @@ async function forceRemoveWorktreeDirAdmitted(repo, worktreePath, { label, log =
return null;
};
heldReason = await check();
+ if (!heldReason) await beforeMutation();
if (!heldReason) await execGit(['worktree', 'remove', worktreePath, '--force'], repo, { timeout: WORKTREE_ADD_TIMEOUT_MS }).catch(async (err) => {
lastFailure = errorExcerpt(err);
if (label) console.log(`⚠️ ${label}: ${err.message}`);
@@ -448,6 +451,7 @@ async function forceRemoveWorktreeDirAdmitted(repo, worktreePath, { label, log =
// bypass a lock acquired after our last read, even with a caller guard.
const directory = await lstat(worktreePath).then(() => 'present', err => err.code === 'ENOENT' ? 'absent' : 'unknown');
if (directory !== 'absent') { heldReason = 'worktree-removal-refused'; return; }
+ await beforeMutation();
await execGit(['worktree', 'prune'], repo).catch((pruneErr) => {
lastFailure = errorExcerpt(pruneErr);
if (logAll) console.log(`⚠️ Worktree prune failed for ${subject}: ${pruneErr.message}`);
diff --git a/server/services/worktreeManager.test.js b/server/services/worktreeManager.test.js
index d2f2f9f40a..deeaa1a9cf 100644
--- a/server/services/worktreeManager.test.js
+++ b/server/services/worktreeManager.test.js
@@ -50,6 +50,7 @@ vi.mock('./git.js', () => ({
const {
shouldRefuseDefaultBranchMerge,
deleteCompletedLocalBranch,
+ forceRemoveWorktreeDir,
isHumanClaimWorktree,
classifyWorktreeDirt,
isGitLockError,
@@ -86,6 +87,48 @@ describe('completed branch archival boundary', () => {
afterEach(() => {
preserveBranchMock.mockReset().mockResolvedValue({ archiveDir: '/branch-archive', verify: async () => true });
});
+ it('runs the trusted receiver guard after archival verification and preserves its refusal reason without pushing', async () => {
+ let ready = false;
+ preserveBranchMock.mockResolvedValue({ verify: async () => { ready = true; return true; } });
+ const beforeMutation = vi.fn(async () => {
+ expect(ready).toBe(true);
+ throw new Error('daemon-paused');
+ });
+ await expect(deleteCompletedLocalBranch('/repo', 'done', head, { beforeMutation })).rejects.toThrow('daemon-paused');
+ expect(beforeMutation).toHaveBeenCalledOnce();
+ expect(execGitMock.mock.calls.some(([args]) => args[0] === 'push')).toBe(false);
+ });
+ it('checks automatic permission after final registration refresh before checkout removal', async () => {
+ let reads = 0, verified = false;
+ execGitMock.mockImplementation(async args => {
+ if (args[0] === 'worktree' && args[1] === 'list') {
+ reads++;
+ return { stdout: `worktree /checkout\nHEAD ${head}\nbranch refs/heads/done\n\n`, exitCode: 0 };
+ }
+ return { stdout: '', exitCode: 0 };
+ });
+ const beforeMutation = vi.fn(async () => {
+ expect(verified).toBe(true); expect(reads).toBe(2);
+ throw new Error('daemon-paused');
+ });
+ expect(await forceRemoveWorktreeDir('/repo', '/checkout', {
+ verifyRemoval: async () => { verified = true; return null; }, beforeMutation,
+ })).toMatchObject({ removed: false, reason: 'daemon-paused' });
+ expect(beforeMutation).toHaveBeenCalledOnce();
+ expect(execGitMock.mock.calls.some(([args]) => args[0] === 'worktree' && ['remove', 'prune'].includes(args[1]))).toBe(false);
+ });
+ it('rechecks automatic permission before pruning an already absent checkout after a failed removal', async () => {
+ execGitMock.mockImplementation(async args => {
+ if (args[0] === 'worktree' && args[1] === 'remove') throw new Error('remove interrupted');
+ return { stdout: '', exitCode: 0 };
+ });
+ lstat.mockRejectedValueOnce(Object.assign(new Error('missing'), { code: 'ENOENT' }));
+ const beforeMutation = vi.fn().mockResolvedValueOnce(undefined).mockRejectedValueOnce(new Error('daemon-stopped'));
+ expect(await forceRemoveWorktreeDir('/repo', '/checkout', { verifyRemoval: async () => null, beforeMutation }))
+ .toMatchObject({ removed: false, reason: 'daemon-stopped' });
+ expect(beforeMutation).toHaveBeenCalledTimes(2);
+ expect(execGitMock.mock.calls.some(([args]) => args[0] === 'worktree' && args[1] === 'prune')).toBe(false);
+ });
it.each(['archive failure', 'archive verification failure'])('never sends the deletion when there is an %s', async failure => {
if (failure === 'archive failure') preserveBranchMock.mockRejectedValue(new Error('archive disk full'));
else preserveBranchMock.mockResolvedValue({ archiveDir: '/branch-archive', verify: async () => false });