Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: recover initialized checkouts after reviewed metadata transitions

[._.]/ Adam Eivy273cb969d337
diff --git a/docs/features/branch-reconciliation.md b/docs/features/branch-reconciliation.md
index 1f7233f124..84bbb174a1 100644
--- a/docs/features/branch-reconciliation.md
+++ b/docs/features/branch-reconciliation.md
@@ -107,7 +107,7 @@ when the OS replaces it twice. These retained original bytes are recovery
evidence; the backend never installs them onto the copy. Missing attributes,
changes to any other attribute, and unrecorded additions still refuse preservation.
No provenance or quarantine attribute is removed or rewritten by this backend.
-Original and quarantined metadata stay exact. Every later
+Normal retirement and rollback require exact original and quarantined metadata. Every later
archive verification or move is bound to the exact manifest hash journaled after
preservation. If that journal write is interrupted before any move, status
exposes an unverified-archive hold and a digest for source-verified cancellation;
@@ -117,10 +117,61 @@ Git operations, insufficient space and unsupported metadata hold the operation.
A full parent store copy and a second standalone restore require bounded free
space; there is no cache deletion fallback.
+### Reviewed recovery after a move changes metadata
+
+On Darwin, an exclusive directory rename can remove residual
+`com.apple.decmpfs` attributes even when `UF_COMPRESSED` was unset. The normal
+transaction refuses further moves when this changes its captured original
+manifest. A source content change is never treated as that metadata transition.
+
+While the interrupted transaction still owns its exact repository reservation:
+
+1. `POST /review-metadata-restore` accepts `appId`, `transactionId` and the current
+ `journalDigest`. It holds the native executor lock and existing reservation,
+ checks current ownership and the unchanged branch, verifies the standalone
+ archive, and creates immutable private evidence for the full observed delta.
+2. Independently inspect that evidence before accepting the returned review
+ digest. Only removed residual compression attributes on regular files whose
+ compressed flag was unset, and a provenance addition on an original root
+ directory, are eligible. Original raw compression bytes must still match the
+ captured hashes in the verified archive. All content, inode identities,
+ timestamps, permissions, ownership, ACLs, flags, other attributes and paths
+ remain exact. Nothing rewrites the original manifest or installs attributes.
+3. `POST /restore-metadata` takes the same identifiers and `review: {digest}`.
+ It repeats the review under both locks, journals the accepted evidence, and
+ restores original inodes with exclusive renames. Post-move observations are
+ retained. A changed review, unknown metadata transition, new holder, changed
+ branch or uncertain reservation refuses further work.
+4. Success is `restored-with-metadata-transitions`, with archives and the
+ metadata evidence retained. The exact reservation is released only after
+ final proof. This restores a checkout; it never deletes or recreates a ref,
+ proves a PR merged, or authorizes another retirement. Obtain a fresh
+ retirement review after restoration.
+
+If the installed server predates this recovery capability, run the supported
+`scripts/recover-initialized-checkout.js` from an independently reviewed checkout
+of this version. It uses the same service, native lock and existing reservation;
+it does not patch the running server or clear a reservation manually:
+
+```sh
+node scripts/recover-initialized-checkout.js review /absolute/request.json \
+ --install-root /absolute/PortOS --output /absolute/new-review.jsonl
+node scripts/recover-initialized-checkout.js restore /absolute/approved-request.json \
+ --install-root /absolute/PortOS --output /absolute/new-restore.jsonl
+```
+
+The request file contains only the API fields above. Each output must be new;
+it records a durable pending intent before execution and then a response. A
+pending-only record means inspect the transaction before any retry. The CLI
+requires an existing self-install registry and trusted owner state, and refuses
+an absent or uncertain reservation. It never initializes a different data tree.
+After an interrupted recovery, read status again and obtain a new review for
+the new journal digest rather than replaying an old approval.
+
Managed checkout creation/adoption/removal, claim binding, the full workspace
preparation through agent registration/spawn, and supported Git API writers
-share admission under the canonical common Git directory. Read-only merge proof
-may add immutable commit objects; concurrent additions conservatively invalidate
+share admission under the canonical common Git directory. Synthetic read-only
+merge proofs use a private temporary object store. Concurrent Git writes conservatively invalidate
an exact archive snapshot rather than authorize removing changed evidence. A pending
reservation survives restart. Normal writers release their permits only after
settlement; timeout, overflow or signal termination retain an uncertain permit
diff --git a/scripts/lib/initialized_checkout_archive.py b/scripts/lib/initialized_checkout_archive.py
index 3b20893462..9c399e4cd7 100644
--- a/scripts/lib/initialized_checkout_archive.py
+++ b/scripts/lib/initialized_checkout_archive.py
@@ -18,6 +18,7 @@ import shutil
import stat
import subprocess
import sys
+import uuid
MAX_FILES = 200000
MAX_BYTES = 20 * 1024 ** 3
@@ -762,26 +763,195 @@ def verify_saved_archive(root, saved):
return proof_repositories(root, facts)
+def original_sources(root, facts):
+ located = placements(root, facts)
+ sources = {}
+ for label in ('working', 'administration'):
+ state = located[label]['state']
+ if state == 'ambiguous':
+ fail('ambiguous-source-placement:' + label)
+ original = facts['release']['path'] if label == 'working' else facts['privateGitDir']
+ path = Path(original) if state == 'original' else root / 'quarantine' / label
+ sources[label] = manifest(path)
+ return located, sources
+
+
+def metadata_recovery_changes(root, saved, located, sources):
+ """Prove only the observed, reversible-rename metadata differences.
+
+ Removed residual compression bytes remain in the verified archive. No
+ attribute is installed on an original, and all non-attribute fields,
+ including filesystem identity and UF_COMPRESSED, remain exact.
+ """
+ changes, total = [], 0
+ for label in ('working', 'administration'):
+ expected, actual = saved['facts']['sources'][label], sources[label]
+ if len(expected) != len(actual):
+ fail('metadata-recovery-source-layout-changed:' + label)
+ original = saved['facts']['release']['path'] if label == 'working' else saved['facts']['privateGitDir']
+ current = Path(original) if located[label]['state'] == 'original' else root / 'quarantine' / label
+ for before, after in zip(expected, actual):
+ if {k: v for k, v in before.items() if k != 'xattrs'} != {
+ k: v for k, v in after.items() if k != 'xattrs'}:
+ fail('metadata-recovery-source-content-or-identity-changed:' + label)
+ old, new = before['xattrs'], after['xattrs']
+ for name in sorted(set(old) | set(new)):
+ if old.get(name) == new.get(name):
+ continue
+ change = {'part': label, 'path': before['path'], 'attribute': name,
+ 'beforeSha256': old.get(name), 'afterSha256': new.get(name)}
+ if (name == 'com.apple.decmpfs' and name in old and name not in new
+ and before['kind'] == 'file' and not before['flags'] & 0x20):
+ raw = raw_xattrs(root / 'archive' / label / before['path']).get(name)
+ if raw is None or hashlib.sha256(raw).hexdigest() != old[name]:
+ fail('metadata-recovery-preimage-missing-or-changed')
+ change['beforeBase64'] = base64.b64encode(raw).decode('ascii')
+ elif (name == 'com.apple.provenance' and name not in old and name in new
+ and before['path'] == '' and before['kind'] == 'directory'):
+ raw = raw_xattrs(current).get(name)
+ if raw is None or hashlib.sha256(raw).hexdigest() != new[name]:
+ fail('metadata-recovery-observation-changed')
+ change['afterBase64'] = base64.b64encode(raw).decode('ascii')
+ else:
+ fail('metadata-recovery-unsupported-attribute-change:' + label + ':' + name)
+ total += len(raw)
+ if total > 16 * 1024 ** 2:
+ fail('metadata-recovery-evidence-budget-exceeded')
+ changes.append(change)
+ return changes
+
+
+def immutable_recovery_evidence(root, prefix, evidence):
+ value = digest(evidence)
+ name = prefix + value + '.json'
+ path = root / name
+ encoded = (json.dumps(evidence, sort_keys=True) + '\n').encode('utf-8')
+ if len(encoded) > 256 * 1024 ** 2:
+ fail('metadata-recovery-evidence-budget-exceeded')
+ def existing_matches():
+ with os.fdopen(os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK), 'rb') as source:
+ info = os.fstat(source.fileno())
+ if (not stat.S_ISREG(info.st_mode) or info.st_nlink != 1
+ or stat.S_IMODE(info.st_mode) != 0o600 or info.st_uid != os.getuid()
+ or info.st_size != len(encoded) or source.read(len(encoded) + 1) != encoded):
+ fail('metadata-recovery-evidence-changed')
+ if os.path.lexists(path):
+ existing_matches()
+ else:
+ # A stopped write must not occupy the deterministic proof name. Keep
+ # abandoned staging files as evidence; only a fully fsynced record may
+ # become canonical, with no replacement of an existing destination.
+ staged = root / ('.' + name + '.' + uuid.uuid4().hex + '.staging')
+ private_json(staged, evidence)
+ try:
+ exclusive_rename(staged, path)
+ except OSError as error:
+ if error.errno != errno.EEXIST:
+ raise
+ existing_matches()
+ fsync_dir(root)
+ return {'file': name, 'sha256': hashlib.sha256(encoded).hexdigest(), 'digest': value}
+
+
+def load_metadata_recovery(root, saved, request):
+ accepted = request['metadataRecovery']
+ if (not isinstance(accepted, dict)
+ or set(accepted) != {'version', 'evidenceFile', 'evidenceSha256', 'digest'}
+ or accepted['version'] != 1
+ or not isinstance(accepted['digest'], str)
+ or not re.fullmatch(r'[0-9a-f]{64}', accepted['digest'])
+ or accepted['evidenceFile'] != 'metadata-recovery-' + accepted['digest'] + '.json'):
+ fail('metadata-recovery-proof-invalid')
+ path = root / accepted['evidenceFile']
+ with os.fdopen(os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK), 'rb') as source:
+ info = os.fstat(source.fileno())
+ if (not stat.S_ISREG(info.st_mode) or info.st_nlink != 1
+ or stat.S_IMODE(info.st_mode) != 0o600 or info.st_uid != os.getuid()
+ or info.st_size > 256 * 1024 ** 2):
+ fail('metadata-recovery-evidence-unsafe')
+ encoded = source.read(256 * 1024 ** 2 + 1)
+ if hashlib.sha256(encoded).hexdigest() != accepted['evidenceSha256']:
+ fail('metadata-recovery-evidence-changed')
+ evidence = json.loads(encoded)
+ if (digest(evidence) != accepted['digest']
+ or set(evidence) != {'version', 'manifestSha256', 'transactionIdentity',
+ 'originalFingerprint', 'initialPlacements', 'sources', 'changes'}
+ or evidence['version'] != 1
+ or evidence['manifestSha256'] != request.get('manifestSha256')
+ or evidence['transactionIdentity'] != saved['transactionIdentity']
+ or evidence['originalFingerprint'] != saved['facts']['fingerprint']
+ or set(evidence['initialPlacements']) != {'working', 'administration'}
+ or any(value not in ('original', 'quarantine') for value in evidence['initialPlacements'].values())):
+ fail('metadata-recovery-proof-invalid')
+ return evidence
+
+
+def verify_metadata_recovery(root, saved, request, located, sources):
+ evidence = load_metadata_recovery(root, saved, request)
+ if sources != evidence['sources']:
+ fail('metadata-recovery-review-changed')
+ for label in ('working', 'administration'):
+ # A restoration can only advance toward original placement. The outer
+ # service additionally binds the initial placement to its journal step.
+ if located[label]['state'] not in (evidence['initialPlacements'][label], 'original'):
+ fail('metadata-recovery-placement-regressed')
+ if metadata_recovery_changes(root, saved, located, sources) != evidence['changes']:
+ fail('metadata-recovery-review-changed')
+
+
+def review_metadata_recovery(request):
+ if not re.fullmatch(r'[0-9a-f]{64}', request.get('manifestSha256', '')):
+ fail('metadata-recovery-manifest-proof-required')
+ root, saved = load_transaction(request)
+ verify_saved_archive(root, saved)
+ located, sources = original_sources(root, saved['facts'])
+ changes = metadata_recovery_changes(root, saved, located, sources)
+ verify_live_common(root, saved, restoring=True)
+ if original_sources(root, saved['facts']) != (located, sources):
+ fail('metadata-recovery-observation-changed')
+ evidence = {'version': 1, 'manifestSha256': request['manifestSha256'],
+ 'transactionIdentity': saved['transactionIdentity'],
+ 'originalFingerprint': saved['facts']['fingerprint'],
+ 'initialPlacements': {label: row['state'] for label, row in located.items()},
+ 'sources': sources, 'changes': changes}
+ stored = immutable_recovery_evidence(root, 'metadata-recovery-', evidence)
+ return {'metadataRecovery': {'version': 1, 'evidenceFile': stored['file'],
+ 'evidenceSha256': stored['sha256'], 'digest': stored['digest']},
+ 'initialPlacements': evidence['initialPlacements'],
+ 'changeCounts': {'residualCompressionRemoved': sum(row['attribute'] == 'com.apple.decmpfs' for row in changes),
+ 'rootProvenanceAdded': sum(row['attribute'] == 'com.apple.provenance' for row in changes)},
+ 'changedPaths': len({(row['part'], row['path']) for row in changes})}
+
+
def verify(request):
root, saved = load_transaction(request)
facts = saved['facts']
+ recovering = 'metadataRecovery' in request
+ if recovering and request.get('mode') != 'restore':
+ fail('metadata-recovery-restore-only')
if request.get('mode') == 'standalone':
proof = verify_saved_archive(root, saved)
return {'verified': True, 'standalone': True, 'fingerprint': facts['fingerprint'],
'restorePath': str(root / 'restore/working'), 'restoreProof': proof}
- located = placements(root, facts)
- for label in ('working', 'administration'):
- state = located[label]['state']
- if state == 'ambiguous':
- fail('ambiguous-source-placement:' + label)
- path = Path(facts['release']['path'] if label == 'working' else facts['privateGitDir']) if state == 'original' else root / 'quarantine' / label
- if manifest(path) != facts['sources'][label]:
- fail('source-content-or-identity-changed:' + label)
+ located, sources = original_sources(root, facts)
+ if recovering:
+ verify_metadata_recovery(root, saved, request, located, sources)
+ else:
+ for label in ('working', 'administration'):
+ if sources[label] != facts['sources'][label]:
+ fail('source-content-or-identity-changed:' + label)
+ verify_saved_archive(root, saved)
+ verify_live_common(root, saved, restoring=request.get('mode') == 'restore')
+ return {'verified': True, 'placements': located, 'fingerprint': facts['fingerprint'],
+ **({'metadataRecoveryAccepted': True} if recovering else {})}
+
+
+def verify_live_common(root, saved, restoring):
+ facts = saved['facts']
common = safe_path(facts['commonGitDir'])
check_admin(common, common=True)
if identity(common) != facts['identity']['common']:
fail('common-store-incarnation-changed')
- restoring = request.get('mode') == 'restore'
mutable_shared = {'objects', 'refs', 'logs', 'packed-refs'} if restoring else set()
for name, expected in facts['sources']['common'].items():
if name not in mutable_shared and manifest(common / name, True) != expected:
@@ -789,7 +959,6 @@ def verify(request):
if sorted(p.name for p in common.iterdir() if p.name in COMMON_PARTS and p.name not in mutable_shared) != sorted(
name for name in facts['sources']['common'] if name not in mutable_shared):
fail('common-store-layout-changed')
- verify_saved_archive(root, saved)
if restoring:
# Other branches may legitimately advance after quarantine. Rollback
# never rewrites shared refs or objects: prove the original private index
@@ -802,13 +971,23 @@ def verify(request):
'fsck', '--full', '--strict', objects=common / 'objects')
prove_index_blobs(root / 'restore/administration', root / 'restore/working',
objects=common / 'objects')
- return {'verified': True, 'placements': located, 'fingerprint': facts['fingerprint']}
+
+
+def exclusive_rename(source, destination):
+ libc = ctypes.CDLL(None, use_errno=True)
+ libc.renamex_np.argtypes = [ctypes.c_char_p, ctypes.c_char_p, ctypes.c_uint]
+ libc.renamex_np.restype = ctypes.c_int
+ # RENAME_EXCL = 0x00000004, sys/stdio.h; never replace any occupied path.
+ if libc.renamex_np(os.fsencode(source), os.fsencode(destination), 0x00000004):
+ raise OSError(ctypes.get_errno(), 'exclusive-rename-failed')
def move(request):
part, direction = request['part'], request['direction']
if part not in ('working', 'administration') or direction not in ('quarantine', 'restore'):
fail('invalid-move')
+ if 'metadataRecovery' in request and direction != 'restore':
+ fail('metadata-recovery-restore-only')
verified = verify({**request, 'mode': 'restore' if direction == 'restore' else 'fresh'})
root, saved = load_transaction(request)
facts = saved['facts']
@@ -825,17 +1004,24 @@ def move(request):
safe_path(str(destination.parent))
if identity(source) != facts['identity'][part] or os.path.lexists(destination):
fail('move-lease-or-destination-changed')
- libc = ctypes.CDLL(None, use_errno=True)
- libc.renamex_np.argtypes = [ctypes.c_char_p, ctypes.c_char_p, ctypes.c_uint]
- libc.renamex_np.restype = ctypes.c_int
- # RENAME_EXCL = 0x00000004, sys/stdio.h; never replace any occupied path.
- if libc.renamex_np(os.fsencode(source), os.fsencode(destination), 0x00000004):
- raise OSError(ctypes.get_errno(), 'exclusive-rename-failed')
+ exclusive_rename(source, destination)
fsync_dir(source.parent)
fsync_dir(destination.parent)
if identity(destination) != facts['identity'][part]:
fail('post-move-identity-uncertain')
- return {'moved': True, 'part': part, 'direction': direction, 'placements': placements(root, facts)}
+ result = {'moved': True, 'part': part, 'direction': direction, 'placements': placements(root, facts)}
+ if 'metadataRecovery' in request:
+ located, sources = original_sources(root, facts)
+ observation = immutable_recovery_evidence(root, 'metadata-recovery-postmove-', {
+ 'version': 1, 'manifestSha256': request['manifestSha256'],
+ 'acceptedMetadataRecovery': request['metadataRecovery'],
+ 'part': part, 'placements': located, 'sources': sources})
+ try:
+ verify({**request, 'mode': 'restore'})
+ except Exception as error:
+ fail('metadata-recovery-postmove-changed:' + observation['file'] + ':' + str(error))
+ result['postMoveObservation'] = observation
+ return result
def main():
@@ -858,10 +1044,11 @@ def main():
elif action == 'status':
root, saved = load_transaction(request)
result = {'placements': placements(root, saved['facts']), 'fingerprint': saved['facts']['fingerprint']}
- elif action in ('preserve', 'verify', 'move'):
+ elif action in ('preserve', 'verify', 'move', 'review-metadata-recovery'):
fd = lock_file(transaction(request['transactionDir']), 'backend.lock')
try:
- result = {'preserve': preserve, 'verify': verify, 'move': move}[action](request)
+ result = {'preserve': preserve, 'verify': verify, 'move': move,
+ 'review-metadata-recovery': review_metadata_recovery}[action](request)
finally:
os.close(fd)
else:
diff --git a/scripts/lib/initialized_checkout_archive_test.py b/scripts/lib/initialized_checkout_archive_test.py
index 8b35cc2dc2..60bb3e870e 100644
--- a/scripts/lib/initialized_checkout_archive_test.py
+++ b/scripts/lib/initialized_checkout_archive_test.py
@@ -136,13 +136,13 @@ class NativeArchive(unittest.TestCase):
# started; sockets and stores belong exclusively to this fixture.
def bind(path):
listener = socket.socket(socket.AF_UNIX)
+ self.addCleanup(listener.close)
previous = Path.cwd()
try:
os.chdir(path.parent)
listener.bind(path.name)
finally:
os.chdir(previous)
- self.addCleanup(listener.close)
return listener
excluded = (common / 'fsmonitor--daemon.ipc',
@@ -440,6 +440,181 @@ class NativeArchive(unittest.TestCase):
self.assertEqual(archive.identity(self.work), before)
self.assertEqual(self.git(self.work, 'rev-parse', 'HEAD'), self.request['release']['head'])
+ def metadata_recovery_fixture(self):
+ subprocess.run(['/usr/bin/xattr', '-w', 'com.apple.decmpfs', 'fixture-residual',
+ str(self.work / 'base.txt')], check=True)
+ self.preserve()
+ before = (self.txn / 'manifest.json').read_bytes()
+ archive.move({**self.request, 'part': 'working', 'direction': 'quarantine'})
+ moved = self.txn / 'quarantine/working/base.txt'
+ # This deliberately mutates only the new disposable fixture to reproduce
+ # metadata removal after rename; recovery never performs an xattr write.
+ if 'com.apple.decmpfs' in archive.raw_xattrs(moved):
+ subprocess.run(['/usr/bin/xattr', '-d', 'com.apple.decmpfs', str(moved)], check=True)
+ with self.assertRaisesRegex(RuntimeError, 'source-content-or-identity-changed'):
+ archive.verify({**self.request, 'mode': 'restore'})
+ return before
+
+ def test_metadata_recovery_preserves_preimages_and_restores_same_inodes_across_processes(self):
+ before = self.metadata_recovery_fixture()
+ def process(request):
+ result = subprocess.run([sys.executable, str(Path(archive.__file__))],
+ input=json.dumps(request) + '\n', env=self.env,
+ capture_output=True, text=True, check=True)
+ return json.loads(result.stdout)
+ review = process({**self.request, 'command': 'review-metadata-recovery'})
+ proof = review['metadataRecovery']
+ evidence_path = self.txn / proof['evidenceFile']
+ evidence = json.loads(evidence_path.read_text())
+ self.assertEqual(archive.sha(evidence_path), proof['evidenceSha256'])
+ self.assertEqual(archive.digest(evidence), proof['digest'])
+ removed = next(row for row in evidence['changes'] if row['attribute'] == 'com.apple.decmpfs')
+ self.assertEqual(archive.base64.b64decode(removed['beforeBase64']), b'fixture-residual')
+ self.assertEqual(review['initialPlacements'], {'working': 'quarantine', 'administration': 'original'})
+ accepted = {**self.request, 'metadataRecovery': proof}
+ for request in ({**accepted, 'command': 'verify'},
+ {**accepted, 'command': 'move', 'part': 'working', 'direction': 'quarantine'}):
+ result = subprocess.run([sys.executable, str(Path(archive.__file__))],
+ input=json.dumps(request) + '\n', env=self.env,
+ capture_output=True, text=True)
+ self.assertEqual(result.returncode, 1)
+ self.assertEqual(json.loads(result.stdout)['reason'], 'metadata-recovery-restore-only')
+ unchanged_identity = archive.identity(self.txn / 'quarantine/working')
+ self.assertTrue(process({**accepted, 'command': 'move', 'part': 'administration',
+ 'direction': 'restore'})['alreadyRestored'])
+ moved = process({**accepted, 'command': 'move', 'part': 'working', 'direction': 'restore'})
+ observation = moved['postMoveObservation']
+ self.assertEqual(archive.sha(self.txn / observation['file']), observation['sha256'])
+ self.assertTrue(process({**accepted, 'command': 'verify', 'mode': 'restore'})['metadataRecoveryAccepted'])
+ self.assertEqual(archive.identity(self.work), unchanged_identity)
+ self.assertNotIn('com.apple.decmpfs', archive.raw_xattrs(self.work / 'base.txt'))
+ self.assertEqual(archive.raw_xattrs(self.txn / 'archive/working/base.txt')['com.apple.decmpfs'], b'fixture-residual')
+ self.assertEqual((self.txn / 'manifest.json').read_bytes(), before)
+ self.assertTrue(archive.verify({**self.request, 'mode': 'standalone'})['standalone'])
+ self.assertEqual(self.git(self.work, 'rev-parse', 'HEAD'), self.request['release']['head'])
+ # The old transaction remains exact rather than silently adopting the
+ # reviewed transition as its new default source baseline.
+ with self.assertRaisesRegex(RuntimeError, 'source-content-or-identity-changed'):
+ archive.verify({**self.request, 'mode': 'restore'})
+
+ def test_metadata_recovery_publishes_complete_evidence_without_overwriting_interrupted_or_foreign_records(self):
+ before = self.metadata_recovery_fixture()
+ write = archive.private_json
+ partial = []
+ def interrupt_staging(path, evidence):
+ if path.name.endswith('.staging'):
+ with os.fdopen(os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600), 'wb') as out:
+ out.write(b'{"interrupted":')
+ out.flush()
+ os.fsync(out.fileno())
+ partial.append(path)
+ raise RuntimeError('fixture interrupted staging write')
+ return write(path, evidence)
+ with patch.object(archive, 'private_json', interrupt_staging):
+ with self.assertRaisesRegex(RuntimeError, 'fixture interrupted staging'):
+ archive.review_metadata_recovery(self.request)
+ self.assertEqual(list(self.txn.glob('metadata-recovery-*.json')), [])
+ reviewed = archive.review_metadata_recovery(self.request)
+ evidence_path = self.txn / reviewed['metadataRecovery']['evidenceFile']
+ self.assertEqual(archive.sha(evidence_path), reviewed['metadataRecovery']['evidenceSha256'])
+ self.assertEqual(partial[0].read_bytes(), b'{"interrupted":')
+ self.assertEqual((self.txn / 'manifest.json').read_bytes(), before)
+ self.assertTrue(archive.verify({**self.request, 'mode': 'restore',
+ 'metadataRecovery': reviewed['metadataRecovery']})['verified'])
+
+ rename = archive.exclusive_rename
+ evidence = {'fixture': 'concurrent complete evidence'}
+ def racing_publication(staged, destination):
+ write(destination, evidence)
+ return rename(staged, destination)
+ with patch.object(archive, 'exclusive_rename', racing_publication):
+ shared = archive.immutable_recovery_evidence(self.txn, 'fixture-concurrent-', evidence)
+ self.assertEqual(json.loads((self.txn / shared['file']).read_text()), evidence)
+ self.assertEqual(len(list(self.txn.glob('.fixture-concurrent-*.staging'))), 1)
+ foreign = {'fixture': 'foreign evidence, retain unchanged'}
+ def conflicting_publication(staged, destination):
+ write(destination, foreign)
+ return rename(staged, destination)
+ with patch.object(archive, 'exclusive_rename', conflicting_publication):
+ with self.assertRaisesRegex(RuntimeError, 'metadata-recovery-evidence-changed'):
+ archive.immutable_recovery_evidence(self.txn, 'fixture-conflicting-', evidence)
+ collision = self.txn / ('fixture-conflicting-' + archive.digest(evidence) + '.json')
+ self.assertEqual(json.loads(collision.read_text()), foreign)
+ self.assertEqual(len(list(self.txn.glob('.fixture-conflicting-*.staging'))), 1)
+
+ def test_metadata_recovery_rejects_stale_review_and_unclassified_source_changes(self):
+ self.metadata_recovery_fixture()
+ review = archive.review_metadata_recovery(self.request)
+ accepted = {**self.request, 'mode': 'restore', 'metadataRecovery': review['metadataRecovery']}
+ original_manifest = archive.manifest
+ def observe(patch_row):
+ def observed(path, reject_links=False):
+ rows = original_manifest(path, reject_links)
+ if path == self.txn / 'quarantine/working':
+ rows = copy.deepcopy(rows)
+ row = next(value for value in rows if value['path'] == 'base.txt')
+ patch_row(row)
+ return rows
+ return observed
+ mutations = {
+ 'content': lambda row: row.update(sha256='0' * 64),
+ 'mode': lambda row: row.update(mode=row['mode'] ^ 0o100),
+ 'flags': lambda row: row.update(flags=row['flags'] | 0x20),
+ 'acl': lambda row: row.update(acl='changed'),
+ 'identity': lambda row: row['filesystemIdentity'].update(ino=0),
+ 'security attribute': lambda row: row['xattrs'].update({'com.apple.quarantine': '0' * 64}),
+ 'nonroot provenance': lambda row: row['xattrs'].update({'com.apple.provenance': '0' * 64}),
+ }
+ for name, mutation in mutations.items():
+ with self.subTest(change=name), patch.object(archive, 'manifest', observe(mutation)):
+ with self.assertRaisesRegex(RuntimeError, 'metadata-recovery-review-changed'):
+ archive.verify(accepted)
+ with self.assertRaisesRegex(RuntimeError, 'metadata-recovery-source-content-or-identity-changed|metadata-recovery-unsupported-attribute-change'):
+ archive.review_metadata_recovery(self.request)
+ evidence_path = self.txn / review['metadataRecovery']['evidenceFile']
+ encoded = evidence_path.read_bytes()
+ evidence_path.write_bytes(encoded + b' ')
+ with self.assertRaisesRegex(RuntimeError, 'metadata-recovery-evidence-changed'):
+ archive.verify(accepted)
+ evidence_path.write_bytes(encoded)
+ attrs = archive.raw_xattrs
+ def changed_preimage(path):
+ values = attrs(path)
+ return {**values, 'com.apple.decmpfs': b'wrong'} if path == self.txn / 'archive/working/base.txt' else values
+ with patch.object(archive, 'raw_xattrs', changed_preimage):
+ with self.assertRaisesRegex(RuntimeError, 'metadata-recovery-preimage-missing-or-changed'):
+ archive.verify(accepted)
+ self.assertFalse(self.work.exists())
+ self.assertTrue((self.txn / 'quarantine/working/base.txt').exists())
+
+ def test_metadata_recovery_records_unexpected_postmove_change_and_requires_new_review(self):
+ before = self.metadata_recovery_fixture()
+ first = archive.review_metadata_recovery(self.request)
+ attrs = archive.raw_xattrs
+ def moved_provenance(path):
+ values = attrs(path)
+ # Model a platform-generated root-only addition observed after the
+ # exclusive rename, without writing security metadata on any file.
+ return {**values, 'com.apple.provenance': b'postmove provenance'} if path == self.work else values
+ with patch.object(archive, 'raw_xattrs', moved_provenance):
+ with self.assertRaisesRegex(RuntimeError, 'metadata-recovery-postmove-changed:'):
+ archive.move({**self.request, 'metadataRecovery': first['metadataRecovery'],
+ 'part': 'working', 'direction': 'restore'})
+ observations = list(self.txn.glob('metadata-recovery-postmove-*.json'))
+ self.assertEqual(len(observations), 1)
+ self.assertTrue(self.work.exists())
+ self.assertFalse((self.txn / 'quarantine/working').exists())
+ with self.assertRaisesRegex(RuntimeError, 'metadata-recovery-review-changed'):
+ archive.verify({**self.request, 'mode': 'restore',
+ 'metadataRecovery': first['metadataRecovery']})
+ refreshed = archive.review_metadata_recovery(self.request)
+ self.assertNotEqual(refreshed['metadataRecovery']['digest'], first['metadataRecovery']['digest'])
+ self.assertEqual(refreshed['changeCounts']['rootProvenanceAdded'], 1)
+ accepted = {**self.request, 'metadataRecovery': refreshed['metadataRecovery']}
+ self.assertTrue(archive.move({**accepted, 'part': 'working', 'direction': 'restore'})['alreadyRestored'])
+ self.assertTrue(archive.verify({**accepted, 'mode': 'restore'})['metadataRecoveryAccepted'])
+ self.assertEqual((self.txn / 'manifest.json').read_bytes(), before)
+
def test_fresh_process_refuses_foreign_rollback_target(self):
self.preserve()
archive.move({**self.request, 'part': 'working', 'direction': 'quarantine'})
diff --git a/scripts/recover-initialized-checkout.js b/scripts/recover-initialized-checkout.js
new file mode 100644
index 0000000000..0375040d96
--- /dev/null
+++ b/scripts/recover-initialized-checkout.js
@@ -0,0 +1,151 @@
+#!/usr/bin/env node
+/**
+ * Restore-only bootstrap for an interrupted initialized checkout transaction.
+ * Run reviewed code from an independent checkout when its install's durable
+ * retirement cut prevents the normal update. Never edits the install's code.
+ *
+ * node scripts/recover-initialized-checkout.js review|restore request.json \
+ * --install-root /absolute/install --output /absolute/new-private-record.jsonl
+ *
+ * Output is an exclusive 0600 JSONL journal: a durable pending intent precedes
+ * the service call; a completed response is appended. A lone pending line means
+ * inspect the transaction before retrying. Full transition evidence stays in
+ * the service's private transaction artifact; stdout contains a small summary.
+ */
+import { constants } from 'node:fs';
+import { lstat, open, realpath } from 'node:fs/promises';
+import { dirname, isAbsolute, join, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { isDirectlyInvoked } from './lib/directInvocation.js';
+
+const CODE_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');
+const MAX_REQUEST_BYTES = 16 * 1024;
+const MAX_RESPONSE_BYTES = 1024 * 1024;
+const HASH = /^[a-f0-9]{64}$/;
+const ID = /^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/;
+const fail = code => { throw Object.assign(new Error(code), { recoveryCode: code }); };
+const object = value => value !== null && typeof value === 'object' && !Array.isArray(value);
+const exactKeys = (value, keys) => object(value) && Object.keys(value).sort().join(',') === [...keys].sort().join(',');
+
+export function parseRecoveryArgs(argv) {
+ const [action, requestPath, ...options] = argv;
+ if (!['review', 'restore'].includes(action) || !requestPath || requestPath.startsWith('--')) fail('recovery-usage');
+ const parsed = { action, requestPath };
+ for (let i = 0; i < options.length; i += 2) {
+ const key = { '--install-root': 'installRoot', '--output': 'outputPath' }[options[i]];
+ if (!key || parsed[key] || !options[i + 1] || options[i + 1].startsWith('--')) fail('recovery-usage');
+ parsed[key] = options[i + 1];
+ }
+ if (!parsed.installRoot || !parsed.outputPath || !isAbsolute(parsed.installRoot) || !isAbsolute(parsed.outputPath)) fail('recovery-absolute-path-required');
+ return parsed;
+}
+
+async function canonicalDirectory(path) {
+ if (!isAbsolute(path) || resolve(path) !== path || await realpath(path) !== path || !(await lstat(path)).isDirectory()) fail('recovery-path-not-canonical');
+ return path;
+}
+
+async function readJson(path, limit) {
+ const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW);
+ try {
+ const before = await handle.stat();
+ if (!before.isFile() || before.size > limit) fail('recovery-json-file-invalid');
+ const raw = await handle.readFile('utf8');
+ const after = await handle.stat();
+ if (Buffer.byteLength(raw) > limit || before.size !== after.size || before.mtimeMs !== after.mtimeMs || before.ctimeMs !== after.ctimeMs) fail('recovery-json-file-changed');
+ try { return JSON.parse(raw); } catch { fail('recovery-json-invalid'); }
+ } finally { await handle.close(); }
+}
+
+function validateRequest(action, request) {
+ const keys = ['appId', 'transactionId', 'journalDigest', ...(action === 'restore' ? ['review'] : [])];
+ if (!exactKeys(request, keys) || typeof request.appId !== 'string' || !/^[a-zA-Z0-9_-]{1,128}$/.test(request.appId)
+ || typeof request.transactionId !== 'string' || !ID.test(request.transactionId)
+ || typeof request.journalDigest !== 'string' || !HASH.test(request.journalDigest)
+ || (action === 'restore' && (!exactKeys(request.review, ['digest'])
+ || typeof request.review.digest !== 'string' || !HASH.test(request.review.digest)))) fail('recovery-request-invalid');
+ return request;
+}
+
+async function prepareInstall(installRoot, appId) {
+ await canonicalDirectory(installRoot);
+ // The safety reader calls ensureDirectories. Requiring every existing input
+ // prevents this bootstrap from initializing an empty or mistyped install.
+ for (const relative of ['data', 'data/cos', 'data/cos/agents', 'data/cos/reports', 'data/cos/scripts', 'data/cos/initialized-checkout-retirements']) {
+ await canonicalDirectory(join(installRoot, relative));
+ }
+ const state = await readJson(join(installRoot, 'data/cos/state.json'), 64 * 1024 * 1024);
+ if (!object(state) || !object(state.agents)) fail('recovery-ownership-unreadable');
+ const context = async requestedId => {
+ if (requestedId !== appId) fail('recovery-app-changed');
+ const apps = await readJson(join(installRoot, 'data/apps.json'), 16 * 1024 * 1024);
+ const app = object(apps?.apps) && Object.hasOwn(apps.apps, appId) ? apps.apps[appId] : null;
+ if (!object(app) || (Object.hasOwn(app, 'id') && app.id !== appId) || app.repoPath !== installRoot) fail('recovery-app-install-mismatch');
+ await canonicalDirectory(app.repoPath);
+ if (app.forgeAccount != null && typeof app.forgeAccount !== 'string') fail('recovery-app-invalid');
+ return { repoPath: app.repoPath, forgeAccount: app.forgeAccount ?? null };
+ };
+ await context(appId);
+ return { context, root: () => join(installRoot, 'data/cos/initialized-checkout-retirements') };
+}
+
+async function loadRecovery(deps, installRoot) {
+ // No PATHS-dependent import occurs until the explicit pin is in place.
+ const { PATHS } = await import('../server/lib/paths.js');
+ if (await realpath(PATHS.installRoot) !== installRoot || await realpath(PATHS.root) !== await realpath(CODE_ROOT)) fail('recovery-data-root-pin-refused');
+ const { _createInitializedCheckoutRecovery } = await import('../server/services/initializedCheckoutRecovery.js');
+ return _createInitializedCheckoutRecovery(deps);
+}
+
+const safeReason = value => typeof value === 'string' && /^[a-z0-9:-]{1,160}$/.test(value) ? value : 'recovery-failed';
+
+/** IO/factory injection lets tests exercise the CLI boundary without live data. */
+export async function runRecovery(argv, { load = loadRecovery, stdout = value => process.stdout.write(value) } = {}) {
+ let output;
+ let record;
+ const previousRoot = process.env.PORTOS_DATA_ROOT;
+ try {
+ const options = parseRecoveryArgs(argv);
+ const request = validateRequest(options.action, await readJson(resolve(options.requestPath), MAX_REQUEST_BYTES));
+ const deps = await prepareInstall(options.installRoot, request.appId);
+ await canonicalDirectory(dirname(options.outputPath));
+ if (resolve(options.outputPath) !== options.outputPath) fail('recovery-path-not-canonical');
+ output = await open(options.outputPath, constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL | constants.O_NOFOLLOW, 0o600);
+ record = { startedAt: new Date().toISOString(), action: options.action, installRoot: options.installRoot,
+ codeRoot: CODE_ROOT, request, outcome: 'pending-or-uncertain' };
+ await output.writeFile(`${JSON.stringify(record)}\n`);
+ await output.sync();
+ if (process.platform !== 'win32') {
+ const parent = await open(dirname(options.outputPath), constants.O_RDONLY);
+ try { await parent.sync(); } finally { await parent.close(); }
+ }
+ process.env.PORTOS_DATA_ROOT = options.installRoot;
+ const service = await load(deps, options.installRoot);
+ const response = await service[options.action === 'review' ? 'reviewMetadataRestore' : 'restoreMetadata'](request);
+ const completion = { completedAt: new Date().toISOString(), outcome: 'response-recorded', response };
+ const serialized = JSON.stringify(completion);
+ if (Buffer.byteLength(serialized) > MAX_RESPONSE_BYTES) fail('recovery-response-exceeds-budget');
+ await output.writeFile(`${serialized}\n`);
+ await output.sync();
+ const summary = { ok: response?.ok === true, output: options.outputPath, transactionId: request.transactionId };
+ if (response?.reason) summary.reason = safeReason(response.reason);
+ for (const key of ['phase', 'journalDigest']) if (typeof response?.[key] === 'string' && response[key].length <= 128) summary[key] = response[key];
+ if (HASH.test(response?.review?.digest ?? '')) summary.reviewDigest = response.review.digest;
+ stdout(`${JSON.stringify(summary)}\n`);
+ return response?.ok === true ? 0 : 1;
+ } catch (error) {
+ const reason = error.recoveryCode || (error.code === 'EEXIST' ? 'recovery-output-already-exists' : 'recovery-failed');
+ if (output && record) {
+ await output.writeFile(`${JSON.stringify({ completedAt: new Date().toISOString(), outcome: 'inspect-before-any-retry', reason })}\n`).catch(() => {});
+ await output.sync().catch(() => {});
+ }
+ stdout(`${JSON.stringify({ ok: false, reason })}\n`);
+ return 1;
+ } finally {
+ await output?.close();
+ if (previousRoot === undefined) delete process.env.PORTOS_DATA_ROOT;
+ else process.env.PORTOS_DATA_ROOT = previousRoot;
+ }
+}
+
+if (isDirectlyInvoked(import.meta.url)) process.exitCode = await runRecovery(process.argv.slice(2));
diff --git a/scripts/recover-initialized-checkout.test.js b/scripts/recover-initialized-checkout.test.js
new file mode 100644
index 0000000000..7d69597378
--- /dev/null
+++ b/scripts/recover-initialized-checkout.test.js
@@ -0,0 +1,108 @@
+import { afterEach, describe, expect, it, vi } from 'vitest';
+import { mkdtemp, mkdir, readFile, realpath, rm, stat, symlink, writeFile } from 'node:fs/promises';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { runRecovery } from './recover-initialized-checkout.js';
+
+const roots = [];
+afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
+const request = { appId: 'portos-default', transactionId: '12345678-1234-1234-1234-123456789abc', journalDigest: 'a'.repeat(64) };
+
+async function fixture(action = 'review') {
+ const root = await realpath(await mkdtemp(join(tmpdir(), 'portos-recovery-cli-')));
+ roots.push(root);
+ for (const relative of ['data/cos/agents', 'data/cos/reports', 'data/cos/scripts', 'data/cos/initialized-checkout-retirements']) {
+ await mkdir(join(root, relative), { recursive: true });
+ }
+ // Persisted registries key apps by ID; getAppById adds the id at read time.
+ const apps = { apps: { 'portos-default': { repoPath: root } } };
+ await writeFile(join(root, 'data/apps.json'), JSON.stringify(apps));
+ await writeFile(join(root, 'data/cos/state.json'), JSON.stringify({ agents: {} }));
+ const body = { ...request, ...(action === 'restore' ? { review: { digest: 'b'.repeat(64) } } : {}) };
+ const input = join(root, 'request.json');
+ const output = join(root, 'result.jsonl');
+ await writeFile(input, JSON.stringify(body));
+ return { root, apps, input, output, body, args: [action, input, '--install-root', root, '--output', output] };
+}
+
+describe('initialized checkout recovery CLI', () => {
+ it.each(['review', 'restore'])('pins the existing install, journals intent first, and invokes only guarded %s', async action => {
+ const f = await fixture(action);
+ const before = process.env.PORTOS_DATA_ROOT;
+ const stdout = vi.fn();
+ const method = action === 'review' ? 'reviewMetadataRestore' : 'restoreMetadata';
+ const load = vi.fn(async (deps, root) => {
+ expect(root).toBe(f.root);
+ expect(process.env.PORTOS_DATA_ROOT).toBe(f.root);
+ expect(deps.root()).toBe(join(f.root, 'data/cos/initialized-checkout-retirements'));
+ expect(await deps.context(request.appId)).toEqual({ repoPath: f.root, forgeAccount: null });
+ const intent = JSON.parse((await readFile(f.output, 'utf8')).trim());
+ expect(intent).toMatchObject({ outcome: 'pending-or-uncertain', request: f.body });
+ if (process.platform !== 'win32') expect((await stat(f.output)).mode & 0o777).toBe(0o600);
+ return { [method]: async input => {
+ expect(input).toEqual(f.body);
+ return { ok: true, review: { digest: 'c'.repeat(64), privateEvidence: 'never print this' } };
+ } };
+ });
+ expect(await runRecovery(f.args, { load, stdout })).toBe(0);
+ expect(load).toHaveBeenCalledOnce();
+ expect(process.env.PORTOS_DATA_ROOT).toBe(before);
+ const lines = (await readFile(f.output, 'utf8')).trim().split('\n').map(JSON.parse);
+ expect(lines).toHaveLength(2);
+ expect(lines[1]).toMatchObject({ outcome: 'response-recorded', response: { ok: true } });
+ expect(stdout.mock.calls[0][0]).not.toContain('never print this');
+ expect(JSON.parse(stdout.mock.calls[0][0]).reviewDigest).toBe('c'.repeat(64));
+ });
+
+ it.each(['missing-install', 'missing-agents', 'invalid-state', 'wrong-app-root', 'conflicting-app-id', 'inherited-app-id', 'invalid-json', 'extra-request-key', 'missing-review', 'unknown-option', 'existing-output'])(
+ 'refuses %s before invoking recovery', async change => {
+ const f = await fixture('restore');
+ if (change === 'missing-install') f.args[3] = join(f.root, 'absent');
+ if (change === 'missing-agents') await rm(join(f.root, 'data/cos/agents'), { recursive: true });
+ if (change === 'invalid-state') await writeFile(join(f.root, 'data/cos/state.json'), '{"agents":null}');
+ if (change === 'wrong-app-root') {
+ f.apps.apps['portos-default'].repoPath = join(f.root, 'other');
+ await writeFile(join(f.root, 'data/apps.json'), JSON.stringify(f.apps));
+ }
+ if (change === 'conflicting-app-id') {
+ f.apps.apps['portos-default'].id = 'different-app';
+ await writeFile(join(f.root, 'data/apps.json'), JSON.stringify(f.apps));
+ }
+ if (change === 'inherited-app-id') await writeFile(f.input, JSON.stringify({ ...f.body, appId: 'constructor' }));
+ if (change === 'invalid-json') await writeFile(f.input, '{');
+ if (change === 'extra-request-key') await writeFile(f.input, JSON.stringify({ ...f.body, force: true }));
+ if (change === 'missing-review') await writeFile(f.input, JSON.stringify(request));
+ if (change === 'unknown-option') f.args.push('--force');
+ if (change === 'existing-output') await writeFile(f.output, 'preserved');
+ const load = vi.fn();
+ expect(await runRecovery(f.args, { load, stdout: () => {} })).toBe(1);
+ expect(load).not.toHaveBeenCalled();
+ if (change === 'existing-output') expect(await readFile(f.output, 'utf8')).toBe('preserved');
+ if (change === 'missing-agents') await expect(stat(join(f.root, 'data/cos/agents'))).rejects.toMatchObject({ code: 'ENOENT' });
+ });
+
+ it.skipIf(process.platform === 'win32')('refuses a symlinked install component before loading service code', async () => {
+ const f = await fixture();
+ const target = join(f.root, 'actual-agents');
+ await mkdir(target);
+ await rm(join(f.root, 'data/cos/agents'), { recursive: true });
+ await symlink(target, join(f.root, 'data/cos/agents'));
+ const load = vi.fn();
+ expect(await runRecovery(f.args, { load, stdout: () => {} })).toBe(1);
+ expect(load).not.toHaveBeenCalled();
+ });
+
+ it('rechecks app binding and preserves uncertainty on a thrown service operation', async () => {
+ const f = await fixture('restore');
+ const stdout = vi.fn();
+ const load = async deps => ({ restoreMetadata: async () => {
+ f.apps.apps['portos-default'].repoPath = '/changed';
+ await writeFile(join(f.root, 'data/apps.json'), JSON.stringify(f.apps));
+ await deps.context(request.appId);
+ } });
+ expect(await runRecovery(f.args, { load, stdout })).toBe(1);
+ const lines = (await readFile(f.output, 'utf8')).trim().split('\n').map(JSON.parse);
+ expect(lines[0].outcome).toBe('pending-or-uncertain');
+ expect(lines[1]).toMatchObject({ outcome: 'inspect-before-any-retry', reason: 'recovery-app-install-mismatch' });
+ });
+});
diff --git a/server/lib/hostControlRoutes.js b/server/lib/hostControlRoutes.js
index 5716f3401b..6edcbb2cdc 100644
--- a/server/lib/hostControlRoutes.js
+++ b/server/lib/hostControlRoutes.js
@@ -420,6 +420,8 @@ export const HOST_CONTROL_ROUTES = Object.freeze([
'POST /api/cos/schedule/branch-reconcile/initialized-checkouts/retire',
'GET /api/cos/schedule/branch-reconcile/initialized-checkouts/status',
'POST /api/cos/schedule/branch-reconcile/initialized-checkouts/restore',
+ 'POST /api/cos/schedule/branch-reconcile/initialized-checkouts/review-metadata-restore',
+ 'POST /api/cos/schedule/branch-reconcile/initialized-checkouts/restore-metadata',
'POST /api/cos/schedule/branch-reconcile/initialized-checkouts/verify-archive',
'POST /api/cos/schedule/branch-reconcile/initialized-checkouts/review-branch',
'POST /api/cos/schedule/branch-reconcile/initialized-checkouts/retire-branch',
diff --git a/server/lib/validation.js b/server/lib/validation.js
index f2c38ba29b..8c05f98cde 100644
--- a/server/lib/validation.js
+++ b/server/lib/validation.js
@@ -2882,6 +2882,9 @@ export const initializedCheckoutRestoreSchema = initializedCheckoutTransactionSc
export const initializedCheckoutBranchRetireSchema = initializedCheckoutRestoreSchema.extend({
review: heldCheckoutRetireBodySchema.shape.review.unwrap(),
}).strict();
+export const initializedCheckoutMetadataRestoreSchema = initializedCheckoutRestoreSchema.extend({
+ review: z.object({ digest: z.string().regex(/^[a-f0-9]{64}$/) }).strict(),
+}).strict();
// Per-app task-type overrides (routes/apps/taskTypes.js)
export const appTaskTypeToggleBodySchema = z.object({ enabled: z.boolean() });
diff --git a/server/routes/cosScheduleRoutes.js b/server/routes/cosScheduleRoutes.js
index 22be49c25a..2e3991ac62 100644
--- a/server/routes/cosScheduleRoutes.js
+++ b/server/routes/cosScheduleRoutes.js
@@ -9,7 +9,7 @@ import { z } from 'zod';
import * as taskSchedule from '../services/taskSchedule.js';
import { logCosScheduleUpdate } from '../services/userActionScheduleLog.js';
import { asyncHandler, ServerError } from '../lib/errorHandler.js';
-import { sanitizeTaskMetadata, taskDataInputsSchema, validateRequest, parsePagination, cosScheduleTargetBodySchema, heldCheckoutRetireBodySchema, heldCheckoutReviewBodySchema, initializedCheckoutRetireBodySchema, initializedCheckoutTransactionSchema, initializedCheckoutRestoreSchema, initializedCheckoutBranchRetireSchema } from '../lib/validation.js';
+import { sanitizeTaskMetadata, taskDataInputsSchema, validateRequest, parsePagination, cosScheduleTargetBodySchema, heldCheckoutRetireBodySchema, heldCheckoutReviewBodySchema, initializedCheckoutRetireBodySchema, initializedCheckoutTransactionSchema, initializedCheckoutRestoreSchema, initializedCheckoutBranchRetireSchema, initializedCheckoutMetadataRestoreSchema } from '../lib/validation.js';
import { promptSourceSchema, PROMPT_SOURCES, scheduleExecutionFieldsSchema } from '../lib/cosValidation.js';
import { EFFORT_LEVELS } from '../lib/providerModels.js';
import { INTERVAL_TYPES, decodeIntervalType, isCronExpression, isKnownIntervalType } from '../services/taskScheduleConstants.js';
@@ -342,6 +342,20 @@ router.post('/schedule/branch-reconcile/initialized-checkouts/restore', asyncHan
res.status(result.ok ? 200 : 409).json(result);
}));
+router.post('/schedule/branch-reconcile/initialized-checkouts/review-metadata-restore', asyncHandler(async (req, res) => {
+ const input = validateRequest(initializedCheckoutRestoreSchema, req.body);
+ const { reviewInitializedCheckoutMetadataRestore } = await import('../services/initializedCheckoutRecovery.js');
+ const result = await reviewInitializedCheckoutMetadataRestore(input);
+ res.status(result.ok ? 200 : 409).json(result);
+}));
+
+router.post('/schedule/branch-reconcile/initialized-checkouts/restore-metadata', asyncHandler(async (req, res) => {
+ const input = validateRequest(initializedCheckoutMetadataRestoreSchema, req.body);
+ const { restoreInitializedCheckoutMetadata } = await import('../services/initializedCheckoutRecovery.js');
+ const result = await restoreInitializedCheckoutMetadata(input);
+ res.status(result.ok ? 200 : 409).json(result);
+}));
+
router.post('/schedule/branch-reconcile/initialized-checkouts/verify-archive', asyncHandler(async (req, res) => {
const input = validateRequest(initializedCheckoutRestoreSchema, req.body);
const { verifyInitializedCheckoutArchive } = await import('../services/initializedCheckoutRecovery.js');
diff --git a/server/routes/cosScheduleRoutes.test.js b/server/routes/cosScheduleRoutes.test.js
index 567d26fe55..b4a02120b1 100644
--- a/server/routes/cosScheduleRoutes.test.js
+++ b/server/routes/cosScheduleRoutes.test.js
@@ -10,7 +10,7 @@ vi.mock('../services/maintenanceRun.js', () => maintenance);
const recovery = vi.hoisted(() => ({ prepareInventoriedCheckoutRetirement: vi.fn(), retireInventoriedCheckout: vi.fn() }));
vi.mock('../services/heldCheckoutRecovery.js', () => recovery);
-const initialized = vi.hoisted(() => ({ reviewInitializedCheckout: vi.fn(), retireInitializedCheckout: vi.fn(), initializedCheckoutStatus: vi.fn(), restoreInitializedCheckout: vi.fn(), verifyInitializedCheckoutArchive: vi.fn(), reviewInitializedCheckoutBranch: vi.fn(), retireInitializedCheckoutBranch: vi.fn() }));
+const initialized = vi.hoisted(() => ({ reviewInitializedCheckout: vi.fn(), retireInitializedCheckout: vi.fn(), initializedCheckoutStatus: vi.fn(), restoreInitializedCheckout: vi.fn(), verifyInitializedCheckoutArchive: vi.fn(), reviewInitializedCheckoutBranch: vi.fn(), retireInitializedCheckoutBranch: vi.fn(), reviewInitializedCheckoutMetadataRestore: vi.fn(), restoreInitializedCheckoutMetadata: vi.fn() }));
vi.mock('../services/initializedCheckoutRecovery.js', () => initialized);
vi.mock('../services/taskSchedule.js', () => ({
@@ -94,6 +94,21 @@ describe('CoS Schedule Routes', () => {
expect(initialized.retireInitializedCheckoutBranch).toHaveBeenCalledWith(branchInput);
});
+ it('separately requires a fresh digest for metadata restoration and rejects caller-supplied allowances', async () => {
+ const base = '/api/cos/schedule/branch-reconcile/initialized-checkouts';
+ const input = { appId: 'app', transactionId: '2f800be9-6824-4726-813d-8d9ea0b2c1dc', journalDigest: 'a'.repeat(64) };
+ initialized.reviewInitializedCheckoutMetadataRestore.mockResolvedValue({ ok: true, review: { digest: 'b'.repeat(64) } });
+ expect((await request(app).post(`${base}/review-metadata-restore`).send(input)).status).toBe(200);
+ expect(initialized.reviewInitializedCheckoutMetadataRestore).toHaveBeenCalledWith(input);
+ expect((await request(app).post(`${base}/restore-metadata`).send(input)).status).toBe(400);
+ const approved = { ...input, review: { digest: 'b'.repeat(64) } };
+ expect((await request(app).post(`${base}/restore-metadata`).send({ ...approved, ignoreXattrs: true })).status).toBe(400);
+ expect(initialized.restoreInitializedCheckoutMetadata).not.toHaveBeenCalled();
+ initialized.restoreInitializedCheckoutMetadata.mockResolvedValue({ ok: false, reason: 'metadata-recovery-review-changed' });
+ expect((await request(app).post(`${base}/restore-metadata`).send(approved)).status).toBe(409);
+ expect(initialized.restoreInitializedCheckoutMetadata).toHaveBeenCalledWith(approved);
+ });
+
describe('GET /api/cos/schedule', () => {
it('surfaces unavailable schedule inputs as errors and recovers on the next read', async () => {
taskSchedule.getScheduleStatus
diff --git a/server/services/initializedCheckoutRecovery.js b/server/services/initializedCheckoutRecovery.js
index 9a4830cd79..ba499b281d 100644
--- a/server/services/initializedCheckoutRecovery.js
+++ b/server/services/initializedCheckoutRecovery.js
@@ -354,6 +354,9 @@ export function _createInitializedCheckoutRecovery(deps = {}) {
...(unboundPreservation ? { recoveryRequired: true, recoveryAction: 'restore', reason: 'preservation-proof-unbound' } : {}),
...(state.phase === 'branch-retirement-intent' ? { branchDeletion: 'uncertain', branchRef, recoveryRequired: true } : {}),
...(state.phase === 'branch-retired' ? { receiptPending: true, recoveryRequired: true } : {}),
+ ...(state.metadataRestoreReview ? { metadataRecovery: state.metadataRestoreReview.metadataRecovery,
+ metadataChangeCounts: state.metadataRestoreReview.changeCounts,
+ metadataRestored: state.phase === 'restored-with-metadata-transitions' } : {}),
...(['branch-retired', 'retired'].includes(state.phase) ? { recoveryAction: 'retire-branch' } : {}) };
};
@@ -419,7 +422,81 @@ export function _createInitializedCheckoutRecovery(deps = {}) {
});
} finally { await executor.release(); }
}).catch(error => ({ ok: false, reason: error.message, transactionId: input.transactionId, recoveryRequired: true }));
- return { review, retire, status, restore, verifyArchive, reviewBranch, retireBranch };
+
+ const metadataRestoreSafety = async ({ ctx, state }) => {
+ const policy = await ensureOwner(ctx, state.release, false);
+ const list = deps.listWorktrees || (await import('./worktreeManager.js')).listWorktrees;
+ if ((await list(ctx.repoPath)).some(tree => String(tree.branch || '').replace(/^refs\/heads\//, '') === state.release.branch
+ && tree.path !== state.release.path)) fail('branch-has-new-holder');
+ const head = await (deps.execGit || execGit)(['rev-parse', '--verify', `refs/heads/${state.release.branch}`],
+ ctx.repoPath, { ignoreExitCode: true });
+ if (head.exitCode !== 0 || head.stdout.trim() !== state.release.head) fail('branch-ref-changed');
+ return policy;
+ };
+ const metadataReview = async bound => {
+ const { directory, state, ctx } = bound;
+ const allowed = ['quarantine-working-intent', 'quarantined-working', 'quarantine-administration-intent',
+ 'quarantined-administration', 'quarantined', 'restore-administration-intent', 'restored-administration',
+ 'restore-working-intent', 'restored-working', 'metadata-restore-approved', 'metadata-restore-administration-intent',
+ 'metadata-restored-administration', 'metadata-restore-working-intent', 'metadata-restored-working',
+ 'restored-with-metadata-transitions'];
+ if (!allowed.includes(state.phase) || state.branchDeleted || !state.proof?.manifestSha256) fail('metadata-recovery-phase-not-ready');
+ const policy = await metadataRestoreSafety(bound);
+ const proof = await native({ command: 'review-metadata-recovery', transactionDir: directory,
+ manifestSha256: state.proof.manifestSha256 });
+ const evidence = { version: 1, mode: 'restore-metadata-only', transactionId: state.id,
+ repoPath: ctx.repoPath, release: state.release, journalDigest: hash(state), phase: state.phase, policy,
+ metadataRecovery: proof.metadataRecovery, initialPlacements: proof.initialPlacements,
+ changeCounts: proof.changeCounts, changedPaths: proof.changedPaths };
+ return { digest: hash(evidence), ...evidence };
+ };
+ // The CLI uses this same path before an update can be deployed. Its local
+ // ownership tail is not a cross-process lock: native execution.lock and the
+ // exact existing publication cut must both remain held for the entire action.
+ const withMetadataRestore = (input, work) => fenced(async () => {
+ const directory = location(input.transactionId);
+ const executor = await native({ command: 'lock', transactionDir: directory }, { keepOpen: true });
+ try {
+ const bound = await loadBoundTransaction(input);
+ const gate = await admission();
+ const reservation = await gate.readCheckoutReservation(bound.ctx.repoPath, bound.state.id);
+ return await gate.withCheckoutReservation(reservation, async () => {
+ executor.assertHeld();
+ if (hash(await readJournal(directory)) !== hash(bound.state)) fail('retirement-journal-changed');
+ return work(bound, { executor, gate, reservation });
+ });
+ } finally { await executor.release(); }
+ }).catch(error => ({ ok: false, reason: error.message, transactionId: input.transactionId, recoveryRequired: true }));
+ const reviewMetadataRestore = input => withMetadataRestore(input,
+ async bound => ({ ok: true, review: await metadataReview(bound) }));
+ const restoreMetadata = input => withMetadataRestore(input, async (bound, { executor, gate, reservation }) => {
+ const current = await metadataReview(bound);
+ if (current.digest !== input.review?.digest) fail('metadata-recovery-review-changed');
+ const { directory } = bound;
+ let state = await appendStep(directory, bound.state, 'metadata-restore-approved',
+ { metadataRestoreReview: current, reservation });
+ const proof = { transactionDir: directory, manifestSha256: state.proof.manifestSha256,
+ metadataRecovery: current.metadataRecovery };
+ for (const part of ['administration', 'working']) {
+ executor.assertHeld();
+ const policy = await metadataRestoreSafety({ ...bound, state });
+ if (JSON.stringify(policy) !== JSON.stringify(current.policy)) fail('retirement-evidence-changed');
+ state = await appendStep(directory, state, `metadata-restore-${part}-intent`);
+ const moved = await native({ command: 'move', ...proof, part, direction: 'restore' });
+ state = await appendStep(directory, state, `metadata-restored-${part}`, {
+ metadataRestoreObservations: [...(state.metadataRestoreObservations || []), { part, proof: moved.postMoveObservation }],
+ });
+ }
+ await native({ command: 'verify', ...proof, mode: 'restore' });
+ const finalPolicy = await metadataRestoreSafety({ ...bound, state });
+ if (JSON.stringify(finalPolicy) !== JSON.stringify(current.policy)) fail('retirement-evidence-changed');
+ executor.assertHeld();
+ state = await appendStep(directory, state, 'restored-with-metadata-transitions');
+ await gate.releaseCheckoutReservation(reservation);
+ return { ok: true, transactionId: state.id, phase: state.phase, restored: true, archivesRetained: true,
+ metadataTransitionsPreserved: true, metadataRecovery: current.metadataRecovery, archiveDir: directory };
+ });
+ return { review, retire, status, restore, verifyArchive, reviewBranch, retireBranch, reviewMetadataRestore, restoreMetadata };
}
const recovery = _createInitializedCheckoutRecovery();
@@ -432,3 +509,5 @@ export const verifyInitializedCheckoutArchive = recovery.verifyArchive;
export const reviewInitializedCheckoutBranch = recovery.reviewBranch;
export const retireInitializedCheckoutBranch = recovery.retireBranch;
+export const reviewInitializedCheckoutMetadataRestore = recovery.reviewMetadataRestore;
+export const restoreInitializedCheckoutMetadata = recovery.restoreMetadata;
diff --git a/server/services/initializedCheckoutRecovery.test.js b/server/services/initializedCheckoutRecovery.test.js
index d89828a659..8f9fbcbede 100644
--- a/server/services/initializedCheckoutRecovery.test.js
+++ b/server/services/initializedCheckoutRecovery.test.js
@@ -52,7 +52,8 @@ beforeEach(async () => {
root = await mkdtemp(join(tmpdir(), 'initialized-recovery-'));
const facts = { prUrl, prHead: release.head, defaultHead: 'b'.repeat(40) };
const snapshot = { fingerprint: 'c'.repeat(64), administrationFingerprint: 'd'.repeat(64), identity: { working: 1, administration: 2, common: 3 } };
- fixture = { cut: null, facts, snapshot, headExists: true, placement: { working: 'original', administration: 'original' }, failPart: null };
+ fixture = { cut: null, facts, snapshot, headExists: true, placement: { working: 'original', administration: 'original' }, failPart: null,
+ metadataToken: 'f'.repeat(64) };
const admission = {
withCheckoutPublication: async (_repo, work) => { if (fixture.cut) throw new Error('checkout-retirement-reserved'); return work(); },
reserveCheckoutRetirement: vi.fn(async (_repo, id) => {
@@ -75,13 +76,21 @@ beforeEach(async () => {
if (fixture.failProofJournal) fixture.failNextJournal = true;
return { ok: true, manifestSha256: 'e'.repeat(64), restoreProof: { verified: true } };
}
+ if (input.command === 'review-metadata-recovery') return {
+ metadataRecovery: { version: 1, digest: fixture.metadataToken, evidenceFile: 'private-proof.json', evidenceSha256: fixture.metadataToken },
+ initialPlacements: { ...fixture.placement }, changeCounts: { residualCompressionRemoved: 3, rootProvenanceAdded: 1 }, changedPaths: 4,
+ };
if (['verify', 'move', 'status'].includes(input.command) && input.manifestSha256 !== 'e'.repeat(64)) throw new Error('archive-manifest-proof-changed');
+ if (fixture.metadataDrift && ['verify', 'move'].includes(input.command) && input.mode !== 'standalone'
+ && input.metadataRecovery?.digest !== fixture.metadataToken) throw new Error('source-content-or-identity-changed:working');
if (input.command === 'move') {
if (fixture.failPart === input.part && input.direction === 'quarantine') throw new Error('move-refused');
+ if (fixture.failRestorePart === input.part && input.direction === 'restore') throw new Error('restore-interrupted');
fixture.placement[input.part] = input.direction === 'quarantine' ? 'quarantined' : 'original';
}
if (fixture.failArchive && input.command === 'verify') throw new Error('archive-changed');
return { ok: true, placement: fixture.placement, fingerprint: fixture.snapshot.fingerprint,
+ ...(input.metadataRecovery ? { postMoveObservation: { file: 'post-move.json', sha256: fixture.metadataToken } } : {}),
placements: Object.fromEntries(Object.entries(fixture.placement).map(([part, state]) => [part, { state: state === 'quarantined' ? 'quarantine' : state }])) };
});
const deps = { root: () => root, fenced: work => work(), admission, native,
@@ -165,6 +174,93 @@ describe('initialized checkout transaction', () => {
expect(await fixture.service.restore({ appId: 'app', transactionId: result.transactionId, journalDigest: status.journalDigest })).toMatchObject({ ok: true });
expect((await readdir(join(root, result.transactionId))).length).toBeGreaterThan(0);
});
+ const interruptedMetadata = async () => {
+ fixture.failPart = 'administration';
+ const retired = await fixture.service.retire(await reviewed());
+ fixture.metadataDrift = true;
+ const status = await fixture.service.status({ appId: 'app', transactionId: retired.transactionId });
+ return { appId: 'app', transactionId: retired.transactionId, journalDigest: status.journalDigest };
+ };
+ it('restores originals only through a separately approved metadata proof and preserves the immutable baseline', async () => {
+ const input = await interruptedMetadata();
+ expect(await fixture.service.restore(input)).toMatchObject({ ok: false, reason: 'source-content-or-identity-changed:working' });
+ const prepared = await fixture.service.reviewMetadataRestore(input);
+ expect(prepared).toMatchObject({ ok: true, review: { mode: 'restore-metadata-only', changeCounts: { residualCompressionRemoved: 3 } } });
+ const restored = await fixture.service.restoreMetadata({ ...input, review: { digest: prepared.review.digest } });
+ expect(restored).toMatchObject({ ok: true, restored: true, phase: 'restored-with-metadata-transitions', archivesRetained: true });
+ expect(fixture.placement).toEqual({ working: 'original', administration: 'original' });
+ expect(fixture.cut).toBeNull();
+ expect(fixture.headExists).toBe(true);
+ expect(fixture.deps.deleteRef).not.toHaveBeenCalled();
+ const dir = join(root, input.transactionId);
+ expect(await readFile(join(dir, 'manifest.json'), 'utf8')).toBe('{}');
+ const journals = (await readdir(dir)).filter(name => /^\d{6}\.json$/.test(name)).sort();
+ const last = JSON.parse(await readFile(join(dir, journals.at(-1)), 'utf8'));
+ expect(last.metadataRestoreReview.metadataRecovery.digest).toBe(fixture.metadataToken);
+ expect(last.metadataRestoreObservations).toHaveLength(2);
+ });
+ it('refuses changed metadata approval or a missing existing reservation before restoring', async () => {
+ const input = await interruptedMetadata();
+ const prepared = await fixture.service.reviewMetadataRestore(input);
+ fixture.metadataToken = '9'.repeat(64);
+ const before = fixture.deps.native.mock.calls.filter(([v]) => v.command === 'move').length;
+ expect(await fixture.service.restoreMetadata({ ...input, review: { digest: prepared.review.digest } }))
+ .toMatchObject({ ok: false, reason: 'metadata-recovery-review-changed' });
+ expect(fixture.deps.native.mock.calls.filter(([v]) => v.command === 'move')).toHaveLength(before);
+ fixture.cut = null;
+ expect(await fixture.service.reviewMetadataRestore(input)).toMatchObject({ ok: false, recoveryRequired: true });
+ expect(fixture.deps.admission.reserveCheckoutRetirement).toHaveBeenCalledTimes(1);
+ });
+ it('requires a new journal-bound review after a metadata restore interruption', async () => {
+ const input = await interruptedMetadata();
+ const prepared = await fixture.service.reviewMetadataRestore(input);
+ fixture.failRestorePart = 'working';
+ expect(await fixture.service.restoreMetadata({ ...input, review: { digest: prepared.review.digest } }))
+ .toMatchObject({ ok: false, reason: 'restore-interrupted' });
+ expect(fixture.cut).not.toBeNull();
+ const restarted = _createInitializedCheckoutRecovery(fixture.deps);
+ expect(await restarted.restoreMetadata({ ...input, review: { digest: prepared.review.digest } }))
+ .toMatchObject({ ok: false, reason: 'retirement-journal-changed' });
+ const status = await restarted.status(input);
+ const fresh = { ...input, journalDigest: status.journalDigest };
+ const second = await restarted.reviewMetadataRestore(fresh);
+ fixture.failRestorePart = null;
+ expect(await restarted.restoreMetadata({ ...fresh, review: { digest: second.review.digest } }))
+ .toMatchObject({ ok: true, restored: true });
+ expect(fixture.cut).toBeNull();
+ expect(fixture.deps.deleteRef).not.toHaveBeenCalled();
+ });
+ it('rechecks current ownership before each metadata restoration move', async () => {
+ const input = await interruptedMetadata();
+ const prepared = await fixture.service.reviewMetadataRestore(input);
+ fixture.deps.ownership.mockResolvedValueOnce({ ok: true, policy: null })
+ .mockResolvedValue({ ok: false, reason: 'active-owner' });
+ const before = fixture.deps.native.mock.calls.filter(([v]) => v.command === 'move').length;
+ expect(await fixture.service.restoreMetadata({ ...input, review: { digest: prepared.review.digest } }))
+ .toMatchObject({ ok: false, reason: 'active-owner' });
+ expect(fixture.deps.native.mock.calls.filter(([v]) => v.command === 'move')).toHaveLength(before);
+ expect(fixture.cut).not.toBeNull();
+ });
+ it('retains the reservation when allowed policy changes during final restoration verification', async () => {
+ const input = await interruptedMetadata();
+ const prepared = await fixture.service.reviewMetadataRestore(input);
+ fixture.deps.ownership.mockResolvedValueOnce({ ok: true, policy: null })
+ .mockResolvedValueOnce({ ok: true, policy: null })
+ .mockResolvedValueOnce({ ok: true, policy: null })
+ .mockResolvedValue({ ok: true, policy: { reason: null, policies: [] } });
+ expect(await fixture.service.restoreMetadata({ ...input, review: { digest: prepared.review.digest } }))
+ .toMatchObject({ ok: false, reason: 'retirement-evidence-changed' });
+ expect(fixture.placement).toEqual({ working: 'original', administration: 'original' });
+ expect(fixture.cut).not.toBeNull();
+ expect(fixture.deps.deleteRef).not.toHaveBeenCalled();
+ const status = await fixture.service.status(input);
+ expect(status.phase).toBe('metadata-restored-working');
+ const fresh = { ...input, journalDigest: status.journalDigest };
+ const second = await fixture.service.reviewMetadataRestore(fresh);
+ expect(await fixture.service.restoreMetadata({ ...fresh, review: { digest: second.review.digest } }))
+ .toMatchObject({ ok: true, restored: true });
+ expect(fixture.cut).toBeNull();
+ });
it('exposes only source-verified cancellation after durable preservation loses its proof journal', async () => {
fixture.failProofJournal = true;
const result = await fixture.service.retire(await reviewed());