Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

feat: recover unsettled output publications with inspected evidence (#13346)

Adam Eivy1f632f215359
diff --git a/docs/STORAGE.md b/docs/STORAGE.md
index 12342c1f30..cbed33c48b 100644
--- a/docs/STORAGE.md
+++ b/docs/STORAGE.md
@@ -110,12 +110,92 @@ Draining. Recovery archives stay in the same backed-up, machine-local subtree:
snapshots retain them as evidence and restores preserve the current subtree
byte-for-byte, including archives, rather than installing a snapshot's authority.
+### Unsettled generic output publications
+
+Use the local operator command below after journal transaction recovery, with a
+normal maintenance hold in place. It never steals an interrupted transaction lock,
+resumes a hold, or settles an exclusive maintenance owner. Run from the install
+checkout. From a reviewed isolated checkout, append
+`--install-root <absolute-install-root>` (the parent of `data/`) to each command.
+This explicit target does not depend on the inherited `PORTOS_DATA_ROOT` pin,
+which worktree code deliberately ignores.
+
+```bash
+# Read-only inventory, including unrelated owners and the exact selected operation.
+node scripts/reconcile-output-publication.mjs diagnose
+node scripts/reconcile-output-publication.mjs diagnose <operation-id>
+
+# Read-only inspection: fingerprint the explicit evidence specification below.
+# Save these private JSON files outside the source tree; do not publish them.
+node scripts/reconcile-output-publication.mjs inspect <operation-id> evidence-spec.json > inspected-request.json
+
+# Review the request, then explicitly acknowledge uncertainty for that owner only.
+node scripts/reconcile-output-publication.mjs reconcile inspected-request.json --confirm-disposition
+```
+
+Generic reservations contain no parent, destination, run, or batch identity. The
+operator must trace the owning workflow and explain how its records relate to
+the **selected** reservation; never assign evidence merely by PID or age. Diagnose
+lists admitted owner IDs and their source modules from `backupAssetOwners.js`.
+Relevant examples:
+
+| Owner | Evidence to inspect |
+|---|---|
+| `toolkit-recording-publication`, `runtime-recording-publication-tails`, `agent-run-recording-publication` | Matching run metadata, output, retained agent spools (`output.txt` / `raw.txt`), and durable failure/warning records. Trace recording creation, spool writes and finalization; terminal metadata alone does not prove a complete transcript. |
+| `cos-agent-archive-and-index-publication` | Agent/archive identity, archive/index records, retained output and the publication failure. |
+| `media-job-completion`, `image-generation-completion-tails`, `generated-video-shared-finalizer` | Exact job/run identity, sidecars/history or inspected database-record evidence, and output integrity or the durable publication failure. A provider completion alone does not prove local publication. |
+
+Other admitted owners require the same record-plus-output/failure inspection of
+their listed source modules. If the relationship cannot be established, keep the
+reservation and recover through its owning service. For DB-primary records, save
+a bounded, private inspection artifact with exact record identity under `data/`;
+do not copy credentials or a whole database into an evidence file. Preserve large
+media separately and use its owning service's integrity/failure evidence here.
+
+Example `evidence-spec.json` (invented paths; replace every explanation after
+actually inspecting the records):
+
+```json
+{
+ "ownerId": "toolkit-recording-publication",
+ "binding": "Describe the exact run identity and evidence linking its interrupted publication to the selected reservation.",
+ "reason": "Explain the inspected failure and why this specific uncertainty is being acknowledged.",
+ "files": [
+ { "path": "runs/example-run/metadata.json", "role": "record", "note": "Describe the inspected run identity and unchanged outcome." },
+ { "path": "runs/example-run/publication-failure.txt", "role": "failure", "note": "Describe the inspected publication failure and retained output limitations." }
+ ]
+}
+```
+
+Evidence uses distinct ordinary data-relative files, without symlinks or journal
+files: a nonempty owner `record` plus nonempty `output` or `failure`, at most
+16 files / 16 MiB total. The request binds SHA-256 hashes and lengths, inspection
+notes, owner, relationship, reason, exact hold, original operation and uncertainty
+stamp. Preserve both evidence files and the inspected request. Empty output,
+process exit and elapsed time never establish saving. A live or unproved admitting
+process refuses disposition; process exit is only a prerequisite, never evidence
+of completeness.
+
+Reconciliation rechecks evidence under the journal transaction lock, fsyncs a
+no-replace `data/workflow-maintenance/publication-<operation-id>-<stamp>.json`
+receipt, then retires only that exact owner. Repeating the **same** request reuses
+the matching receipt, including a receipt-first interruption after normal journal
+recovery. A missing owner cannot mint a receipt. Stale holds/operations/stamps,
+missing/changed evidence, incompatible receipts, exclusive ownership, and failed
+receipt persistence retain ownership. Refusals name the next inspection/recovery
+step; never edit the request's hashes to bypass them. The hold and every other
+operation remain; run/task outcomes and transcript warnings stay unchanged.
+`acknowledged-uncertainty` with `outputCompleteness: "unknown"` records an operator
+disposition, **not** successful saving. These machine-local receipts share the
+backup/restore exclusions described below for abandoned-agent receipts; they
+have no seed, migration, federation or automatic expiry.
+
### Explicitly abandoned duplicate agents
A local operator may reconcile a user-killed duplicate that saved no output with
`node scripts/reconcile-abandoned-agent.mjs <agent-id> <run-id> --confirm-abandoned <reason>`.
-Begin a normal maintenance hold first. When using a reviewed isolated checkout,
-`PORTOS_DATA_ROOT` names the install root (the parent of `data/`). This is an
+Begin a normal maintenance hold first and run from the install checkout;
+worktree code deliberately ignores the `PORTOS_DATA_ROOT` pin. This is an
explicit abandonment decision, never automatic expiry or a successful-run verdict.
The command refuses present agent/runner ownership, live recorded PIDs, retained
agent output (including archives), worktrees/branches, nonempty run output,
diff --git a/scripts/outputPublicationRecovery.test.js b/scripts/outputPublicationRecovery.test.js
new file mode 100644
index 0000000000..8c4c719cc9
--- /dev/null
+++ b/scripts/outputPublicationRecovery.test.js
@@ -0,0 +1,212 @@
+import { afterEach, beforeEach, expect, it } from 'vitest';
+import * as fs from 'node:fs';
+import { randomUUID } from 'node:crypto';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { spawnSync } from 'node:child_process';
+import { fileURLToPath } from 'node:url';
+import { createMaintenanceAdmission } from '../server/lib/maintenanceAdmission.js';
+import { runPublicationRecovery } from './reconcile-output-publication.mjs';
+
+let root, data, gate, deps, operationId, hold, spec, specPath, requestPath, statePath;
+const readState = () => JSON.parse(fs.readFileSync(statePath, 'utf8'));
+const receiptPath = request => join(gate.directory, `publication-${request.operation.id}-${request.operation.uncertaintyStamp}.json`);
+const inspect = () => runPublicationRecovery(['inspect', operationId, specPath], deps);
+const reconcile = (request, overrides = {}) => {
+ fs.writeFileSync(requestPath, JSON.stringify(request));
+ return runPublicationRecovery(['reconcile', requestPath, '--confirm-disposition'], { ...deps, ...overrides });
+};
+
+beforeEach(() => {
+ root = fs.mkdtempSync(join(tmpdir(), 'publication-recovery-'));
+ data = join(root, 'data');
+ fs.mkdirSync(join(data, 'runs', 'example-run'), { recursive: true });
+ fs.writeFileSync(join(data, 'runs/example-run/metadata.json'), JSON.stringify({ id: 'example-run', success: false, transcriptWarning: 'Synthetic missing output warning' }));
+ fs.writeFileSync(join(data, 'runs/example-run/failure.txt'), 'Synthetic publication callback failed; retained output completeness is unknown.');
+ gate = createMaintenanceAdmission(data);
+ const owner = gate.admit('settlement', 'Output publication');
+ owner.markUnsettled();
+ operationId = owner.id;
+ gate.admit('provider', 'unrelated-owner');
+ hold = gate.begin({ reason: 'Inspect synthetic publication', owner: 'Operator' }).hold;
+ deps = { data, gate, pidDead: () => true };
+ spec = { ownerId: 'toolkit-recording-publication',
+ binding: 'Synthetic run identity and failure record link this inspected recording to the selected publication.',
+ reason: 'Acknowledge this inspected failure while preserving the incomplete transcript warning.',
+ files: [
+ { path: 'runs/example-run/metadata.json', role: 'record', note: 'Inspected synthetic identity, failed outcome and retained warning.' },
+ { path: 'runs/example-run/failure.txt', role: 'failure', note: 'Inspected the synthetic callback failure; saving remains unknown.' },
+ ] };
+ specPath = join(root, 'spec.json');
+ requestPath = join(root, 'request.json');
+ statePath = join(gate.directory, 'state.json');
+ fs.writeFileSync(specPath, JSON.stringify(spec));
+});
+afterEach(() => fs.rmSync(root, { recursive: true, force: true }));
+
+it('diagnoses selected and unrelated reservations through the executable command without writing', () => {
+ const before = fs.readFileSync(statePath);
+ const result = spawnSync(process.execPath, [fileURLToPath(new URL('./reconcile-output-publication.mjs', import.meta.url)), 'diagnose', operationId, '--install-root', root],
+ { env: { ...process.env, PORTOS_DATA_ROOT: root }, encoding: 'utf8' });
+ expect(result.stderr).toBe('');
+ expect(result.status, result.stdout).toBe(0);
+ const diagnosis = JSON.parse(result.stdout);
+ expect(diagnosis.selected).toMatchObject({ id: operationId, unsettled: true, uncertaintyStamp: expect.any(String) });
+ expect(diagnosis.operations).toEqual(readState().operations);
+ expect(diagnosis.owners).toContainEqual(expect.objectContaining({ id: spec.ownerId }));
+ expect(diagnosis.nextStep).toContain('owning workflow');
+ expect(fs.readFileSync(statePath)).toEqual(before);
+});
+
+it('publishes and fsyncs evidence before retiring only the selected owner, preserving warnings and outcomes', () => {
+ const before = fs.readFileSync(statePath);
+ const originalRecords = spec.files.map(file => fs.readFileSync(join(data, file.path)));
+ const request = inspect();
+ expect(fs.readFileSync(statePath)).toEqual(before);
+ expect(request.operation).toEqual(readState().operations[0]);
+ expect(request.evidence[0]).toMatchObject({ path: spec.files[0].path, sha256: expect.stringMatching(/^[a-f0-9]{64}$/) });
+ let directorySynced = false;
+ let fileSynced = false;
+ let linked = false;
+ let inspectedUnderLock = false;
+ const io = { ...fs,
+ openSync(path, ...args) {
+ if (path.endsWith('metadata.json')) {
+ expect(fs.existsSync(join(gate.directory, 'transaction'))).toBe(true);
+ inspectedUnderLock = true;
+ }
+ return fs.openSync(path, ...args);
+ },
+ linkSync(...args) { expect(fileSynced).toBe(true); fs.linkSync(...args); linked = true; },
+ fsyncSync(fd) {
+ fs.fsyncSync(fd);
+ if (fs.fstatSync(fd).isFile()) fileSynced = true;
+ if (linked && fs.fstatSync(fd).isDirectory()) directorySynced = true;
+ },
+ unlinkSync(path) {
+ expect(directorySynced).toBe(process.platform !== 'win32');
+ expect(readState().operations).toContainEqual(request.operation);
+ expect(JSON.parse(fs.readFileSync(receiptPath(request))).request).toEqual(request);
+ fs.unlinkSync(path);
+ },
+ };
+ expect(reconcile(request, { io })).toMatchObject({ disposition: 'acknowledged-uncertainty', outputCompleteness: 'unknown', replayed: false });
+ expect(inspectedUnderLock).toBe(true);
+ expect(gate.status()).toMatchObject({ state: 'draining', hold });
+ expect(gate.operations()).toEqual([expect.objectContaining({ resource: 'unrelated-owner' })]);
+ expect(spec.files.map(file => fs.readFileSync(join(data, file.path)))).toEqual(originalRecords);
+ const receiptBytes = fs.readFileSync(receiptPath(request));
+ expect(reconcile(request).replayed).toBe(true);
+ expect(fs.readFileSync(receiptPath(request))).toEqual(receiptBytes);
+ fs.writeFileSync(statePath, before); // Disposable receipt-first interruption fixture.
+ expect(reconcile(request).replayed).toBe(false);
+ expect(fs.readFileSync(receiptPath(request))).toEqual(receiptBytes);
+});
+
+it.each(['hold', 'operation', 'stamp'])('refuses stale %s identity without publishing a receipt or retiring ownership', field => {
+ const request = inspect();
+ if (field === 'hold') request.hold.revision++;
+ if (field === 'operation') request.operation.startedAt = new Date(0).toISOString();
+ if (field === 'stamp') request.operation.uncertaintyStamp = randomUUID();
+ expect(() => reconcile(request)).toThrow(field === 'hold' ? 'hold changed' : 'reservation changed');
+ expect(gate.operations()).toHaveLength(2);
+ expect(gate.status().hold).toEqual(hold);
+ expect(fs.existsSync(receiptPath(request))).toBe(false);
+});
+
+it.each(['changed', 'missing', 'empty'])('refuses %s evidence and explains the next step', mode => {
+ const request = inspect();
+ const path = join(data, spec.files[1].path);
+ if (mode === 'missing') fs.unlinkSync(path);
+ else fs.writeFileSync(path, mode === 'empty' ? '' : 'Changed synthetic failure evidence.');
+ expect(() => reconcile(request)).toThrow(/Inspect the owning workflow records/);
+ expect(gate.operations()).toHaveLength(2);
+ expect(gate.status().state).toBe('draining');
+ expect(fs.existsSync(receiptPath(request))).toBe(false);
+});
+
+it.each(['absent', 'duplicate', 'journal', 'unbound'])('rejects insufficient %s evidence before mutation', mode => {
+ if (mode === 'absent') spec.files = [];
+ if (mode === 'duplicate') spec.files[1].path = spec.files[0].path;
+ if (mode === 'journal') spec.files[1].path = 'workflow-maintenance/state.json';
+ if (mode === 'unbound') spec.binding = '';
+ fs.writeFileSync(specPath, JSON.stringify(spec));
+ expect(() => inspect()).toThrow(/Inspect the owning workflow records/);
+ expect(gate.operations()).toHaveLength(2);
+});
+
+it('requires confirmation and refuses a live admitting process', () => {
+ const request = inspect();
+ fs.writeFileSync(requestPath, JSON.stringify(request));
+ expect(() => runPublicationRecovery(['reconcile', requestPath], deps)).toThrow('Usage:');
+ const liveDeps = { data, gate };
+ expect(() => runPublicationRecovery(['reconcile', requestPath, '--confirm-disposition'], liveDeps)).toThrow('admitting process is live');
+ expect(gate.operations()).toHaveLength(2);
+});
+
+it('never creates a receipt for missing ownership or spends an old receipt on a replacement', () => {
+ const request = inspect();
+ const state = readState();
+ state.operations[0].id = randomUUID();
+ fs.writeFileSync(statePath, JSON.stringify(state));
+ expect(() => reconcile(request)).toThrow('missing owner cannot create a receipt');
+ expect(gate.operations()).toEqual(state.operations);
+});
+
+it('refuses a malformed existing receipt without overwriting it or stranding the journal', () => {
+ const request = inspect();
+ fs.writeFileSync(receiptPath(request), '{');
+ expect(() => reconcile(request)).toThrow('No valid matching receipt');
+ expect(fs.readFileSync(receiptPath(request), 'utf8')).toBe('{');
+ expect(gate.status().state).toBe('draining');
+ expect(gate.operations()).toHaveLength(2);
+});
+
+it('rejects changed receipt bindings on replay and retains any replacement owner', () => {
+ const request = inspect();
+ reconcile(request);
+ const replacement = gate.recoverOwned('settlement', 'Output publication');
+ replacement.markUnsettled();
+ expect(reconcile(request).replayed).toBe(true);
+ expect(gate.operations()).toContainEqual(expect.objectContaining({ id: replacement.id, unsettled: true }));
+ request.reason = 'A different disposition must not reuse the original receipt.';
+ expect(() => reconcile(request)).toThrow('receipt binds different');
+ expect(gate.operations()).toHaveLength(2);
+});
+
+it('refuses recovery when an exclusive coordinator owns the hold', () => {
+ const request = inspect();
+ const state = readState();
+ fs.writeFileSync(statePath, JSON.stringify({ ...state, operations: [] }));
+ const exclusive = gate.claimReady({ id: hold.id, revision: hold.revision, operation: {
+ operationId: randomUUID(), requestId: randomUUID(), peerInstanceId: randomUUID(), hostInstanceId: randomUUID(),
+ grantId: randomUUID(), grantGeneration: 1, scope: 'execution-v1', pairBinding: 'a'.repeat(64),
+ intent: { action: 'portos.update' }, receiverVersion: '1.2.3', evidenceDigest: 'b'.repeat(64),
+ } }, gate.observeIdle());
+ fs.writeFileSync(statePath, JSON.stringify({ ...readState(), operations: state.operations }));
+ expect(() => reconcile(request)).toThrow('hold changed');
+ expect(gate.getExclusive()).toEqual(exclusive);
+ expect(gate.operations()).toEqual(state.operations);
+ expect(fs.existsSync(receiptPath(request))).toBe(false);
+});
+
+it.each(['write', 'file-sync', 'link', ...(process.platform === 'win32' ? [] : ['directory-sync'])])('preserves ownership after failed receipt %s and recovers receipt-first interruptions', stage => {
+ const request = inspect();
+ let linked = false;
+ const failure = () => { throw Object.assign(new Error('Synthetic receipt persistence failure'), { code: 'EIO' }); };
+ const io = { ...fs,
+ writeFileSync(...args) { if (stage === 'write') failure(); return fs.writeFileSync(...args); },
+ linkSync(...args) { if (stage === 'link') failure(); fs.linkSync(...args); linked = true; },
+ fsyncSync(fd) { if (stage === 'file-sync' || (stage === 'directory-sync' && linked)) failure(); return fs.fsyncSync(fd); },
+ };
+ expect(() => reconcile(request, { io })).toThrow('needs recovery');
+ expect(readState().operations).toContainEqual(request.operation);
+ expect(gate.status().state).toBe('unavailable');
+ expect(() => runPublicationRecovery(['diagnose'], deps)).toThrow('Recover its interrupted transaction');
+ // Simulate completed journal recovery ONLY in the disposable fixture.
+ fs.rmdirSync(join(gate.directory, 'transaction'));
+ gate = createMaintenanceAdmission(data);
+ deps.gate = gate;
+ expect(reconcile(request).outputCompleteness).toBe('unknown');
+ expect(gate.operations()).toEqual([expect.objectContaining({ resource: 'unrelated-owner' })]);
+});
diff --git a/scripts/reconcile-output-publication.mjs b/scripts/reconcile-output-publication.mjs
new file mode 100644
index 0000000000..6ce8182027
--- /dev/null
+++ b/scripts/reconcile-output-publication.mjs
@@ -0,0 +1,43 @@
+#!/usr/bin/env node
+import { readFileSync, statSync } from 'node:fs';
+import { isAbsolute, join } from 'node:path';
+import { parseArgs } from 'node:util';
+import { diagnoseOutputPublications, inspectOutputPublication, reconcileOutputPublication } from '../server/services/outputPublicationRecovery.js';
+import { createMaintenanceAdmission } from '../server/lib/maintenanceAdmission.js';
+import { isWorktreeRoot } from '../server/lib/dataRoot.js';
+import { PATHS } from '../server/lib/paths.js';
+import { isDirectlyInvoked } from './lib/directInvocation.js';
+
+export function runPublicationRecovery(argv, deps) {
+ const { values, positionals } = parseArgs({ args: argv, allowPositionals: true,
+ options: { 'install-root': { type: 'string' }, 'confirm-disposition': { type: 'boolean' } } });
+ const root = values['install-root'];
+ if (root !== undefined) {
+ if (!isAbsolute(root) || !statSync(join(root, 'data')).isDirectory()) throw new Error('--install-root must name an existing absolute install root containing data/.');
+ const data = join(root, 'data');
+ deps = { data, gate: createMaintenanceAdmission(data) };
+ } else if (!deps && isWorktreeRoot(PATHS.root)) {
+ throw new Error('A worktree ignores PORTOS_DATA_ROOT. Pass --install-root <absolute-install-root> explicitly, or run from the install checkout.');
+ }
+ const [action, first, second, ...extra] = positionals;
+ const confirmed = values['confirm-disposition'] === true;
+ if (action === 'diagnose' && second === undefined && !confirmed) return diagnoseOutputPublications(first, deps);
+ if (action === 'inspect' && first && second && extra.length === 0 && !confirmed)
+ return inspectOutputPublication(first, JSON.parse(readFileSync(second, 'utf8')), deps);
+ if (action === 'reconcile' && first && second === undefined && confirmed) {
+ const receipt = reconcileOutputPublication(JSON.parse(readFileSync(first, 'utf8')), { ...deps, confirmDisposition: true });
+ return { disposition: receipt.disposition, operationId: receipt.request.operation.id,
+ uncertaintyStamp: receipt.request.operation.uncertaintyStamp, outputCompleteness: receipt.outputCompleteness,
+ replayed: receipt.replayed ?? false };
+ }
+ throw new Error('Usage: node scripts/reconcile-output-publication.mjs diagnose [operation-id] | inspect <operation-id> <evidence-spec.json> | reconcile <inspected-request.json> --confirm-disposition [--install-root <absolute-install-root>]');
+}
+
+if (isDirectlyInvoked(import.meta.url)) {
+ try { console.log(JSON.stringify(runPublicationRecovery(process.argv.slice(2)), null, 2)); }
+ catch (error) {
+ const log = error.code === 'MAINTENANCE_STALE' ? console.log : console.error;
+ log(`${error.code === 'MAINTENANCE_STALE' ? '🚫' : '❌'} ${error.message}`);
+ process.exitCode = 1;
+ }
+}
diff --git a/server/services/outputPublicationRecovery.js b/server/services/outputPublicationRecovery.js
new file mode 100644
index 0000000000..84c45b8cb3
--- /dev/null
+++ b/server/services/outputPublicationRecovery.js
@@ -0,0 +1,161 @@
+/** Local operator disposition of legacy publications with no destination identity. */
+import * as fs from 'node:fs';
+import { createHash, randomUUID } from 'node:crypto';
+import { join, resolve } from 'node:path';
+import { isDeepStrictEqual } from 'node:util';
+import { z } from 'zod';
+import { maintenance } from '../lib/maintenanceAdmission.js';
+import { BACKUP_ASSET_OWNERS } from '../lib/backupAssetOwners.js';
+import { PATHS } from '../lib/paths.js';
+
+const nextStep = 'Inspect the owning workflow records and output/failure evidence; run inspect again after any change. Keep the maintenance hold and original warnings/outcomes.';
+const refuse = message => { throw Object.assign(new Error(`${message} ${nextStep}`), { code: 'MAINTENANCE_STALE' }); };
+const owners = BACKUP_ASSET_OWNERS.filter(owner => owner.status === 'admitted');
+const MAX_EVIDENCE_BYTES = 16 * 1024 * 1024;
+const operationSchema = z.object({
+ id: z.string().uuid(), kind: z.literal('settlement'), resource: z.literal('Output publication'),
+ pid: z.number().int().positive(), startedAt: z.string().datetime(), unsettled: z.literal(true),
+ uncertaintyStamp: z.string().uuid(),
+}).strict();
+const holdSchema = z.object({ id: z.string().uuid(), revision: z.number().int().positive() }).strict();
+const fileSchema = z.object({
+ path: z.string().min(1).max(1024), role: z.enum(['record', 'output', 'failure']),
+ note: z.string().trim().min(10).max(1000),
+}).strict();
+const evidenceSchema = fileSchema.extend({ bytes: z.number().int().nonnegative(), sha256: z.string().regex(/^[a-f0-9]{64}$/) });
+const inspectionFields = {
+ ownerId: z.string().refine(id => owners.some(owner => owner.id === id), 'Choose an admitted owner from diagnose.'),
+ binding: z.string().trim().min(20).max(2000), reason: z.string().trim().min(10).max(1000),
+};
+const inspectionSchema = z.object({ ...inspectionFields, files: z.array(fileSchema).min(2).max(16) }).strict();
+const requestSchema = z.object({
+ version: z.literal(1), hold: holdSchema, operation: operationSchema, ...inspectionFields,
+ evidence: z.array(evidenceSchema).min(2).max(16),
+}).strict();
+const receiptSchema = z.object({
+ version: z.literal(1), disposition: z.literal('acknowledged-uncertainty'),
+ outputCompleteness: z.literal('unknown'), at: z.string().datetime(), request: requestSchema,
+}).strict();
+const parse = (schema, input) => {
+ const result = schema.safeParse(input);
+ if (!result.success) refuse(`Invalid recovery evidence: ${result.error.message}`);
+ return result.data;
+};
+const isDead = pid => {
+ try { process.kill(pid, 0); return false; } catch (error) { return error.code === 'ESRCH'; }
+};
+
+function inventory(gate) {
+ const status = gate.status();
+ if (status.state === 'unavailable')
+ refuse('Maintenance journal is unavailable. Recover its interrupted transaction through the journal recovery procedure first; this command never removes its lock.');
+ return { status, operations: gate.operations() };
+}
+
+export function diagnoseOutputPublications(operationId, { gate = maintenance } = {}) {
+ if (operationId !== undefined) parse(z.string().uuid(), operationId);
+ const { status, operations } = inventory(gate);
+ const selected = operationId === undefined ? null : operations.find(op => op.id === operationId);
+ if (operationId !== undefined && !selected) refuse('Selected reservation is missing. Diagnose the current inventory; only an existing matching receipt may replay it.');
+ return { version: 1, maintenance: status, operations, selected, nextStep,
+ owners: owners.map(({ id, modules }) => ({ id, modules })) };
+}
+
+function fingerprint(file, data, io) {
+ const parts = file.path.split('/');
+ if (file.path.includes('\\') || parts.some(part => !part || part === '.' || part === '..')
+ || parts[0] === 'workflow-maintenance' || parts[0].includes(':'))
+ refuse('Evidence must name ordinary data-relative files outside workflow-maintenance.');
+ let fd;
+ try {
+ const path = join(io.realpathSync(data), ...parts);
+ if (io.realpathSync(path) !== resolve(path)) refuse('Evidence symlinks are not accepted.');
+ fd = io.openSync(path, fs.constants.O_RDONLY | (fs.constants.O_NOFOLLOW ?? 0));
+ const before = io.fstatSync(fd);
+ if (!before.isFile() || before.size > MAX_EVIDENCE_BYTES) refuse('Evidence must be regular files, at most 16 MiB in total. Preserve larger artifacts and use bounded owning-service evidence.');
+ const raw = io.readFileSync(fd);
+ const after = io.fstatSync(fd);
+ const current = io.lstatSync(path);
+ if (raw.length !== before.size || ['dev', 'ino', 'size', 'mtimeMs', 'ctimeMs'].some(key => before[key] !== after[key] || after[key] !== current[key]))
+ refuse('Evidence changed while being inspected.');
+ return { ...file, bytes: raw.length, sha256: createHash('sha256').update(raw).digest('hex') };
+ } catch (error) {
+ if (error.code === 'MAINTENANCE_STALE') throw error;
+ refuse(`Evidence is missing or unreadable (${error.code ?? 'read failed'}).`);
+ } finally { if (fd !== undefined) io.closeSync(fd); }
+}
+
+function inspectFiles(files, data, io) {
+ if (new Set(files.map(file => file.path)).size !== files.length) refuse('Evidence paths must be distinct.');
+ const evidence = files.map(file => fingerprint(file, data, io));
+ if (evidence.reduce((sum, file) => sum + file.bytes, 0) > MAX_EVIDENCE_BYTES) refuse('Evidence exceeds 16 MiB; preserve the originals and use bounded owning-service evidence.');
+ if (!evidence.some(file => file.role === 'record' && file.bytes > 0)
+ || !evidence.some(file => ['output', 'failure'].includes(file.role) && file.bytes > 0))
+ refuse('Require a nonempty owner record and nonempty output or failure evidence. Empty output is not proof of saving.');
+ return evidence;
+}
+
+export function inspectOutputPublication(operationId, input, { gate = maintenance, data = PATHS.data, io = fs } = {}) {
+ const spec = parse(inspectionSchema, input);
+ const { maintenance: status, selected } = diagnoseOutputPublications(operationId, { gate });
+ if (!status.hold) refuse('Begin a normal maintenance hold before inspection.');
+ const operation = parse(operationSchema, selected);
+ const evidence = inspectFiles(spec.files, data, io);
+ return { version: 1, hold: { id: status.hold.id, revision: status.hold.revision },
+ operation: { ...selected, ...operation }, ownerId: spec.ownerId, binding: spec.binding, reason: spec.reason, evidence };
+}
+
+function syncDirectory(path, io) {
+ if (process.platform === 'win32') return;
+ const fd = io.openSync(path, 'r');
+ try { io.fsyncSync(fd); } finally { io.closeSync(fd); }
+}
+
+export function reconcileOutputPublication(input, { confirmDisposition = false, gate = maintenance,
+ data = PATHS.data, io = fs, pidDead = isDead } = {}) {
+ if (confirmDisposition !== true) refuse('Explicit --confirm-disposition is required; output completeness will remain unknown.');
+ const request = parse(requestSchema, input);
+ // The admission capability compares the original serialized operation, including key order.
+ request.operation = { ...input.operation };
+ const receiptPath = join(gate.directory, `publication-${request.operation.id}-${request.operation.uncertaintyStamp}.json`);
+ const verify = () => {
+ if (!pidDead(request.operation.pid)) refuse('The admitting process is live or its exit is unproved. Recover the owning workflow before disposition.');
+ const evidence = inspectFiles(request.evidence.map(({ path, role, note }) => ({ path, role, note })), data, io);
+ if (!isDeepStrictEqual(evidence, request.evidence)) refuse('Evidence changed since inspection.');
+ };
+ const readReceipt = () => {
+ let receipt;
+ try { receipt = parse(receiptSchema, JSON.parse(io.readFileSync(receiptPath, 'utf8'))); }
+ catch { refuse('No valid matching receipt exists. Preserve existing evidence and diagnose the reservation; a missing owner cannot create a receipt.'); }
+ if (!isDeepStrictEqual(receipt.request, request)) refuse('The existing receipt binds different evidence, reason, hold, or operation.');
+ // A receipt-first interrupted write may have linked the file before directory fsync.
+ const fd = io.openSync(receiptPath, 'r+');
+ try { io.fsyncSync(fd); } finally { io.closeSync(fd); }
+ syncDirectory(gate.directory, io);
+ return receipt;
+ };
+ try { return gate.reconcilePublicationRefusal({ hold: request.hold, expected: request.operation,
+ replay: () => { verify(); return { ...readReceipt(), replayed: true }; },
+ publish: () => {
+ verify();
+ try { io.lstatSync(receiptPath); return readReceipt(); }
+ catch (error) { if (error.code !== 'ENOENT') throw error; }
+ const receipt = { version: 1, disposition: 'acknowledged-uncertainty', outputCompleteness: 'unknown',
+ at: new Date().toISOString(), request };
+ const pending = `${receiptPath}.${randomUUID()}.pending`;
+ const fd = io.openSync(pending, 'wx', 0o600);
+ try { io.writeFileSync(fd, JSON.stringify(receipt) + '\n'); io.fsyncSync(fd); } finally { io.closeSync(fd); }
+ // No replacement: an existing evidence receipt is authority, never scratch.
+ io.linkSync(pending, receiptPath);
+ syncDirectory(gate.directory, io);
+ io.unlinkSync(pending);
+ syncDirectory(gate.directory, io);
+ return receipt;
+ },
+ }); } catch (error) {
+ if (error.code === 'MAINTENANCE_STALE' && !error.message.endsWith(nextStep)) refuse(error.message);
+ if (error.code === 'MAINTENANCE_UNAVAILABLE')
+ throw Object.assign(new Error('Maintenance state needs recovery; ownership remains held. Recover the interrupted journal transaction before replaying this exact request; preserve its receipt and pending evidence.'), { code: error.code });
+ throw error;
+ }
+}