Beautiful, AI-native markdown IDE and LLM wiki
TypeScript 2nd-brainagent-skillsclaudecodexcompany-braindocsknowledge-knowledge-baseknowledge-managementllmllm-wikillm-wiki-karpathymarkdownmarkdown-editormdmdxnotespkmsecond-brainskills
PRD-9724 Compare every Markdown write against a durable receipt (#7120)
mike-inkeep5a435ff9c17b
diff --git a/.changeset/compare-before-every-save.md b/.changeset/compare-before-every-save.mdnew file mode 100644index 000000000..c8f85b5cc--- /dev/null+++ b/.changeset/compare-before-every-save.md@@ -0,0 +1,5 @@+---+'@inkeep/open-knowledge': patch+---++Saving a Markdown document in the app, by an agent, while idle or at quit now checks the file first and merges an outside edit instead of writing over it. Agent writes over a changed file are merged instead of refused with `urn:ok:error:disk-divergence`, which is retired.diff --git a/docs/content/reference/mcp.mdx b/docs/content/reference/mcp.mdxindex ad14e7567..c18404f48 100644--- a/docs/content/reference/mcp.mdx+++ b/docs/content/reference/mcp.mdx@@ -118,6 +118,8 @@ A doc can land in conflict through a Git merge, a pull that collides with a locaA write that first detects the stale file during its final disk flush returns a different 409, `urn:ok:error:stale-external-write`. The edit reached collaborative state and the recovery snapshot, but the disk write did not happen. Ask the user to resolve the conflict, then re-read the document. Do not repeat the operation: compare the resolved content with the intended result and make only any remaining edit.+When OpenKnowledge saves a Markdown document, it first compares the file with the version it last agreed with the disk. If another tool changed the file, that change is merged into the document before the save, so the write succeeds and the file holds both edits; re-read the document before editing it further. This covers saves of edits made in the app, idle saves, the save at quit, the in-app agent's whole-document write and the agent write routes. Other writers, such as rename, delete, templates, skills, and Mermaid, Excalidraw and plain-text documents, do not run this comparison yet. The stale-external-write case above is the exception: when the file holds a version that an earlier agent write replaced, the write returns that 409 instead. If the save cannot finish safely, the write returns `urn:ok:error:save-held` (503). The body carries `reason` (what holds the save), `retryWhen` (what the server's own save waits for; it is never a cue to repeat the request) and `nextStep: re-read`; in a batch response the same fields appear on that entry's `error`. When the file cannot be read (`pending-unreadable`, with the errno in `causeCode`), keeps changing (`pending-exhausted`, `pending-base-moved`) or the outside change could not be merged (`pending-intake-failed`), the next edit or the idle save within a few minutes saves it (`retryWhen: next-save`). When OpenKnowledge has no agreed version of the file to merge with (`pending-intake-failed-no-base`, or any hold on such a document), the edit is not saved: the next edit to the document replaces it with the file on disk and moves the held edit to the document history (`retryWhen: replaced-on-next-edit`); re-read after that and re-apply the edit if it is still needed. When merging raised a conflict (`pending-conflict`), it is saved after the user resolves the conflict in the app (`conflict-resolved`). In every case, do not repeat the write now: re-read the document before editing it further.+`conflict.kind` says where the conflict came from, and the kind decides what a resolve does. `merge-native` is a git merge that left unmerged stages in the index. Resolving one stages your choice with git and commits the merge once no `merge-native` conflict is left. `working-tree` is a pull-only overlay collision, pinned to the origin blob it hit. `reconcile` is OpenKnowledge's own three-way merge of the editor against disk, including protected stale saves; no git object holds those stages, so OpenKnowledge snapshots them when it detects the conflict. Resolving a `working-tree` or `reconcile` conflict commits nothing.A `reconcile` conflict also carries a `reason`: `refused-conflict-markers`, `refused-no-base`, `refused-too-large`, `disk-markers`, or `stale-external-write`; `merged-with-markers` appears only on entries recorded by earlier versions, or when both kept versions could not be applied to the open document. On `stale-external-write` the compatibility field `conflictKind` is `stale-external-write`. Every other row uses `conflictKind: "git"`. An outside edit to the same block as unsaved edits is not a conflict: both versions are kept in the document. `refused-too-large` means a merge past the size cap or one that failed OpenKnowledge's fidelity check, and is raised only when the outside version could not be saved to history; otherwise the OpenKnowledge version stays and the outside version goes to the document's history. The server log names the failed check.diff --git a/docs/content/reference/what-open-knowledge-writes.mdx b/docs/content/reference/what-open-knowledge-writes.mdxindex 4588142a9..8d0140dc7 100644--- a/docs/content/reference/what-open-knowledge-writes.mdx+++ b/docs/content/reference/what-open-knowledge-writes.mdx@@ -154,6 +154,7 @@ The app runs a collaboration server for the open project. It writes runtime stat| `.ok/local/conflicts.json` | Every conflict the app is holding a document for, including conflicts that involve no remote. A row for OpenKnowledge's own merge of the editor against disk embeds up to three full copies of the document body, so **keep this file while any conflict is open**. Deleting it loses those copies. If the app cannot read the file, it renames it to `conflicts.json.corrupt-<timestamp>` and leaves it in place | In-project, gitignored || `.ok/local/stale-external-writes.json` | Recovery snapshot for a rejected stale external save. It can hold the acknowledged, rejected, and retained document text. Protection expires after 30 minutes, but an unresolved snapshot stays until you resolve the conflict | In-project, gitignored || `.ok/local/doc-notices.json` | The notice a document shows after both versions of an edit were kept, or a version was saved to the timeline instead of merged: document names, times, and timeline entry ids, never document text. A notice is removed when you dismiss it, when its document is renamed or deleted, when the version it names leaves the timeline, or when more than 200 newer notices push it out, and each branch keeps its own. If the app cannot read the file, it starts with no notices | In-project, gitignored |+| `.ok/local/receipts.jsonl` | One record per document OpenKnowledge has opened or saved: a hash of the version it and the file last agreed on, its size, and the file's identity and modification time. No document text. OpenKnowledge keeps the same record in memory while it runs, and a save of a Markdown document compares the file with that record first, so an outside edit is merged instead of written over; other writers do not check it yet. After a restart the file is only used to count documents changed while OpenKnowledge was closed. Safe to delete: the next load rebuilds it | In-project, gitignored || `.ok/local/state.json`, `.ok/local/last-spawn-error.log` | State-schema manifest and the last detached-spawn error log | In-project, gitignored || `.ok/local/acp-agents.json` | Read, not written: your own list of [custom in-app agents](../features/editor.mdx#add-your-own-in-app-agent), re-read every time the agent catalog is built | In-project, gitignored || `.ok/local/acp-launch-failures.log` | In-app agent launches that failed: when, which agent, why, and the last lines the agent printed before it exited, so a bug report shows the cause without a re-run. Starts over once it passes 256 KB | In-project, gitignored |@@ -297,6 +298,7 @@ On Windows, Claude Desktop and OpenCode configs live under `%APPDATA%\Claude\` a| `.ok/local/conflicts.json` | Every conflict the app is holding a document for, including conflicts that involve no remote. A row for OpenKnowledge's own merge of the editor against disk embeds up to three full copies of the document body, so **keep this file while any conflict is open**. Deleting it loses those copies. If the app cannot read the file, it renames it to `conflicts.json.corrupt-<timestamp>` and leaves it in place | In-project, gitignored || `.ok/local/stale-external-writes.json` | Recovery snapshot for a rejected stale external save. It can hold the acknowledged, rejected, and retained document text. Protection expires after 30 minutes, but an unresolved snapshot stays until you resolve the conflict | In-project, gitignored || `.ok/local/doc-notices.json` | The notice a document shows after both versions of an edit were kept, or a version was saved to the timeline instead of merged: document names, times, and timeline entry ids, never document text. A notice is removed when you dismiss it, when its document is renamed or deleted, when the version it names leaves the timeline, or when more than 200 newer notices push it out, and each branch keeps its own. If the app cannot read the file, it starts with no notices | In-project, gitignored |+| `.ok/local/receipts.jsonl` | One record per document OpenKnowledge has opened or saved: a hash of the version it and the file last agreed on, its size, and the file's identity and modification time. No document text. OpenKnowledge keeps the same record in memory while it runs, and a save of a Markdown document compares the file with that record first, so an outside edit is merged instead of written over; other writers do not check it yet. After a restart the file is only used to count documents changed while OpenKnowledge was closed. Safe to delete: the next load rebuilds it | In-project, gitignored || `.ok/local/state.json`, `.ok/local/last-spawn-error.log` | State-schema manifest and the last detached-spawn error log | In-project, gitignored || `.ok/local/acp-agents.json` | Read, not written: your own list of [custom in-app agents](../features/editor.mdx#add-your-own-in-app-agent), re-read every time the agent catalog is built | In-project, gitignored || `.ok/local/acp-launch-failures.log` | In-app agent launches that failed: when, which agent, why, and the last lines the agent printed before it exited, so a bug report shows the cause without a re-run. Starts over once it passes 256 KB | In-project, gitignored |diff --git a/packages/app/tests/foundation/fresh-slot.node.test.ts b/packages/app/tests/foundation/fresh-slot.node.test.tsindex 9aaf3d1e2..89c6a1a9f 100644--- a/packages/app/tests/foundation/fresh-slot.node.test.ts+++ b/packages/app/tests/foundation/fresh-slot.node.test.ts@@ -35,6 +35,7 @@ function freshSlot(): string {write(root, '.ok/local/state.json', '{}');write(root, '.ok/local/acp-registry-cache.json', '{}');write(root, '.ok/local/stale-external-writes.json', '{}');+ write(root, '.ok/local/receipts.jsonl', '');write(root, '.ok/local/removed-docs.json', '{}');write(root, '.git/HEAD', 'ref: refs/heads/main\n');return root;diff --git a/packages/app/tests/integration/disk-divergence-backstop.test.ts b/packages/app/tests/integration/disk-divergence-backstop.test.tsindex 819dd50dd..b2b8bd44b 100644--- a/packages/app/tests/integration/disk-divergence-backstop.test.ts+++ b/packages/app/tests/integration/disk-divergence-backstop.test.ts@@ -1,4 +1,5 @@import { randomUUID } from 'node:crypto';+import { getMetrics } from '@inkeep/open-knowledge-server';import { afterEach, describe, expect, test } from 'vitest';import {agentWriteMd,@@ -49,8 +50,8 @@ async function agentUndoRaw(});}-describe('PRD-6832 β L3: store-time divergence backstop', () => {- test('reverts on TOCTOU divergence (409); disk wins; retry re-applies exactly once', async () => {+describe('store-time compare: an outside write between the pre-write merge and the store', () => {+ test('is merged with the agent write instead of refusing it, and the agent line lands once', async () => {server = await createTestServer({ debounce: 50, maxDebounce: 200 });const { port, contentDir } = server;const docName = `l3-content-${randomUUID()}`;@@ -58,35 +59,24 @@ describe('PRD-6832 β L3: store-time divergence backstop', () => {const seed = await writeMd(port, '# V1\n\nbody-v1\n', { docName, position: 'replace' });expect(seed.status).toBe(200);await pollUntil(() => readTestDoc(contentDir, docName).includes('body-v1'));+ const mergesBefore = getMetrics().storeDiskCompare.changed ?? 0;process.env.OK_TEST_STORE_DIVERGENCE = docName;-- const attempt1 = await writeMd(port, 'AGENT-APPEND-XYZ\n', {- docName,- position: 'append',- });- expect(attempt1.status).toBe(409);- const body1 = (await attempt1.json()) as { type?: string };- expect(body1.type).toBe('urn:ok:error:disk-divergence');-- const afterRevert = readTestDoc(contentDir, docName);- expect(afterRevert).toContain(INJECTED_MARKER);- expect(afterRevert).not.toContain('AGENT-APPEND-XYZ');-+ const attempt = await writeMd(port, 'AGENT-APPEND-XYZ\n', { docName, position: 'append' });+ expect(attempt.status).toBe(200);delete process.env.OK_TEST_STORE_DIVERGENCE;- const attempt2 = await writeMd(port, 'AGENT-APPEND-XYZ\n', {- docName,- position: 'append',- });- expect(attempt2.status).toBe(200);- const afterRetry = readTestDoc(contentDir, docName);- expect(afterRetry).toContain(INJECTED_MARKER);- expect(afterRetry).toContain('AGENT-APPEND-XYZ');- expect(afterRetry.split('AGENT-APPEND-XYZ').length - 1).toBe(1);+ const merged = '# NATIVE\n\nnative-divergence-injected\n\nAGENT-APPEND-XYZ\n';+ await pollUntil(() => readTestDoc(contentDir, docName) === merged);+ expect(server.instance.hocuspocus.documents.get(docName)?.getText('source').toString()).toBe(+ merged,+ );+ expect(merged.split('AGENT-APPEND-XYZ')).toHaveLength(2);+ expect(getMetrics().storeDiskCompare.changed ?? 0).toBeGreaterThan(mergesBefore);+ expect(server.instance.conflicts.findByDocName(docName)).toBeUndefined();});- test('undo: L3 reverts on TOCTOU divergence (409); native survives; undo NOT applied', async () => {+ test('an agent undo whose store meets an outside write merges it and the undo lands', async () => {server = await createTestServer({ debounce: 50, maxDebounce: 200 });const { port, contentDir } = server;const docName = `l3-undo-${randomUUID()}`;@@ -102,25 +92,17 @@ describe('PRD-6832 β L3: store-time divergence backstop', () => {await pollUntil(() => readTestDoc(contentDir, docName).includes('UNDO-ME-LINE'));process.env.OK_TEST_STORE_DIVERGENCE = docName;-const undoRes = await agentUndoRaw(port, { docName, connectionId: 'agent-u1', scope: 'last' });- expect(undoRes.status).toBe(409);- const body = (await undoRes.json()) as { type?: string };- expect(body.type).toBe('urn:ok:error:disk-divergence');-- const afterRevert = readTestDoc(contentDir, docName);- expect(afterRevert).toContain(INJECTED_MARKER);- expect(afterRevert).not.toContain('base-body');-+ expect(undoRes.status).toBe(200);delete process.env.OK_TEST_STORE_DIVERGENCE;- await agentWriteMd(port, 'RECOVERY-LINE\n', { docName, position: 'append', agentId: 'u1' });- await pollUntil(() => readTestDoc(contentDir, docName).includes('RECOVERY-LINE'));- const afterRecovery = readTestDoc(contentDir, docName);- expect(afterRecovery).toContain(INJECTED_MARKER);- expect(afterRecovery).toContain('RECOVERY-LINE');++ const merged = '# NATIVE\n\nnative-divergence-injected\n';+ await pollUntil(() => readTestDoc(contentDir, docName) === merged);+ expect(readTestDoc(contentDir, docName)).not.toContain('UNDO-ME-LINE');+ expect(server.instance.conflicts.findByDocName(docName)).toBeUndefined();});- test('gate: an unmarked human/client store is NEVER reverted by L3', async () => {+ test('a person store over the same outside write keeps the person edit', async () => {server = await createTestServer({ debounce: 50, maxDebounce: 200 });const { port, contentDir } = server;const docName = `l3-gate-human-${randomUUID()}`;@@ -142,6 +124,7 @@ describe('PRD-6832 β L3: store-time divergence backstop', () => {await awaitDocQuiescence(client.doc, { timeoutMs: 3000 });await pollUntil(() => readTestDoc(contentDir, docName).includes(HUMAN_MARK), 8000);+ expect(readTestDoc(contentDir, docName)).toContain(INJECTED_MARKER);} finally {delete process.env.OK_TEST_STORE_DIVERGENCE;await client.cleanup();diff --git a/packages/app/tests/integration/lint-fix-live-preservation.test.ts b/packages/app/tests/integration/lint-fix-live-preservation.test.tsindex 5df57185e..233347108 100644--- a/packages/app/tests/integration/lint-fix-live-preservation.test.ts+++ b/packages/app/tests/integration/lint-fix-live-preservation.test.ts@@ -434,7 +434,7 @@ describe('lint fix live-write preservation', () => {},);- test('a disk divergence returns 409 and realigns the live document to the injected disk state', async () => {+ test('an outside write at the lint fix store keeps both versions instead of refusing the fix', async () => {server = await createTestServer({ markdownlintEnabled: true });const docName = `lint-divergence-${randomUUID()}`;const file = join(server.contentDir, `${docName}.md`);@@ -448,27 +448,26 @@ describe('lint fix live-write preservation', () => {agentName: 'Divergence Agent',summary: 'Divergent durability',});- expect(response.status).toBe(409);- expect(await response.json()).toMatchObject({ type: 'urn:ok:error:disk-divergence' });+ expect(response.status).toBe(200);+ delete process.env.OK_TEST_STORE_DIVERGENCE;await pollUntil(() => readFileSync(file, 'utf-8').includes('native-divergence-injected'));- await pollUntil(() =>- Boolean(- getServerState(server as TestServer, docName)- ?.ytext.toString()- .includes('native-divergence-injected'),- ),+ await pollUntil(+ () =>+ getServerState(server as TestServer, docName)?.ytext.toString() ===+ readFileSync(file, 'utf-8'),);- const state = getServerState(server, docName);- if (state === null) throw new Error('Expected a realigned server document');- expect(state.ytext.toString()).toBe(readFileSync(file, 'utf-8'));- expect(state.ytext.toString()).toBe('# NATIVE\n\nnative-divergence-injected\n');+ expect(readFileSync(file, 'utf-8')).toBe(+ '# NATIVE\n\n indented with a hard tab\n\nnative-divergence-injected\n',+ );+ expect(server.instance.conflicts.findByDocName(docName)).toBeUndefined();expect(contributors()).toEqual([expect.objectContaining({id: 'agent-lint-divergence',docs: [docName],summaries: ['Divergent durability'],}),+ expect.objectContaining({ id: 'file-system', docs: [docName] }),]);expect(server.instance.agentPresenceBroadcaster.getPresenceMap()['agent-lint-divergence'],diff --git a/packages/app/tests/integration/older-own-write-echo.test.ts b/packages/app/tests/integration/older-own-write-echo.test.tsindex 3c1dfe863..c47352277 100644--- a/packages/app/tests/integration/older-own-write-echo.test.ts+++ b/packages/app/tests/integration/older-own-write-echo.test.ts@@ -221,6 +221,27 @@ describe('older own-write observations', () => {);});+ test('restoring the acknowledged bytes after an external rollback clears the stale-write record with the conflict', async () => {+ const { server: active, docName, file, post, observe } = await startOrderedServer();+ expect((await post('Before\n')).status).toBe(200);+ expect((await post('After\n')).status).toBe(200);+ await observe();+ writeFileSync(file, 'Before\n');+ await observe();+ expect(active.instance.conflicts.findByDocName(docName)).toMatchObject({+ reason: 'stale-external-write',+ });+ expect(active.instance.durabilityState.getStaleExternalWrite(docName)).toBeDefined();++ writeFileSync(file, 'Before\n\nAfter\n');+ await observe();++ expect(active.instance.conflicts.findByDocName(docName)).toBeUndefined();+ expect(active.instance.durabilityState.getStaleExternalWrite(docName)).toBeUndefined();+ expect((await post('Final\n')).status).toBe(200);+ expect(readFileSync(file, 'utf8')).toBe('Before\n\nAfter\n\nFinal\n');+ });+test('a newer external edit still reconciles before the next write', async () => {const { server: active, docName, file, post, observe } = await startOrderedServer();expect((await post('Before\n')).status).toBe(200);diff --git a/packages/app/tests/integration/persistence-divergence-realign.test.ts b/packages/app/tests/integration/persistence-divergence-realign.test.tsdeleted file mode 100644index c758fd2f4..000000000--- a/packages/app/tests/integration/persistence-divergence-realign.test.ts+++ /dev/null@@ -1,192 +0,0 @@-import { execFileSync } from 'node:child_process';-import { randomUUID } from 'node:crypto';-import { existsSync, readFileSync } from 'node:fs';-import { join } from 'node:path';-import { parseCheckpoint } from '@inkeep/open-knowledge-core';-import { afterEach, describe, expect, test } from 'vitest';-import { HARNESS_BOOT_TIMEOUT_MS } from './harness-boot-timeout';-import { createTestClient, createTestServer, type TestServer } from './test-harness.ts';--const INJECTED_MARKER = 'native-divergence-injected';--const HUMAN_PENDING_LINE = 'Zzz human paragraph typed before the realign.';--const AGENT_LINE = 'AGENT-APPEND-XYZ';--interface LossRingEvent {- event: string;- docName: string;- site?: string;- direction?: string;- writerId?: string | null;- lostLen?: number;- digest?: string;- checkpointSha?: string;-}--let server: TestServer | undefined;--afterEach(async () => {- delete process.env.OK_TEST_STORE_DIVERGENCE;- if (server) {- await server.cleanup();- server = undefined;- }-});--function readLossEvents(contentDir: string): LossRingEvent[] {- const path = join(contentDir, '.ok', 'local', 'loss-capture', 'loss-current.jsonl');- if (!existsSync(path)) return [];- return readFileSync(path, 'utf-8')- .split('\n')- .filter((line) => line.length > 0)- .flatMap((line) => {- try {- return [JSON.parse(line) as LossRingEvent];- } catch {- return [];- }- });-}--function shadowDirFor(s: TestServer): string {- const dir = join(s.contentDir, '.git', 'ok');- if (!existsSync(dir)) {- throw new Error(`shadow repo not found at ${dir}; the rig needs gitEnabled: true`);- }- return dir;-}--function shadowGitRaw(s: TestServer, args: string[]): string {- return execFileSync('git', ['--git-dir', shadowDirFor(s), ...args], {- encoding: 'utf-8',- }).toString();-}--function listCheckpoints(s: TestServer): Array<{ sha: string; kind: string | null }> {- const shas = shadowGitRaw(s, ['for-each-ref', '--format=%(objectname)', 'refs/checkpoints'])- .split('\n')- .map((x) => x.trim())- .filter(Boolean);- return shas.map((sha) => ({- sha,- kind: parseCheckpoint(shadowGitRaw(s, ['log', '-1', '--format=%B', sha]))?.kind ?? null,- }));-}--async function agentWriteMd(- port: number,- markdown: string,- opts: { docName: string; position: 'append' | 'prepend' | 'replace' },-): Promise<Response> {- return fetch(`http://127.0.0.1:${port}/api/agent-write-md`, {- method: 'POST',- headers: { 'Content-Type': 'application/json' },- body: JSON.stringify({ markdown, ...opts }),- });-}--async function pollUntil(- predicate: () => boolean,- { timeoutMs = 10_000, pollMs = 50 }: { timeoutMs?: number; pollMs?: number } = {},-): Promise<void> {- const deadline = Date.now() + timeoutMs;- while (Date.now() < deadline) {- if (predicate()) return;- await new Promise((r) => setTimeout(r, pollMs));- }- throw new Error(`pollUntil timed out after ${timeoutMs}ms`);-}--describe('L3 agent-write divergence realign — content-loss conventions', () => {- test(- 'the realign checkpoints the live document and reports the loss before discarding it',- async () => {- server = await createTestServer({- gitEnabled: true,- debounce: 300_000,- maxDebounce: 600_000,- });- const { port, contentDir } = server;- const docName = `realign-${randomUUID().slice(0, 8)}`;- const docPath = join(contentDir, `${docName}.md`);-- const seed = await agentWriteMd(port, '# V1\n\nbody-v1\n', { docName, position: 'replace' });- expect(seed.status).toBe(200);- await pollUntil(- () => existsSync(docPath) && readFileSync(docPath, 'utf-8').includes('body-v1'),- );-- const client = await createTestClient(port, docName, { skipInvariantWatcher: true });- await pollUntil(() => client.ytext.toString().includes('body-v1'));- const before = client.ytext.toString();- client.doc.transact(() => {- client.ytext.insert(before.length, `\n${HUMAN_PENDING_LINE}\n`);- });- await pollUntil(() =>- Boolean(- server?.instance.hocuspocus.documents- .get(docName)- ?.getText('source')- .toString()- .includes(HUMAN_PENDING_LINE),- ),- );- expect(readFileSync(docPath, 'utf-8')).not.toContain(HUMAN_PENDING_LINE);-- process.env.OK_TEST_STORE_DIVERGENCE = docName;-- const attempt = await agentWriteMd(port, `${AGENT_LINE}\n`, { docName, position: 'append' });- expect(attempt.status).toBe(409);- expect(((await attempt.json()) as { type?: string }).type).toBe(- 'urn:ok:error:disk-divergence',- );-- await pollUntil(() => readFileSync(docPath, 'utf-8').includes(INJECTED_MARKER));- expect(readFileSync(docPath, 'utf-8')).not.toContain(AGENT_LINE);-- await pollUntil(- () =>- !server?.instance.hocuspocus.documents- .get(docName)- ?.getText('source')- .toString()- .includes(HUMAN_PENDING_LINE),- );-- const rig = server;- await pollUntil(() =>- listCheckpoints(rig).some((c) => c.kind === 'persistence-divergence-realign'),- );- const anchors = listCheckpoints(rig).filter(- (c) => c.kind === 'persistence-divergence-realign',- );- expect(anchors).toHaveLength(1);- const anchor = anchors[0];- if (!anchor) throw new Error('unreachable: anchor asserted above');-- const blob = shadowGitRaw(rig, ['show', `${anchor.sha}:${docName}`]);- expect(blob).toContain(HUMAN_PENDING_LINE);-- const ring = readLossEvents(contentDir).filter(- (e) => e.site === 'persistence-divergence-realign' && e.docName === docName,- );- const trips = ring.filter((e) => e.event === 'detector-trip');- expect(trips).toHaveLength(1);- expect(trips[0]?.direction).toBe('b');- expect(trips[0]?.lostLen ?? 0).toBeGreaterThanOrEqual(HUMAN_PENDING_LINE.length);- expect(JSON.stringify(trips[0])).not.toContain(HUMAN_PENDING_LINE);- await pollUntil(() =>- readLossEvents(contentDir).some(- (e) =>- e.site === 'persistence-divergence-realign' &&- e.event === 'checkpoint-write' &&- e.checkpointSha === anchor.sha,- ),- );-- await client.cleanup();- },- HARNESS_BOOT_TIMEOUT_MS + 60_000,- );-});diff --git a/packages/app/tests/integration/persistence-staleness-watchdog.test.ts b/packages/app/tests/integration/persistence-staleness-watchdog.test.tsindex fea67c07c..cf064394d 100644--- a/packages/app/tests/integration/persistence-staleness-watchdog.test.ts+++ b/packages/app/tests/integration/persistence-staleness-watchdog.test.ts@@ -74,7 +74,7 @@ describe('persistence staleness watchdog (integration)', () => {expect(getMetrics().persistenceStalenessForcedStores).toBe(forcedBefore);});- test('never writes over an external native edit that lands under unflushed edits', async () => {+ test('forces a store that keeps both versions when an outside edit lands under unflushed edits', async () => {server = await createTestServer({ stalenessGraceMs: 250, stalenessSweepIntervalMs: 100 });const docName = `staleness-external-${randomUUID()}`;const filePath = join(server.contentDir, `${docName}.md`);@@ -87,6 +87,7 @@ describe('persistence staleness watchdog (integration)', () => {);expect(flushed).toBe(true);+ const forcedBefore = getMetrics().persistenceStalenessForcedStores;const stoodDownBefore = getMetrics().persistenceStalenessStoodDown;process.env.OK_TEST_STORE_FAULT = docName;const second = await writeMd(server.port, docName, '# version two\n');@@ -94,23 +95,11 @@ describe('persistence staleness watchdog (integration)', () => {delete process.env.OK_TEST_STORE_FAULT;writeFileSync(filePath, '# external native edit\n', 'utf-8');- const watchdogStoodDown = await pollUntil(- () => getMetrics().persistenceStalenessStoodDown > stoodDownBefore,- 2_000,- );- if (!watchdogStoodDown) {- const state = await fetch(- `http://127.0.0.1:${server.port}/api/document?docName=${encodeURIComponent(docName)}`,- );- expect(state.ok).toBe(true);- expect((await state.text()).includes('external native edit')).toBe(true);- }-- const deadline = Date.now() + 1_500;- while (Date.now() < deadline) {- expect(readFileSync(filePath, 'utf-8')).toContain('# external native edit\n');- await wait(50);- }+ const bothKept = '# version two\n\n# external native edit\n';+ const landed = await pollUntil(() => readFileSync(filePath, 'utf-8') === bothKept, 10_000);+ expect(landed).toBe(true);+ expect(getMetrics().persistenceStalenessForcedStores).toBeGreaterThan(forcedBefore);+ expect(getMetrics().persistenceStalenessStoodDown).toBe(stoodDownBefore);expect(server.instance.conflicts.findByDocName(docName)).toBeUndefined();});});diff --git a/packages/app/tests/integration/template-concurrent-write-no-loss.test.ts b/packages/app/tests/integration/template-concurrent-write-no-loss.test.tsindex ac305f7cf..913c471cf 100644--- a/packages/app/tests/integration/template-concurrent-write-no-loss.test.ts+++ b/packages/app/tests/integration/template-concurrent-write-no-loss.test.ts@@ -128,7 +128,8 @@ describe('template concurrent-write reconcile on the content branch', () => {?.getText('source').toString();expect(docSource).toContain(AUTHOR_PENDING_LINE);- expect(docSource).not.toContain(FOREIGN_LINE);+ expect(docSource).toContain(FOREIGN_LINE);+ expect(readFileSync(tplFile, 'utf-8')).toBe(docSource);expect(listCheckpoints(rig).filter((c) => c.kind === 'managed-artifact-reconcile'),diff --git a/packages/app/tests/integration/write-receipt-compare.test.ts b/packages/app/tests/integration/write-receipt-compare.test.tsnew file mode 100644index 000000000..559ceacb4--- /dev/null+++ b/packages/app/tests/integration/write-receipt-compare.test.ts@@ -0,0 +1,746 @@+import { randomUUID } from 'node:crypto';+import { mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import { afterEach, describe, expect, test, vi } from 'vitest';+import type * as Y from 'yjs';+import { applyAgentMarkdownWrite } from '../../../server/src/agent-sessions';+import { DerivedDocumentIndex } from '../../../server/src/derived-document-index';+import * as diskContentIntake from '../../../server/src/disk-content-intake';+import { _resetDocExtensionsForTests } from '../../../server/src/doc-extensions';+import * as fsTraced from '../../../server/src/fs-traced';+import { getMetrics } from '../../../server/src/metrics';+import { contentHash } from '../../../server/src/version-hash';+import { createManualClock } from '../../../server/src/watcher-clock.test-helper';+import { createWatcherDouble } from '../../../server/src/watcher-double.test-helper';+import {+ createTestClient,+ createTestServer,+ pollUntil,+ type TestClient,+ type TestServer,+} from './test-harness';++const SEED = '# Doc\n\nfirst paragraph\n\nsecond paragraph\n';++let server: TestServer | undefined;+let client: TestClient | undefined;+const ownedDirs: string[] = [];++afterEach(async () => {+ vi.restoreAllMocks();+ delete process.env.OK_TEST_COMPARE_READ_FAULT;+ delete process.env.OK_TEST_STORE_RESTAT_RACE;+ delete process.env.OK_TEST_STORE_FAULT;+ await client?.cleanup();+ client = undefined;+ await server?.cleanup();+ server = undefined;+ for (const dir of ownedDirs.splice(0)) rmSync(dir, { recursive: true, force: true });+});++function compareCount(outcome: string): number {+ return (getMetrics().storeDiskCompare as Record<string, number | undefined>)[outcome] ?? 0;+}++async function startWithDoc(+ options: Parameters<typeof createTestServer>[0] = {},+): Promise<{ server: TestServer; docName: string; file: string }> {+ const started = await createTestServer({+ watcherBackend: createWatcherDouble().backend,+ debounce: 50,+ maxDebounce: 200,+ ...options,+ });+ server = started;+ const docName = `receipt-${randomUUID()}`;+ const file = join(started.contentDir, `${docName}.md`);+ writeFileSync(file, SEED);+ return { server: started, docName, file };+}++async function openClient(+ port: number,+ docName: string,+ options?: Parameters<typeof createTestClient>[2],+): Promise<TestClient> {+ client = await createTestClient(port, docName, options);+ const opened = client;+ await pollUntil(() => opened.ytext.toString() === SEED, 5_000, 25, 'the seeded document');+ return opened;+}++function typeAfter(doc: Y.Doc, text: Y.Text, anchor: string, insertion: string): void {+ const at = text.toString().indexOf(anchor);+ if (at < 0) throw new Error(`anchor not found: ${anchor}`);+ doc.transact(() => text.insert(at + anchor.length, insertion));+}++function serverText(target: TestServer, docName: string): string | undefined {+ return target.instance.hocuspocus.documents.get(docName)?.getText('source').toString();+}++async function expectDiskAndLiveToSettleOn(+ target: TestServer,+ docName: string,+ file: string,+ expected: string,+): Promise<void> {+ await pollUntil(+ () => readFileSync(file, 'utf-8') === expected,+ 8_000,+ 25,+ 'the merged bytes on disk',+ );+ await pollUntil(() => serverText(target, docName) === expected, 5_000, 25, 'the live text');+ expect(target.instance.durabilityState.getReceipt(docName)?.hash).toBe(contentHash(expected));+}++describe('compare before every Markdown write', () => {+ test('a person save over an outside edit in another block merges both, byte for byte', async () => {+ const { server: s, docName, file } = await startWithDoc();+ const c = await openClient(s.port, docName);+ const changedBefore = compareCount('changed');++ writeFileSync(file, '# Doc\n\nfirst paragraph changed outside\n\nsecond paragraph\n');+ typeAfter(c.doc, c.ytext, 'second paragraph', ' typed here');++ await expectDiskAndLiveToSettleOn(+ s,+ docName,+ file,+ '# Doc\n\nfirst paragraph changed outside\n\nsecond paragraph typed here\n',+ );+ expect(compareCount('changed')).toBeGreaterThan(changedBefore);+ });++ test('an outside change the bridge normalizer would tolerate is kept, not written over', async () => {+ const { server: s, docName, file } = await startWithDoc();+ const c = await openClient(s.port, docName);++ writeFileSync(file, '# Doc\n\nfirst paragraph \n\nsecond paragraph\n');+ typeAfter(c.doc, c.ytext, 'second paragraph', ' typed here');++ await expectDiskAndLiveToSettleOn(+ s,+ docName,+ file,+ '# Doc\n\nfirst paragraph \n\nsecond paragraph typed here\n',+ );+ });++ test('an agent store over an outside change its pre-write merge skipped merges it first', async () => {+ const { server: s, docName, file } = await startWithDoc();+ await openClient(s.port, docName);++ writeFileSync(file, '# Doc\n\nfirst paragraph \n\nsecond paragraph\n');+ const response = await fetch(`http://127.0.0.1:${s.port}/api/agent-write-md`, {+ method: 'POST',+ headers: { 'Content-Type': 'application/json' },+ body: JSON.stringify({ docName, markdown: '\nagent line\n', position: 'append' }),+ });+ expect(response.status).toBe(200);++ await expectDiskAndLiveToSettleOn(+ s,+ docName,+ file,+ '# Doc\n\nfirst paragraph \n\nsecond paragraph\n\nagent line\n',+ );+ });++ test('the staleness watchdog forces a store that merges an outside edit instead of standing down', async () => {+ const {+ server: s,+ docName,+ file,+ } = await startWithDoc({+ stalenessGraceMs: 250,+ stalenessSweepIntervalMs: 100,+ });+ await openClient(s.port, docName);+ const forcedBefore = getMetrics().persistenceStalenessForcedStores;++ process.env.OK_TEST_STORE_FAULT = docName;+ const failed = await fetch(`http://127.0.0.1:${s.port}/api/agent-write-md`, {+ method: 'POST',+ headers: { 'Content-Type': 'application/json' },+ body: JSON.stringify({ docName, markdown: '\nagent line\n', position: 'append' }),+ });+ expect(failed.status).toBe(507);+ delete process.env.OK_TEST_STORE_FAULT;+ writeFileSync(file, '# Doc\n\nfirst paragraph changed outside\n\nsecond paragraph\n');++ await expectDiskAndLiveToSettleOn(+ s,+ docName,+ file,+ '# Doc\n\nfirst paragraph changed outside\n\nsecond paragraph\n\nagent line\n',+ );+ expect(getMetrics().persistenceStalenessForcedStores).toBeGreaterThan(forcedBefore);+ });++ test('a whole-document agent replace saved by the ordinary store merges an outside edit', async () => {+ const { server: s, docName, file } = await startWithDoc();+ const session = await s.instance.sessionManager.getSession(docName, 'acp-replace', {+ displayName: 'ACP agent',+ colorSeed: 'acp-replace',+ clientName: 'acp',+ });+ await pollUntil(+ () => serverText(s, docName) === SEED,+ 5_000,+ 25,+ 'the session to load the document',+ );++ writeFileSync(file, '# Doc\n\nfirst paragraph changed outside\n\nsecond paragraph\n');+ session.dc.document.transact(() => {+ applyAgentMarkdownWrite(+ session.dc.document,+ '# Doc\n\nfirst paragraph\n\nsecond paragraph rewritten by the agent\n',+ 'replace',+ );+ }, session.origin);++ await expectDiskAndLiveToSettleOn(+ s,+ docName,+ file,+ '# Doc\n\nfirst paragraph changed outside\n\nsecond paragraph rewritten by the agent\n',+ );+ });++ test('the shutdown flush merges an outside edit into the pending save', async () => {+ const contentDir = realpathSync(mkdtempSync(join(tmpdir(), 'ok-receipt-shutdown-')));+ ownedDirs.push(contentDir);+ const {+ server: s,+ docName,+ file,+ } = await startWithDoc({+ contentDir,+ keepContentDir: true,+ debounce: 60_000,+ maxDebounce: 120_000,+ });+ await s.instance.hocuspocus.openDirectConnection(docName);+ const live = s.instance.hocuspocus.documents.get(docName);+ if (!live) throw new Error('document not loaded');+ await pollUntil(() => live.getText('source').toString() === SEED, 5_000, 25, 'the load');++ typeAfter(live, live.getText('source'), 'second paragraph', ' typed here');+ writeFileSync(file, '# Doc\n\nfirst paragraph changed outside\n\nsecond paragraph\n');+ expect(readFileSync(file, 'utf-8')).not.toContain('typed here');++ await s.cleanup();+ server = undefined;++ expect(readFileSync(file, 'utf-8')).toBe(+ '# Doc\n\nfirst paragraph changed outside\n\nsecond paragraph typed here\n',+ );+ });++ test('a change made while the server was stopped is found at load from the persisted receipt', async () => {+ const contentDir = realpathSync(mkdtempSync(join(tmpdir(), 'ok-receipt-restart-')));+ ownedDirs.push(contentDir);+ const first = await startWithDoc({ contentDir, keepContentDir: true });+ const c = await openClient(first.server.port, first.docName, { resetOnCleanup: false });+ typeAfter(c.doc, c.ytext, 'second paragraph', ' saved before quitting');+ const saved = '# Doc\n\nfirst paragraph\n\nsecond paragraph saved before quitting\n';+ await pollUntil(() => readFileSync(first.file, 'utf-8') === saved, 8_000, 25, 'the save');+ await c.cleanup();+ client = undefined;+ await first.server.cleanup();+ server = undefined;++ const receiptLog = readFileSync(join(contentDir, '.ok', 'local', 'receipts.jsonl'), 'utf-8');+ expect(receiptLog).toContain(contentHash(saved));++ const editedWhileStopped =+ '# Doc\n\nfirst paragraph edited while stopped\n\nsecond paragraph saved before quitting\n';+ writeFileSync(first.file, editedWhileStopped);+ const changedAtLoadBefore = getMetrics().receiptAtLoad.changed ?? 0;+ const mergesBefore = compareCount('changed');++ server = await createTestServer({+ contentDir,+ keepContentDir: true,+ watcherBackend: createWatcherDouble().backend,+ debounce: 50,+ maxDebounce: 200,+ });+ client = await createTestClient(server.port, first.docName);+ const reopened = client;+ await pollUntil(+ () => reopened.ytext.toString() === editedWhileStopped,+ 5_000,+ 25,+ 'the disk version at load',+ );+ expect(getMetrics().receiptAtLoad.changed ?? 0).toBe(changedAtLoadBefore + 1);+ expect(server.instance.durabilityState.getReconciledBase(first.docName)).toBe(+ editedWhileStopped,+ );++ typeAfter(reopened.doc, reopened.ytext, 'saved before quitting', ' and after');+ const final =+ '# Doc\n\nfirst paragraph edited while stopped\n\nsecond paragraph saved before quitting and after\n';+ await expectDiskAndLiveToSettleOn(server, first.docName, first.file, final);+ expect(compareCount('changed')).toBe(mergesBefore);+ });++ test('an .mdx receipt survives a cold restart, counts a change made while stopped and recognises a late echo', async () => {+ const contentDir = realpathSync(mkdtempSync(join(tmpdir(), 'ok-receipt-mdx-restart-')));+ ownedDirs.push(contentDir);+ const docName = `receipt-mdx-${randomUUID()}`;+ const file = join(contentDir, `${docName}.mdx`);+ writeFileSync(file, SEED);+ server = await createTestServer({+ contentDir,+ keepContentDir: true,+ watcherBackend: createWatcherDouble().backend,+ debounce: 50,+ maxDebounce: 200,+ });+ const first = await openClient(server.port, docName, { resetOnCleanup: false });+ typeAfter(first.doc, first.ytext, 'second paragraph', ' saved before quitting');+ const saved = '# Doc\n\nfirst paragraph\n\nsecond paragraph saved before quitting\n';+ await pollUntil(() => readFileSync(file, 'utf-8') === saved, 8_000, 25, 'the save');+ await first.cleanup();+ client = undefined;+ await server.cleanup();+ server = undefined;++ const edited =+ '# Doc\n\nfirst paragraph edited while stopped\n\nsecond paragraph saved before quitting\n';+ writeFileSync(file, edited);+ _resetDocExtensionsForTests();+ const changedBefore = getMetrics().receiptAtLoad.changed ?? 0;+ const clock = createManualClock(Date.now());+ const double = createWatcherDouble();+ server = await createTestServer({+ contentDir,+ keepContentDir: true,+ watcherBackend: double.backend,+ watcherClock: clock,+ debounce: 50,+ maxDebounce: 200,+ });+ const restarted = server;+ await new Promise((resolve) => setImmediate(resolve));+ client = await createTestClient(restarted.port, docName);+ const reopened = client;+ await pollUntil(() => reopened.ytext.toString() === edited, 5_000, 25, 'the load');+ expect(getMetrics().receiptAtLoad.changed ?? 0).toBe(changedBefore + 1);++ typeAfter(reopened.doc, reopened.ytext, 'quitting', ' and after');+ const final =+ '# Doc\n\nfirst paragraph edited while stopped\n\nsecond paragraph saved before quitting and after\n';+ await pollUntil(+ () => readFileSync(file, 'utf-8') === final,+ 8_000,+ 25,+ 'the save after restart',+ );+ clock.advance(25_000);+ const echoesBefore = getMetrics().receiptIntakeSkip.echo ?? 0;+ double.subscriptionFor(restarted.contentDir).deliver([{ type: 'update', path: file }]);+ await pollUntil(+ () => {+ clock.advance(50);+ return (getMetrics().receiptIntakeSkip.echo ?? 0) > echoesBefore;+ },+ 5_000,+ 25,+ 'the late echo to reach the intake',+ );+ expect(readFileSync(join(contentDir, '.ok', 'local', 'receipts.jsonl'), 'utf-8')).toContain(+ contentHash(final),+ );+ });++ test('a store whose merge of the outside edit fails to apply holds the save and never writes over the outside bytes', async () => {+ const { server: s, docName, file } = await startWithDoc();+ const c = await openClient(s.port, docName);+ const outside = '# Doc\n\nfirst paragraph changed outside\n\nsecond paragraph\n';+ const realApply = diskContentIntake.applyDiskContentToDoc;+ vi.spyOn(diskContentIntake, 'applyDiskContentToDoc').mockImplementation((...args) => {+ if (args[1].includes('changed outside')) throw new Error('injected apply failure');+ return realApply(...args);+ });+ const heldBefore = compareCount('pending-intake-failed');++ writeFileSync(file, outside);+ typeAfter(c.doc, c.ytext, 'second paragraph', ' typed here');++ await pollUntil(+ () => compareCount('pending-intake-failed') > heldBefore,+ 8_000,+ 25,+ 'the held save',+ );+ expect(readFileSync(file, 'utf-8')).toBe(outside);+ expect(serverText(s, docName)).toContain('typed here');+ expect(serverText(s, docName)).not.toContain('changed outside');+ expect(s.instance.durabilityState.getReceipt(docName)?.hash).not.toBe(contentHash(outside));+ expect(s.instance.durabilityState.isStoreRefused(docName)).toBe(true);++ vi.restoreAllMocks();+ typeAfter(c.doc, c.ytext, 'typed here', ' and again');+ await expectDiskAndLiveToSettleOn(+ s,+ docName,+ file,+ '# Doc\n\nfirst paragraph changed outside\n\nsecond paragraph typed here and again\n',+ );+ });++ test('a save with no agreed version whose ingest of the outside file fails is held and leaves no receipt for the outside bytes', async () => {+ const started = await createTestServer({+ watcherBackend: createWatcherDouble().backend,+ debounce: 300_000,+ maxDebounce: 600_000,+ });+ server = started;+ const docName = `receipt-no-base-${randomUUID()}`;+ const file = join(started.contentDir, `${docName}.md`);+ client = await createTestClient(started.port, docName);+ const c = client;+ c.doc.transact(() => c.ytext.insert(0, '# Typed\n\nbefore the file existed\n'));+ await pollUntil(+ () => serverText(started, docName)?.includes('before the file existed') === true,+ 5_000,+ 25,+ 'the typing to reach the server',+ );+ const outside = '# Doc\n\ncreated outside\n';+ writeFileSync(file, outside);+ const realApply = diskContentIntake.applyDiskContentToDoc;+ vi.spyOn(diskContentIntake, 'applyDiskContentToDoc').mockImplementation((...args) => {+ if (args[1].includes('created outside')) throw new Error('injected apply failure');+ return realApply(...args);+ });+ const heldBefore = compareCount('pending-intake-failed-no-base');++ await runPendingStore(started, docName).catch(() => undefined);++ expect(compareCount('pending-intake-failed-no-base')).toBe(heldBefore + 1);+ expect(readFileSync(file, 'utf-8')).toBe(outside);+ expect(started.instance.durabilityState.getReceipt(docName)).toBeUndefined();+ expect(started.instance.durabilityState.getReconciledBase(docName)).toBeUndefined();+ expect(serverText(started, docName)).toContain('before the file existed');++ vi.restoreAllMocks();+ c.doc.transact(() => c.ytext.insert(c.ytext.length, '\nmore typing\n'));+ await pollUntil(+ () => serverText(started, docName)?.includes('more typing') === true,+ 5_000,+ 25,+ 'the second edit to reach the server',+ );+ await runPendingStore(started, docName).catch(() => undefined);+ await pollUntil(() => serverText(started, docName) === outside, 5_000, 25, 'the ingest');+ expect(readFileSync(file, 'utf-8')).toBe(outside);+ expect(started.instance.durabilityState.getReceipt(docName)?.hash).toBe(contentHash(outside));+ });++ test('an outside file the intake raises a conflict for holds the save once as a conflict, not as a file that kept changing', async () => {+ const { server: s, docName, file } = await startWithDoc();+ const c = await openClient(s.port, docName);+ const conflictsBefore = getMetrics().conflictCount;+ const heldBefore = compareCount('pending-conflict');+ const exhaustedBefore = compareCount('pending-exhausted');+ const marked =+ '# Doc\n\n<<<<<<< HEAD\nfirst paragraph\n=======\nfirst paragraph elsewhere\n>>>>>>> other\n\nsecond paragraph\n';++ writeFileSync(file, marked);+ typeAfter(c.doc, c.ytext, 'second paragraph', ' typed here');++ await pollUntil(+ () => compareCount('pending-conflict') > heldBefore,+ 8_000,+ 25,+ 'the held save',+ );+ expect(compareCount('pending-exhausted')).toBe(exhaustedBefore);+ expect(getMetrics().conflictCount).toBe(conflictsBefore + 1);+ expect(s.instance.conflicts.findByDocName(docName)).toBeDefined();+ expect(readFileSync(file, 'utf-8')).toBe(marked);+ expect(serverText(s, docName)).toContain('typed here');+ });++ test('an index failure after the outside edit was taken in does not hold the save', async () => {+ const { server: s, docName, file } = await startWithDoc();+ const c = await openClient(s.port, docName);+ const heldBefore = compareCount('pending-intake-failed');+ const outside = '# Doc\n\nfirst paragraph changed outside\n\nsecond paragraph\n';+ const realUpsert = DerivedDocumentIndex.prototype.recordDiskUpsert;+ vi.spyOn(DerivedDocumentIndex.prototype, 'recordDiskUpsert').mockImplementation(function (+ this: DerivedDocumentIndex,+ documentName,+ markdown,+ ) {+ if (documentName === docName && markdown === outside) {+ return Promise.reject(new Error('injected index failure'));+ }+ return realUpsert.call(this, documentName, markdown);+ });++ writeFileSync(file, outside);+ typeAfter(c.doc, c.ytext, 'second paragraph', ' typed here');++ await expectDiskAndLiveToSettleOn(+ s,+ docName,+ file,+ '# Doc\n\nfirst paragraph changed outside\n\nsecond paragraph typed here\n',+ );+ expect(compareCount('pending-intake-failed')).toBe(heldBefore);+ });++ test('a file that keeps changing across every run holds the save with the file untouched and the edit kept', async () => {+ const { server: s, docName, file } = await startWithDoc();+ const c = await openClient(s.port, docName);+ const exhaustedBefore = compareCount('pending-exhausted');+ let outsideWrites = 0;+ const realWrite = fsTraced.tracedWriteFile;+ vi.spyOn(fsTraced, 'tracedWriteFile').mockImplementation(async (...args) => {+ await realWrite(...args);+ if (!String(args[0]).startsWith(`${file}.tmp.`)) return;+ outsideWrites++;+ writeFileSync(file, `# Doc\n\noutside write ${outsideWrites}\n\nsecond paragraph\n`);+ });++ typeAfter(c.doc, c.ytext, 'second paragraph', ' typed here');++ await pollUntil(+ () => compareCount('pending-exhausted') > exhaustedBefore,+ 8_000,+ 25,+ 'the exhausted save',+ );+ expect(readFileSync(file, 'utf-8')).toBe(+ `# Doc\n\noutside write ${outsideWrites}\n\nsecond paragraph\n`,+ );+ expect(serverText(s, docName)).toContain('typed here');+ expect(s.instance.durabilityState.isStoreRefused(docName)).toBe(true);+ });++ test('an echo of our own save delivered after the tracker window is recognised by the receipt', async () => {+ const clock = createManualClock(Date.now());+ const double = createWatcherDouble();+ const {+ server: s,+ docName,+ file,+ } = await startWithDoc({+ watcherBackend: double.backend,+ watcherClock: clock,+ });+ const c = await openClient(s.port, docName);+ typeAfter(c.doc, c.ytext, 'second paragraph', ' typed here');+ const saved = '# Doc\n\nfirst paragraph\n\nsecond paragraph typed here\n';+ await pollUntil(() => readFileSync(file, 'utf-8') === saved, 8_000, 25, 'the save');++ clock.advance(25_000);+ const echoesBefore = getMetrics().receiptIntakeSkip.echo ?? 0;+ const reconcilesBefore = getMetrics().reconcileCount;+ double.subscriptionFor(s.contentDir).deliver([{ type: 'update', path: file }]);+ await pollUntil(+ () => {+ clock.advance(50);+ return (getMetrics().receiptIntakeSkip.echo ?? 0) > echoesBefore;+ },+ 5_000,+ 25,+ 'the late echo to reach the intake',+ );++ expect(getMetrics().reconcileCount).toBe(reconcilesBefore);+ expect(serverText(s, docName)).toBe(saved);+ expect(readFileSync(file, 'utf-8')).toBe(saved);+ });++ test('a compare read that is briefly busy is retried and the save lands', async () => {+ const { server: s, docName, file } = await startWithDoc();+ const c = await openClient(s.port, docName);+ const retriesBefore = getMetrics().storeDiskCompareReadRetries;++ process.env.OK_TEST_COMPARE_READ_FAULT = `${docName}:EBUSY:2`;+ typeAfter(c.doc, c.ytext, 'second paragraph', ' typed here');++ await expectDiskAndLiveToSettleOn(+ s,+ docName,+ file,+ '# Doc\n\nfirst paragraph\n\nsecond paragraph typed here\n',+ );+ expect(getMetrics().storeDiskCompareReadRetries).toBe(retriesBefore + 2);+ });++ test('a compare read that stays busy leaves the save pending and the file untouched', async () => {+ const { server: s, docName, file } = await startWithDoc();+ const c = await openClient(s.port, docName);+ const pendingBefore = compareCount('pending-unreadable');++ process.env.OK_TEST_COMPARE_READ_FAULT = `${docName}:EBUSY:1000`;+ typeAfter(c.doc, c.ytext, 'second paragraph', ' typed here');++ await pollUntil(+ () => compareCount('pending-unreadable') > pendingBefore,+ 8_000,+ 25,+ 'the pending save',+ );+ expect(readFileSync(file, 'utf-8')).toBe(SEED);+ expect(serverText(s, docName)).toContain('typed here');+ expect(s.instance.durabilityState.isStoreRefused(docName)).toBe(true);+ });++ test('a write that lands between the compare and the rename is refused and merged on the next run', async () => {+ const { server: s, docName, file } = await startWithDoc();+ const c = await openClient(s.port, docName);+ const refusedBefore = getMetrics().storeRestatRefused;++ process.env.OK_TEST_STORE_RESTAT_RACE = docName;+ typeAfter(c.doc, c.ytext, 'first paragraph', ' typed here');++ await expectDiskAndLiveToSettleOn(+ s,+ docName,+ file,+ '# Doc\n\nfirst paragraph typed here\n\nsecond paragraph\n\nrestat-race-injected\n',+ );+ expect(getMetrics().storeRestatRefused).toBe(refusedBefore + 1);+ });+});++interface HistoryVersion {+ side: string;+ content: string;+}++async function historyVersions(+ target: TestServer,+ docName: string,+ kind: string,+ count: number,+): Promise<HistoryVersion[]> {+ let rows: Array<{ sha: string; checkpoint?: { kind?: string; metadata?: { side?: string } } }> =+ [];+ await pollUntil(+ async () => {+ const res = await fetch(+ `http://127.0.0.1:${target.port}/api/history?docName=${encodeURIComponent(docName)}&limit=100`,+ );+ const body = (await res.json()) as { entries?: typeof rows };+ rows = (body.entries ?? []).filter((entry) => entry.checkpoint?.kind === kind);+ return rows.length >= count;+ },+ 10_000,+ 50,+ `${count} ${kind} history entries`,+ );+ const versions: HistoryVersion[] = [];+ for (const row of rows) {+ const res = await fetch(+ `http://127.0.0.1:${target.port}/api/history/${row.sha}?docName=${encodeURIComponent(docName)}`,+ );+ const body = (await res.json()) as { content: string };+ versions.push({ side: row.checkpoint?.metadata?.side ?? '', content: body.content });+ }+ return versions.sort((a, b) => a.side.localeCompare(b.side));+}++async function startWithUnsavedEdit(+ saved: string,+ edit: { find: string; replace: string },+): Promise<{ server: TestServer; docName: string; file: string }> {+ const started = await createTestServer({+ gitEnabled: true,+ watcherBackend: createWatcherDouble().backend,+ debounce: 300_000,+ maxDebounce: 600_000,+ });+ server = started;+ const docName = `receipt-collide-${randomUUID()}`;+ const file = join(started.contentDir, `${docName}.md`);+ writeFileSync(file, saved);+ client = await createTestClient(started.port, docName);+ const opened = client;+ await pollUntil(() => opened.ytext.toString() === saved, 5_000, 25, 'the seeded document');+ opened.doc.transact(() => {+ const at = opened.ytext.toString().indexOf(edit.find);+ opened.ytext.delete(at, edit.find.length);+ opened.ytext.insert(at, edit.replace);+ });+ await pollUntil(+ () => serverText(started, docName)?.includes(edit.replace) === true,+ 5_000,+ 25,+ 'the unsaved edit to reach the server',+ );+ return { server: started, docName, file };+}++async function runPendingStore(target: TestServer, docName: string): Promise<void> {+ const id = `onStoreDocument-${docName}`;+ if (target.instance.hocuspocus.debouncer.isDebounced(id)) {+ await target.instance.hocuspocus.debouncer.executeNow(id);+ }+}++describe('compare before every Markdown write: an outside edit to the block being saved', () => {+ test('a person save keeps both versions of the block, live and on disk, and both originals in history', async () => {+ const saved = '# Doc\n\nShared paragraph.\n\nTail.\n';+ const {+ server: s,+ docName,+ file,+ } = await startWithUnsavedEdit(saved, {+ find: 'Shared paragraph.',+ replace: 'Shared paragraph, edited here.',+ });+ const bothKeptBefore = getMetrics().reconcileBothKept;+ const outside = '# Doc\n\nShared paragraph, edited outside.\n\nTail.\n';++ writeFileSync(file, outside);+ await runPendingStore(s, docName);++ const bothKept =+ '# Doc\n\nShared paragraph, edited here.\n\nShared paragraph, edited outside.\n\nTail.\n';+ await expectDiskAndLiveToSettleOn(s, docName, file, bothKept);+ expect(getMetrics().reconcileBothKept).toBe(bothKeptBefore + 1);+ expect(s.instance.conflicts.list()).toEqual([]);+ expect(await historyVersions(s, docName, 'external-change-both-kept', 2)).toEqual([+ { side: 'ours', content: '# Doc\n\nShared paragraph, edited here.\n\nTail.\n' },+ { side: 'theirs', content: outside },+ ]);+ });++ test('a person save whose merge fails the fidelity check keeps the version here and moves the outside version to history', async () => {+ const saved = '# Doc\n\nA\n\nB\n\nC\n';+ const {+ server: s,+ docName,+ file,+ } = await startWithUnsavedEdit(saved, {+ find: 'B\n\nC',+ replace: 'X\n\nB edited\n\nC',+ });+ const keptLiveBefore = getMetrics().reconcileKeptLive;+ const outside = '# Doc\n\nA\n\nX\n\nB\n\nC\n';++ writeFileSync(file, outside);+ await runPendingStore(s, docName);++ const here = '# Doc\n\nA\n\nX\n\nB edited\n\nC\n';+ await expectDiskAndLiveToSettleOn(s, docName, file, here);+ expect(getMetrics().reconcileKeptLive).toBe(keptLiveBefore + 1);+ expect(s.instance.conflicts.list()).toEqual([]);+ expect(await historyVersions(s, docName, 'external-change-kept-live', 1)).toEqual([+ { side: 'theirs', content: outside },+ ]);+ });+});diff --git a/packages/app/tests/stress/_helpers/reuse/fresh-slot.ts b/packages/app/tests/stress/_helpers/reuse/fresh-slot.tsindex 56663183e..9ffb0679d 100644--- a/packages/app/tests/stress/_helpers/reuse/fresh-slot.ts+++ b/packages/app/tests/stress/_helpers/reuse/fresh-slot.ts@@ -24,6 +24,7 @@ const SERVER_WRITTEN_TREE: SlotTree = {'acp-registry-cache.json': SERVER_OWNED,cache: SERVER_OWNED,'principal.json': SERVER_OWNED,+ 'receipts.jsonl': SERVER_OWNED,'removed-docs.json': SERVER_OWNED,'server.lock': SERVER_OWNED,'stale-external-writes.json': SERVER_OWNED,diff --git a/packages/core/src/handoff/urn-ipc-registry.ts b/packages/core/src/handoff/urn-ipc-registry.tsindex e45c45ea5..424911920 100644--- a/packages/core/src/handoff/urn-ipc-registry.ts+++ b/packages/core/src/handoff/urn-ipc-registry.ts@@ -60,6 +60,7 @@ export const URN_HTTP_ONLY: ReadonlySet<ProblemType> = new Set<ProblemType>(['urn:ok:error:disk-divergence','urn:ok:error:stale-external-write','urn:ok:error:store-refused',+ 'urn:ok:error:save-held','urn:ok:error:doc-removed','urn:ok:error:branch-switched','urn:ok:error:doc-in-conflict',diff --git a/packages/core/src/schemas/api/_envelope.ts b/packages/core/src/schemas/api/_envelope.tsindex 6d446a981..adba83d05 100644--- a/packages/core/src/schemas/api/_envelope.ts+++ b/packages/core/src/schemas/api/_envelope.ts@@ -84,6 +84,7 @@ export const ProblemTypeSchema = z.enum(['urn:ok:error:disk-divergence','urn:ok:error:stale-external-write','urn:ok:error:store-refused',+ 'urn:ok:error:save-held','urn:ok:error:doc-removed','urn:ok:error:branch-switched','urn:ok:error:doc-not-found',diff --git a/packages/server/src/api-agent-write-batch-persistence-failure.test.ts b/packages/server/src/api-agent-write-batch-persistence-failure.test.tsindex 0c696128b..02c5ce555 100644--- a/packages/server/src/api-agent-write-batch-persistence-failure.test.ts+++ b/packages/server/src/api-agent-write-batch-persistence-failure.test.ts@@ -20,6 +20,7 @@ import {DocumentDurabilityState,OK_DOC_REMOVED,OK_STORE_DISCARDED,+ OK_STORE_HELD,OK_STORE_REFUSED,} from './document-durability-state.ts';@@ -138,6 +139,37 @@ describe('agent-write-batch — store-refused flush failure rendering', () => {expect(results[0]?.error?.detail).not.toContain('will be lost');});+ test('a held save renders the save-held contract on the batch endpoint', async () => {+ const docName = 'batch-held';+ durabilityState.recordStoreFailure(docName, {+ code: OK_STORE_HELD,+ message: `${docName} was not saved: its file changed and could not be merged, so a conflict was raised`,+ held: { reason: 'pending-conflict' },+ });++ const response = await postBatch({+ docs: [{ docName, markdown: '# Batch held\n\nUpdated body.\n', position: 'replace' }],+ agentId: 'claude-1',+ agentName: 'Claude',+ });++ expect(response.status).toBe(200);+ const { results } = JSON.parse(response.body) as {+ results: Array<{ status: string; error?: Record<string, unknown> }>;+ };+ expect(results[0]?.status).toBe('error');+ expect(results[0]?.error).toMatchObject({+ type: 'urn:ok:error:save-held',+ reason: 'pending-conflict',+ retryWhen: 'conflict-resolved',+ nextStep: 're-read',+ });+ expect(String(results[0]?.error?.detail)).toContain(+ 'after the conflict on this document is resolved',+ );+ expect(String(results[0]?.error?.detail)).toContain('Do not repeat this write');+ });+test('a doc-removed flush failure keeps the doc-removed contract on the batch endpoint', async () => {const docName = 'batch-doc-removed';durabilityState.recordStoreFailure(docName, {diff --git a/packages/server/src/api-extension.persistence-failure.test.ts b/packages/server/src/api-extension.persistence-failure.test.tsindex 59506af6a..e0200e02a 100644--- a/packages/server/src/api-extension.persistence-failure.test.ts+++ b/packages/server/src/api-extension.persistence-failure.test.ts@@ -3,6 +3,7 @@ import { describe, expect, test } from 'vitest';import { respondPersistenceFailure } from './api-extension.ts';import {OK_STORE_DISCARDED,+ OK_STORE_HELD,OK_STORE_REFUSED,type StoreFailure,} from './document-durability-state.ts';@@ -65,6 +66,108 @@ describe('respondPersistenceFailure — store-refused', () => {});});+describe('respondPersistenceFailure: a held save', () => {+ test('an OK_STORE_HELD failure renders a 503 save-held problem with its reason, cause, retry rule and a re-read next step', () => {+ const { res, writeHeadCalls, endCalls } = makeMockRes();+ const failure: StoreFailure = {+ code: OK_STORE_HELD,+ message: 'notes was not saved: its file could not be read for the compare before the write',+ held: { reason: 'pending-unreadable', causeCode: 'EBUSY' },+ };++ respondPersistenceFailure(res, failure, 'agent-write-md');++ expect(writeHeadCalls[0]?.status).toBe(503);+ const body = JSON.parse(endCalls[0] ?? '{}') as Record<string, unknown>;+ expect(body).toMatchObject({+ type: 'urn:ok:error:save-held',+ title: 'Edit applied in memory; its save to disk is held.',+ reason: 'pending-unreadable',+ retryWhen: 'next-save',+ causeCode: 'EBUSY',+ nextStep: 're-read',+ });+ const detail = String(body.detail);+ expect(detail).toContain('kept in the document');+ expect(detail).toContain('once the file can be read again');+ expect(detail).toContain('the idle save within a few minutes');+ expect(detail).toContain('Do not repeat this write');+ expect(detail).not.toContain('will be lost');+ });++ test.each([+ { held: { reason: 'pending-intake-failed-no-base' as const } },+ { held: { reason: 'pending-unreadable' as const, causeCode: 'EBUSY', noBase: true as const } },+ ])(+ 'a hold on a document with no agreed version ($held.reason) never says the edit is saved',+ ({ held }) => {+ const { res, endCalls } = makeMockRes();+ respondPersistenceFailure(+ res,+ { code: OK_STORE_HELD, message: 'notes was not saved', held },+ 'agent-write-md',+ );+ const body = JSON.parse(endCalls[0] ?? '{}') as Record<string, unknown>;+ expect(body).toMatchObject({+ type: 'urn:ok:error:save-held',+ reason: held.reason,+ retryWhen: 'replaced-on-next-edit',+ nextStep: 're-read',+ });+ const detail = String(body.detail);+ expect(detail).toContain('This edit is not saved to the file');+ expect(detail).toContain('replaces the document with the file on disk');+ expect(detail).toContain('document history (timeline)');+ expect(detail).toContain('re-apply this edit');+ expect(detail).not.toContain('kept in the document');+ expect(detail).not.toMatch(/saves it|writes it|is saved after/);+ },+ );++ test.each([+ {+ reason: 'pending-conflict' as const,+ retryWhen: 'conflict-resolved',+ says: 'after the conflict on this document is resolved',+ neverSays: 'idle save',+ },+ {+ reason: 'pending-intake-failed' as const,+ retryWhen: 'next-save',+ says: 'the idle save within a few minutes',+ neverSays: 'conflict',+ },+ ])('a $reason hold states its own retry rule', ({ reason, retryWhen, says, neverSays }) => {+ const { res, endCalls } = makeMockRes();+ respondPersistenceFailure(+ res,+ { code: OK_STORE_HELD, message: 'notes was not saved', held: { reason } },+ 'agent-write-md',+ );+ const body = JSON.parse(endCalls[0] ?? '{}') as Record<string, unknown>;+ expect(body).toMatchObject({ reason, retryWhen, nextStep: 're-read' });+ expect(String(body.detail)).toContain(says);+ expect(String(body.detail)).not.toContain(neverSays);+ });++ test('a held save with no errno names its reason and carries no cause code', () => {+ const { res, endCalls } = makeMockRes();+ const failure: StoreFailure = {+ code: OK_STORE_HELD,+ message: 'notes was not saved: its file kept changing while it was being saved',+ held: { reason: 'pending-exhausted' },+ };++ respondPersistenceFailure(res, failure, 'agent-patch');++ const body = JSON.parse(endCalls[0] ?? '{}') as Record<string, unknown>;+ expect(body.type).toBe('urn:ok:error:save-held');+ expect(body.reason).toBe('pending-exhausted');+ expect(body).not.toHaveProperty('causeCode');+ expect(String(body.detail)).toContain('once the file stops changing');+ });+});+describe('respondPersistenceFailure: store discarded by a branch switch', () => {test('an OK_STORE_DISCARDED failure renders a 409 telling the agent the edit was not saved', () => {const { res, writeHeadCalls, endCalls } = makeMockRes();diff --git a/packages/server/src/api-extension.ts b/packages/server/src/api-extension.tsindex 53d62fa81..9d01d89e3 100644--- a/packages/server/src/api-extension.ts+++ b/packages/server/src/api-extension.ts@@ -230,6 +230,7 @@ import {OK_DOC_REMOVED,OK_PATH_UNRESOLVABLE,OK_STORE_DISCARDED,+ OK_STORE_HELD,OK_STORE_REFUSED,type StoreFailure,} from './document-durability-state.ts';@@ -360,6 +361,7 @@ import {incrementAgentWriteCalls,incrementSummariesProvided,incrementSummariesTruncated,+ type StorePendingReason,} from './metrics.ts';import { isWithinDir, toPosix } from './path-utils.ts';import { assertProjectContentSubtree } from './project-content-scope.ts';@@ -1463,6 +1465,64 @@ function discardedStoreProblem(failure: StoreFailure) {} satisfies BatchEntryError;}+type HeldSaveRetry = 'next-save' | 'next-edit' | 'conflict-resolved' | 'replaced-on-next-edit';++const HELD_SAVE_RULES: Readonly<+ Record<StorePendingReason, { readonly retryWhen: HeldSaveRetry; readonly sentence: string }>+> = {+ 'pending-unreadable': {+ retryWhen: 'next-save',+ sentence:+ 'A later save writes it once the file can be read again: the next edit to this document, or the idle save within a few minutes.',+ },+ 'pending-exhausted': {+ retryWhen: 'next-save',+ sentence:+ 'A later save writes it once the file stops changing: the next edit to this document, or the idle save within a few minutes.',+ },+ 'pending-base-moved': {+ retryWhen: 'next-save',+ sentence:+ 'A later save writes it once the outside changes have been taken in: the next edit to this document, or the idle save within a few minutes.',+ },+ 'pending-intake-failed': {+ retryWhen: 'next-save',+ sentence:+ 'A later save merges the outside change and writes it: the next edit to this document, or the idle save within a few minutes.',+ },+ 'pending-intake-failed-no-base': {+ retryWhen: 'replaced-on-next-edit',+ sentence:+ 'This edit is not saved to the file, and no save runs on its own: OpenKnowledge has no agreed version of the file to merge it with. The next edit to this document replaces the document with the file on disk and moves this edit, with any typing since, to the document history (timeline). Do not repeat this write now: after that next edit, re-read the document and re-apply this edit if it is still needed.',+ },+ 'pending-no-intake': {+ retryWhen: 'next-edit',+ sentence: 'The next edit to this document tries the save again.',+ },+ 'pending-conflict': {+ retryWhen: 'conflict-resolved',+ sentence:+ 'It is saved after the conflict on this document is resolved; ask the user to resolve it in the OpenKnowledge app.',+ },+};++function heldSaveProblem(failure: StoreFailure) {+ const reason = failure.held?.reason ?? 'pending-unreadable';+ const noBase = failure.held?.noBase === true || reason === 'pending-intake-failed-no-base';+ const rule = HELD_SAVE_RULES[noBase ? 'pending-intake-failed-no-base' : reason];+ return {+ type: 'urn:ok:error:save-held' as const,+ title: 'Edit applied in memory; its save to disk is held.',+ detail: noBase+ ? `${failure.message}. ${rule.sentence}`+ : `${failure.message}. The edit is kept in the document. ${rule.sentence} Do not repeat this write: re-read the document before editing it further.`,+ reason,+ retryWhen: rule.retryWhen,+ ...(failure.held?.causeCode === undefined ? {} : { causeCode: failure.held.causeCode }),+ nextStep: 're-read' as const,+ } satisfies BatchEntryError;+}+function refusedStoreProblem(failure: StoreFailure) {return {type: 'urn:ok:error:store-refused' as const,@@ -1482,6 +1542,11 @@ export function respondPersistenceFailure(errorResponse(res, 503, type, title, { handler, detail });return;}+ if (failure.code === OK_STORE_HELD) {+ const { type, title, detail, ...extensions } = heldSaveProblem(failure);+ errorResponse(res, 503, type, title, { handler, detail, extensions });+ return;+ }if (failure.code === OK_DOC_REMOVED) {const { type, title, detail } = removedDocProblem(failure);errorResponse(res, 409, type, title, { handler, detail });@@ -1922,7 +1987,6 @@ export function createApiExtension(options: ApiExtensionOptions): Extension & {type FlushOutcome =| { kind: 'failure'; failure: StoreFailure }- | { kind: 'divergence' }| { kind: 'stale-external-write' }| null;@@ -1975,20 +2039,9 @@ export function createApiExtension(options: ApiExtensionOptions): Extension & {if (durabilityState.takeStaleExternalWriteFreeze(docName)) {return { kind: 'stale-external-write' };}- if (durabilityState.takeStoreDivergence(docName)) return { kind: 'divergence' };return null;}- function respondDiskDivergence(res: ServerResponse, handler: string): void {- errorResponse(- res,- 409,- 'urn:ok:error:disk-divergence',- 'The document changed on disk after your edit was prepared; your edit was NOT applied, to avoid overwriting the newer on-disk content. Re-read the document and retry.',- { handler },- );- }-function staleExternalWriteProblem(docName: string) {return {type: 'urn:ok:error:stale-external-write' as const,@@ -3729,10 +3782,6 @@ export function createApiExtension(options: ApiExtensionOptions): Extension & {respondPersistenceFailure(res, flushOutcome.failure, 'agent-write');return;}- if (flushOutcome?.kind === 'divergence') {- respondDiskDivergence(res, 'agent-write');- return;- }if (flushOutcome?.kind === 'stale-external-write') {respondStaleExternalWrite(res, 'agent-write', docName);return;@@ -4038,6 +4087,8 @@ export function createApiExtension(options: ApiExtensionOptions): Extension & {flushErrors.set(p.docName, pathFaultProblem(flushOutcome.failure));} else if (flushOutcome.failure.code === OK_STORE_REFUSED) {flushErrors.set(p.docName, refusedStoreProblem(flushOutcome.failure));+ } else if (flushOutcome.failure.code === OK_STORE_HELD) {+ flushErrors.set(p.docName, heldSaveProblem(flushOutcome.failure));} else if (flushOutcome.failure.code === OK_STORE_DISCARDED) {flushErrors.set(p.docName, discardedStoreProblem(flushOutcome.failure));} else {@@ -4050,12 +4101,6 @@ export function createApiExtension(options: ApiExtensionOptions): Extension & {detail: `${flushOutcome.failure.code ?? 'unknown error'}: ${flushOutcome.failure.message}. The content was NOT saved and will be lost if the server restarts.`,});}- } else if (flushOutcome?.kind === 'divergence') {- flushErrors.set(p.docName, {- type: 'urn:ok:error:disk-divergence',- title:- 'The document changed on disk after your edit was prepared; your edit was NOT applied. Re-read the document and retry.',- });} else if (flushOutcome?.kind === 'stale-external-write') {flushErrors.set(p.docName, staleExternalWriteProblem(p.docName));} else {@@ -5145,7 +5190,6 @@ export function createApiExtension(options: ApiExtensionOptions): Extension & {buildAgentActor,flushDiskAndDetectOutcome,respondPersistenceFailure,- respondDiskDivergence,respondStaleExternalWrite,flushDocToDisk,log,@@ -5178,7 +5222,6 @@ export function createApiExtension(options: ApiExtensionOptions): Extension & {flushDiskAndDetectOutcome,respondStaleExternalWrite,respondPersistenceFailure,- respondDiskDivergence,attributeOkArtifactWrite,okArtifactKey,commitOkArtifactWrite,@@ -5227,7 +5270,6 @@ export function createApiExtension(options: ApiExtensionOptions): Extension & {flushDiskAndDetectOutcome,respondStaleExternalWrite,respondPersistenceFailure,- respondDiskDivergence,okArtifactKey,resolveSkillsRoot,extractActorIdentityFromQuery,@@ -5415,7 +5457,6 @@ export function createApiExtension(options: ApiExtensionOptions): Extension & {scheduleOkArtifactFlush,flushDiskAndDetectOutcome,respondPersistenceFailure,- respondDiskDivergence,respondStaleExternalWrite,registerWrittenDocInFileIndex,captureAndCloseDocuments,@@ -5446,7 +5487,6 @@ export function createApiExtension(options: ApiExtensionOptions): Extension & {flushDiskAndDetectOutcome,respondStaleExternalWrite,respondPersistenceFailure,- respondDiskDivergence,skillInstallOps,skillPlacementOps,signalChannel,@@ -5514,7 +5554,6 @@ export function createApiExtension(options: ApiExtensionOptions): Extension & {countNormalizedSummary,flushDiskAndDetectOutcome,respondPersistenceFailure,- respondDiskDivergence,flushDocToDisk,agentFocusBroadcaster,onAgentWrite,diff --git a/packages/server/src/config-persistence.test.ts b/packages/server/src/config-persistence.test.tsindex d1bb5979b..e08b647b8 100644--- a/packages/server/src/config-persistence.test.ts+++ b/packages/server/src/config-persistence.test.ts@@ -571,6 +571,7 @@ describe('persistence extension dispatch — config-doc integration', () => {const rejections: Array<{ docName: string; error: ConfigValidationError }> = [];const handle = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: fx.projectDir,projectDir: fx.projectDir,gitEnabled: false,@@ -591,6 +592,7 @@ describe('persistence extension dispatch — config-doc integration', () => {test('config-doc onStoreDocument validates + writes disk', async () => {const { createPersistenceExtension } = await import('./persistence.ts');const handle = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: fx.projectDir,projectDir: fx.projectDir,gitEnabled: false,@@ -616,6 +618,7 @@ describe('persistence extension dispatch — config-doc integration', () => {const notified: string[] = [];const handle = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: fx.projectDir,projectDir: fx.projectDir,gitEnabled: false,@@ -646,6 +649,7 @@ describe('persistence extension dispatch — config-doc integration', () => {const notified: string[] = [];const handle = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: fx.projectDir,projectDir: fx.projectDir,gitEnabled: false,@@ -680,6 +684,7 @@ describe('persistence extension dispatch — config-doc integration', () => {const rejections: Array<{ docName: string; error: ConfigValidationError }> = [];const handle = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: fx.projectDir,projectDir: fx.projectDir,gitEnabled: false,@@ -713,6 +718,7 @@ describe('persistence extension dispatch — config-doc integration', () => {const { createPersistenceExtension } = await import('./persistence.ts');const rejections: Array<{ docName: string; error: ConfigValidationError }> = [];const handle = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: fx.projectDir,projectDir: fx.projectDir,gitEnabled: false,diff --git a/packages/server/src/disk-receipts.test.ts b/packages/server/src/disk-receipts.test.tsnew file mode 100644index 000000000..b4e4502ef--- /dev/null+++ b/packages/server/src/disk-receipts.test.ts@@ -0,0 +1,185 @@+import {+ existsSync,+ lstatSync,+ mkdirSync,+ mkdtempSync,+ readFileSync,+ rmSync,+ symlinkSync,+ writeFileSync,+} from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import { afterEach, beforeEach, describe, expect, test } from 'vitest';+import { DiskReceiptLog } from './disk-receipts.ts';+import { getMetrics, resetMetrics } from './metrics.ts';+import { contentHash } from './version-hash.ts';++let dir: string;+let path: string;++beforeEach(() => {+ resetMetrics();+ dir = mkdtempSync(join(tmpdir(), 'ok-disk-receipts-'));+ path = join(dir, '.ok', 'local', 'receipts.jsonl');+});++afterEach(() => {+ rmSync(dir, { recursive: true, force: true });+});++function persistedLines(): Record<string, unknown>[] {+ return readFileSync(path, 'utf-8')+ .split('\n')+ .filter((line) => line !== '')+ .map((line) => JSON.parse(line) as Record<string, unknown>);+}++const observation = {+ size: 7n,+ mtimeNs: 1_700_000_000_123_456_789n,+ dev: 16777220n,+ ino: 2n ** 60n,+};++describe('DiskReceiptLog', () => {+ test('a recorded receipt carries the content hash, byte size and file identity, and survives a reload', () => {+ const log = new DiskReceiptLog({ path, now: () => 42 });+ log.recordOwnWrite('notes/a', 'héllo\n', observation);++ const reloaded = new DiskReceiptLog({ path });+ expect(reloaded.get('notes/a')).toEqual({+ hash: contentHash('héllo\n'),+ size: Buffer.byteLength('héllo\n'),+ mtimeNs: '1700000000123456789',+ dev: '16777220',+ ino: (2n ** 60n).toString(),+ at: 42,+ via: 'own-write',+ });+ });++ test('a receipt recorded without an observation has no identity or timestamp', () => {+ const log = new DiskReceiptLog({ path });+ log.recordAgreed('doc', 'agreed bytes\n');+ expect(log.get('doc')).toMatchObject({ mtimeNs: null, dev: null, ino: null, via: 'agreed' });+ });++ test('the last record for a document wins and a delete removes it across a reload', () => {+ const log = new DiskReceiptLog({ path });+ log.recordAgreed('kept', 'one\n');+ log.recordAgreed('kept', 'two\n');+ log.recordAgreed('gone', 'bytes\n');+ log.forget('gone');++ const reloaded = new DiskReceiptLog({ path });+ expect(reloaded.get('kept')?.hash).toBe(contentHash('two\n'));+ expect(reloaded.get('gone')).toBeUndefined();+ });++ test('recording the same bytes again appends nothing', () => {+ const log = new DiskReceiptLog({ path });+ const sameObservation = { ...observation, size: BigInt(Buffer.byteLength('same\n')) };+ log.recordAgreed('doc', 'same\n', sameObservation);+ log.recordAgreed('doc', 'same\n');+ log.recordAgreed('doc', 'same\n', sameObservation);+ expect(persistedLines()).toHaveLength(1);+ });++ test('loading compacts superseded lines and tombstones away', () => {+ const log = new DiskReceiptLog({ path });+ log.recordAgreed('a', '1\n');+ log.recordAgreed('a', '2\n');+ log.recordAgreed('b', '1\n');+ log.forget('b');+ expect(persistedLines()).toHaveLength(4);++ new DiskReceiptLog({ path });+ expect(persistedLines()).toEqual([+ expect.objectContaining({ doc: 'a', hash: contentHash('2\n') }),+ ]);+ });++ test('torn and malformed records are skipped and counted, records from another schema version are kept unread, and the rest still load', () => {+ mkdirSync(join(dir, '.ok', 'local'), { recursive: true });+ const valid = {+ v: 1,+ doc: 'good',+ hash: contentHash('ok\n'),+ size: 3,+ mtimeNs: null,+ dev: null,+ ino: null,+ at: 1,+ via: 'agreed',+ };+ const future = JSON.stringify({ v: 2, doc: 'future', digest: 'from a newer build' });+ writeFileSync(+ path,+ [+ JSON.stringify(valid),+ future,+ JSON.stringify({ ...valid, doc: 'bad-hash', hash: 'nope' }),+ '{"v":1,"doc":"torn","ha',+ ].join('\n'),+ );++ const log = new DiskReceiptLog({ path });+ expect(log.get('good')?.hash).toBe(contentHash('ok\n'));+ expect(log.get('future')).toBeUndefined();+ expect(log.get('bad-hash')).toBeUndefined();+ expect(log.get('torn')).toBeUndefined();+ expect(getMetrics().receiptRecordsUnreadable).toBe(2);+ expect(getMetrics().receiptRecordsOtherVersion).toBe(1);+ expect(readFileSync(path, 'utf-8').split('\n')).toContain(future);++ log.recordAgreed('later', 'x\n');+ new DiskReceiptLog({ path });+ expect(readFileSync(path, 'utf-8').split('\n')).toContain(future);+ });++ test('a symbolic link at the receipt log path is neither followed nor replaced, and receipts stay in memory', () => {+ mkdirSync(join(dir, '.ok', 'local'), { recursive: true });+ const target = join(dir, 'outside-target.jsonl');+ symlinkSync(target, path);++ const log = new DiskReceiptLog({ path });+ log.recordAgreed('doc', 'bytes\n');+ log.forget('doc');+ log.recordOwnWrite('doc', 'again\n');++ expect(existsSync(target)).toBe(false);+ expect(lstatSync(path).isSymbolicLink()).toBe(true);+ expect(log.get('doc')?.hash).toBe(contentHash('again\n'));+ expect(getMetrics().receiptLogRefused).toBe(1);+ });++ test('a receipt log that cannot be persisted keeps receipts in memory and counts the failure', () => {+ writeFileSync(join(dir, 'blocker'), 'a file where the state folder should be');+ const log = new DiskReceiptLog({ path: join(dir, 'blocker', 'receipts.jsonl') });+ log.recordAgreed('doc', 'bytes\n');+ expect(log.get('doc')?.hash).toBe(contentHash('bytes\n'));+ expect(getMetrics().receiptPersistFailures).toBe(1);+ });++ test('recent own writes are remembered by count, newest kept, and forgotten with the receipt', () => {+ const log = new DiskReceiptLog();+ for (let i = 0; i < 10; i++) log.recordOwnWrite('doc', `v${i}\n`);+ expect(log.isRecentOwnWrite('doc', contentHash('v9\n'))).toBe(true);+ expect(log.isRecentOwnWrite('doc', contentHash('v2\n'))).toBe(true);+ expect(log.isRecentOwnWrite('doc', contentHash('v1\n'))).toBe(false);+ log.recordAgreed('doc', 'outside\n');+ expect(log.isRecentOwnWrite('doc', contentHash('outside\n'))).toBe(false);+ log.forget('doc');+ expect(log.isRecentOwnWrite('doc', contentHash('v9\n'))).toBe(false);+ });++ test('pruning drops receipts whose file is gone and rewrites the log without them', () => {+ const log = new DiskReceiptLog({ path });+ log.recordAgreed('present', 'a\n');+ log.recordAgreed('missing', 'b\n');+ expect(log.pruneGone((docName) => docName === 'missing')).toBe(1);+ expect(log.get('missing')).toBeUndefined();+ expect(persistedLines().map((line) => line.doc)).toEqual(['present']);+ });+});diff --git a/packages/server/src/disk-receipts.ts b/packages/server/src/disk-receipts.tsnew file mode 100644index 000000000..c5a2a769a--- /dev/null+++ b/packages/server/src/disk-receipts.ts@@ -0,0 +1,371 @@+import { closeSync, constants, existsSync, lstatSync, openSync, readFileSync } from 'node:fs';+import { dirname } from 'node:path';+import { atomicTempPath } from '@inkeep/open-knowledge-core/server';+import {+ tracedAppendNoFollowSync,+ tracedMkdirSync,+ tracedRenameSync,+ tracedUnlinkSync,+ tracedWriteFileSync,+} from './fs-traced.ts';+import { errnoCode } from './http/handler-utils.ts';+import { getLogger } from './logger.ts';+import {+ incrementReceiptLogRefused,+ incrementReceiptPersistFailures,+ incrementReceiptRecordsOtherVersion,+ incrementReceiptRecordsUnreadable,+} from './metrics.ts';+import { contentHash } from './version-hash.ts';++const DISK_RECEIPT_SCHEMA_VERSION = 1;++const RECENT_OWN_WRITE_LIMIT = 8;+const MIN_COMPACTION_LINES = 2048;+const COMPACTION_FACTOR = 4;++export interface FileObservation {+ readonly size: bigint;+ readonly mtimeNs: bigint;+ readonly dev: bigint;+ readonly ino: bigint;+}++export interface DiskReceipt {+ readonly hash: string;+ readonly size: number;+ readonly mtimeNs: string | null;+ readonly dev: string | null;+ readonly ino: string | null;+ readonly at: number;+ readonly via: 'own-write' | 'agreed';+}++interface PersistedReceipt extends DiskReceipt {+ readonly v: typeof DISK_RECEIPT_SCHEMA_VERSION;+ readonly doc: string;+}++interface PersistedTombstone {+ readonly v: typeof DISK_RECEIPT_SCHEMA_VERSION;+ readonly doc: string;+ readonly gone: true;+ readonly at: number;+}++interface DiskReceiptLogOptions {+ path?: string;+ now?: () => number;+}++const log = getLogger('disk-receipts');++function isNullableDecimal(value: unknown): value is string | null {+ return value === null || (typeof value === 'string' && /^\d+$/.test(value));+}++type ParsedLine = PersistedReceipt | PersistedTombstone | 'other-version' | null;++function parseLine(line: string): ParsedLine {+ let value: unknown;+ try {+ value = JSON.parse(line);+ } catch {+ return null;+ }+ if (!value || typeof value !== 'object') return null;+ const record = value as Record<string, unknown>;+ if (typeof record.doc !== 'string') return null;+ if (typeof record.v === 'number' && record.v !== DISK_RECEIPT_SCHEMA_VERSION) {+ return 'other-version';+ }+ if (record.v !== DISK_RECEIPT_SCHEMA_VERSION) return null;+ if (typeof record.at !== 'number') return null;+ if (record.gone === true) {+ return { v: DISK_RECEIPT_SCHEMA_VERSION, doc: record.doc, gone: true, at: record.at };+ }+ if (+ typeof record.hash !== 'string' ||+ !/^[0-9a-f]{64}$/.test(record.hash) ||+ typeof record.size !== 'number' ||+ !isNullableDecimal(record.mtimeNs) ||+ !isNullableDecimal(record.dev) ||+ !isNullableDecimal(record.ino) ||+ (record.via !== 'own-write' && record.via !== 'agreed')+ ) {+ return null;+ }+ return {+ v: DISK_RECEIPT_SCHEMA_VERSION,+ doc: record.doc,+ hash: record.hash,+ size: record.size,+ mtimeNs: record.mtimeNs,+ dev: record.dev,+ ino: record.ino,+ at: record.at,+ via: record.via,+ };+}++function sameObservation(receipt: DiskReceipt, observation: FileObservation): boolean {+ return (+ receipt.size === Number(observation.size) &&+ receipt.mtimeNs === observation.mtimeNs.toString() &&+ receipt.dev === observation.dev.toString() &&+ receipt.ino === observation.ino.toString()+ );+}++export class DiskReceiptLog {+ private readonly receipts = new Map<string, DiskReceipt>();+ private readonly otherVersionLines: string[] = [];+ private readonly recentOwnWrites = new Map<string, string[]>();+ private readonly path: string | undefined;+ private readonly now: () => number;+ private linesSinceCompaction = 0;+ private directoryReady = false;+ private persistFailures = 0;+ private refused = false;++ constructor(options: DiskReceiptLogOptions = {}) {+ this.path = options.path;+ this.now = options.now ?? Date.now;+ this.load();+ }++ get(docName: string): DiskReceipt | undefined {+ return this.receipts.get(docName);+ }++ isRecentOwnWrite(docName: string, hash: string): boolean {+ return this.recentOwnWrites.get(docName)?.includes(hash) ?? false;+ }++ recordAgreed(docName: string, content: string, observation?: FileObservation): void {+ this.record(docName, content, 'agreed', observation);+ }++ recordOwnWrite(docName: string, content: string, observation?: FileObservation): void {+ this.record(docName, content, 'own-write', observation);+ const hash = contentHash(content);+ const recent = (this.recentOwnWrites.get(docName) ?? []).filter((entry) => entry !== hash);+ recent.push(hash);+ this.recentOwnWrites.set(docName, recent.slice(-RECENT_OWN_WRITE_LIMIT));+ }++ forget(docName: string): void {+ this.recentOwnWrites.delete(docName);+ if (!this.receipts.delete(docName)) return;+ this.append({ v: DISK_RECEIPT_SCHEMA_VERSION, doc: docName, gone: true, at: this.now() });+ }++ pruneGone(isGone: (docName: string) => boolean): number {+ let pruned = 0;+ for (const docName of this.receipts.keys()) {+ if (!isGone(docName)) continue;+ this.receipts.delete(docName);+ this.recentOwnWrites.delete(docName);+ pruned++;+ }+ if (pruned > 0) this.compact();+ return pruned;+ }++ private record(+ docName: string,+ content: string,+ via: DiskReceipt['via'],+ observation: FileObservation | undefined,+ ): void {+ const hash = contentHash(content);+ const existing = this.receipts.get(docName);+ if (+ existing?.hash === hash &&+ (observation === undefined || sameObservation(existing, observation))+ ) {+ return;+ }+ const receipt: DiskReceipt = {+ hash,+ size: Buffer.byteLength(content, 'utf8'),+ mtimeNs: observation ? observation.mtimeNs.toString() : null,+ dev: observation ? observation.dev.toString() : null,+ ino: observation ? observation.ino.toString() : null,+ at: this.now(),+ via,+ };+ this.receipts.set(docName, receipt);+ this.append({ v: DISK_RECEIPT_SCHEMA_VERSION, doc: docName, ...receipt });+ }++ private append(record: PersistedReceipt | PersistedTombstone): void {+ if (!this.path || this.refused) return;+ try {+ if (!this.directoryReady) {+ tracedMkdirSync(dirname(this.path), { recursive: true });+ this.directoryReady = true;+ if (this.refuseUnsafeLogPath()) return;+ }+ tracedAppendNoFollowSync(this.path, `${JSON.stringify(record)}\n`, 0o600);+ this.linesSinceCompaction++;+ } catch (err) {+ if (errnoCode(err) === 'ELOOP') {+ this.refuse('symlink');+ return;+ }+ this.reportPersistFailure(err, 'append');+ return;+ }+ if (+ this.linesSinceCompaction >+ Math.max(MIN_COMPACTION_LINES, COMPACTION_FACTOR * this.receipts.size)+ ) {+ this.compact();+ }+ }++ private compact(): void {+ if (!this.path || this.refused) return;+ const lines: string[] = [...this.otherVersionLines];+ for (const [docName, receipt] of this.receipts) {+ lines.push(JSON.stringify({ v: DISK_RECEIPT_SCHEMA_VERSION, doc: docName, ...receipt }));+ }+ const temporaryPath = atomicTempPath(this.path);+ try {+ tracedMkdirSync(dirname(this.path), { recursive: true });+ this.directoryReady = true;+ tracedWriteFileSync(temporaryPath, lines.length > 0 ? `${lines.join('\n')}\n` : '', {+ encoding: 'utf-8',+ mode: 0o600,+ flag: 'wx',+ });+ tracedRenameSync(temporaryPath, this.path);+ this.linesSinceCompaction = 0;+ } catch (err) {+ this.reportPersistFailure(err, 'compact');+ try {+ if (existsSync(temporaryPath)) tracedUnlinkSync(temporaryPath);+ } catch (cleanupErr) {+ log.warn({ err: cleanupErr }, '[disk-receipts] could not remove the compaction temp file');+ }+ }+ }++ private refuseUnsafeLogPath(): boolean {+ if (!this.path) return false;+ let stats: ReturnType<typeof lstatSync> | undefined;+ try {+ stats = lstatSync(this.path, { throwIfNoEntry: false });+ } catch (err) {+ if (errnoCode(err) === 'ENOTDIR') return false;+ this.reportPersistFailure(err, 'append');+ return false;+ }+ if (stats === undefined || stats.isFile()) return false;+ this.refuse(stats.isSymbolicLink() ? 'symlink' : 'not-a-file');+ return true;+ }++ private refuse(reason: 'symlink' | 'not-a-file'): void {+ this.refused = true;+ incrementReceiptLogRefused();+ log.warn(+ { reason },+ `[disk-receipts] the receipt log path is a ${reason === 'symlink' ? 'symbolic link' : 'non-regular file'}; it is neither read nor written, and receipts stay in memory for this run`,+ );+ }++ private readLog(): string | null {+ if (!this.path) return null;+ if (this.refuseUnsafeLogPath()) return null;+ let fd: number;+ try {+ fd = openSync(this.path, constants.O_RDONLY | (constants.O_NOFOLLOW ?? 0));+ } catch (err) {+ const code = errnoCode(err);+ if (code === 'ENOENT') return null;+ if (code === 'ELOOP') {+ this.refuse('symlink');+ return null;+ }+ incrementReceiptRecordsUnreadable();+ log.warn(+ { err },+ '[disk-receipts] receipt log unreadable; every document starts without a receipt',+ );+ return null;+ }+ try {+ return readFileSync(fd, 'utf-8');+ } catch (err) {+ incrementReceiptRecordsUnreadable();+ log.warn(+ { err },+ '[disk-receipts] receipt log unreadable; every document starts without a receipt',+ );+ return null;+ } finally {+ closeSync(fd);+ }+ }++ private load(): void {+ const raw = this.readLog();+ if (raw === null) return;+ let lines = 0;+ let unreadable = 0;+ for (const line of raw.split('\n')) {+ if (line.trim() === '') continue;+ lines++;+ const record = parseLine(line);+ if (record === null) {+ unreadable++;+ continue;+ }+ if (record === 'other-version') {+ this.otherVersionLines.push(line);+ continue;+ }+ if ('gone' in record) {+ this.receipts.delete(record.doc);+ continue;+ }+ this.receipts.set(record.doc, {+ hash: record.hash,+ size: record.size,+ mtimeNs: record.mtimeNs,+ dev: record.dev,+ ino: record.ino,+ at: record.at,+ via: record.via,+ });+ }+ if (unreadable > 0) {+ incrementReceiptRecordsUnreadable(unreadable);+ log.warn(+ { unreadable, lines },+ `[disk-receipts] skipped ${unreadable} unreadable receipt record(s); those documents start without a receipt`,+ );+ }+ if (this.otherVersionLines.length > 0) {+ incrementReceiptRecordsOtherVersion(this.otherVersionLines.length);+ log.info(+ { otherVersion: this.otherVersionLines.length },+ `[disk-receipts] kept ${this.otherVersionLines.length} receipt record(s) written by another schema version without reading them`,+ );+ }+ if (lines !== this.receipts.size + this.otherVersionLines.length) this.compact();+ }++ private reportPersistFailure(err: unknown, operation: 'append' | 'compact'): void {+ incrementReceiptPersistFailures();+ this.persistFailures++;+ if (this.persistFailures === 1 || this.persistFailures % 100 === 0) {+ log.warn(+ { err, operation, failures: this.persistFailures },+ '[disk-receipts] could not persist the receipt log; receipts stay in memory for this run',+ );+ }+ }+}diff --git a/packages/server/src/document-durability-state.test.ts b/packages/server/src/document-durability-state.test.tsindex ccd41f2cf..fb23e7c2f 100644--- a/packages/server/src/document-durability-state.test.ts+++ b/packages/server/src/document-durability-state.test.ts@@ -13,6 +13,7 @@ import {import { tmpdir } from 'node:os';import { join } from 'node:path';import { describe, expect, test, vi } from 'vitest';+import { DiskReceiptLog } from './disk-receipts.ts';import {_resetDocExtensionsForTests,docNameToRelativePath,@@ -30,6 +31,7 @@ import {import * as tracedFs from './fs-traced.ts';import { getLogger } from './logger.ts';import { getMetrics, resetMetrics } from './metrics.ts';+import { contentHash } from './version-hash.ts';describe('DocumentDurabilityState', () => {test('starts with an empty main scope and no transient coordination state', () => {@@ -41,7 +43,6 @@ describe('DocumentDurabilityState', () => {expect(state.isBatchInProgress()).toBe(false);expect(state.consumeAgentWriteStore('doc')).toBe(false);expect(state.takeStoreFailure('doc')).toBeNull();- expect(state.takeStoreDivergence('doc')).toBe(false);});test('retains reconciled bases independently for each visited branch', () => {@@ -76,7 +77,6 @@ describe('DocumentDurabilityState', () => {first.beginInFlightFlush('doc', 'first flush');first.markAgentWriteStore('doc');first.recordStoreFailure('doc', { code: 'ENOSPC', message: 'full' });- first.recordStoreDivergence('doc');first.openDeferredStore('doc');expect(second.getReconciledBase('doc')).toBeUndefined();@@ -84,7 +84,6 @@ describe('DocumentDurabilityState', () => {expect(second.peekInFlightFlush('doc')).toBeUndefined();expect(second.consumeAgentWriteStore('doc')).toBe(false);expect(second.takeStoreFailure('doc')).toBeNull();- expect(second.takeStoreDivergence('doc')).toBe(false);await expect(second.awaitDeferredStoreOutcome('doc')).resolves.toBeUndefined();});@@ -246,18 +245,15 @@ describe('DocumentDurabilityState', () => {expect(state.peekInFlightFlush('doc')).toBeUndefined();});- test('consumes agent markers, failures, and divergences once', () => {+ test('consumes agent markers and failures once', () => {const state = new DocumentDurabilityState();state.markAgentWriteStore('doc');state.recordStoreFailure('doc', { message: 'write failed' });- state.recordStoreDivergence('doc');expect(state.consumeAgentWriteStore('doc')).toBe(true);expect(state.consumeAgentWriteStore('doc')).toBe(false);expect(state.takeStoreFailure('doc')).toEqual({ message: 'write failed' });expect(state.takeStoreFailure('doc')).toBeNull();- expect(state.takeStoreDivergence('doc')).toBe(true);- expect(state.takeStoreDivergence('doc')).toBe(false);});test('clears a store failure without consuming another document failure', () => {@@ -1477,3 +1473,74 @@ describe('DocumentDurabilityStateError names the running channel CLI', () => {}});});++describe('the durable receipt follows the acknowledged base', () => {+ test('setting the base records it as agreed, a store records it as an own write, and a delete forgets it', () => {+ const state = new DocumentDurabilityState();+ state.setReconciledBase('doc', 'loaded\n');+ expect(state.getReceipt('doc')).toMatchObject({+ hash: contentHash('loaded\n'),+ via: 'agreed',+ });++ const observation = { size: 7n, mtimeNs: 5n, dev: 1n, ino: 9n };+ state.recordSuccessfulStore('doc', 'stored\n', undefined, observation);+ expect(state.getReceipt('doc')).toMatchObject({+ hash: contentHash('stored\n'),+ via: 'own-write',+ ino: '9',+ });+ expect(state.isRecentOwnWrite('doc', contentHash('loaded\n'))).toBe(false);+ expect(state.isRecentOwnWrite('doc', contentHash('stored\n'))).toBe(true);++ state.deleteReconciledBase('doc');+ expect(state.getReceipt('doc')).toBeUndefined();+ });++ test('a store that displaces an agent-acknowledged version still records the written bytes', () => {+ const state = new DocumentDurabilityState();+ state.setReconciledBase('doc', 'before\n');+ state.recordSuccessfulStore('doc', 'after\n', 'before\n');+ expect(state.getReceipt('doc')?.hash).toBe(contentHash('after\n'));+ expect(state.getReconciledBase('doc')).toBe('after\n');+ });++ test('the check at load reports a missing, unchanged or changed receipt and counts it', () => {+ resetMetrics();+ const dir = mkdtempSync(join(tmpdir(), 'ok-durability-receipts-'));+ try {+ const path = join(dir, 'receipts.jsonl');+ const before = new DocumentDurabilityState('main', {+ receipts: new DiskReceiptLog({ path }),+ });+ expect(before.checkReceiptAtLoad('doc', 'first\n')).toBe('missing');+ before.setReconciledBase('doc', 'first\n');++ const restarted = new DocumentDurabilityState('main', {+ receipts: new DiskReceiptLog({ path }),+ });+ expect(restarted.getReconciledBase('doc')).toBeUndefined();+ expect(restarted.checkReceiptAtLoad('doc', 'first\n')).toBe('unchanged');+ expect(restarted.checkReceiptAtLoad('doc', 'edited while closed\n')).toBe('changed');+ expect(getMetrics().receiptAtLoad).toEqual({ missing: 1, unchanged: 1, changed: 1 });+ } finally {+ rmSync(dir, { recursive: true, force: true });+ }+ });++ test('an outside change to a document that is not loaded refreshes the receipt it already has', () => {+ const receipts = new DiskReceiptLog();+ receipts.recordAgreed('closed', 'v1\n');+ const state = new DocumentDurabilityState('main', { receipts });++ state.refreshReceiptWithoutBase('closed', 'v2\n');+ expect(state.getReceipt('closed')?.hash).toBe(contentHash('v2\n'));++ state.refreshReceiptWithoutBase('never-loaded', 'bytes\n');+ expect(state.getReceipt('never-loaded')).toBeUndefined();++ state.setReconciledBase('loaded', 'v1\n');+ state.refreshReceiptWithoutBase('loaded', 'v2\n');+ expect(state.getReceipt('loaded')?.hash).toBe(contentHash('v1\n'));+ });+});diff --git a/packages/server/src/document-durability-state.ts b/packages/server/src/document-durability-state.tsindex b528d448f..38d831fae 100644--- a/packages/server/src/document-durability-state.ts+++ b/packages/server/src/document-durability-state.ts@@ -3,6 +3,7 @@ import { existsSync, readFileSync } from 'node:fs';import { dirname } from 'node:path';import { currentCliCommandName } from '@inkeep/open-knowledge-core';import { atomicTempPath } from '@inkeep/open-knowledge-core/server';+import { type DiskReceipt, DiskReceiptLog, type FileObservation } from './disk-receipts.ts';import {tracedMkdirSync,tracedRenameSync,@@ -10,13 +11,24 @@ import {tracedWriteFileSync,} from './fs-traced.ts';import { getLogger } from './logger.ts';-import { incrementInFlightFlushExpired, incrementStaleExternalWriteRefused } from './metrics.ts';+import {+ incrementInFlightFlushExpired,+ incrementReceiptAtLoad,+ incrementStaleExternalWriteRefused,+ type ReceiptAtLoad,+ type StorePendingReason,+} from './metrics.ts';import { contentHash } from './version-hash.ts';export interface StoreFailure {code?: string;message: string;removal?: StoreFailureRemoval;+ held?: {+ readonly reason: StorePendingReason;+ readonly causeCode?: string;+ readonly noBase?: true;+ };}export type StoreFailureRemoval =@@ -52,6 +64,7 @@ export function assertNeverDeferredStoreWait(wait: never): never {export const OK_DOC_REMOVED = 'OK_DOC_REMOVED';export const OK_PATH_UNRESOLVABLE = 'OK_PATH_UNRESOLVABLE';export const OK_STORE_REFUSED = 'OK_STORE_REFUSED';+export const OK_STORE_HELD = 'OK_STORE_HELD';export const OK_STORE_DISCARDED = 'OK_STORE_DISCARDED';const IN_FLIGHT_FLUSH_TTL_MS = 60_000;@@ -169,6 +182,7 @@ export interface DocumentDurabilityStateOptions {onStaleExternalWriteChange?: () => void;onStoreRefused?: (docName: string) => void;hasResolvedExtension?: (docName: string) => boolean;+ receipts?: DiskReceiptLog;}export class DocumentDurabilityStateError extends Error {@@ -280,13 +294,13 @@ export class DocumentDurabilityState {private readonly agentWriteStores = new Set<string>();private readonly refusedStoreDocNames = new Set<string>();private readonly storeFailures = new Map<string, StoreFailure>();- private readonly storeDivergences = new Set<string>();private readonly staleExternalWriteFreezes = new Set<string>();private readonly persistencePath: string | undefined;private readonly fileForDocName: (docName: string) => string;private readonly onStaleExternalWriteChange: (() => void) | undefined;private readonly onStoreRefused: ((docName: string) => void) | undefined;private readonly hasResolvedExtension: ((docName: string) => boolean) | undefined;+ private readonly receipts: DiskReceiptLog;private cachedDocumentsByBranch = new Map<string, Map<string, CachedDurabilityDocument>>();private lastPersistedParts = snapshotParts(new Map());private nextDisplacedExpiryAt = Number.POSITIVE_INFINITY;@@ -304,6 +318,7 @@ export class DocumentDurabilityState {this.onStaleExternalWriteChange = options.onStaleExternalWriteChange;this.onStoreRefused = options.onStoreRefused;this.hasResolvedExtension = options.hasResolvedExtension;+ this.receipts = options.receipts ?? new DiskReceiptLog();this.reconciledBaseByBranch.set(initialBranch, new Map());this.restore();}@@ -324,7 +339,34 @@ export class DocumentDurabilityState {return this.reconciledBaseByBranch.get(this.activeBranch)?.get(docName);}- setReconciledBase(docName: string, content: string): void {+ getReceipt(docName: string): DiskReceipt | undefined {+ return this.receipts.get(docName);+ }++ isRecentOwnWrite(docName: string, hash: string): boolean {+ return this.receipts.isRecentOwnWrite(docName, hash);+ }++ checkReceiptAtLoad(docName: string, diskContent: string): ReceiptAtLoad {+ const receipt = this.receipts.get(docName);+ const outcome: ReceiptAtLoad =+ receipt === undefined+ ? 'missing'+ : receipt.hash === contentHash(diskContent)+ ? 'unchanged'+ : 'changed';+ incrementReceiptAtLoad(outcome);+ return outcome;+ }++ refreshReceiptWithoutBase(docName: string, diskContent: string): void {+ if (this.getReconciledBase(docName) !== undefined) return;+ if (this.receipts.get(docName) === undefined) return;+ this.receipts.recordAgreed(docName, diskContent);+ }++ setReconciledBase(docName: string, content: string, observation?: FileObservation): void {+ this.receipts.recordAgreed(docName, content, observation);let bases = this.reconciledBaseByBranch.get(this.activeBranch);if (!bases) {bases = new Map();@@ -339,7 +381,28 @@ export class DocumentDurabilityState {this.mutateDocument(docName, () => bases.set(docName, content));}- recordSuccessfulStore(docName: string, content: string, displacedContent?: string): void {+ restoreReconciledBase(docName: string, previous: string | undefined): void {+ if (previous !== undefined) {+ this.setReconciledBase(docName, previous);+ return;+ }+ this.receipts.forget(docName);+ const bases = this.reconciledBaseByBranch.get(this.activeBranch);+ if (!bases?.has(docName)) return;+ if (!this.isPersistedDocument(docName)) {+ bases.delete(docName);+ return;+ }+ this.mutateDocument(docName, () => bases.delete(docName));+ }++ recordSuccessfulStore(+ docName: string,+ content: string,+ displacedContent?: string,+ observation?: FileObservation,+ ): void {+ this.receipts.recordOwnWrite(docName, content, observation);if (displacedContent === undefined) {this.setReconciledBase(docName, content);return;@@ -376,6 +439,7 @@ export class DocumentDurabilityState {* reads that same base and returns early without it.*/deleteReconciledBase(docName: string): void {+ this.receipts.forget(docName);this.refusedStoreDocNames.delete(docName);this.storeFailures.delete(docName);const persist = this.isPersistedDocument(docName);@@ -954,14 +1018,6 @@ export class DocumentDurabilityState {return failure;}- recordStoreDivergence(docName: string): void {- this.storeDivergences.add(docName);- }-- takeStoreDivergence(docName: string): boolean {- return this.storeDivergences.delete(docName);- }-recordStaleExternalWriteFreeze(docName: string): void {this.staleExternalWriteFreezes.add(docName);}diff --git a/packages/server/src/fs-traced.ts b/packages/server/src/fs-traced.tsindex 592a9494d..1bc88ca1e 100644--- a/packages/server/src/fs-traced.ts+++ b/packages/server/src/fs-traced.ts@@ -1,18 +1,22 @@-import type { RmOptions, WriteFileOptions } from 'node:fs';+import type { BigIntStats, RmOptions, WriteFileOptions } from 'node:fs';import {appendFileSync,chmodSync,closeSync,+ constants,cpSync,+ fstatSync,linkSync,mkdirSync,openSync,+ readFileSync,renameSync,rmdirSync,rmSync,symlinkSync,unlinkSync,writeFileSync,+ writeSync,} from 'node:fs';import { appendFile, mkdir, rename, rm, writeFile } from 'node:fs/promises';import { basename, sep } from 'node:path';@@ -146,6 +150,35 @@ export function tracedAppendFileSync();}+export function tracedAppendNoFollowSync(path: string, data: string, mode: number): void {+ withSpanSync(+ 'fs.appendFileSync',+ { attributes: buildAttrs('appendFileSync', path, { 'fs.bytes': byteLength(data) }) },+ () => {+ const fd = openSync(+ path,+ constants.O_WRONLY | constants.O_APPEND | constants.O_CREAT | (constants.O_NOFOLLOW ?? 0),+ mode,+ );+ try {+ const bytes = Buffer.from(data, 'utf8');+ let offset = 0;+ while (offset < bytes.length) {+ const written = writeSync(fd, bytes, offset, bytes.length - offset);+ if (written <= 0) {+ throw Object.assign(new Error(`no bytes written appending to ${path}`), {+ code: 'EIO',+ });+ }+ offset += written;+ }+ } finally {+ closeSync(fd);+ }+ },+ );+}+export function tracedMkdirSync(path: string,options?: Parameters<typeof mkdirSync>[1],@@ -185,6 +218,30 @@ function tracedOpenSync(path: string, flags: string, mode: number): number {);}+export interface ObservedFileRead {+ readonly bytes: Buffer;+ readonly before: BigIntStats;+ readonly after: BigIntStats;+}++export function tracedReadObservedSync(path: string): ObservedFileRead {+ return withSpanSync(+ 'fs.readObservedSync',+ { attributes: buildAttrs('readObservedSync', path) },+ () => {+ const fd = openSync(path, 'r');+ try {+ const before = fstatSync(fd, { bigint: true });+ const bytes = readFileSync(fd);+ const after = fstatSync(fd, { bigint: true });+ return { bytes, before, after };+ } finally {+ closeSync(fd);+ }+ },+ );+}+export function tracedUnlinkSync(path: string): void {withSpanSync('fs.unlinkSync', { attributes: buildAttrs('unlinkSync', path) }, () => {unlinkSync(path);diff --git a/packages/server/src/http/agent-write-routes.test.ts b/packages/server/src/http/agent-write-routes.test.tsindex 1983e8454..66c6c2724 100644--- a/packages/server/src/http/agent-write-routes.test.ts+++ b/packages/server/src/http/agent-write-routes.test.ts@@ -46,7 +46,6 @@ function buildGroup(overrides: Partial<Deps> = {}) {countNormalizedSummary: notDispatched,flushDiskAndDetectOutcome: notDispatched,respondPersistenceFailure: notDispatched,- respondDiskDivergence: notDispatched,flushDocToDisk: notDispatched,agentFocusBroadcaster: undefined,onAgentWrite: undefined,diff --git a/packages/server/src/http/agent-write-routes.ts b/packages/server/src/http/agent-write-routes.tsindex feb6f9ab3..a4e416271 100644--- a/packages/server/src/http/agent-write-routes.ts+++ b/packages/server/src/http/agent-write-routes.ts@@ -189,13 +189,9 @@ export interface AgentWriteRouteDeps {flushDiskAndDetectOutcome: (docName: string,) => Promise<- | { kind: 'failure'; failure: StoreFailure }- | { kind: 'divergence' }- | { kind: 'stale-external-write' }- | null+ { kind: 'failure'; failure: StoreFailure } | { kind: 'stale-external-write' } | null>;respondPersistenceFailure: (res: ServerResponse, failure: StoreFailure, handler: string) => void;- respondDiskDivergence: (res: ServerResponse, handler: string) => void;flushDocToDisk: (docName: string, label: string) => void;agentFocusBroadcaster: AgentFocusBroadcaster | undefined;onAgentWrite: (() => void) | undefined;@@ -251,7 +247,6 @@ export function createAgentWriteRoutes(deps: AgentWriteRouteDeps): ApiRouteGroupcountNormalizedSummary,flushDiskAndDetectOutcome,respondPersistenceFailure,- respondDiskDivergence,flushDocToDisk,agentFocusBroadcaster,onAgentWrite,@@ -470,10 +465,6 @@ export function createAgentWriteRoutes(deps: AgentWriteRouteDeps): ApiRouteGrouprespondPersistenceFailure(res, flushOutcome.failure, 'agent-write-md');return;}- if (flushOutcome?.kind === 'divergence') {- respondDiskDivergence(res, 'agent-write-md');- return;- }if (flushOutcome?.kind === 'stale-external-write') {respondStaleExternalWrite(res, 'agent-write-md', resolvedDocName);return;@@ -740,10 +731,6 @@ export function createAgentWriteRoutes(deps: AgentWriteRouteDeps): ApiRouteGrouprespondPersistenceFailure(res, flushOutcome.failure, 'frontmatter-patch');return;}- if (flushOutcome?.kind === 'divergence') {- respondDiskDivergence(res, 'frontmatter-patch');- return;- }if (flushOutcome?.kind === 'stale-external-write') {respondStaleExternalWrite(res, 'frontmatter-patch', resolvedDocName);return;@@ -1051,10 +1038,6 @@ export function createAgentWriteRoutes(deps: AgentWriteRouteDeps): ApiRouteGrouprespondPersistenceFailure(res, flushOutcome.failure, 'agent-patch');return;}- if (flushOutcome?.kind === 'divergence') {- respondDiskDivergence(res, 'agent-patch');- return;- }if (flushOutcome?.kind === 'stale-external-write') {respondStaleExternalWrite(res, 'agent-patch', docName);return;@@ -1221,10 +1204,6 @@ export function createAgentWriteRoutes(deps: AgentWriteRouteDeps): ApiRouteGrouprespondPersistenceFailure(res, flushOutcome.failure, 'agent-undo');return;}- if (flushOutcome?.kind === 'divergence') {- respondDiskDivergence(res, 'agent-undo');- return;- }if (flushOutcome?.kind === 'stale-external-write') {respondStaleExternalWrite(res, 'agent-undo', docName);return;@@ -1671,10 +1650,6 @@ export function createAgentWriteRoutes(deps: AgentWriteRouteDeps): ApiRouteGrouprespondPersistenceFailure(res, flushOutcome.failure, 'rollback');return;}- if (flushOutcome?.kind === 'divergence') {- respondDiskDivergence(res, 'rollback');- return;- }if (flushOutcome?.kind === 'stale-external-write') {respondStaleExternalWrite(res, 'rollback', docName);return;diff --git a/packages/server/src/http/folder-template-routes.test.ts b/packages/server/src/http/folder-template-routes.test.tsindex a3d43f12d..b8c1a86c5 100644--- a/packages/server/src/http/folder-template-routes.test.ts+++ b/packages/server/src/http/folder-template-routes.test.ts@@ -32,7 +32,6 @@ function buildGroup() {scheduleOkArtifactFlush: () => {},flushDiskAndDetectOutcome: () => Promise.resolve(null),respondPersistenceFailure: () => {},- respondDiskDivergence: () => {},respondStaleExternalWrite: () => {},registerWrittenDocInFileIndex: () => {},captureAndCloseDocuments: () => Promise.resolve(new Map()),diff --git a/packages/server/src/http/folder-template-routes.ts b/packages/server/src/http/folder-template-routes.tsindex b87599a30..4c32f79b6 100644--- a/packages/server/src/http/folder-template-routes.ts+++ b/packages/server/src/http/folder-template-routes.ts@@ -107,13 +107,9 @@ export interface FolderTemplateRouteDeps {flushDiskAndDetectOutcome: (docName: string,) => Promise<- | { kind: 'failure'; failure: StoreFailure }- | { kind: 'divergence' }- | { kind: 'stale-external-write' }- | null+ { kind: 'failure'; failure: StoreFailure } | { kind: 'stale-external-write' } | null>;respondPersistenceFailure: (res: ServerResponse, failure: StoreFailure, handler: string) => void;- respondDiskDivergence: (res: ServerResponse, handler: string) => void;respondStaleExternalWrite: (res: ServerResponse, handler: string, docName: string) => void;registerWrittenDocInFileIndex: (docName: string, content: string) => void;captureAndCloseDocuments: (@@ -161,7 +157,6 @@ export function createFolderTemplateRoutes(deps: FolderTemplateRouteDeps): ApiRoscheduleOkArtifactFlush,flushDiskAndDetectOutcome,respondPersistenceFailure,- respondDiskDivergence,respondStaleExternalWrite,registerWrittenDocInFileIndex,captureAndCloseDocuments,@@ -707,10 +702,6 @@ export function createFolderTemplateRoutes(deps: FolderTemplateRouteDeps): ApiRorespondPersistenceFailure(res, templateFlush.failure, 'template-put');return;}- if (templateFlush?.kind === 'divergence') {- respondDiskDivergence(res, 'template-put');- return;- }if (templateFlush?.kind === 'stale-external-write') {respondStaleExternalWrite(res, 'template-put', templateDocName);return;@@ -1322,10 +1313,6 @@ export function createFolderTemplateRoutes(deps: FolderTemplateRouteDeps): ApiRorespondPersistenceFailure(res, templateFlush.failure, 'template-import');return;}- if (templateFlush?.kind === 'divergence') {- respondDiskDivergence(res, 'template-import');- return;- }if (templateFlush?.kind === 'stale-external-write') {respondStaleExternalWrite(res, 'template-import', templateDocName);return;diff --git a/packages/server/src/http/lint-write-routes.test.ts b/packages/server/src/http/lint-write-routes.test.tsindex 48fd48cfd..fcd7e3ca4 100644--- a/packages/server/src/http/lint-write-routes.test.ts+++ b/packages/server/src/http/lint-write-routes.test.ts@@ -48,7 +48,6 @@ function buildGroup(overrides: Partial<Deps> = {}) {buildAgentActor: notDispatched,flushDiskAndDetectOutcome: notDispatched,respondPersistenceFailure: notDispatched,- respondDiskDivergence: notDispatched,respondStaleExternalWrite: notDispatched,flushDocToDisk: notDispatched,log: loggerFactory.getLogger('test'),diff --git a/packages/server/src/http/lint-write-routes.ts b/packages/server/src/http/lint-write-routes.tsindex c759ade25..2af2fb2a1 100644--- a/packages/server/src/http/lint-write-routes.ts+++ b/packages/server/src/http/lint-write-routes.ts@@ -99,13 +99,9 @@ export interface LintWriteRouteDeps {flushDiskAndDetectOutcome: (docName: string,) => Promise<- | { kind: 'failure'; failure: StoreFailure }- | { kind: 'divergence' }- | { kind: 'stale-external-write' }- | null+ { kind: 'failure'; failure: StoreFailure } | { kind: 'stale-external-write' } | null>;respondPersistenceFailure: (res: ServerResponse, failure: StoreFailure, handler: string) => void;- respondDiskDivergence: (res: ServerResponse, handler: string) => void;respondStaleExternalWrite: (res: ServerResponse, handler: string, docName: string) => void;flushDocToDisk: (docName: string, label: string) => void;log: PinoLogger;@@ -131,7 +127,6 @@ export function createLintWriteRoutes(deps: LintWriteRouteDeps): ApiRouteGroup {buildAgentActor,flushDiskAndDetectOutcome,respondPersistenceFailure,- respondDiskDivergence,respondStaleExternalWrite,flushDocToDisk,log,@@ -393,10 +388,6 @@ export function createLintWriteRoutes(deps: LintWriteRouteDeps): ApiRouteGroup {respondPersistenceFailure(res, flushOutcome.failure, 'lint-fix');return;}- if (flushOutcome?.kind === 'divergence') {- respondDiskDivergence(res, 'lint-fix');- return;- }if (flushOutcome?.kind === 'stale-external-write') {respondStaleExternalWrite(res, 'lint-fix', resolvedDocName);return;diff --git a/packages/server/src/http/skills-document-routes.test.ts b/packages/server/src/http/skills-document-routes.test.tsindex 6c7423891..c6b9326ce 100644--- a/packages/server/src/http/skills-document-routes.test.ts+++ b/packages/server/src/http/skills-document-routes.test.ts@@ -32,7 +32,6 @@ function buildGroup() {flushDiskAndDetectOutcome: () => Promise.resolve(null),respondStaleExternalWrite: () => {},respondPersistenceFailure: () => {},- respondDiskDivergence: () => {},okArtifactKey: () => '',resolveSkillsRoot: () => '/nonexistent-skills',extractActorIdentityFromQuery: () => ({ kind: 'invalid-summary' }),diff --git a/packages/server/src/http/skills-document-routes.ts b/packages/server/src/http/skills-document-routes.tsindex a160a5ff6..fc075a10c 100644--- a/packages/server/src/http/skills-document-routes.ts+++ b/packages/server/src/http/skills-document-routes.ts@@ -161,13 +161,9 @@ export interface SkillsDocumentRouteDeps {flushDiskAndDetectOutcome: (docName: string,) => Promise<- | { kind: 'failure'; failure: StoreFailure }- | { kind: 'divergence' }- | { kind: 'stale-external-write' }- | null+ { kind: 'failure'; failure: StoreFailure } | { kind: 'stale-external-write' } | null>;respondPersistenceFailure: (res: ServerResponse, failure: StoreFailure, handler: string) => void;- respondDiskDivergence: (res: ServerResponse, handler: string) => void;okArtifactKey: (kind: 'template' | 'folder-frontmatter' | 'folder' | 'skill',folder: string,@@ -241,7 +237,6 @@ export function createSkillsDocumentRoutes(deps: SkillsDocumentRouteDeps): ApiRoflushDiskAndDetectOutcome,respondStaleExternalWrite,respondPersistenceFailure,- respondDiskDivergence,okArtifactKey,resolveSkillsRoot,extractActorIdentityFromQuery,@@ -628,10 +623,6 @@ export function createSkillsDocumentRoutes(deps: SkillsDocumentRouteDeps): ApiRorespondPersistenceFailure(res, flushOutcome.failure, 'skill-put');return;}- if (flushOutcome?.kind === 'divergence') {- respondDiskDivergence(res, 'skill-put');- return;- }if (flushOutcome?.kind === 'stale-external-write') {respondStaleExternalWrite(res, 'skill-put', docName);return;diff --git a/packages/server/src/http/skills-file-routes.test.ts b/packages/server/src/http/skills-file-routes.test.tsindex ebb5a77b3..821a541b4 100644--- a/packages/server/src/http/skills-file-routes.test.ts+++ b/packages/server/src/http/skills-file-routes.test.ts@@ -32,7 +32,6 @@ function buildGroup() {flushDiskAndDetectOutcome: () => Promise.resolve(null),respondStaleExternalWrite: () => {},respondPersistenceFailure: () => {},- respondDiskDivergence: () => {},okArtifactKey: () => '',resolveSkillsRoot: () => '/nonexistent-skills',extractActorIdentityFromQuery: () => ({ kind: 'invalid-summary' }),diff --git a/packages/server/src/http/skills-file-routes.ts b/packages/server/src/http/skills-file-routes.tsindex 8814a16c4..483837c26 100644--- a/packages/server/src/http/skills-file-routes.ts+++ b/packages/server/src/http/skills-file-routes.ts@@ -95,13 +95,9 @@ export interface SkillsFileRouteDeps {flushDiskAndDetectOutcome: (docName: string,) => Promise<- | { kind: 'failure'; failure: StoreFailure }- | { kind: 'divergence' }- | { kind: 'stale-external-write' }- | null+ { kind: 'failure'; failure: StoreFailure } | { kind: 'stale-external-write' } | null>;respondPersistenceFailure: (res: ServerResponse, failure: StoreFailure, handler: string) => void;- respondDiskDivergence: (res: ServerResponse, handler: string) => void;okArtifactKey: (kind: 'template' | 'folder-frontmatter' | 'folder' | 'skill',folder: string,@@ -197,7 +193,6 @@ export function createSkillsFileRoutes(deps: SkillsFileRouteDeps): ApiRouteGroupflushDiskAndDetectOutcome,respondStaleExternalWrite,respondPersistenceFailure,- respondDiskDivergence,attributeOkArtifactWrite,okArtifactKey,commitOkArtifactWrite,@@ -449,10 +444,6 @@ export function createSkillsFileRoutes(deps: SkillsFileRouteDeps): ApiRouteGrouprespondPersistenceFailure(res, flushOutcome.failure, 'skill-file-put');return;}- if (flushOutcome?.kind === 'divergence') {- respondDiskDivergence(res, 'skill-file-put');- return;- }if (flushOutcome?.kind === 'stale-external-write') {respondStaleExternalWrite(res, 'skill-file-put', refDocName);return;diff --git a/packages/server/src/http/skills-install-routes.test.ts b/packages/server/src/http/skills-install-routes.test.tsindex 3affb415b..6c21de302 100644--- a/packages/server/src/http/skills-install-routes.test.ts+++ b/packages/server/src/http/skills-install-routes.test.ts@@ -27,7 +27,6 @@ function buildGroup(overrides: Partial<SkillsInstallRouteDeps> = {}) {flushDiskAndDetectOutcome: () => Promise.resolve(null),respondStaleExternalWrite: unexpectedCall,respondPersistenceFailure: unexpectedCall,- respondDiskDivergence: unexpectedCall,skillInstallOps: {resolveFork: unexpectedCall,applyAddRemove: unexpectedCall,@@ -127,21 +126,6 @@ describe('skill-install surfaces every flush outcome instead of installing overexpect(captured.body).not.toContain('INVALID_SKILL_SOURCE');});- test('a diverged disk write answers with the divergence rather than validating and installing', async () => {- const seen: string[] = [];- const captured = await dispatchInstall({- flushDiskAndDetectOutcome: () => Promise.resolve({ kind: 'divergence' }),- respondDiskDivergence: (res, handler) => {- seen.push(handler);- res.writeHead(409, { 'Content-Type': 'application/problem+json' });- res.end(JSON.stringify({ type: 'urn:ok:error:disk-divergence' }));- },- });- expect(seen).toEqual(['skill-install']);- expect(captured.status).toBe(409);- expect(captured.body).not.toContain('INVALID_SKILL_SOURCE');- });-test('a clean flush falls through to validation, which rejects a SKILL.md-less source', async () => {const captured = await dispatchInstall({});expect(captured.status).toBe(400);diff --git a/packages/server/src/http/skills-install-routes.ts b/packages/server/src/http/skills-install-routes.tsindex 3a73ff28f..6b48d77a5 100644--- a/packages/server/src/http/skills-install-routes.ts+++ b/packages/server/src/http/skills-install-routes.ts@@ -50,14 +50,10 @@ export interface SkillsInstallRouteDeps {shippedBundleSkillMd: (name: string, scope?: 'project' | 'global') => string | null;respondStaleExternalWrite: (res: ServerResponse, handler: string, docName: string) => void;respondPersistenceFailure: (res: ServerResponse, failure: StoreFailure, handler: string) => void;- respondDiskDivergence: (res: ServerResponse, handler: string) => void;flushDiskAndDetectOutcome: (docName: string,) => Promise<- | { kind: 'failure'; failure: StoreFailure }- | { kind: 'divergence' }- | { kind: 'stale-external-write' }- | null+ { kind: 'failure'; failure: StoreFailure } | { kind: 'stale-external-write' } | null>;skillInstallOps: SkillInstallOpsService;skillPlacementOps: SkillPlacementOpsService;@@ -81,7 +77,6 @@ export function createSkillsInstallRoutes(deps: SkillsInstallRouteDeps): ApiRoutflushDiskAndDetectOutcome,respondStaleExternalWrite,respondPersistenceFailure,- respondDiskDivergence,skillInstallOps,skillPlacementOps,signalChannel,@@ -142,10 +137,6 @@ export function createSkillsInstallRoutes(deps: SkillsInstallRouteDeps): ApiRoutrespondPersistenceFailure(res, liveSkillFlush.failure, 'skill-install');return;}- if (liveSkillFlush?.kind === 'divergence') {- respondDiskDivergence(res, 'skill-install');- return;- }if (liveSkillFlush?.kind === 'stale-external-write') {respondStaleExternalWrite(res, 'skill-install', liveSkillDoc);return;diff --git a/packages/server/src/metrics.test.ts b/packages/server/src/metrics.test.tsindex 2cb851ea0..f395fd22d 100644--- a/packages/server/src/metrics.test.ts+++ b/packages/server/src/metrics.test.ts@@ -78,8 +78,6 @@ describe('reconciliation metrics', () => {const metrics = getMetrics();expect(metrics.staleExternalWriteRefused).toBe(1);expect(metrics.conflictCount).toBe(0);- expect(metrics.persistenceDivergenceRealign).toBe(0);- expect(metrics.persistenceDivergenceRealignCheckpointCreated).toBe(0);resetMetrics();expect(getMetrics().staleExternalWriteRefused).toBe(0);});@@ -161,7 +159,10 @@ describe('reconciliation metrics', () => {key === 'bridgeToleranceApplied' ||key === 'mapDrivenSpliceFallback' ||key === 'mapDrivenSpliceMemoSkips' ||- key === 'observerAResidualMergeSpliceUnavailable'+ key === 'observerAResidualMergeSpliceUnavailable' ||+ key === 'storeDiskCompare' ||+ key === 'receiptIntakeSkip' ||+ key === 'receiptAtLoad') {expect(value).toEqual({});} else {diff --git a/packages/server/src/metrics.ts b/packages/server/src/metrics.tsindex ec9f8ac20..b2350f5ad 100644--- a/packages/server/src/metrics.ts+++ b/packages/server/src/metrics.ts@@ -19,6 +19,27 @@ export type MapDrivenSpliceMemoSkipReason =| 'position-not-numeric'| 'compose-failed';+export type StoreDiskCompareOutcome =+ | 'match'+ | 'changed'+ | 'drift'+ | 'absent'+ | 'stale-refused'+ | StorePendingReason;++export type StorePendingReason =+ | 'pending-unreadable'+ | 'pending-exhausted'+ | 'pending-base-moved'+ | 'pending-no-intake'+ | 'pending-intake-failed'+ | 'pending-intake-failed-no-base'+ | 'pending-conflict';++export type ReceiptIntakeSkip = 'echo' | 'superseded';++export type ReceiptAtLoad = 'unchanged' | 'changed' | 'missing';+export interface ReconciliationMetrics {reconcileCount: number;conflictCount: number;@@ -113,9 +134,15 @@ export interface ReconciliationMetrics {persistenceDuplicationSpared: number;persistenceDuplicationBaselineMissing: number;persistenceDuplicationBaselineRefusals: number;- persistenceDivergenceRealign: number;- persistenceDivergenceRealignCheckpointCreated: number;- persistenceDivergenceRealignDeduped: number;+ storeDiskCompare: Partial<Record<StoreDiskCompareOutcome, number>>;+ storeDiskCompareReadRetries: number;+ storeRestatRefused: number;+ receiptIntakeSkip: Partial<Record<ReceiptIntakeSkip, number>>;+ receiptAtLoad: Partial<Record<ReceiptAtLoad, number>>;+ receiptPersistFailures: number;+ receiptRecordsUnreadable: number;+ receiptRecordsOtherVersion: number;+ receiptLogRefused: number;managedArtifactReconcile: number;managedArtifactReconcileCheckpointCreated: number;managedArtifactReconcileDeduped: number;@@ -234,9 +261,15 @@ const counters: ReconciliationMetrics = {persistenceDuplicationSpared: 0,persistenceDuplicationBaselineMissing: 0,persistenceDuplicationBaselineRefusals: 0,- persistenceDivergenceRealign: 0,- persistenceDivergenceRealignCheckpointCreated: 0,- persistenceDivergenceRealignDeduped: 0,+ storeDiskCompare: {},+ storeDiskCompareReadRetries: 0,+ storeRestatRefused: 0,+ receiptIntakeSkip: {},+ receiptAtLoad: {},+ receiptPersistFailures: 0,+ receiptRecordsUnreadable: 0,+ receiptRecordsOtherVersion: 0,+ receiptLogRefused: 0,managedArtifactReconcile: 0,managedArtifactReconcileCheckpointCreated: 0,managedArtifactReconcileDeduped: 0,@@ -562,16 +595,40 @@ export function incrementPersistenceDuplicationBaselineRefusals(): void {counters.persistenceDuplicationBaselineRefusals++;}-export function incrementPersistenceDivergenceRealign(): void {- counters.persistenceDivergenceRealign++;+export function incrementStoreDiskCompare(outcome: StoreDiskCompareOutcome): void {+ counters.storeDiskCompare[outcome] = (counters.storeDiskCompare[outcome] ?? 0) + 1;+}++export function incrementStoreDiskCompareReadRetries(): void {+ counters.storeDiskCompareReadRetries++;+}++export function incrementStoreRestatRefused(): void {+ counters.storeRestatRefused++;+}++export function incrementReceiptIntakeSkip(kind: ReceiptIntakeSkip): void {+ counters.receiptIntakeSkip[kind] = (counters.receiptIntakeSkip[kind] ?? 0) + 1;+}++export function incrementReceiptAtLoad(outcome: ReceiptAtLoad): void {+ counters.receiptAtLoad[outcome] = (counters.receiptAtLoad[outcome] ?? 0) + 1;+}++export function incrementReceiptPersistFailures(): void {+ counters.receiptPersistFailures++;+}++export function incrementReceiptRecordsUnreadable(count = 1): void {+ counters.receiptRecordsUnreadable += count;}-export function incrementPersistenceDivergenceRealignCheckpointCreated(): void {- counters.persistenceDivergenceRealignCheckpointCreated++;+export function incrementReceiptRecordsOtherVersion(count: number): void {+ counters.receiptRecordsOtherVersion += count;}-export function incrementPersistenceDivergenceRealignDeduped(): void {- counters.persistenceDivergenceRealignDeduped++;+export function incrementReceiptLogRefused(): void {+ counters.receiptLogRefused++;}export function incrementManagedArtifactReconcile(): void {@@ -724,6 +781,9 @@ export function getMetrics(): ReconciliationMetrics {...counters,cc1LastSeq: { ...counters.cc1LastSeq },bridgeToleranceApplied: { ...counters.bridgeToleranceApplied },+ storeDiskCompare: { ...counters.storeDiskCompare },+ receiptIntakeSkip: { ...counters.receiptIntakeSkip },+ receiptAtLoad: { ...counters.receiptAtLoad },mapDrivenSpliceFallback: { ...counters.mapDrivenSpliceFallback },mapDrivenSpliceMemoSkips: { ...counters.mapDrivenSpliceMemoSkips },observerAResidualMergeSpliceUnavailable: {@@ -819,9 +879,15 @@ export function resetMetrics(): void {counters.persistenceDuplicationSpared = 0;counters.persistenceDuplicationBaselineMissing = 0;counters.persistenceDuplicationBaselineRefusals = 0;- counters.persistenceDivergenceRealign = 0;- counters.persistenceDivergenceRealignCheckpointCreated = 0;- counters.persistenceDivergenceRealignDeduped = 0;+ counters.storeDiskCompare = {};+ counters.storeDiskCompareReadRetries = 0;+ counters.storeRestatRefused = 0;+ counters.receiptIntakeSkip = {};+ counters.receiptAtLoad = {};+ counters.receiptPersistFailures = 0;+ counters.receiptRecordsUnreadable = 0;+ counters.receiptRecordsOtherVersion = 0;+ counters.receiptLogRefused = 0;counters.managedArtifactReconcile = 0;counters.managedArtifactReconcileCheckpointCreated = 0;counters.managedArtifactReconcileDeduped = 0;diff --git a/packages/server/src/path-identity.test.ts b/packages/server/src/path-identity.test.tsindex 746f7bec0..8c8fa12ee 100644--- a/packages/server/src/path-identity.test.ts+++ b/packages/server/src/path-identity.test.ts@@ -172,6 +172,7 @@ describe('directory identity across server components', () => {writeFileSync(join(contentDir, `${docName}.md`), seed);const state = new DocumentDurabilityState();const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir,projectDir,gitEnabled: false,@@ -209,6 +210,7 @@ describe('directory identity across server components', () => {writeFileSync(outsidePath, seed);const state = new DocumentDurabilityState();const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir,projectDir,gitEnabled: false,@@ -246,6 +248,7 @@ describe('directory identity across server components', () => {const shadow = await initShadowRepo(projectDir);const state = new DocumentDurabilityState();const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir,projectDir,gitEnabled: true,@@ -577,6 +580,7 @@ describe('one directory identity for watcher keys and linked documents', () => {symlinkSync(target, join(canonicalContentDir, `${docName}.md`));}const loader = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir,projectDir,gitEnabled: false,@@ -584,6 +588,7 @@ describe('one directory identity for watcher keys and linked documents', () => {});const storeState = new DocumentDurabilityState();const storer = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir,projectDir,gitEnabled: false,@@ -606,6 +611,7 @@ describe('one directory identity for watcher keys and linked documents', () => {const document = new Y.Doc();try {composeAndWriteRawBody(document, edited, 'agent');+ storeState.setReconciledBase(documentName, original);storeState.markAgentWriteStore(documentName);await storer.forceStore(document, documentName);} finally {diff --git a/packages/server/src/persistence-defer-hold.test.ts b/packages/server/src/persistence-defer-hold.test.tsindex 1f38da673..923d80fc7 100644--- a/packages/server/src/persistence-defer-hold.test.ts+++ b/packages/server/src/persistence-defer-hold.test.ts@@ -76,6 +76,7 @@ async function createStoreHarness(});const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: projectRoot,projectDir: projectRoot,gitEnabled: false,diff --git a/packages/server/src/persistence-deferred-store.test.ts b/packages/server/src/persistence-deferred-store.test.tsindex 95fe2508f..997c19230 100644--- a/packages/server/src/persistence-deferred-store.test.ts+++ b/packages/server/src/persistence-deferred-store.test.ts@@ -102,6 +102,7 @@ describe('batch-gated L1 persistence', () => {writeFileSync(docPath, 'initial\n', 'utf-8');const acks: Array<{ docName: string; sv: Uint8Array }> = [];const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -142,6 +143,7 @@ describe('batch-gated L1 persistence', () => {previousMarkdown: string | null;}> = [];const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -169,6 +171,7 @@ describe('batch-gated L1 persistence', () => {const docPath = join(tmpDir, `${docName}.md`);writeFileSync(docPath, 'clean\n', 'utf-8');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -198,6 +201,7 @@ describe('batch-gated L1 persistence', () => {const goodPath = join(tmpDir, `${goodDocName}.md`);writeFileSync(goodPath, 'good base\n', 'utf-8');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -234,6 +238,7 @@ describe('batch-gated L1 persistence', () => {writeFileSync(docPath, 'base\n', 'utf-8');let resetAttempts = 0;const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -268,6 +273,7 @@ describe('batch-gated L1 persistence', () => {writeFileSync(docPath, 'branch A base\n', 'utf-8');const acks: Array<{ docName: string; sv: Uint8Array }> = [];const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -302,6 +308,7 @@ describe('batch-gated L1 persistence', () => {const docPath = join(tmpDir, `${docName}.md`);writeFileSync(docPath, 'branch A base\n', 'utf-8');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -341,6 +348,7 @@ describe('batch-gated L1 persistence', () => {let secondPendingAfterDiscardCall: Promise<unknown> | undefined;const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -403,6 +411,7 @@ describe('quiescence gate — deferCount cleanup on disk-write error', () => {writeFileSync(docPath, 'initial\n', 'utf-8');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -431,6 +440,7 @@ describe('quiescence gate — deferCount cleanup on disk-write error', () => {writeFileSync(docPath, 'initial\n', 'utf-8');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -512,6 +522,7 @@ describe('Y.Text-is-truth wiring (FR-33 / FR-35)', () => {const docPath = join(tmpDir, `${docName}.md`);writeFileSync(docPath, '', 'utf-8');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -537,6 +548,7 @@ describe('Y.Text-is-truth wiring (FR-33 / FR-35)', () => {writeFileSync(docPath, '__cold__\n', 'utf-8');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -586,11 +598,12 @@ describe('FR-9 — deferred-store-failed event + counter', () => {rmSync(tmpDir, { recursive: true, force: true });});- test('emits deferred-store-failed with errorClass=disk-write under default redaction', async () => {+ test('emits deferred-store-failed with errorClass=store-held-pending under default redaction when the file cannot be compared', async () => {const docName = 'fr9-disk-write-default';const docPath = join(tmpDir, `${docName}.md`);writeFileSync(docPath, 'initial\n', 'utf-8');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -617,7 +630,7 @@ describe('FR-9 — deferred-store-failed event + counter', () => {const ev = events[0] as Record<string, unknown>;expect(ev.event).toBe('deferred-store-failed');expect(ev['doc.name']).toBe(docName);- expect(ev.errorClass).toMatch(/^(disk-write|traced-rename)$/);+ expect(ev.errorClass).toBe('store-held-pending');expect(typeof ev.errorMessageHash).toBe('string');expect((ev.errorMessageHash as string).length).toBe(8);expect(typeof ev.timestamp).toBe('string');@@ -632,6 +645,7 @@ describe('FR-9 — deferred-store-failed event + counter', () => {const docPath = join(tmpDir, `${docName}.md`);writeFileSync(docPath, 'initial\n', 'utf-8');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -664,6 +678,7 @@ describe('FR-9 — deferred-store-failed event + counter', () => {const docPath = join(tmpDir, `${docName}.md`);writeFileSync(docPath, 'initial\n', 'utf-8');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -722,6 +737,7 @@ describe('FR-9 — deferred-store-failed event + counter', () => {const docPath = join(tmpDir, `${docName}.md`);writeFileSync(docPath, 'initial\n', 'utf-8');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -770,6 +786,7 @@ describe('FR-9 — deferred-store-failed event + counter', () => {const docPath = join(tmpDir, `${docName}.md`);writeFileSync(docPath, 'initial\n', 'utf-8');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -884,6 +901,7 @@ describe('forceStore dispatch (staleness watchdog entry point)', () => {const docPath = join(tmpDir, `${docName}.md`);writeFileSync(docPath, 'initial\n', 'utf-8');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -906,6 +924,7 @@ describe('forceStore dispatch (staleness watchdog entry point)', () => {const docPath = join(tmpDir, `${docName}.md`);writeFileSync(docPath, 'initial\n', 'utf-8');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -929,6 +948,7 @@ describe('forceStore dispatch (staleness watchdog entry point)', () => {test('short-circuits synthetic docs without touching queues', async () => {const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,diff --git a/packages/server/src/persistence-flush-during-commit.test.ts b/packages/server/src/persistence-flush-during-commit.test.tsindex 3b4be38ce..25545b5d8 100644--- a/packages/server/src/persistence-flush-during-commit.test.ts+++ b/packages/server/src/persistence-flush-during-commit.test.ts@@ -81,6 +81,7 @@ test.each([async ({ commitDebounceMs, waitForTimer }) => {writeFileSync(join(contentDir, 'notes.md'), 'initial\n');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir,projectDir: tmpDir,contentRoot: 'content',diff --git a/packages/server/src/persistence-large-file-cap.test.ts b/packages/server/src/persistence-large-file-cap.test.tsindex b9d94d292..36411d795 100644--- a/packages/server/src/persistence-large-file-cap.test.ts+++ b/packages/server/src/persistence-large-file-cap.test.ts@@ -33,7 +33,10 @@ describe('persistence large-file cap', () => {const filePath = join(tmpDir, 'big.md');writeFileSync(filePath, '');truncateSync(filePath, DOCUMENT_OPEN_BYTE_LIMIT + 1);- const persistence = createPersistenceExtension({ contentDir: tmpDir });+ const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,+ contentDir: tmpDir,+ });const document = new Y.Doc();await expect(loadDocument(persistence, document, 'big')).rejects.toBeInstanceOf(@@ -46,7 +49,10 @@ describe('persistence large-file cap', () => {test('allows a document at exactly the byte limit', async () => {const filePath = join(tmpDir, 'exact.md');writeFileSync(filePath, 'a'.repeat(DOCUMENT_OPEN_BYTE_LIMIT));- const persistence = createPersistenceExtension({ contentDir: tmpDir });+ const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,+ contentDir: tmpDir,+ });const document = new Y.Doc();await expect(loadDocument(persistence, document, 'exact')).resolves.toBeUndefined();diff --git a/packages/server/src/persistence-publication-receipt.test-helper.ts b/packages/server/src/persistence-publication-receipt.test-helper.tsindex 4ef65429b..bce0da8a1 100644--- a/packages/server/src/persistence-publication-receipt.test-helper.ts+++ b/packages/server/src/persistence-publication-receipt.test-helper.ts@@ -69,6 +69,7 @@ export async function createReceiptFixture(initial = 'initial\n') {durability.switchReconciledBaseScope('main');writeFileSync(docPath, initial, { flag: 'wx' });const options: ReceiptOptions = {+ ingestDiskAtStore: null,contentDir,projectDir: contentDir,gitEnabled: false,diff --git a/packages/server/src/persistence-publication-receipt.test.ts b/packages/server/src/persistence-publication-receipt.test.tsindex 8f5f35b95..0f98129f4 100644--- a/packages/server/src/persistence-publication-receipt.test.ts+++ b/packages/server/src/persistence-publication-receipt.test.ts@@ -23,6 +23,7 @@ describe('successful publication receipt ownership', () => {test('ordinary publication remains usable without a receipt consumer', async () => {const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: fixture.contentDir,projectDir: fixture.contentDir,gitEnabled: false,@@ -41,6 +42,7 @@ describe('successful publication receipt ownership', () => {throw new Error('Receipt consumer cannot accept the occurrence');};const options: ReceiptOptions = {+ ingestDiskAtStore: null,contentDir: fixture.contentDir,projectDir: fixture.contentDir,gitEnabled: false,diff --git a/packages/server/src/persistence-removed-doc-publish.test.ts b/packages/server/src/persistence-removed-doc-publish.test.tsindex f548dfd76..86e75da3d 100644--- a/packages/server/src/persistence-removed-doc-publish.test.ts+++ b/packages/server/src/persistence-removed-doc-publish.test.ts@@ -52,6 +52,7 @@ describe('persistence publish guard — a removed document is never republished'function persistenceWith(isRecentlyRemoved?: (name: string) => boolean) {return createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,diff --git a/packages/server/src/persistence-rename-backfill.test.ts b/packages/server/src/persistence-rename-backfill.test.tsindex dc340fdaa..8804f63e5 100644--- a/packages/server/src/persistence-rename-backfill.test.ts+++ b/packages/server/src/persistence-rename-backfill.test.ts@@ -81,6 +81,7 @@ describe('rename-log commitSha backfill', () => {writeFileSync(join(contentDir, 'alpha.md'), '# alpha\n');writeFileSync(join(contentDir, 'gamma.md'), '# gamma\n');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir,projectDir: tmpDir,contentRoot: 'content',diff --git a/packages/server/src/persistence-staleness-watchdog.test.ts b/packages/server/src/persistence-staleness-watchdog.test.tsindex d46159d42..b6020e9d5 100644--- a/packages/server/src/persistence-staleness-watchdog.test.ts+++ b/packages/server/src/persistence-staleness-watchdog.test.ts@@ -375,20 +375,16 @@ describe('exclusions', () => {});describe('external-edit stand-down (disk authority)', () => {- test('does not overwrite disk bytes the base does not account for', async () => {+ test('forces the store when disk differs from the base, leaving the merge to the store compare', async () => {seedWedgedDoc(rig, 'external-doc');rig.disk.set('external-doc', '# external native edit\n');await rig.watchdog.sweep();- expect(rig.forceCalls).toEqual([]);+ expect(rig.forceCalls).toEqual(['external-doc']);expect(getMetrics().persistenceStalenessDetected).toBe(1);- expect(getMetrics().persistenceStalenessForcedStores).toBe(0);- expect(getMetrics().persistenceStalenessStoodDown).toBe(1);- expect(rig.disk.get('external-doc')).toBe('# external native edit\n');-- await rig.watchdog.sweep();- expect(getMetrics().persistenceStalenessStoodDown).toBe(1);+ expect(getMetrics().persistenceStalenessForcedStores).toBe(1);+ expect(getMetrics().persistenceStalenessStoodDown).toBe(0);});test('does not resurrect a file deleted out-of-band', async () => {@@ -452,14 +448,14 @@ describe('external-edit stand-down (disk authority)', () => {test('a stand-down re-arms when the memory content changes', async () => {const doc = seedWedgedDoc(rig, 'rearm-doc');- rig.disk.set('rearm-doc', '# external native edit\n');+ rig.disk.delete('rearm-doc');await rig.watchdog.sweep();await rig.watchdog.sweep();expect(rig.forceCalls).toEqual([]);expect(getMetrics().persistenceStalenessDetected).toBe(1);- rig.bases.set('rearm-doc', '# external native edit\n');+ rig.disk.set('rearm-doc', '# old on disk\n');doc.getText('source').delete(0, doc.getText('source').length);doc.getText('source').insert(0, '# newer memory edit\n');diff --git a/packages/server/src/persistence-staleness-watchdog.ts b/packages/server/src/persistence-staleness-watchdog.tsindex 08bb2f64a..d454b47aa 100644--- a/packages/server/src/persistence-staleness-watchdog.ts+++ b/packages/server/src/persistence-staleness-watchdog.ts@@ -152,8 +152,6 @@ export function createPersistenceStalenessWatchdog(if (diskBytes !== null) standDownReason = 'disk-file-never-loaded';} else if (diskBytes === null) {standDownReason = 'disk-file-missing';- } else if (normalizeBridge(diskBytes) !== normalizedBase) {- standDownReason = 'disk-diverged-from-base';}} catch (err) {if (err instanceof StructuralDiskReadError) {diff --git a/packages/server/src/persistence-store-publish-order.test.ts b/packages/server/src/persistence-store-publish-order.test.tsindex 01d86a3cc..3bf72ff65 100644--- a/packages/server/src/persistence-store-publish-order.test.ts+++ b/packages/server/src/persistence-store-publish-order.test.ts@@ -48,6 +48,7 @@ describe('persistence publish ordering — a store must not publish bytes olderfunction createPersistence(): ReturnType<typeof createPersistenceExtension> {return createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,diff --git a/packages/server/src/persistence-store-superseded-drop.test.ts b/packages/server/src/persistence-store-superseded-drop.test.tsindex b0d38600d..cb1e9fd03 100644--- a/packages/server/src/persistence-store-superseded-drop.test.ts+++ b/packages/server/src/persistence-store-superseded-drop.test.ts@@ -63,6 +63,7 @@ describe('persistence superseded-store drop — a store that loses the publish rfunction createPersistence(): ReturnType<typeof createPersistenceExtension> {return createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,diff --git a/packages/server/src/persistence-tripwire-paste.test.ts b/packages/server/src/persistence-tripwire-paste.test.tsindex eee548c87..4061f98c3 100644--- a/packages/server/src/persistence-tripwire-paste.test.ts+++ b/packages/server/src/persistence-tripwire-paste.test.ts@@ -411,7 +411,7 @@ describe('persistence tripwire with an absent reconciled base', () => {}}, 40_000);- test('a doubling with a settled write behind it is still spared when the base is absent', async () => {+ test('a doubling with a settled write behind it is spared by the tripwire but never written over a file with no base', async () => {rig = await setupRig('ok-tripwire-absent-spare-');const docName = 'absent-base-spare';const docPath = join(rig.tmpDir, `${docName}.md`);@@ -473,17 +473,25 @@ describe('persistence tripwire with an absent reconciled base', () => {.filter((s) => s.includes('"event":"ok-persistence-duplication-spared"'));expect(spared).toHaveLength(1);- await waitWithinTestBudget(- `the doubled paste to reach ${docName}.md on disk`,- () => occurrences(readFileSync(docPath, 'utf-8'), settledMarker) === 2,- { timeoutMs: 8_000 },+ await vi.waitFor(+ () => expect(getMetrics().storeDiskCompare.changed ?? 0).toBeGreaterThanOrEqual(1),+ { timeout: 10_000, interval: 25 },);- const doubledDisk = readFileSync(docPath, 'utf-8');await expectStable(`${docName}.md on disk`, () => readFileSync(docPath, 'utf-8'), {durationMs: 700,});- expect(readFileSync(docPath, 'utf-8')).toBe(doubledDisk);+ expect(readFileSync(docPath, 'utf-8')).toBe(settledStored);+ expect(server.durabilityState.getReconciledBase(docName)).toBe(settledStored);expect(getMetrics().persistenceDuplicationReset).toBe(0);+ await vi.waitFor(+ async () => {+ const hist = await getDocumentHistory(rig.shadow, { docName }, '');+ expect(+ hist.entries.some((e) => e.checkpoint?.kind === 'external-change-rescue-disk-update'),+ ).toBe(true);+ },+ { timeout: 10_000, interval: 50 },+ );conn.disconnect();} finally {@@ -504,6 +512,7 @@ describe('persistence tripwire with an absent reconciled base', () => {const durabilityState = new DocumentDurabilityState();const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: rig.tmpDir,projectDir: rig.tmpDir,gitEnabled: false,@@ -550,6 +559,7 @@ describe('persistence tripwire with an absent reconciled base', () => {const durabilityState = new DocumentDurabilityState();const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: rig.tmpDir,projectDir: rig.tmpDir,gitEnabled: false,@@ -588,6 +598,7 @@ describe('persistence tripwire with an absent reconciled base', () => {const durabilityState = new DocumentDurabilityState();const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: rig.tmpDir,projectDir: rig.tmpDir,gitEnabled: false,@@ -625,6 +636,7 @@ describe('persistence tripwire with an absent reconciled base', () => {const durabilityState = new DocumentDurabilityState();const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: rig.tmpDir,projectDir: rig.tmpDir,gitEnabled: false,@@ -661,6 +673,7 @@ describe('persistence tripwire with an absent reconciled base', () => {const durabilityState = new DocumentDurabilityState();const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: rig.tmpDir,projectDir: rig.tmpDir,gitEnabled: false,@@ -721,6 +734,7 @@ describe('persistence tripwire with an absent reconciled base', () => {const durabilityState = new DocumentDurabilityState();const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: rig.tmpDir,projectDir: rig.tmpDir,gitEnabled: false,@@ -773,6 +787,7 @@ describe('persistence tripwire with an absent reconciled base', () => {const durabilityState = new DocumentDurabilityState();const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: rig.tmpDir,projectDir: rig.tmpDir,gitEnabled: false,@@ -813,6 +828,7 @@ describe('persistence tripwire with an absent reconciled base', () => {const durabilityState = new DocumentDurabilityState();const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: rig.tmpDir,projectDir: rig.tmpDir,gitEnabled: false,diff --git a/packages/server/src/persistence-tripwire-read-failure.test.ts b/packages/server/src/persistence-tripwire-read-failure.test.tsindex af9b6da2d..b7e075580 100644--- a/packages/server/src/persistence-tripwire-read-failure.test.ts+++ b/packages/server/src/persistence-tripwire-read-failure.test.ts@@ -53,6 +53,7 @@ describe('tripwire reset readFileSync failure', () => {mkdirSync(join(contentDir, `${docName}.md`));const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir,projectDir: contentDir,gitEnabled: false,diff --git a/packages/server/src/persistence-tripwire-symlink.test.ts b/packages/server/src/persistence-tripwire-symlink.test.tsindex 04b64e99e..cdc3a6444 100644--- a/packages/server/src/persistence-tripwire-symlink.test.ts+++ b/packages/server/src/persistence-tripwire-symlink.test.ts@@ -69,6 +69,7 @@ describe('tripwire reset symlink-escape', () => {symlinkSync(secretPath, docPath);const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir,projectDir: contentDir,gitEnabled: false,@@ -109,6 +110,7 @@ describe('tripwire reset symlink-escape', () => {symlinkSync('.ok/local/principal.json', join(contentDir, `${docName}.md`));const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir,projectDir: contentDir,gitEnabled: false,diff --git a/packages/server/src/persistence.test.ts b/packages/server/src/persistence.test.tsindex e7be16994..5e136c1ab 100644--- a/packages/server/src/persistence.test.ts+++ b/packages/server/src/persistence.test.ts@@ -256,6 +256,7 @@ describe('persistence refuses links into private repository state', () => {function createPersistence(durabilityState = new DocumentDurabilityState()) {return createPersistenceExtension({+ ingestDiskAtStore: null,contentDir,projectDir: contentDir,gitEnabled: false,@@ -267,10 +268,14 @@ describe('persistence refuses links into private repository state', () => {const path = join(contentDir, 'note.md');writeFileSync(path, '# Before\n');let owned = true;+ const durabilityState = new DocumentDurabilityState();+ durabilityState.setReconciledBase('note', '# Before\n');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir,projectDir: contentDir,gitEnabled: false,+ durabilityState,assertContentPath: () => {if (!owned) throw new ContentScopeAdmissionError();},@@ -459,7 +464,9 @@ describe('persistence refuses links into private repository state', () => {const linkPath = join(contentDir, 'notes', 'alias.md');writeFileSync(targetPath, '# Before\n');symlinkSync('../shared/a.md', linkPath);- const persistence = createPersistence();+ const durabilityState = new DocumentDurabilityState();+ durabilityState.setReconciledBase('notes/alias', '# Before\n');+ const persistence = createPersistence(durabilityState);const document = new Y.Doc();composeAndWriteRawBody(document, '# After\n', 'test');@@ -509,6 +516,7 @@ describe('durable derived-index projection', () => {recordDurableStore,};const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir,projectDir: contentDir,gitEnabled: false,@@ -531,6 +539,7 @@ describe('durable derived-index projection', () => {const durabilityState = new DocumentDurabilityState();const recordDurableStore = vi.fn();const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir,projectDir: contentDir,gitEnabled: false,@@ -560,6 +569,7 @@ describe('durable derived-index projection', () => {});await derivedDocumentIndex.close();const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir,projectDir: contentDir,gitEnabled: false,diff --git a/packages/server/src/persistence.ts b/packages/server/src/persistence.tsindex 76a7b0444..2807d794f 100644--- a/packages/server/src/persistence.ts+++ b/packages/server/src/persistence.ts@@ -65,6 +65,7 @@ import {} from './derived-document-index.ts';import { resolveDirectoryRoot, resolveNativePath } from './directory-root.ts';import { applyDiskContentToDoc, FILE_WATCHER_ORIGIN } from './disk-content-intake.ts';+import type { FileObservation } from './disk-receipts.ts';import {assertNeverStorePublishOutcome,type DeferredStoreOutcome,@@ -72,14 +73,24 @@ import {DocumentDurabilityState,OK_DOC_REMOVED,OK_PATH_UNRESOLVABLE,+ OK_STORE_HELD,OK_STORE_REFUSED,type StoreFailure,type StoreFailureRemoval,+ type StorePublishOutcome,} from './document-durability-state.ts';import { refuseStaleExternalWrite } from './external-change.ts';import { contentHash, registerWrite } from './file-watcher.ts';import { resolvesIntoPrivateState } from './fs-safety.ts';-import { tracedMkdir, tracedRenameSync, tracedUnlinkSync, tracedWriteFile } from './fs-traced.ts';+import {+ type ObservedFileRead,+ tracedAppendFileSync,+ tracedMkdir,+ tracedReadObservedSync,+ tracedRenameSync,+ tracedUnlinkSync,+ tracedWriteFile,+} from './fs-traced.ts';import { errnoCode } from './http/handler-utils.ts';import { getLogger } from './logger.ts';import {@@ -110,9 +121,6 @@ import {incrementManagedArtifactReconcileDeduped,incrementPersistenceDeferHold,incrementPersistenceDiskWrite,- incrementPersistenceDivergenceRealign,- incrementPersistenceDivergenceRealignCheckpointCreated,- incrementPersistenceDivergenceRealignDeduped,incrementPersistenceDuplicationBaselineMissing,incrementPersistenceDuplicationBaselineRefusals,incrementPersistenceDuplicationReset,@@ -128,6 +136,10 @@ import {incrementPersistenceSkipNonQuiescent,incrementPersistenceStoreRemovedDoc,incrementPersistenceStoreSuperseded,+ incrementStoreDiskCompare,+ incrementStoreDiskCompareReadRetries,+ incrementStoreRestatRefused,+ type StorePendingReason,} from './metrics.ts';import { toPosix } from './path-utils.ts';import { classifyDuplication } from './persistence-tripwire.ts';@@ -149,9 +161,85 @@ import {saveInMemoryCheckpoint,shadowGit,} from './shadow-repo.ts';+import {+ assertNeverDiskCompare,+ COMPARE_READ_RETRY_DELAYS_MS,+ compareDiskBeforeReplace,+ type DiskCompareOutcome,+ restatFile,+ StoreRestatFailedError,+ StoreRestatRefusedError,+ statObservation,+} from './store-compare.ts';import { getMeter, setActiveSpanAttributes, withSpan } from './telemetry.ts';const log = getLogger('persistence');+const compareLog = getLogger('store-compare');++const MAX_STORE_COMPARE_RUNS = 3;++const consumedCompareReadFaults = new Map<string, number>();++function compareReaderFor(documentName: string): (path: string) => ObservedFileRead {+ const fault =+ process.env.NODE_ENV === 'test' ? process.env.OK_TEST_COMPARE_READ_FAULT : undefined;+ if (fault === undefined) return tracedReadObservedSync;+ const [docName, code, count] = fault.split(':');+ if (docName !== documentName || code === undefined) return tracedReadObservedSync;+ return (path) => {+ const consumed = consumedCompareReadFaults.get(fault) ?? 0;+ if (consumed >= Number(count ?? '1')) return tracedReadObservedSync(path);+ consumedCompareReadFaults.set(fault, consumed + 1);+ const err = new Error(`OK_TEST_COMPARE_READ_FAULT: simulated ${code} reading ${path}`);+ throw Object.assign(err, { code });+ };+}++interface StoreRunControl {+ agentTriggered: boolean | undefined;+ rerun: 'base-moved' | 'disk-changed' | null;+}++type StoreIntakeResult =+ | { readonly kind: 'taken-in' }+ | { readonly kind: 'conflict' }+ | { readonly kind: 'failed'; readonly error: unknown };++type StoreDiskIntake = (+ docName: string,+ absPath: string,+ content: string,+) => Promise<StoreIntakeResult>;++const STORE_PENDING_DESCRIPTIONS: Readonly<Record<StorePendingReason, string>> = {+ 'pending-unreadable': 'its file could not be read for the compare before the write',+ 'pending-exhausted': 'its file kept changing while it was being saved',+ 'pending-base-moved': 'its acknowledged version kept moving while it was being saved',+ 'pending-no-intake': 'its file changed and this persistence has no intake to merge it',+ 'pending-intake-failed': 'its file changed and merging the outside version into it failed',+ 'pending-intake-failed-no-base':+ 'its file changed and taking the outside version in failed, with no agreed version to merge against',+ 'pending-conflict': 'its file changed and could not be merged, so a conflict was raised',+};++function assertNeverStoreIntake(intake: never): never {+ throw new Error(`unhandled store intake result: ${(intake as { kind: string }).kind}`);+}++class StoreHeldPendingError extends Error {+ readonly reason: StorePendingReason;+ readonly code = OK_STORE_HELD;+ readonly causeCode: string | undefined;+ readonly noBase: boolean;++ constructor(docName: string, reason: StorePendingReason, noBase: boolean, cause?: unknown) {+ super(`${docName} was not saved: ${STORE_PENDING_DESCRIPTIONS[reason]}`, { cause });+ this.name = 'StoreHeldPendingError';+ this.reason = reason;+ this.noBase = noBase;+ this.causeCode = cause === undefined ? undefined : errnoCode(cause);+ }+}export class DocumentOpenSizeLimitError extends Error {readonly docName: string;@@ -240,6 +328,7 @@ const DEFERRED_STORE_ERROR_CLASSES = ['parse-fallback','traced-rename','duplication-baseline-unavailable',+ 'store-held-pending','unknown',] as const;type DeferredStoreErrorClass = (typeof DEFERRED_STORE_ERROR_CLASSES)[number];@@ -267,6 +356,7 @@ export function classifyDeferredStoreError(err: unknown): DeferredStoreErrorClasif (err instanceof DuplicationBaselineUnavailableError) {return 'duplication-baseline-unavailable';}+ if (err instanceof StoreHeldPendingError) return 'store-held-pending';const e = err as { code?: unknown; message?: unknown };const message = typeof e.message === 'string' ? e.message : '';if (message.startsWith('symlink-escape:')) return 'disk-write';@@ -324,6 +414,7 @@ export interface PersistenceOptions {isRecentlyRemoved?: (docName: string) => boolean;lifecycleOf?: (document: Y.Doc, docName: string) => LifecycleView | null;ephemeral?: boolean;+ ingestDiskAtStore: StoreDiskIntake | null;}export function captureDocSnapshotForPersistence(document: Y.Doc): {@@ -380,6 +471,17 @@ function probeDocumentFileOnDisk(path: string): DocumentFileProbe {}function toStoreFailure(err: unknown): StoreFailure {+ if (err instanceof StoreHeldPendingError) {+ return {+ code: OK_STORE_HELD,+ message: err.message,+ held: {+ reason: err.reason,+ ...(err.causeCode === undefined ? {} : { causeCode: err.causeCode }),+ ...(err.noBase ? { noBase: true as const } : {}),+ },+ };+ }let code: string | undefined;try {const c = (err as NodeJS.ErrnoException | null)?.code;@@ -411,7 +513,7 @@ export interface PersistenceHandle {readonly managedArtifactCtx: ManagedArtifactCtx;}-export function createPersistenceExtension(options?: PersistenceOptions): PersistenceHandle {+export function createPersistenceExtension(options: PersistenceOptions): PersistenceHandle {const durabilityState = options?.durabilityState ?? new DocumentDurabilityState();const contentDir = resolveDirectoryRoot(resolve(options?.contentDir ?? process.cwd()), {root: 'content',@@ -843,7 +945,6 @@ export function createPersistenceExtension(options?: PersistenceOptions): Persisconst PERSISTENCE_PREWRITE_SITE = 'persistence-prewrite';const PERSISTENCE_DUPLICATION_SITE = 'persistence-duplication-reset';- const PERSISTENCE_REALIGN_SITE = 'persistence-divergence-realign';const MANAGED_ARTIFACT_RECONCILE_SITE = 'managed-artifact-reconcile';const lastFloorCheckpointPayload = new WeakMap<Y.Doc, string>();@@ -1056,78 +1157,6 @@ export function createPersistenceExtension(options?: PersistenceOptions): Persis});}- const lastRealignCheckpointPayload = new WeakMap<Y.Doc, string>();-- function checkpointBeforeDivergenceRealign(- document: Y.Doc,- documentName: string,- liveMarkdown: string,- diskContent: string,- ): void {- incrementPersistenceDivergenceRealign();- const ring = options?.getLossRing?.();- const shadow = shadowRef?.current;- if (!shadow) {- void ring?.record({- event: LOSS_EVENT_CHECKPOINT_WRITE,- docName: documentName,- writerId: null,- direction: 'b',- site: PERSISTENCE_REALIGN_SITE,- lostLen: liveMarkdown.length,- });- return;- }- if (lastRealignCheckpointPayload.get(document) === liveMarkdown) {- incrementPersistenceDivergenceRealignDeduped();- return;- }- lastRealignCheckpointPayload.set(document, liveMarkdown);- const branch = getCurrentBranch?.() ?? 'main';- queueMicrotask(() => {- saveInMemoryCheckpoint(shadow, contentRoot, {- kind: 'persistence-divergence-realign',- docName: documentName,- contents: liveMarkdown,- label: `Before divergence realign @ ${new Date().toISOString()}`,- branch,- metadata: { diskBytes: diskContent.length, discardedBytes: liveMarkdown.length },- })- .then((sha) => {- incrementPersistenceDivergenceRealignCheckpointCreated();- void ring?.record({- event: LOSS_EVENT_CHECKPOINT_WRITE,- docName: documentName,- writerId: null,- direction: 'b',- site: PERSISTENCE_REALIGN_SITE,- lostLen: liveMarkdown.length,- checkpointSha: sha,- });- console.warn(- JSON.stringify({- event: 'persistence-divergence-realign-checkpoint-created',- docName: documentName,- sha,- kind: 'persistence-divergence-realign',- timestamp: new Date().toISOString(),- }),- );- })- .catch((checkpointErr: unknown) => {- if (lastRealignCheckpointPayload.get(document) === liveMarkdown) {- lastRealignCheckpointPayload.delete(document);- }- const e =- checkpointErr instanceof Error ? checkpointErr : new Error(String(checkpointErr));- log.warn(- { documentName, err: e },- '[persistence] divergence-realign checkpoint write failed',- );- });- });- }-const lastManagedReconcileCheckpointPayload = new WeakMap<Y.Doc, string>();function checkpointBeforeManagedArtifactReconcile(@@ -1360,15 +1389,62 @@ export function createPersistenceExtension(options?: PersistenceOptions): Persisreturn 'refused';}- async function storeDocumentNow({- document,- documentName,- lastTransactionOrigin,- }: {+ function holdStorePending(+ documentName: string,+ reason: StorePendingReason,+ cause?: unknown,+ ): never {+ const held = new StoreHeldPendingError(+ documentName,+ reason,+ durabilityState.getReconciledBase(documentName) === undefined,+ cause,+ );+ incrementStoreDiskCompare(reason);+ compareLog.warn(+ { documentName, reason, code: held.causeCode, err: cause },+ `[store-compare] ${held.message}; the save stays pending and the file is untouched`,+ );+ durabilityState.markStoreRefused(documentName);+ persistenceDeferCounts.delete(documentName);+ throw held;+ }++ async function storeDocumentNow(store: {document: Y.Doc;documentName: string;lastTransactionOrigin: unknown;}): Promise<void> {+ let agentTriggered: boolean | undefined;+ for (let run = 1; ; run++) {+ const control: StoreRunControl = { agentTriggered, rerun: null };+ await storeDocumentRun(store, control);+ if (control.rerun === null) return;+ agentTriggered = control.agentTriggered;+ if (run >= MAX_STORE_COMPARE_RUNS) {+ const reason = control.rerun === 'base-moved' ? 'pending-base-moved' : 'pending-exhausted';+ try {+ holdStorePending(store.documentName, reason);+ } catch (held) {+ durabilityState.recordStoreFailure(store.documentName, toStoreFailure(held));+ throw held;+ }+ }+ }+ }++ async function storeDocumentRun(+ {+ document,+ documentName,+ lastTransactionOrigin,+ }: {+ document: Y.Doc;+ documentName: string;+ lastTransactionOrigin: unknown;+ },+ control: StoreRunControl,+ ): Promise<void> {ensureHistograms();const started = Date.now();let inFlightFlushValue: string | undefined;@@ -1384,7 +1460,8 @@ export function createPersistenceExtension(options?: PersistenceOptions): Persis`[persistence] Skipped store for ${documentName}: lifecycle=${lifecycleStatus}`,);if (isRemovalLifecycleStatus(lifecycleStatus)) {- const callerAwaitingFlush = durabilityState.consumeAgentWriteStore(documentName);+ const callerAwaitingFlush =+ control.agentTriggered ?? durabilityState.consumeAgentWriteStore(documentName);const base = durabilityState.getReconciledBase(documentName);const candidate = document.getText('source').toString();const baseline = base ?? frozenStoreBaselines.get(documentName);@@ -1416,6 +1493,7 @@ export function createPersistenceExtension(options?: PersistenceOptions): Persis);incrementPersistenceSkipNonQuiescent();persistenceDeferCounts.set(documentName, deferCount + 1);+ if (control.agentTriggered) durabilityState.markAgentWriteStore(documentName);return;}console.warn(@@ -1429,7 +1507,9 @@ export function createPersistenceExtension(options?: PersistenceOptions): PersisincrementPersistenceForceFlushDuringBurst();}- const agentTriggeredStore = durabilityState.consumeAgentWriteStore(documentName);+ const agentTriggeredStore =+ control.agentTriggered ?? durabilityState.consumeAgentWriteStore(documentName);+ control.agentTriggered = agentTriggeredStore;const { sv: stateVectorAtRead, json } = captureDocSnapshotForPersistence(document);const ytextSnapshot = document.getText('source').toString();@@ -1779,89 +1859,6 @@ export function createPersistenceExtension(options?: PersistenceOptions): Persisawait tracedWriteFile(canonicalPath, '# NATIVE\n\nnative-divergence-injected\n', 'utf-8');}- let diskBeforeWrite: string | null = null;- if (agentTriggeredStore && currentBase !== undefined) {- try {- if (existsSync(canonicalPath)) {- diskBeforeWrite = readFileSync(canonicalPath, 'utf-8');- }- } catch (err) {- diskBeforeWrite = null;- log.warn(- { err, documentName },- '[persistence] L3 disk-read failed; divergence check skipped for this store',- );- }- if (- diskBeforeWrite !== null &&- normalizeBridge(diskBeforeWrite) !== normalizeBridge(currentBase)- ) {- const diskContent = diskBeforeWrite;- if (- refuseStaleExternalWrite(- durabilityState,- document,- documentName,- diskContent,- options?.conflicts,- markdown,- )- ) {- durabilityState.recordStaleExternalWriteFreeze(documentName);- persistenceDeferCounts.delete(documentName);- return;- }- console.warn(- JSON.stringify({- event: 'agent-write-content-divergence',- 'doc.name': documentName,- outcome: 'reverted',- diskBytes: diskContent.length,- baseBytes: currentBase.length,- candidateBytes: markdown.length,- }),- );- checkpointBeforeDivergenceRealign(document, documentName, markdown, diskContent);- const lossRing = options?.getLossRing?.();- const detect: DeriveLossDetectOptions = {- baselineFullMd: diskContent,- report: (obs) => {- try {- const dropped = detectPairedIntakeLoss(obs);- if (dropped.length === 0) return;- void lossRing?.record({- event: LOSS_EVENT_DETECTOR_TRIP,- docName: documentName,- writerId: FILE_SYSTEM_WRITER.id,- direction: 'b',- site: PERSISTENCE_REALIGN_SITE,- lostLen: dropped.reduce((n, s) => n + s.length, 0),- digest: fnv1aDigest(dropped.join('\n')),- });- } catch (detectErr) {- log.warn(- { documentName, err: detectErr },- '[persistence] divergence-realign loss detection failed',- );- }- },- };- try {- document.transact(() => {- applyDiskContent(document, diskContent, undefined, undefined, undefined, detect);- }, FILE_WATCHER_ORIGIN);- } catch (err) {- durabilityState.recordStoreFailure(documentName, toStoreFailure(err));- persistenceDeferCounts.delete(documentName);- throw err;- }- durabilityState.setReconciledBase(documentName, diskContent);- durabilityState.recordStoreDivergence(documentName);- persistenceDeferCounts.delete(documentName);- return;- }- }-if (documentHadFileOnDisk(documentName, currentBase) &&resolveStorePathPresence(@@ -1876,6 +1873,7 @@ export function createPersistenceExtension(options?: PersistenceOptions): Persis}const tmpPath = atomicTempPath(canonicalPath);+ let storedObservation: FileObservation | undefined;try {if (process.env.NODE_ENV === 'test' && process.env.OK_TEST_STORE_FAULT === documentName) {const faultErr = new Error(@@ -1885,33 +1883,182 @@ export function createPersistenceExtension(options?: PersistenceOptions): Persisthrow faultErr;}await tracedWriteFile(tmpPath, markdown, 'utf-8');- if (- documentHadFileOnDisk(documentName, currentBase) &&- resolveStorePathPresence(- documentName,- canonicalPath,- currentBase,- markdown,- agentTriggeredStore,- ) === 'refused'- ) {+ const discardTemp = (reason: string): void => {try {tracedUnlinkSync(tmpPath);} catch (cleanupErr) {if (errnoCode(cleanupErr) !== 'ENOENT') {log.warn({ err: cleanupErr, docName: documentName, path: tmpPath },- '[persistence] could not remove the temp file after refusing the publish',+ `[persistence] could not remove the temp file after ${reason}`,);}}+ };+ const readForCompare = compareReaderFor(documentName);+ let compared: DiskCompareOutcome;+ for (let attempt = 0; ; attempt++) {+ if (+ documentHadFileOnDisk(documentName, currentBase) &&+ resolveStorePathPresence(+ documentName,+ canonicalPath,+ currentBase,+ markdown,+ agentTriggeredStore,+ ) === 'refused'+ ) {+ discardTemp('refusing the publish');+ return;+ }+ compared = compareDiskBeforeReplace(+ {+ path: canonicalPath,+ receiptHash: durabilityState.getReceipt(documentName)?.hash,+ baseHash: currentBase === undefined ? undefined : contentHash(currentBase),+ candidateHash: contentHash(markdown),+ },+ readForCompare,+ );+ const retryDelayMs = COMPARE_READ_RETRY_DELAYS_MS[attempt];+ if (+ compared.kind !== 'unreadable' ||+ !compared.transient ||+ retryDelayMs === undefined+ ) {+ break;+ }+ incrementStoreDiskCompareReadRetries();+ await new Promise((resolve) => setTimeout(resolve, retryDelayMs));+ }+ if (+ (compared.kind === 'drift' ||+ (compared.kind === 'match' && !compared.alreadyCandidate)) &&+ durabilityState.getReconciledBase(documentName) !== currentBase &&+ durabilityState.getReceipt(documentName)?.via !== 'own-write'+ ) {+ discardTemp('the acknowledged base moved during the store');+ compareLog.debug(+ { documentName },+ `[store-compare] ${documentName}: an intake moved the acknowledged version while the store ran; storing again from a fresh snapshot`,+ );+ control.rerun = 'base-moved';+ return;+ }+ let comparedObservation: FileObservation | null;+ switch (compared.kind) {+ case 'match':+ incrementStoreDiskCompare('match');+ comparedObservation = compared.observation;+ break;+ case 'drift':+ incrementStoreDiskCompare('drift');+ if (currentBase !== undefined) {+ durabilityState.setReconciledBase(documentName, currentBase, compared.observation);+ }+ comparedObservation = compared.observation;+ break;+ case 'absent':+ incrementStoreDiskCompare('absent');+ if (documentHadFileOnDisk(documentName, currentBase)) {+ discardTemp('the file vanished before the publish');+ control.rerun = 'disk-changed';+ return;+ }+ comparedObservation = null;+ break;+ case 'unreadable':+ discardTemp('an unreadable pre-write compare');+ return holdStorePending(documentName, 'pending-unreadable', compared.error);+ case 'changed': {+ discardTemp('finding an outside change before the publish');+ if (+ refuseStaleExternalWrite(+ durabilityState,+ document,+ documentName,+ compared.text,+ options?.conflicts,+ markdown,+ )+ ) {+ incrementStoreDiskCompare('stale-refused');+ if (agentTriggeredStore) {+ durabilityState.recordStaleExternalWriteFreeze(documentName);+ }+ persistenceDeferCounts.delete(documentName);+ return;+ }+ incrementStoreDiskCompare('changed');+ const ingest = options.ingestDiskAtStore;+ if (ingest === null) return holdStorePending(documentName, 'pending-no-intake');+ compareLog.info(+ {+ documentName,+ diskBytes: Buffer.byteLength(compared.text, 'utf8'),+ candidateBytes: Buffer.byteLength(markdown, 'utf8'),+ },+ `[store-compare] ${documentName} changed on disk since the last agreed version; merging it before the write`,+ );+ const intake = await ingest(documentName, canonicalPath, compared.text);+ switch (intake.kind) {+ case 'taken-in':+ persistenceDeferCounts.delete(documentName);+ control.rerun = 'disk-changed';+ return;+ case 'conflict':+ return holdStorePending(documentName, 'pending-conflict');+ case 'failed':+ return holdStorePending(+ documentName,+ currentBase === undefined+ ? 'pending-intake-failed-no-base'+ : 'pending-intake-failed',+ intake.error,+ );+ default:+ return assertNeverStoreIntake(intake);+ }+ }+ default:+ return assertNeverDiskCompare(compared);+ }+ let outcome: StorePublishOutcome;+ try {+ outcome = durabilityState.tryPublishStore(storeAttempt, () => {+ assertProjectContentScope(canonicalPath, contentDir);+ options?.assertContentPath?.(canonicalPath);+ if (+ process.env.NODE_ENV === 'test' &&+ process.env.OK_TEST_STORE_RESTAT_RACE === documentName+ ) {+ delete process.env.OK_TEST_STORE_RESTAT_RACE;+ tracedAppendFileSync(canonicalPath, '\nrestat-race-injected\n', 'utf-8');+ }+ const restat = restatFile(canonicalPath, comparedObservation);+ if (restat.kind === 'failed') {+ throw new StoreRestatFailedError(documentName, restat.error);+ }+ if (restat.kind === 'moved') throw new StoreRestatRefusedError(documentName);+ storedObservation = statObservation(tmpPath);+ tracedRenameSync(tmpPath, canonicalPath);+ });+ } catch (publishErr) {+ if (publishErr instanceof StoreRestatFailedError) {+ discardTemp('a failed re-stat');+ return holdStorePending(documentName, 'pending-unreadable', publishErr.cause);+ }+ if (!(publishErr instanceof StoreRestatRefusedError)) throw publishErr;+ discardTemp('a refused re-stat');+ incrementStoreRestatRefused();+ compareLog.warn(+ { documentName },+ `[store-compare] ${documentName} changed between the compare and the rename; the write was refused and the store runs again`,+ );+ persistenceDeferCounts.delete(documentName);+ control.rerun = 'disk-changed';return;}- const outcome = durabilityState.tryPublishStore(storeAttempt, () => {- assertProjectContentScope(canonicalPath, contentDir);- options?.assertContentPath?.(canonicalPath);- tracedRenameSync(tmpPath, canonicalPath);- });if (outcome !== 'published') {try {tracedUnlinkSync(tmpPath);@@ -1990,8 +2137,9 @@ export function createPersistenceExtension(options?: PersistenceOptions): PersisdocumentName,markdown,agentTriggeredStore && currentBase !== undefined && currentBase !== markdown- ? (diskBeforeWrite ?? currentBase)+ ? currentBase: undefined,+ storedObservation,);docsWithSettledWrite.add(documentName);docsWithFileObservedOnDisk.add(documentName);@@ -2271,7 +2419,15 @@ export function createPersistenceExtension(options?: PersistenceOptions): Persis);}- if (!staleExternalWrite) durabilityState.setReconciledBase(documentName, raw);+ if (!staleExternalWrite) {+ if (durabilityState.checkReceiptAtLoad(documentName, raw) === 'changed') {+ compareLog.info(+ { documentName },+ `[store-compare] ${documentName} changed on disk since its last agreed version; loaded the disk version`,+ );+ }+ durabilityState.setReconciledBase(documentName, raw);+ }},).finally(() => {loadDurationHist?.record((Date.now() - started) / 1000);diff --git a/packages/server/src/reconcile-own-flush-window.test.ts b/packages/server/src/reconcile-own-flush-window.test.tsindex 02606760d..6b61d52da 100644--- a/packages/server/src/reconcile-own-flush-window.test.ts+++ b/packages/server/src/reconcile-own-flush-window.test.ts@@ -102,6 +102,7 @@ async function drivePhantomDivergence(let windowFired = false;let windowGuard: Promise<ReconcileBeforeWriteResult> | undefined;const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -205,6 +206,7 @@ describe('reconcileDiskBeforeAgentWrite — own persistence flush is not foreignwriteFileSync(docPath, BASE_CONTENT, 'utf-8');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -243,6 +245,7 @@ describe('reconcileDiskBeforeAgentWrite — own persistence flush is not foreignlet laterFlush: Promise<void> | undefined;let peekAfterLaterStart: string | undefined;const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -311,6 +314,24 @@ describe('reconcileDiskBeforeAgentWrite — overlapping own flushes are not forelet conflictAfterGuard: boolean | undefined;const persistence = createPersistenceExtension({+ ingestDiskAtStore: async (name, _path, content) => {+ const taken = await reconcileDiskBeforeAgentWrite(+ durabilityState,+ fakeHocuspocusWith(name, document),+ name,+ tmpDir,+ undefined,+ undefined,+ RECONCILE_TEST_CONFLICTS,+ NO_SHADOW_MERGE_HISTORY,+ );+ return taken.reconciled || durabilityState.getReconciledBase(name) === content+ ? { kind: 'taken-in' }+ : {+ kind: 'not-taken-in',+ error: new Error('the pre-write merge did not take the disk version in'),+ };+ },contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,diff --git a/packages/server/src/server-factory.test.ts b/packages/server/src/server-factory.test.tsindex dd03d3e42..f9b4da49c 100644--- a/packages/server/src/server-factory.test.ts+++ b/packages/server/src/server-factory.test.ts@@ -6390,6 +6390,84 @@ describe('createServer() — disk-event reconcile with an absent reconciled base}}, 60_000);+ test('a parked WIP restored after a branch round trip is saved by the next store instead of being merged away', async () => {+ const projectDir = await realpath(mkdtempSync(join(tmpdir(), 'ok-reconcile-park-store-')));+ const git = simpleGit(projectDir);+ await git.init(['--initial-branch=main']);+ configureTestGitRepository(projectDir);+ await git.raw('config', 'user.name', 'Test');+ await git.raw('config', 'user.email', '[email protected]');+ const docName = 'park-store-target';+ const docPath = join(projectDir, `${docName}.md`);+ const mainContent = '# Park store target\n\nMain branch body.\n\nTail.\n';+ const featureContent = '# Park store target\n\nFeature branch body.\n\nTail.\n';+ writeFileSync(docPath, mainContent, 'utf-8');+ await git.add('.');+ await git.commit('main content');+ await git.checkoutLocalBranch('feature');+ writeFileSync(docPath, featureContent, 'utf-8');+ await git.add(['-A']);+ await git.commit('feature content');+ await git.checkout('main');++ const shadow = await initShadowRepo(projectDir);+ rig = {+ tmpDir: projectDir,+ shadow,+ cleanup: () => rmSync(projectDir, { recursive: true, force: true }),+ };+ const server = createServer({+ contentDir: projectDir,+ projectDir,+ quiet: true,+ debounce: 60_000,+ maxDebounce: 120_000,+ gitEnabled: false,+ shadowRepo: shadow,+ });+ try {+ await server.ready;+ const conn = await server.hocuspocus.openDirectConnection(docName);+ const serverDoc = server.hocuspocus.documents.get(docName);+ expect(serverDoc).toBeDefined();+ if (!serverDoc) return;+ await vi.waitFor(+ () => expect(server.durabilityState.getReconciledBase(docName)).toBe(mainContent),+ { timeout: 5_000, interval: 25 },+ );+ const source = serverDoc.getText('source');+ serverDoc.transact(() =>+ source.insert(source.toString().indexOf('Tail.'), 'Unsaved WIP.\n\n'),+ );+ const withWip = '# Park store target\n\nMain branch body.\n\nUnsaved WIP.\n\nTail.\n';+ expect(source.toString()).toBe(withWip);++ await git.checkout('feature');+ await vi.waitFor(() => expect(source.toString()).toBe(featureContent), {+ timeout: 15_000,+ interval: 25,+ });+ await git.checkout('main');+ await vi.waitFor(() => expect(source.toString()).toBe(withWip), {+ timeout: 15_000,+ interval: 25,+ });++ serverDoc.transact(() => source.insert(source.length, '\nTyped after the switch.\n'));+ await server.hocuspocus.flushPendingStores();+ const saved = `${withWip}\nTyped after the switch.\n`;+ await vi.waitFor(() => expect(readFileSync(docPath, 'utf-8')).toBe(saved), {+ timeout: 10_000,+ interval: 25,+ });+ expect(source.toString()).toBe(saved);++ conn.disconnect();+ } finally {+ await server.destroy();+ }+ }, 60_000);+test('a branch-switch reset rescues unflushed WIP on an absent-base doc before overwriting with disk content', async () => {const projectDir = await realpath(mkdtempSync(join(tmpdir(), 'ok-reconcile-bs-rescue-')));const git = simpleGit(projectDir);@@ -6612,7 +6690,7 @@ describe('createServer() — disk-event reconcile with an absent reconciled base}}, 30_000);- test('a failed disk-authoritative ingest keeps the live content, adopts the disk bytes as the base, and does not re-ingest on the next update', async () => {+ test('a failed disk-authoritative ingest keeps the live content, leaves no agreed version for the disk bytes it lacks, and takes the next update in again', async () => {rig = await setupReconcileRig('ok-reconcile-ingest-failure-');const docName = 'ingest-failure-target';const initial = '# Ingest failure target\n\nLive paragraph that exists only in the editor.\n';@@ -6655,9 +6733,12 @@ describe('createServer() — disk-event reconcile with an absent reconciled basewriteFileSync(docPath, theirs, 'utf-8');await vi.waitFor(- () => expect(server.durabilityState.getReconciledBase(docName)).toBe(theirs),+ () =>+ expect(getMetrics().diskAuthoritativeIngestApplyFailures).toBe(applyFailuresBefore + 1),{ timeout: 10_000, interval: 25 },);+ expect(server.durabilityState.getReconciledBase(docName)).toBeUndefined();+ expect(server.durabilityState.getReceipt(docName)).toBeUndefined();expect(serverDoc.getText('source').toString()).toBe(initial);expect(serverDoc.getMap('lifecycle').get('status')).toBeUndefined();expect(getMetrics().diskAuthoritativeIngestApplyFailures).toBe(applyFailuresBefore + 1);@@ -6668,26 +6749,25 @@ describe('createServer() — disk-event reconcile with an absent reconciled basewriteFileSync(docPath, followUp, 'utf-8');- await vi.waitFor(- () =>- expect(serverDoc.getText('source').toString()).toContain('Disk addition after recovery.'),- { timeout: 15_000, interval: 25 },- );- expect(serverDoc.getText('source').toString()).toContain(- 'Live paragraph that exists only in the editor.',- );+ await vi.waitFor(() => expect(serverDoc.getText('source').toString()).toBe(followUp), {+ timeout: 15_000,+ interval: 25,+ });+ expect(server.durabilityState.getReconciledBase(docName)).toBe(followUp);+ expect(getMetrics().diskAuthoritativeIngestCount).toBe(ingestsBefore + 2);await vi.waitFor(async () => {const rescues = await listRescueCheckpoints(rig.shadow, 'main');- expect(rescues.filter((r) => r.docName === docName)).toHaveLength(2);+ expect(rescues.filter((r) => r.docName === docName).map((r) => r.size)).toEqual(+ expect.arrayContaining([initial.length, theirs.length]),+ );},{ timeout: 10_000, interval: 50 },);const rescues = await listRescueCheckpoints(rig.shadow, 'main');const docRescues = rescues.filter((r) => r.docName === docName);- expect(docRescues).toHaveLength(2);- expect(docRescues.map((r) => r.size).sort((a, b) => a - b)).toEqual(- [initial.length, theirs.length].sort((a, b) => a - b),+ expect(new Set(docRescues.map((r) => r.size))).toEqual(+ new Set([initial.length, theirs.length]),);const sg = shadowGit(rig.shadow);const mintKinds = new Set(diff --git a/packages/server/src/server-factory.ts b/packages/server/src/server-factory.tsindex 0882cd70d..bec310944 100644--- a/packages/server/src/server-factory.ts+++ b/packages/server/src/server-factory.ts@@ -1,6 +1,13 @@import { type SpawnSyncReturns, spawnSync } from 'node:child_process';import { randomUUID } from 'node:crypto';-import { existsSync, readFileSync, realpathSync, statSync, writeFileSync } from 'node:fs';+import {+ existsSync,+ lstatSync,+ readFileSync,+ realpathSync,+ statSync,+ writeFileSync,+} from 'node:fs';import { homedir } from 'node:os';import { basename, dirname, join, relative, resolve, sep } from 'node:path';import type { Document, Extension } from '@hocuspocus/server';@@ -123,11 +130,13 @@ import {} from './derived-document-index.ts';import { resolveDirectoryRoot } from './directory-root.ts';import { applyDiskContentToDoc } from './disk-content-intake.ts';+import { DiskReceiptLog } from './disk-receipts.ts';import {canonicalDocName,docNameToRelativePath,getDocExtension,isRegisteredMarkdownDocName,+ SUPPORTED_DOC_EXTENSIONS,stripDocExtension,} from './doc-extensions.ts';import { runDocLineageGuard } from './doc-lineage-guard.ts';@@ -243,6 +252,7 @@ import {incrementDiskAuthoritativeIngestApplyFailures,incrementDiskAuthoritativeIngestUnrescued,incrementPark,+ incrementReceiptIntakeSkip,incrementRecentlyRemovedDocsEviction,incrementReconcile,incrementReconcileInsertDedupSkipped,@@ -329,6 +339,7 @@ import {import { resyncRecordedSkillCopies } from './skill-placements.ts';import { skillStateYamlPath } from './skill-state.ts';import { assertCompatibleStateManifest } from './state-manifest.ts';+import { classifyIntakeAgainstReceipt, receiptIntakeSkipOf } from './store-compare.ts';import { assertRealpathWithinDir } from './symlink-guard.ts';import { SyncEngine } from './sync-engine.ts';import { createSyncHandshakeSpanExtension } from './sync-handshake-span-extension.ts';@@ -590,8 +601,26 @@ export function createServer(options: ServerOptions): ServerInstance {const generatedIndexWarningScope = createGeneratedIndexWarningScope(resolve(contentDir));let cc1Broadcaster: CC1Broadcaster | null = null;const initialBranch = readProjectHeadState(projectDir).branch ?? 'main';+ const diskReceipts = new DiskReceiptLog({+ path: join(getLocalDir(projectDir), 'receipts.jsonl'),+ });+ const receiptFileIsGone = (docName: string): boolean => {+ for (const extension of SUPPORTED_DOC_EXTENSIONS) {+ const candidate = resolve(contentDir, `${docName}${extension}`);+ if (!isWithinDir(candidate, contentDir)) return false;+ try {+ lstatSync(candidate);+ return false;+ } catch (err) {+ const code = errnoCode(err);+ if (code !== 'ENOENT' && code !== 'ENOTDIR') return false;+ }+ }+ return true;+ };const durabilityState = new DocumentDurabilityState(initialBranch, {persistencePath: join(getLocalDir(projectDir), 'stale-external-writes.json'),+ receipts: diskReceipts,onStaleExternalWriteChange: () => {cc1Broadcaster?.signal('sync-status');hydrateStaleExternalWrites();@@ -2208,6 +2237,25 @@ export function createServer(options: ServerOptions): ServerInstance {isRecentlyRemoved: (docName) => recentlyRemovedDocs.has(docName),mdManager: options.mdManager,getLossRing: () => lossRing,+ ingestDiskAtStore: async (docName, absPath, content) => {+ let outcome: DiskIntakeOutcome;+ try {+ outcome = await intakeDiskUpdate({ kind: 'update', path: absPath, docName, content });+ } catch (error) {+ return { kind: 'failed', error };+ }+ switch (outcome.kind) {+ case 'applied':+ case 'nothing-to-take':+ return { kind: 'taken-in' };+ case 'conflict':+ return { kind: 'conflict' };+ case 'failed':+ return { kind: 'failed', error: outcome.error };+ default:+ return assertNeverDiskIntakeOutcome(outcome);+ }+ },};persistence = createPersistenceExtension({ ...persistenceOpts, durabilityState });@@ -3007,8 +3055,9 @@ export function createServer(options: ServerOptions): ServerInstance {const applyDiskAuthoritativeIngest = async (docName: string,theirs: string,- baseAbsent: boolean,- ): Promise<void> => {+ priorBase: string | undefined,+ ): Promise<DiskIntakeOutcome> => {+ const baseAbsent = priorBase === undefined;incrementDiskAuthoritativeIngest();const ours = serializeDoc(docName) ?? '';const branch = getStorageBranch() ?? 'main';@@ -3022,18 +3071,19 @@ export function createServer(options: ServerOptions): ServerInstance {}const oursH = contentHash(ours).slice(0, 6);const baseToken = baseAbsent ? 'absent' : 'blockless';+ let applyError: unknown;let ingested = false;try {applyToDoc(docName, theirs);dissolveNonGitConflicts(docName);ingested = true;} catch (e) {+ applyError = e;incrementDiskAuthoritativeIngestApplyFailures();log.error({ err: e, docName },`[reconcile] failed to apply disk-authoritative ingest to Y.Doc for ${docName}`,);- setReconciledBase(docName, theirs);mintExternalChangeRescue(docName, theirs, branch, 'disk-update', theirsH, true);dissolveNonGitConflicts(docName);let surviving: string | undefined;@@ -3045,6 +3095,8 @@ export function createServer(options: ServerOptions): ServerInstance {`[reconcile] unable to read ${docName} back after a failed disk-authoritative ingest; treating the live content as lost`,);}+ if (surviving === theirs) setReconciledBase(docName, theirs);+ else durabilityState.restoreReconciledBase(docName, priorBase);if (rescue === 'lost' && surviving !== ours) {incrementDiskAuthoritativeIngestUnrescued();log.warn(@@ -3078,10 +3130,231 @@ export function createServer(options: ServerOptions): ServerInstance {},`[reconcile] ${docName} base=${baseToken} reason=no-base result=${result} ours=${oursH} theirs=${theirsH} rescue=${rescue}`,);+ await recordUpsertAfterIntake(docName, theirs);+ return { kind: 'applied' };+ }+ return { kind: 'failed', error: applyError };+ };++ const recordUpsertAfterIntake = async (docName: string, theirs: string): Promise<void> => {+ try {await derivedDocumentIndex.recordDiskUpsert(docName, theirs);+ } catch (err) {+ log.warn(+ { err, docName },+ `[reconcile] ${docName}: the derived index could not record the disk version`,+ );}};+ type DiskIntakeOutcome =+ | { readonly kind: 'applied' }+ | { readonly kind: 'nothing-to-take' }+ | { readonly kind: 'conflict' }+ | { readonly kind: 'failed'; readonly error: unknown };++ function assertNeverDiskIntakeOutcome(outcome: never): never {+ throw new Error(`unhandled disk intake outcome: ${(outcome as { kind: string }).kind}`);+ }++ function assertNeverReconcileOutcome(outcome: never): never {+ throw new Error(`unhandled reconcile outcome: ${(outcome as { kind: string }).kind}`);+ }++ async function intakeDiskUpdate(+ event: Extract<DiskEvent, { kind: 'update' }>,+ ): Promise<DiskIntakeOutcome> {+ const { docName, content: theirs } = event;+ onUpstreamAdd(docName);+ if (indexedMetadataChanged(event.previousIndexedFields, theirs, docName)) {+ scheduleIndexRegeneration(docName);+ }+ if (conflicts.findByDocName(docName)?.kind === 'merge-native') {+ try {+ await conflicts.pruneMergeNativeAgainstGit();+ } catch (err) {+ log.warn({ err, docName }, '[reconcile] conflict prune against git failed');+ }+ }++ const document = hocuspocus.documents.get(docName);+ if (!document) {+ if (!refuseStaleExternalWrite(durabilityState, undefined, docName, theirs, conflicts)) {+ durabilityState.refreshReceiptWithoutBase(docName, theirs);+ }+ await derivedDocumentIndex.recordDiskUpsert(docName, theirs);+ return { kind: 'nothing-to-take' };+ }+ if (reviveFrozenResident(docName, theirs) !== 'not-frozen') {+ await derivedDocumentIndex.recordDiskUpsert(docName, theirs);+ return { kind: 'nothing-to-take' };+ }++ const theirsHash = contentHash(theirs);+ const receiptIntake = classifyIntakeAgainstReceipt({+ receiptHash: durabilityState.getReceipt(docName)?.hash,+ eventHash: theirsHash,+ isRecentOwnWrite: durabilityState.isRecentOwnWrite(docName, theirsHash),+ readDiskHash: () => {+ try {+ return contentHash(readFileSync(event.path, 'utf-8'));+ } catch {+ return null;+ }+ },+ });+ const receiptSkip = receiptIntakeSkipOf(receiptIntake);+ if (receiptSkip !== null && durabilityState.getStaleExternalWrite(docName) === undefined) {+ incrementReceiptIntakeSkip(receiptSkip);+ log.debug(+ { docName, receiptIntake },+ `[reconcile] ${docName}: the disk bytes match its receipt (${receiptIntake}); nothing to take in`,+ );+ conflicts.dissolveReconcile(docName);+ await recordUpsertAfterIntake(docName, theirs);+ return { kind: 'nothing-to-take' };+ }++ const base = getReconciledBase(docName);+ const ours = serializeDoc(docName) ?? base ?? '';++ if (refuseStaleExternalWrite(durabilityState, document, docName, theirs, conflicts, ours)) {+ await recordUpsertAfterIntake(docName, theirs);+ return { kind: 'conflict' };+ }++ if (base === undefined) {+ return applyDiskAuthoritativeIngest(docName, theirs, undefined);+ }++ const result = reconcile({ docName, base, ours, theirs });++ const baseH = contentHash(base).slice(0, 6);+ const oursH = contentHash(ours).slice(0, 6);+ const theirsH = contentHash(theirs).slice(0, 6);+ const oursLen = ours.length;+ const theirsLen = theirs.length;+ const newLen =+ result.kind === 'clean' || result.kind === 'merged' || result.kind === 'both-kept'+ ? result.newContent.length+ : undefined;+ const delta = newLen === undefined ? undefined : newLen - Math.max(oursLen, theirsLen);+ const sizeSuffix = newLen === undefined ? '' : ` newLen=${newLen} delta=${delta}`;+ log.info(+ {+ docName,+ base: baseH,+ ours: oursH,+ theirs: theirsH,+ result: result.kind,+ oursLen,+ theirsLen,+ ...(newLen === undefined ? {} : { newLen, delta }),+ },+ `[reconcile] ${docName} base=${baseH} ours=${oursH} theirs=${theirsH} result=${result.kind} oursLen=${oursLen} theirsLen=${theirsLen}${sizeSuffix}`,+ );++ reportInsertDedupSkipped(docName, result, ours, theirs);++ switch (result.kind) {+ case 'noop':+ conflicts.dissolveReconcile(docName);+ await recordUpsertAfterIntake(docName, theirs);+ return { kind: 'nothing-to-take' };++ case 'clean':+ try {+ applyToDoc(docName, result.newContent);+ } catch (e) {+ log.error(+ { err: e, docName },+ `[reconcile] failed to apply clean content to Y.Doc for ${docName}`,+ );+ setReconciledBase(docName, base);+ conflicts.dissolveReconcile(docName);+ return { kind: 'failed', error: e };+ }+ setReconciledBase(docName, result.newContent);+ incrementReconcile();+ conflicts.dissolveReconcile(docName);+ await recordUpsertAfterIntake(docName, theirs);+ return { kind: 'applied' };++ case 'merged':+ try {+ applyToDoc(docName, result.newContent);+ } catch (e) {+ log.error(+ { err: e, docName },+ `[reconcile] failed to apply merged content to Y.Doc for ${docName}`,+ );+ setReconciledBase(docName, base);+ conflicts.dissolveReconcile(docName);+ return { kind: 'failed', error: e };+ }+ setReconciledBase(docName, theirs);+ incrementReconcile();+ conflicts.dissolveReconcile(docName);+ await recordUpsertAfterIntake(docName, theirs);+ return { kind: 'applied' };++ case 'both-kept': {+ try {+ applyToDoc(docName, result.newContent);+ } catch (e) {+ log.error(+ { err: e, docName },+ `[reconcile] failed to apply both-kept content to Y.Doc for ${docName}; raising the conflict`,+ );+ setReconciledBase(docName, base);+ incrementConflict();+ conflicts.raise({+ kind: 'reconcile',+ file: conflicts.fileOf(docName),+ reason: 'merged-with-markers',+ stages: { base, ours, theirs },+ });+ return { kind: 'failed', error: e };+ }+ mergeHistory.bothKept(docName, { ours, theirs }, result.kept);+ setReconciledBase(docName, theirs);+ incrementReconcile();+ reportReconcileBothKept(docName, result.kept, 'watcher');+ conflicts.dissolveReconcile(docName);+ await recordUpsertAfterIntake(docName, theirs);+ return { kind: 'applied' };+ }++ case 'refused': {+ if (result.reason === 'no-base') {+ return applyDiskAuthoritativeIngest(docName, theirs, base);+ }+ reportReconcileCheckFailed(docName, result, 'watcher');+ if (result.reason === 'check-failed' || result.reason === 'too-large') {+ if (+ await mergeHistory.keptLive(docName, { side: 'theirs', bytes: theirs }, result.reason)+ ) {+ setReconciledBase(docName, theirs);+ reportReconcileKeptLive(docName, result.reason, 'watcher');+ conflicts.dissolveReconcile(docName);+ await recordUpsertAfterIntake(docName, theirs);+ return { kind: 'applied' };+ }+ }+ incrementConflict();+ conflicts.raise({+ kind: 'reconcile',+ file: conflicts.fileOf(docName),+ reason: wireReasonForRefusal(result.reason),+ stages: { base, ours, theirs },+ });+ return { kind: 'conflict' };+ }+ default:+ return assertNeverReconcileOutcome(result);+ }+ }+const bufferedRemovalScopes = new Map<Extract<DiskEvent, { kind: 'delete' | 'rename' }>,ReturnType<typeof beginRemoval>@@ -3110,184 +3383,7 @@ export function createServer(options: ServerOptions): ServerInstance {}case 'update': {- const { docName, content: theirs } = event;- onUpstreamAdd(docName);- if (indexedMetadataChanged(event.previousIndexedFields, theirs, docName)) {- scheduleIndexRegeneration(docName);- }- if (conflicts.findByDocName(docName)?.kind === 'merge-native') {- try {- await conflicts.pruneMergeNativeAgainstGit();- } catch (err) {- log.warn({ err, docName }, '[reconcile] conflict prune against git failed');- }- }-- const document = hocuspocus.documents.get(docName);- if (!document) {- refuseStaleExternalWrite(durabilityState, undefined, docName, theirs, conflicts);- await derivedDocumentIndex.recordDiskUpsert(docName, theirs);- return;- }- if (reviveFrozenResident(docName, theirs) !== 'not-frozen') {- await derivedDocumentIndex.recordDiskUpsert(docName, theirs);- return;- }-- const base = getReconciledBase(docName);- const ours = serializeDoc(docName) ?? base ?? '';-- if (- refuseStaleExternalWrite(durabilityState, document, docName, theirs, conflicts, ours)- ) {- await derivedDocumentIndex.recordDiskUpsert(docName, theirs);- return;- }-- if (base === undefined) {- await applyDiskAuthoritativeIngest(docName, theirs, true);- break;- }-- const result = reconcile({ docName, base, ours, theirs });-- const baseH = contentHash(base).slice(0, 6);- const oursH = contentHash(ours).slice(0, 6);- const theirsH = contentHash(theirs).slice(0, 6);- const oursLen = ours.length;- const theirsLen = theirs.length;- const newLen =- result.kind === 'clean' || result.kind === 'merged' || result.kind === 'both-kept'- ? result.newContent.length- : undefined;- const delta = newLen === undefined ? undefined : newLen - Math.max(oursLen, theirsLen);- const sizeSuffix = newLen === undefined ? '' : ` newLen=${newLen} delta=${delta}`;- log.info(- {- docName,- base: baseH,- ours: oursH,- theirs: theirsH,- result: result.kind,- oursLen,- theirsLen,- ...(newLen === undefined ? {} : { newLen, delta }),- },- `[reconcile] ${docName} base=${baseH} ours=${oursH} theirs=${theirsH} result=${result.kind} oursLen=${oursLen} theirsLen=${theirsLen}${sizeSuffix}`,- );-- reportInsertDedupSkipped(docName, result, ours, theirs);-- switch (result.kind) {- case 'noop':- conflicts.dissolveReconcile(docName);- await derivedDocumentIndex.recordDiskUpsert(docName, theirs);- break;-- case 'clean':- {- let applied = false;- try {- applyToDoc(docName, result.newContent);- setReconciledBase(docName, result.newContent);- incrementReconcile();- conflicts.dissolveReconcile(docName);- applied = true;- } catch (e) {- log.error(- { err: e, docName },- `[reconcile] failed to apply clean content to Y.Doc for ${docName}`,- );- setReconciledBase(docName, theirs);- conflicts.dissolveReconcile(docName);- }- if (applied) {- await derivedDocumentIndex.recordDiskUpsert(docName, theirs);- }- }- break;-- case 'merged':- {- let applied = false;- try {- applyToDoc(docName, result.newContent);- setReconciledBase(docName, theirs);- incrementReconcile();- conflicts.dissolveReconcile(docName);- applied = true;- } catch (e) {- log.error(- { err: e, docName },- `[reconcile] failed to apply merged content to Y.Doc for ${docName}`,- );- setReconciledBase(docName, theirs);- conflicts.dissolveReconcile(docName);- }- if (applied) {- await derivedDocumentIndex.recordDiskUpsert(docName, theirs);- }- }- break;-- case 'both-kept': {- try {- applyToDoc(docName, result.newContent);- } catch (e) {- log.error(- { err: e, docName },- `[reconcile] failed to apply both-kept content to Y.Doc for ${docName}; raising the conflict`,- );- setReconciledBase(docName, theirs);- incrementConflict();- conflicts.raise({- kind: 'reconcile',- file: conflicts.fileOf(docName),- reason: 'merged-with-markers',- stages: { base, ours, theirs },- });- break;- }- mergeHistory.bothKept(docName, { ours, theirs }, result.kept);- setReconciledBase(docName, theirs);- incrementReconcile();- reportReconcileBothKept(docName, result.kept, 'watcher');- conflicts.dissolveReconcile(docName);- await derivedDocumentIndex.recordDiskUpsert(docName, theirs);- break;- }-- case 'refused': {- if (result.reason === 'no-base') {- await applyDiskAuthoritativeIngest(docName, theirs, false);- break;- }- reportReconcileCheckFailed(docName, result, 'watcher');- if (result.reason === 'check-failed' || result.reason === 'too-large') {- if (- await mergeHistory.keptLive(- docName,- { side: 'theirs', bytes: theirs },- result.reason,- )- ) {- setReconciledBase(docName, theirs);- reportReconcileKeptLive(docName, result.reason, 'watcher');- conflicts.dissolveReconcile(docName);- await derivedDocumentIndex.recordDiskUpsert(docName, theirs);- break;- }- }- incrementConflict();- conflicts.raise({- kind: 'reconcile',- file: conflicts.fileOf(docName),- reason: wireReasonForRefusal(result.reason),- stages: { base, ours, theirs },- });- break;- }- }+ await intakeDiskUpdate(event);break;}@@ -5049,12 +5145,12 @@ export function createServer(options: ServerOptions): ServerInstance {case 'merged':case 'clean':applyToDoc(docName, outcome.newContent);- setReconciledBase(docName, outcome.newContent);+ setReconciledBase(docName, currentDisk);restoredCount++;break;case 'both-kept': {applyToDoc(docName, outcome.newContent);- setReconciledBase(docName, outcome.newContent);+ setReconciledBase(docName, currentDisk);mergeHistory.bothKept(docName,{ ours: parked.markdown, theirs: currentDisk },@@ -5313,6 +5409,11 @@ export function createServer(options: ServerOptions): ServerInstance {() => {indexRegenerationReady = true;resolveReady();+ setImmediate(() => {+ const pruned = diskReceipts.pruneGone(receiptFileIsGone);+ if (pruned > 0)+ log.info({ pruned }, `[disk-receipts] dropped ${pruned} receipt(s) whose file is gone`);+ }).unref();deferBootIndexSweep();deferShadowHousekeeping();},diff --git a/packages/server/src/stale-external-write-gate.test.ts b/packages/server/src/stale-external-write-gate.test.tsindex fd65bc111..e7b54bbf3 100644--- a/packages/server/src/stale-external-write-gate.test.ts+++ b/packages/server/src/stale-external-write-gate.test.ts@@ -55,6 +55,7 @@ describe('reconcileDiskBeforeAgentWrite — stale external write gate', () => {async function settleWrite(agentTriggered = true): Promise<void> {writeFileSync(join(tmpDir, `${docName}.md`), STALE_CONTENT, 'utf-8');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -115,7 +116,7 @@ describe('reconcileDiskBeforeAgentWrite — stale external write gate', () => {const after = getMetrics();expect(after.staleExternalWriteRefused).toBe(before.staleExternalWriteRefused + 1);expect(after.conflictCount).toBe(before.conflictCount);- expect(after.persistenceDivergenceRealign).toBe(before.persistenceDivergenceRealign);+ expect(after.storeDiskCompare).toEqual(before.storeDiskCompare);expect(after.bridgeMergeContentLoss).toBe(before.bridgeMergeContentLoss);});@@ -176,6 +177,7 @@ describe('reconcileDiskBeforeAgentWrite — stale external write gate', () => {writeFileSync(join(tmpDir, `${docName}.md`), STALE_CONTENT, 'utf-8');durabilityState.recordDisplacedVersion(docName, STALE_CONTENT);const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -299,6 +301,7 @@ describe('reconcileDiskBeforeAgentWrite — stale external write gate', () => {document.destroy();document = new Y.Doc();const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -325,6 +328,7 @@ describe('reconcileDiskBeforeAgentWrite — stale external write gate', () => {document = new Y.Doc();durabilityState = new DocumentDurabilityState('main', { persistencePath });const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -349,6 +353,7 @@ describe('reconcileDiskBeforeAgentWrite — stale external write gate', () => {writeFileSync(join(tmpDir, `${docName}.md`), STALE_CONTENT, 'utf-8');durabilityState.markAgentWriteStore(docName);const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -366,7 +371,6 @@ describe('reconcileDiskBeforeAgentWrite — stale external write gate', () => {expect(durabilityState.getStaleExternalWrite(docName)).toBeDefined();expect(durabilityState.getReconciledBase(docName)).toBe(ACKNOWLEDGED_CONTENT);expect(durabilityState.takeStaleExternalWriteFreeze(docName)).toBe(true);- expect(durabilityState.takeStoreDivergence(docName)).toBe(false);const retained = document.getText('source').toString();durabilityState.recordDisplacedVersion(docName, 'an older displaced version\n');writeFileSync(join(tmpDir, `${docName}.md`), 'an older displaced version\n', 'utf-8');@@ -377,6 +381,7 @@ describe('reconcileDiskBeforeAgentWrite — stale external write gate', () => {document = new Y.Doc();const restoredState = new DocumentDurabilityState('main', { persistencePath });const restarted = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -456,6 +461,7 @@ describe('reconcileDiskBeforeAgentWrite — stale external write gate', () => {document = new Y.Doc();const restored = new DocumentDurabilityState('main', { persistencePath });const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,@@ -486,6 +492,10 @@ describe('reconcileDiskBeforeAgentWrite — stale external write gate', () => {projectDir: tmpDir,gitEnabled: false,durabilityState,+ ingestDiskAtStore: async (name, _path, content) => {+ durabilityState.setReconciledBase(name, content);+ return { kind: 'taken-in' };+ },});await persistence.extension.onStoreDocument?.({@@ -501,6 +511,7 @@ describe('reconcileDiskBeforeAgentWrite — stale external write gate', () => {test('a blockless acknowledged base conflicts the pre-write reconcile and preserves the live doc', async () => {writeFileSync(join(tmpDir, `${docName}.md`), STALE_CONTENT, 'utf-8');const persistence = createPersistenceExtension({+ ingestDiskAtStore: null,contentDir: tmpDir,projectDir: tmpDir,gitEnabled: false,diff --git a/packages/server/src/store-compare.test.ts b/packages/server/src/store-compare.test.tsnew file mode 100644index 000000000..769adf779--- /dev/null+++ b/packages/server/src/store-compare.test.ts@@ -0,0 +1,278 @@+import {+ type BigIntStats,+ mkdtempSync,+ renameSync,+ rmSync,+ statSync,+ utimesSync,+ writeFileSync,+} from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import { afterEach, beforeEach, describe, expect, test } from 'vitest';+import type { ObservedFileRead } from './fs-traced.ts';+import {+ classifyIntakeAgainstReceipt,+ compareDiskBeforeReplace,+ isTransientReadError,+ observationOf,+ receiptIntakeSkipOf,+ restatFile,+} from './store-compare.ts';+import { contentHash } from './version-hash.ts';++let dir: string;+let path: string;++beforeEach(() => {+ dir = mkdtempSync(join(tmpdir(), 'ok-store-compare-'));+ path = join(dir, 'doc.md');+});++afterEach(() => {+ rmSync(dir, { recursive: true, force: true });+});++const BASE = '# Doc\n\nagreed\n';+const CANDIDATE = '# Doc\n\nagreed\n\ntyped here\n';++function compare(overrides: Partial<Parameters<typeof compareDiskBeforeReplace>[0]> = {}) {+ return compareDiskBeforeReplace({+ path,+ receiptHash: contentHash(BASE),+ baseHash: contentHash(BASE),+ candidateHash: contentHash(CANDIDATE),+ ...overrides,+ });+}++function errno(code: string): Error {+ return Object.assign(new Error(code), { code });+}++describe('compareDiskBeforeReplace', () => {+ test('a file equal to the receipt matches and reports the identity it was read with', () => {+ writeFileSync(path, BASE);+ const outcome = compare();+ expect(outcome.kind).toBe('match');+ if (outcome.kind !== 'match') return;+ expect(outcome.observation).toEqual(observationOf(statSync(path, { bigint: true })));+ });++ test('a file that already holds the candidate matches', () => {+ writeFileSync(path, CANDIDATE);+ expect(compare().kind).toBe('match');+ });++ test('a whitespace-only outside change the bridge normalizer tolerates is still a change', () => {+ writeFileSync(path, '# Doc\n\nagreed \n');+ expect(compare()).toEqual({ kind: 'changed', text: '# Doc\n\nagreed \n' });+ });++ test('a line-ending-only outside change is a change', () => {+ writeFileSync(path, BASE.replaceAll('\n', '\r\n'));+ expect(compare().kind).toBe('changed');+ });++ test('without a receipt the in-memory base is the agreed state', () => {+ writeFileSync(path, BASE);+ expect(compare({ receiptHash: undefined }).kind).toBe('match');+ writeFileSync(path, 'outside\n');+ expect(compare({ receiptHash: undefined }).kind).toBe('changed');+ });++ test('without a receipt or a base any other file is a change', () => {+ writeFileSync(path, BASE);+ expect(compare({ receiptHash: undefined, baseHash: undefined }).kind).toBe('changed');+ });++ test('a file equal to the base but not the receipt is drift, not a change', () => {+ writeFileSync(path, BASE);+ expect(compare({ receiptHash: contentHash('something else') }).kind).toBe('drift');+ });++ test('a missing file is absent', () => {+ expect(compare()).toEqual({ kind: 'absent' });+ });++ test('a directory in the way is unreadable and not transient', () => {+ const outcome = compareDiskBeforeReplace(+ {+ path: dir,+ receiptHash: undefined,+ baseHash: undefined,+ candidateHash: contentHash(CANDIDATE),+ },+ undefined,+ 'linux',+ );+ expect(outcome).toMatchObject({ kind: 'unreadable', transient: false });+ });++ test('a busy file is unreadable and transient', () => {+ const outcome = compareDiskBeforeReplace(+ {+ path,+ receiptHash: undefined,+ baseHash: undefined,+ candidateHash: '',+ },+ () => {+ throw errno('EBUSY');+ },+ );+ expect(outcome).toMatchObject({ kind: 'unreadable', transient: true });+ });++ test('a file that changes during the read is unreadable and transient', () => {+ writeFileSync(path, BASE);+ const before = statSync(path, { bigint: true });+ const changing: ObservedFileRead = {+ bytes: Buffer.from(BASE),+ before,+ after: { ...before, size: before.size + 1n } as BigIntStats,+ };+ const outcome = compareDiskBeforeReplace(+ { path, receiptHash: contentHash(BASE), baseHash: undefined, candidateHash: '' },+ () => changing,+ );+ expect(outcome).toMatchObject({ kind: 'unreadable', transient: true });+ });+});++describe('isTransientReadError', () => {+ test.each([+ ['EBUSY', 'linux', true],+ ['EAGAIN', 'darwin', true],+ ['EMFILE', 'linux', true],+ ['EPERM', 'win32', true],+ ['EACCES', 'win32', true],+ ['EPERM', 'linux', false],+ ['EACCES', 'darwin', false],+ ['EISDIR', 'win32', false],+ ] as const)('%s on %s is transient: %s', (code, platform, expected) => {+ expect(isTransientReadError(errno(code), platform)).toBe(expected);+ });++ test('an error without a code is not transient', () => {+ expect(isTransientReadError(new Error('boom'), 'win32')).toBe(false);+ });+});++describe('restatFile', () => {+ test('an untouched file is the same file', () => {+ writeFileSync(path, BASE);+ expect(restatFile(path, observationOf(statSync(path, { bigint: true })))).toEqual({+ kind: 'same',+ });+ });++ test('a rewrite with a different size moved', () => {+ writeFileSync(path, BASE);+ const observed = observationOf(statSync(path, { bigint: true }));+ writeFileSync(path, `${BASE}more\n`);+ expect(restatFile(path, observed)).toEqual({ kind: 'moved' });+ });++ test('a same-size rewrite with a new modification time moved', () => {+ writeFileSync(path, BASE);+ const observed = observationOf(statSync(path, { bigint: true }));+ writeFileSync(path, BASE.toUpperCase());+ utimesSync(path, new Date(2_000_000_000_000), new Date(2_000_000_000_000));+ expect(restatFile(path, observed)).toEqual({ kind: 'moved' });+ });++ test('a file replaced by another file with the same bytes moved', () => {+ writeFileSync(path, BASE);+ const observed = observationOf(statSync(path, { bigint: true }));+ const other = join(dir, 'other.md');+ writeFileSync(other, BASE);+ renameSync(other, path);+ expect(restatFile(path, observed)).toEqual({ kind: 'moved' });+ });++ test('a file created where none was observed moved, and an absent path is the same absence', () => {+ expect(restatFile(path, null)).toEqual({ kind: 'same' });+ expect(restatFile(join(path, 'below-a-file'), null)).toEqual({ kind: 'same' });+ writeFileSync(path, BASE);+ expect(restatFile(join(path, 'below-a-file'), null)).toEqual({ kind: 'same' });+ expect(restatFile(path, null)).toEqual({ kind: 'moved' });+ });++ test('a stat error that is not an absence is reported as a failure with its errno, not as a race', () => {+ const outcome = restatFile(`${dir}/${'x'.repeat(5000)}`, null);+ expect(outcome.kind).toBe('failed');+ if (outcome.kind !== 'failed') return;+ expect((outcome.error as NodeJS.ErrnoException).code).toBe('ENAMETOOLONG');+ });+});++describe('receiptIntakeSkipOf', () => {+ test('maps every decision to its counter key or to no skip', () => {+ expect(receiptIntakeSkipOf('own-write')).toBe('echo');+ expect(receiptIntakeSkipOf('superseded')).toBe('superseded');+ expect(receiptIntakeSkipOf('outside')).toBeNull();+ });+});++describe('classifyIntakeAgainstReceipt', () => {+ const receiptHash = contentHash('newest own write\n');++ test('an event carrying the receipt bytes is our own write', () => {+ expect(+ classifyIntakeAgainstReceipt({+ receiptHash,+ eventHash: receiptHash,+ isRecentOwnWrite: true,+ readDiskHash: () => {+ throw new Error('no read needed');+ },+ }),+ ).toBe('own-write');+ });++ test('an older own write delivered after a newer one is superseded when disk holds the newer one', () => {+ expect(+ classifyIntakeAgainstReceipt({+ receiptHash,+ eventHash: contentHash('older own write\n'),+ isRecentOwnWrite: true,+ readDiskHash: () => receiptHash,+ }),+ ).toBe('superseded');+ });++ test('an older own write that is on disk again is an outside change', () => {+ const older = contentHash('older own write\n');+ expect(+ classifyIntakeAgainstReceipt({+ receiptHash,+ eventHash: older,+ isRecentOwnWrite: true,+ readDiskHash: () => older,+ }),+ ).toBe('outside');+ });++ test('bytes we never wrote are an outside change', () => {+ expect(+ classifyIntakeAgainstReceipt({+ receiptHash,+ eventHash: contentHash('outside\n'),+ isRecentOwnWrite: false,+ readDiskHash: () => receiptHash,+ }),+ ).toBe('outside');+ });++ test('without a receipt every event is an outside change', () => {+ expect(+ classifyIntakeAgainstReceipt({+ receiptHash: undefined,+ eventHash: receiptHash,+ isRecentOwnWrite: true,+ readDiskHash: () => receiptHash,+ }),+ ).toBe('outside');+ });+});diff --git a/packages/server/src/store-compare.ts b/packages/server/src/store-compare.tsnew file mode 100644index 000000000..f7d8c6d03--- /dev/null+++ b/packages/server/src/store-compare.ts@@ -0,0 +1,176 @@+import { type BigIntStats, statSync } from 'node:fs';+import type { FileObservation } from './disk-receipts.ts';+import { type ObservedFileRead, tracedReadObservedSync } from './fs-traced.ts';+import { errnoCode } from './http/handler-utils.ts';+import type { ReceiptIntakeSkip } from './metrics.ts';+import { contentHash } from './version-hash.ts';++export const COMPARE_READ_RETRY_DELAYS_MS: readonly number[] = [50, 200, 800];++const ABSENT_CODES = new Set(['ENOENT', 'ENOTDIR']);+const TRANSIENT_READ_CODES = new Set(['EBUSY', 'EAGAIN', 'EINTR', 'EMFILE', 'ENFILE']);+const WIN32_TRANSIENT_READ_CODES = new Set(['EPERM', 'EACCES']);++class FileChangedDuringReadError extends Error {+ readonly code = 'EAGAIN';++ constructor() {+ super('the file changed while it was being read for the pre-write compare');+ this.name = 'FileChangedDuringReadError';+ }+}++export class StoreRestatRefusedError extends Error {+ constructor(docName: string) {+ super(`the file for ${docName} changed between the pre-write compare and the rename`);+ this.name = 'StoreRestatRefusedError';+ }+}++export class StoreRestatFailedError extends Error {+ readonly code: string | undefined;++ constructor(docName: string, cause: unknown) {+ super(`the file for ${docName} could not be re-read before the rename`, { cause });+ this.name = 'StoreRestatFailedError';+ this.code = errnoCode(cause);+ }+}++export type DiskCompareOutcome =+ | {+ readonly kind: 'match';+ readonly observation: FileObservation;+ readonly alreadyCandidate: boolean;+ }+ | { readonly kind: 'drift'; readonly observation: FileObservation }+ | { readonly kind: 'changed'; readonly text: string }+ | { readonly kind: 'absent' }+ | { readonly kind: 'unreadable'; readonly error: unknown; readonly transient: boolean };++export function assertNeverDiskCompare(outcome: never): never {+ throw new Error(`unhandled disk compare outcome: ${(outcome as { kind: string }).kind}`);+}++export interface DiskCompareInput {+ readonly path: string;+ readonly receiptHash: string | undefined;+ readonly baseHash: string | undefined;+ readonly candidateHash: string;+}++export function isTransientReadError(+ err: unknown,+ platform: NodeJS.Platform = process.platform,+): boolean {+ if (err instanceof FileChangedDuringReadError) return true;+ const code = errnoCode(err);+ if (code === undefined) return false;+ if (TRANSIENT_READ_CODES.has(code)) return true;+ return platform === 'win32' && WIN32_TRANSIENT_READ_CODES.has(code);+}++export function observationOf(stats: BigIntStats): FileObservation {+ return { size: stats.size, mtimeNs: stats.mtimeNs, dev: stats.dev, ino: stats.ino };+}++function sameFileState(a: FileObservation, b: FileObservation): boolean {+ return a.size === b.size && a.mtimeNs === b.mtimeNs && a.dev === b.dev && a.ino === b.ino;+}++export function compareDiskBeforeReplace(+ input: DiskCompareInput,+ read: (path: string) => ObservedFileRead = tracedReadObservedSync,+ platform: NodeJS.Platform = process.platform,+): DiskCompareOutcome {+ let observed: ObservedFileRead;+ try {+ observed = read(input.path);+ } catch (error) {+ const code = errnoCode(error);+ if (code !== undefined && ABSENT_CODES.has(code)) return { kind: 'absent' };+ return { kind: 'unreadable', error, transient: isTransientReadError(error, platform) };+ }+ const observation = observationOf(observed.before);+ if (+ !sameFileState(observation, observationOf(observed.after)) ||+ BigInt(observed.bytes.byteLength) !== observed.before.size+ ) {+ return { kind: 'unreadable', error: new FileChangedDuringReadError(), transient: true };+ }+ const text = observed.bytes.toString('utf8');+ const diskHash = contentHash(text);+ const expected = input.receiptHash ?? input.baseHash;+ if (diskHash === input.candidateHash) {+ return { kind: 'match', observation, alreadyCandidate: true };+ }+ if (diskHash === expected) {+ return { kind: 'match', observation, alreadyCandidate: false };+ }+ if (input.baseHash !== undefined && diskHash === input.baseHash) {+ return { kind: 'drift', observation };+ }+ return { kind: 'changed', text };+}++export type RestatOutcome =+ | { readonly kind: 'same' }+ | { readonly kind: 'moved' }+ | { readonly kind: 'failed'; readonly error: unknown };++export function restatFile(path: string, observation: FileObservation | null): RestatOutcome {+ let stats: BigIntStats | undefined;+ try {+ stats = statSync(path, { bigint: true });+ } catch (error) {+ const code = errnoCode(error);+ if (code === undefined || !ABSENT_CODES.has(code)) return { kind: 'failed', error };+ stats = undefined;+ }+ if (observation === null) return stats === undefined ? { kind: 'same' } : { kind: 'moved' };+ return stats !== undefined && sameFileState(observation, observationOf(stats))+ ? { kind: 'same' }+ : { kind: 'moved' };+}++export function statObservation(path: string): FileObservation | undefined {+ try {+ const stats = statSync(path, { bigint: true, throwIfNoEntry: false });+ return stats ? observationOf(stats) : undefined;+ } catch {+ return undefined;+ }+}++export type ReceiptIntakeDecision = 'own-write' | 'superseded' | 'outside';++export function classifyIntakeAgainstReceipt(input: {+ readonly receiptHash: string | undefined;+ readonly eventHash: string;+ readonly isRecentOwnWrite: boolean;+ readonly readDiskHash: () => string | null;+}): ReceiptIntakeDecision {+ if (input.receiptHash === undefined) return 'outside';+ if (input.eventHash === input.receiptHash) return 'own-write';+ if (input.isRecentOwnWrite && input.readDiskHash() === input.receiptHash) {+ return 'superseded';+ }+ return 'outside';+}++export function receiptIntakeSkipOf(decision: ReceiptIntakeDecision): ReceiptIntakeSkip | null {+ switch (decision) {+ case 'own-write':+ return 'echo';+ case 'superseded':+ return 'superseded';+ case 'outside':+ return null;+ default:+ return assertNeverIntakeDecision(decision);+ }+}++function assertNeverIntakeDecision(decision: never): never {+ throw new Error(`unhandled receipt intake decision: ${String(decision)}`);+}