Skip to content

Beautiful, AI-native markdown IDE and LLM wiki

fix(open-knowledge): table-cell fidelity across serialize, bridge, and paste (#2409)

Nick Gomez8cd05a669e8f
diff --git a/.changeset/bridge-lazy-continuation-checks.md b/.changeset/bridge-lazy-continuation-checks.md
new file mode 100644
index 000000000..90cb1f574
--- /dev/null
+++ b/.changeset/bridge-lazy-continuation-checks.md
@@ -0,0 +1,9 @@
+---
+"@inkeep/open-knowledge": patch
+---
+
+Fix spurious bridge warnings and re-derive churn on documents containing CommonMark lazy continuations.
+
+Documents whose markdown carries a lazy continuation — an unindented wrapped line inside a list item, a paragraph glued directly under a list's last bullet, or a blockquote continuation without the `> ` prefix — previously emitted recurring `bridge-invariant-violation` (tolerance-class `untracked`) and `bridge-split-brain-rederive` warnings, re-derived the editor fragment on every settlement, and re-ran fragment reconciliation on every save, even though the document parsed correctly and no data was ever at risk.
+
+The bridge health checks now recognize parse-equivalence: when the stored bytes and the canonical serialization parse to the same document, the resting byte difference is reported through the `bridge-tolerance-applied` channel (class `parse-equivalence`) instead of alerting, and no re-derive or reconciliation churn runs. Genuine divergences — where the editor state does not match what the stored markdown parses to — keep alerting exactly as before. Stored bytes are never rewritten; your authored lazy-continuation form stays on disk verbatim.
diff --git a/.changeset/paste-table-cell-integrity.md b/.changeset/paste-table-cell-integrity.md
new file mode 100644
index 000000000..982c8b5c0
--- /dev/null
+++ b/.changeset/paste-table-cell-integrity.md
@@ -0,0 +1,5 @@
+---
+"@inkeep/open-knowledge-core": patch
+---
+
+Pasted HTML tables with block content in a cell (a list, multiple paragraphs, a code block) no longer corrupt the document. The paste conversion now normalizes cells the same way the editor's own serializer does: block structure degrades to `<br />`-joined single-line rows, so the stored markdown stays a valid GFM table instead of fracturing on the next parse. Plain `<br>` line breaks inside pasted cells are preserved as hard breaks instead of collapsing to spaces.
diff --git a/.changeset/table-cell-line-breaks.md b/.changeset/table-cell-line-breaks.md
new file mode 100644
index 000000000..bbe12132a
--- /dev/null
+++ b/.changeset/table-cell-line-breaks.md
@@ -0,0 +1,9 @@
+---
+"@inkeep/open-knowledge": patch
+---
+
+Fix tables corrupting for collaborators when a cell holds multi-line content.
+
+Pressing Enter, Shift+Enter, or adding a list inside a table cell used to serialize raw newlines into the markdown row — the table stayed intact in your own editor but split into a truncated table plus stray bullet lines for everyone else re-opening the file (teammates, restarts, fresh checkouts), and adjacent lines could glue together with their boundary lost. Multi-line cell content now serializes to single-line GFM rows joined by `<br />`, the standard idiom GitHub renders as a line break: extra paragraphs become line breaks, list items keep a `- ` (or `1. `) marker on their own visual line, and nothing in the row ever emits a raw newline. The same guard covers Shift+Enter inside headings.
+
+In the other direction, `<br>`, `<br/>`, and `<br />` in markdown now parse as real line breaks in the editor instead of showing as literal angle-bracket text, and each spelling is preserved byte-for-byte on save. Backslash-escaped `\<br />` and `<br>` inside inline code stay literal, as authored.
diff --git a/packages/app/tests/integration/_fixtures/init-load-byte-stable-corpus.ts b/packages/app/tests/integration/_fixtures/init-load-byte-stable-corpus.ts
index 1cf24580b..7813f922b 100644
--- a/packages/app/tests/integration/_fixtures/init-load-byte-stable-corpus.ts
+++ b/packages/app/tests/integration/_fixtures/init-load-byte-stable-corpus.ts
@@ -397,6 +397,7 @@ export const CORPUS: CorpusEntry[] = [
'entity-ref-preservation',
'pua-sentinel-ranges-reserved',
'doc-start-thematic-break-to-asterisks',
+ 'lazy-continuation-canonicalization',
],
description:
'Mega-combo: 8 byte-unsafe constructs in one .md doc — doc-start thematic, multi-blank, GFM table, math+footnote, alert, PUA, backslash, HTML entity. e2e tier target.',
@@ -452,6 +453,21 @@ export const CORPUS: CorpusEntry[] = [
'PUA: ' +
NL +
NL +
+ // lazy-continuation-canonicalization — a list item whose wrapped line
+ // continues without indent, and a blockquote continuation without the
+ // `> ` prefix (CommonMark 5.2). Both rest byte-divergent from canonical
+ // and must survive load verbatim (health checks tolerate via
+ // parse-equivalence; the byte comparator keeps them residual-bearing).
+ '- lazy list item whose wrapped line' +
+ NL +
+ 'continues without indent.' +
+ NL +
+ NL +
+ '> lazy quote line' +
+ NL +
+ 'continues without the prefix.' +
+ NL +
+ NL +
'Tail.' +
NL,
},
diff --git a/packages/app/tests/integration/c16-lazy-continuation.test.ts b/packages/app/tests/integration/c16-lazy-continuation.test.ts
new file mode 100644
index 000000000..2f7b1c392
--- /dev/null
+++ b/packages/app/tests/integration/c16-lazy-continuation.test.ts
@@ -0,0 +1,168 @@
+/**
+ * C16: CommonMark lazy-continuation docs — multi-client convergence without
+ * bridge health-check churn.
+ *
+ * A doc whose source carries a lazy continuation (an unindented wrapped
+ * line inside a list item) rests byte-divergent beyond every normalizeBridge
+ * class by design; the fragment still IS `parse(ytext)` (precedent #38).
+ * Multi-client coverage per the observer-bridge rule: single-client rigs
+ * miss remote-peer divergence, and this changeset touches
+ * `server-observers.ts`.
+ *
+ * Pins, through the real client → server → client collab pipeline:
+ * - a WYSIWYG edit on a resting lazy-continuation doc converges on every
+ * peer with the authored bytes preserved verbatim (no sanitize, no
+ * canonical rewrite) and no `bridge-split-brain-rederive` emission for
+ * the doc;
+ * - the harness bridge-invariant watcher's parse-equivalence fallback
+ * does NOT swallow genuine divergence — a paired-origin transaction
+ * that moves Y.Text without the fragment still throws.
+ *
+ * Per-test docName isolation via createTestClients(port, { count }) default.
+ * Client lifecycle in try/finally (not afterEach) per R8a.
+ */
+
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import { setTimeout as wait } from 'node:timers/promises';
+import { BridgeInvariantViolationError } from '@inkeep/open-knowledge-core';
+import * as Y from 'yjs';
+import { HARNESS_BOOT_TIMEOUT_MS } from './harness-boot-timeout';
+import {
+ assertBridgeInvariant,
+ createTestClients,
+ createTestServer,
+ pollUntil,
+ serializeFragment,
+ type TestClient,
+ type TestServer,
+} from './test-harness';
+
+let server: TestServer;
+
+beforeAll(async () => {
+ server = await createTestServer();
+}, HARNESS_BOOT_TIMEOUT_MS);
+
+afterAll(async () => {
+ await server.cleanup();
+});
+
+const LAZY_SOURCE =
+ '- Also read at session start: operating rules,\ncontinues without indent on the next line.\n\nBody text stays.\n';
+const PRESERVED_SLICE = 'operating rules,\ncontinues without indent';
+
+/** Append a paragraph with the given text to a client's XmlFragment. */
+function appendParagraph(client: TestClient, text: string): void {
+ const paragraph = new Y.XmlElement('paragraph');
+ const ytext = new Y.XmlText();
+ ytext.applyDelta([{ insert: text }]);
+ paragraph.insert(0, [ytext]);
+ client.fragment.push([paragraph]);
+}
+
+/** Capture structured console.warn events for one docName while `fn` runs
+ * and the post-window settles. Filtering by docName keeps the capture safe
+ * under parallel test files sharing the process console. */
+async function captureDocEvents(
+ docName: string,
+ eventName: string,
+ fn: () => Promise<void>,
+): Promise<Record<string, unknown>[]> {
+ const originalWarn = console.warn;
+ const lines: string[] = [];
+ console.warn = (...args: unknown[]) => {
+ lines.push(args.map(String).join(' '));
+ originalWarn(...args);
+ };
+ try {
+ await fn();
+ } finally {
+ console.warn = originalWarn;
+ }
+ return lines
+ .map((l) => {
+ try {
+ return JSON.parse(l) as Record<string, unknown>;
+ } catch {
+ return null;
+ }
+ })
+ .filter((e): e is Record<string, unknown> => e !== null)
+ .filter((e) => e.event === eventName && e['doc.name'] === docName);
+}
+
+describe('C16: lazy-continuation docs across clients', () => {
+ test('WYSIWYG edit on a resting lazy-continuation doc converges on both peers without rederive churn', async () => {
+ const clients = await createTestClients(server.port, { count: 2 });
+ try {
+ // Client A authors the lazy continuation through the source surface
+ // (W2) — the organic path that leaves Y.Text beyond byte tolerance
+ // while the server derives the fragment from parse(ytext).
+ clients[0].doc.transact(() => {
+ clients[0].ytext.insert(0, LAZY_SOURCE);
+ });
+ await pollUntil(() => clients[1].ytext.toString().includes('Body text stays.'), 5000);
+ await wait(500);
+
+ // Client B edits WYSIWYG-side while the doc rests beyond tolerance.
+ const events = await captureDocEvents(
+ clients[0].docName,
+ 'bridge-split-brain-rederive',
+ async () => {
+ appendParagraph(clients[1], 'C16-CLIENT-B-WYSIWYG');
+ for (const c of clients) {
+ await pollUntil(() => c.ytext.toString().includes('C16-CLIENT-B-WYSIWYG'), 5000);
+ }
+ await wait(500);
+ },
+ );
+ expect(events).toHaveLength(0);
+
+ // Authored source form survives verbatim on every peer alongside the
+ // new content; peers agree byte-for-byte.
+ const ytexts = clients.map((c) => c.ytext.toString());
+ for (const t of ytexts) {
+ expect(t).toContain(PRESERVED_SLICE);
+ expect(t).toContain('C16-CLIENT-B-WYSIWYG');
+ }
+ expect(ytexts[1]).toBe(ytexts[0]);
+ for (const c of clients) {
+ expect(serializeFragment(c.fragment)).toContain('C16-CLIENT-B-WYSIWYG');
+ assertBridgeInvariant(c.ytext, c.fragment);
+ }
+ } finally {
+ for (const c of clients) {
+ await c.cleanup();
+ }
+ }
+ });
+
+ test('harness watcher fallback does not swallow genuine divergence — one-sided paired write still throws', async () => {
+ const clients = await createTestClients(server.port, { count: 1 });
+ try {
+ clients[0].doc.transact(() => {
+ clients[0].ytext.insert(0, 'Settled paragraph.\n');
+ });
+ await pollUntil(
+ () => serializeFragment(clients[0].fragment).includes('Settled paragraph.'),
+ 5000,
+ );
+ await wait(500);
+
+ // A paired-marked origin promises "both CRDTs mutated atomically";
+ // moving ONLY Y.Text under it is a genuine fragment↔Y.Text divergence
+ // (content the fragment lacks), which no parse-equivalence fallback
+ // may bridge — the watcher must keep throwing on the drain.
+ const onesidedPairedOrigin = { context: { paired: true } };
+ expect(() => {
+ clients[0].doc.transact(() => {
+ clients[0].ytext.insert(0, 'Smuggled one-sided content.\n');
+ }, onesidedPairedOrigin);
+ }).toThrow(BridgeInvariantViolationError);
+ } finally {
+ for (const c of clients) {
+ await c.cleanup();
+ }
+ }
+ });
+});
diff --git a/packages/app/tests/integration/test-harness.ts b/packages/app/tests/integration/test-harness.ts
index d4aa261bd..014f75156 100644
--- a/packages/app/tests/integration/test-harness.ts
+++ b/packages/app/tests/integration/test-harness.ts
@@ -32,6 +32,7 @@ import {
type BridgeInvariantViolation,
BridgeInvariantViolationError,
type InvariantViolation,
+ isParseEquivalentBridge,
MarkdownManager,
normalizeBridge,
prependFrontmatter,
@@ -669,6 +670,13 @@ export function serializeFragment(fragment: Y.XmlFragment): string {
return mdManager.serialize(yXmlFragmentToProseMirrorRootNode(fragment, schema).toJSON());
}
+/** Body canonicalizer for the parse-equivalence fallback — the harness's
+ * own parse pipeline (no embed resolution: harness docs resolve none; an
+ * embed-bearing doc degrades toward alerting, never masking). */
+function canonicalizeBodyForHarness(body: string): string {
+ return mdManager.serialize(mdManager.parseWithFallback(body));
+}
+
/** Strip trailing whitespace per line + trailing newlines */
export function stripTrailingWhitespace(s: string): string {
return s
@@ -678,13 +686,21 @@ export function stripTrailingWhitespace(s: string): string {
.replace(/\n+$/, '');
}
-/** Assert bridge invariant: normalized Y.Text === serialized XmlFragment.
+/** Assert bridge invariant: normalized Y.Text === serialized XmlFragment,
+ * with the parse-equivalence fallback for byte forms beyond every
+ * normalizeBridge class whose parse matches the fragment (CommonMark lazy
+ * continuations et al. — fragment ≡ parse(ytext) holds, precedent #38).
* Normalization includes: blank-line count between blocks may normalize
* (ProseMirror schema limitation). Collapse 3+ consecutive newlines to 2. */
export function assertBridgeInvariant(ytext: Y.Text, fragment: Y.XmlFragment): void {
- const textNorm = normalizeBridge(ytext.toString());
- const fragNorm = normalizeBridge(serializeFragment(fragment));
- if (textNorm !== fragNorm) {
+ const ytextStr = ytext.toString();
+ const fragMd = serializeFragment(fragment);
+ const textNorm = normalizeBridge(ytextStr);
+ const fragNorm = normalizeBridge(fragMd);
+ if (
+ textNorm !== fragNorm &&
+ !isParseEquivalentBridge(ytextStr, fragMd, canonicalizeBodyForHarness)
+ ) {
throw new Error(
`Bridge invariant violated.\n` +
` Y.Text (${textNorm.length} chars): ${textNorm.slice(0, 200)}...\n` +
@@ -1149,6 +1165,8 @@ const BRIDGE_ENFORCING_NON_PAIRED_ORIGINS: Set<LocalTransactionOrigin> = new Set
* After every drain that contains AT LEAST ONE transaction with an enforcing
* origin, asserts:
* normalizeBridge(ytext) === normalizeBridge(prependFrontmatter(fm, serialize(fragment)))
+ * with the parse-equivalence fallback for beyond-tolerance byte forms whose
+ * parse matches the fragment (mirrors the server watchdog's canonicalizeBody).
*
* where `normalizeBridge` carries the extended tolerance set (CRLF/BOM/
* doc-start/leading-newline/trailing-newline/per-line trailing whitespace/
@@ -1201,6 +1219,10 @@ export function attachBridgeInvariantWatcher(
const fragNorm = normalizeBridge(fragMd);
if (ytextNorm === fragNorm) return;
+ // Parse-equivalence fallback — mirrors the server watchdog: a byte form
+ // beyond every normalizeBridge class whose parse matches the fragment is
+ // a resting serializer canonicalization, not a violation.
+ if (isParseEquivalentBridge(ytextStr, fragMd, canonicalizeBodyForHarness)) return;
const info: InvariantViolation = {
site: 'test-harness',
diff --git a/packages/app/tests/stress/paste-fidelity.e2e.ts b/packages/app/tests/stress/paste-fidelity.e2e.ts
index 2b58eb5a5..1aa066111 100644
--- a/packages/app/tests/stress/paste-fidelity.e2e.ts
+++ b/packages/app/tests/stress/paste-fidelity.e2e.ts
@@ -335,6 +335,24 @@ test.describe('Paste from vendor HTML → structured content through Branch D',
expect(content).toContain('bold');
}).toPass({ timeout: 5_000 });
});
+
+ test('table with block-content cell pastes as an intact single-line row', async ({ page }) => {
+ // A pasted <td> carrying a list must degrade to one physical GFM row
+ // (breaks as <br />), never emit a raw newline inside the row — a raw
+ // newline fractures the table on every fresh parse of the stored bytes.
+ await pasteWithMimes(page, {
+ 'text/plain': 'fallback text',
+ 'text/html':
+ '<table><tr><th>h</th></tr><tr><td><ul><li>one</li><li>two</li></ul></td></tr></table>',
+ });
+ await expect(async () => {
+ const content = await getYText(page);
+ // Both list items live on the same physical row line.
+ expect(content).toMatch(/\|[^\n]*one[^\n]*two[^\n]*\|/);
+ // No fracture: nothing escaped the table as a top-level block.
+ expect(content).not.toMatch(/^[*-] two/m);
+ }).toPass({ timeout: 5_000 });
+ });
});
// ─── FR-specific WYSIWYG scenarios ───
diff --git a/packages/core/schema-snapshot.json b/packages/core/schema-snapshot.json
index ff2962da2..10d862c45 100644
--- a/packages/core/schema-snapshot.json
+++ b/packages/core/schema-snapshot.json
@@ -141,6 +141,10 @@
"hardBreakStyle": {
"hasDefault": true,
"default": "backslash"
+ },
+ "sourceRaw": {
+ "hasDefault": true,
+ "default": null
}
},
"content": "",
@@ -871,6 +875,11 @@
"version": null,
"hasOptions": false
},
+ {
+ "name": "void-br-promoter",
+ "version": null,
+ "hasOptions": false
+ },
{
"name": "restore-entity-refs",
"version": null,
diff --git a/packages/core/src/bridge/index.ts b/packages/core/src/bridge/index.ts
index 654e99a6f..1606e7531 100644
--- a/packages/core/src/bridge/index.ts
+++ b/packages/core/src/bridge/index.ts
@@ -66,6 +66,11 @@ export {
detectAppliedToleranceClasses,
normalizeBridge,
} from './normalize.ts';
+export {
+ type BridgeToleranceSignal,
+ isParseEquivalentBridge,
+ PARSE_EQUIVALENCE_TOLERANCE,
+} from './parse-equivalence.ts';
export { defaultScheduler, type Scheduler } from './scheduler.ts';
export {
classifySeverity,
diff --git a/packages/core/src/bridge/parse-equivalence.ts b/packages/core/src/bridge/parse-equivalence.ts
new file mode 100644
index 000000000..27ff79a38
--- /dev/null
+++ b/packages/core/src/bridge/parse-equivalence.ts
@@ -0,0 +1,76 @@
+import { stripFrontmatter } from '../extensions/frontmatter.ts';
+import type { BridgeToleranceClass } from './normalize.ts';
+
+export const PARSE_EQUIVALENCE_TOLERANCE = 'parse-equivalence' as const;
+
+export type BridgeToleranceSignal = BridgeToleranceClass | typeof PARSE_EQUIVALENCE_TOLERANCE;
+
+function stripDocBoundary(body: string): string {
+ return body.replace(/^\n+/, '').replace(/\n+$/, '');
+}
+
+/** Leading ordered-list marker digits (after optional indent), canonicalized
+ * before skeletonizing so remark-stringify's lazy-list renumbering
+ * (`1.`/`1.` → `1.`/`2.`) doesn't read as a content substitution in the
+ * preservation probe. Only the digits collapse; delimiter + content stay. */
+const ORDERED_MARKER_DIGITS_RE = /^([ \t]*)\d+([.)])(?=[ \t])/;
+
+function contentSkeleton(body: string): string {
+ return body
+ .split('\n')
+ .map((line) =>
+ line.replace(ORDERED_MARKER_DIGITS_RE, (_m, indent, delim) => `${indent}1${delim}`),
+ )
+ .join('')
+ .replace(/\s+/g, '');
+}
+
+/** Breadcrumb gate for canonicalizeBody throws: warn once per DISTINCT
+ * error message (bounded set) so a systematic parser regression arriving
+ * after an unrelated exotic-doc throw still surfaces its own signature,
+ * without any per-drain flood. */
+const warnedCanonicalizeErrors = new Set<string>();
+const MAX_WARNED_CANONICALIZE_ERRORS = 8;
+
+/** True when `needle` appears in `haystack` as a subsequence (two-pointer,
+ * O(n+m)). Insertions in `haystack` are free; any dropped or substituted
+ * `needle` byte fails. */
+function isSubsequence(needle: string, haystack: string): boolean {
+ let i = 0;
+ for (let j = 0; j < haystack.length && i < needle.length; j++) {
+ if (needle[i] === haystack[j]) i++;
+ }
+ return i === needle.length;
+}
+
+export function isParseEquivalentBridge(
+ left: string,
+ right: string,
+ canonicalizeBody: (body: string) => string,
+): boolean {
+ const leftSplit = stripFrontmatter(left);
+ const rightSplit = stripFrontmatter(right);
+ if (leftSplit.frontmatter !== rightSplit.frontmatter) return false;
+ if (leftSplit.body === rightSplit.body) return true;
+ let canonicalLeftBody: string;
+ try {
+ canonicalLeftBody = canonicalizeBody(leftSplit.body);
+ } catch (err) {
+ const message = err instanceof Error ? err.message : String(err);
+ if (
+ warnedCanonicalizeErrors.size < MAX_WARNED_CANONICALIZE_ERRORS &&
+ !warnedCanonicalizeErrors.has(message)
+ ) {
+ warnedCanonicalizeErrors.add(message);
+ console.warn(
+ '[parse-equivalence] canonicalizeBody threw; treating as not equivalent:',
+ message,
+ );
+ }
+ return false;
+ }
+ if (stripDocBoundary(canonicalLeftBody) !== stripDocBoundary(rightSplit.body)) {
+ return false;
+ }
+ return isSubsequence(contentSkeleton(leftSplit.body), contentSkeleton(canonicalLeftBody));
+}
diff --git a/packages/core/src/bridge/tolerance-telemetry.ts b/packages/core/src/bridge/tolerance-telemetry.ts
index 035040532..c42403fd1 100644
--- a/packages/core/src/bridge/tolerance-telemetry.ts
+++ b/packages/core/src/bridge/tolerance-telemetry.ts
@@ -1,10 +1,10 @@
-import type { BridgeToleranceClass } from './normalize.ts';
+import type { BridgeToleranceSignal } from './parse-equivalence.ts';
export type ToleranceClassSeverity = 'pm-model-caused' | 'parser-caused' | 'serializer-caused';
export interface ToleranceFireRecord {
timestamp: string;
- className: BridgeToleranceClass;
+ className: BridgeToleranceSignal;
documentName: string | undefined;
codeUnitPosition: number;
severity: ToleranceClassSeverity;
@@ -29,9 +29,10 @@ const SEVERITY_BY_CLASS = {
'trailing-whitespace': 'serializer-caused',
'blank-line-collapse': 'serializer-caused',
'trailing-newline': 'serializer-caused',
-} as const satisfies Record<BridgeToleranceClass, ToleranceClassSeverity>;
+ 'parse-equivalence': 'parser-caused',
+} as const satisfies Record<BridgeToleranceSignal, ToleranceClassSeverity>;
-export function classifySeverity(cls: BridgeToleranceClass): ToleranceClassSeverity {
+export function classifySeverity(cls: BridgeToleranceSignal): ToleranceClassSeverity {
return SEVERITY_BY_CLASS[cls];
}
@@ -51,7 +52,7 @@ export function setToleranceTelemetryHook(h: ToleranceTelemetryHook | null): voi
}
export function emitToleranceFire(
- classes: readonly BridgeToleranceClass[],
+ classes: readonly BridgeToleranceSignal[],
left: string,
right: string,
documentName: string | undefined,
diff --git a/packages/core/src/extensions/hard-break-fidelity.ts b/packages/core/src/extensions/hard-break-fidelity.ts
index ebc121194..b58ec1f42 100644
--- a/packages/core/src/extensions/hard-break-fidelity.ts
+++ b/packages/core/src/extensions/hard-break-fidelity.ts
@@ -17,6 +17,9 @@ export const HardBreakFidelity = HardBreak.extend({
return {
...this.parent?.(),
hardBreakStyle: { default: 'backslash' },
+ // Void-HTML-authored breaks (`<br>` / `<br/>` / `<br />`) carry their
+ // exact source spelling so serialization re-emits it byte-identically.
+ sourceRaw: { default: null },
};
},
});
diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts
index 89eb77e85..9570ebbef 100644
--- a/packages/core/src/index.ts
+++ b/packages/core/src/index.ts
@@ -962,6 +962,7 @@ export {
type BridgeMergeContentLossSide,
type BridgeMergeContentLossWhich,
type BridgeToleranceClass,
+ type BridgeToleranceSignal,
bindFrontmatterDoc,
classifySeverity,
type DiffChange,
@@ -989,9 +990,11 @@ export {
findFirstDivergenceIndex,
fnv1aDigest,
type InvariantViolation,
+ isParseEquivalentBridge,
MAX_FM_REGION_BYTES,
mergeThreeWay,
normalizeBridge,
+ PARSE_EQUIVALENCE_TOLERANCE,
type ParsedFmRegion,
parseFencedFmRegion,
parseFmRegion,
diff --git a/packages/core/src/markdown/html-to-mdast.ts b/packages/core/src/markdown/html-to-mdast.ts
index 22a73aa64..78fdfeb02 100644
--- a/packages/core/src/markdown/html-to-mdast.ts
+++ b/packages/core/src/markdown/html-to-mdast.ts
@@ -15,6 +15,7 @@ import { rehypeStripGsheetsWrapper } from './rehype-plugins/strip-gsheets-wrappe
import { rehypeStripMsoStyles } from './rehype-plugins/strip-mso-styles.ts';
import { rehypeStripSlackClasses } from './rehype-plugins/strip-slack-classes.ts';
import { rehypeStripVscodeSpans } from './rehype-plugins/strip-vscode-spans.ts';
+import { flattenTableCellsInTree } from './table-cell-flatten.ts';
interface HtmlToMdastOptions {
additionalCleanupPlugins?: Plugin[];
@@ -92,5 +93,6 @@ export function htmlToMdast(html: string, options?: HtmlToMdastOptions): MdastRo
}
export function mdastToMarkdown(tree: MdastRoot): string {
+ flattenTableCellsInTree(tree);
return String(unified().use(remarkGfm).use(remarkStringify).stringify(tree));
}
diff --git a/packages/core/src/markdown/index.ts b/packages/core/src/markdown/index.ts
index 1b13478a8..58a810193 100644
--- a/packages/core/src/markdown/index.ts
+++ b/packages/core/src/markdown/index.ts
@@ -67,6 +67,7 @@ import {
serializeMd,
} from './pipeline.ts';
import { normalizeDocRelativeAssetUrl } from './resolve-image-url.ts';
+import { flattenCellBlocks } from './table-cell-flatten.ts';
import { toMarkdownHandlers } from './to-markdown-handlers.ts';
import {
decodeInlineWhitespaceNumericCharRefRun,
@@ -601,6 +602,7 @@ function buildMdastToPmHandlers(
handlers.break = (node: Break) =>
n.hardBreak.createAndFill({
hardBreakStyle: node.data?.sourceStyle ?? 'spaces',
+ sourceRaw: node.data?.sourceRaw ?? null,
});
}
@@ -1096,7 +1098,12 @@ function buildPmToMdastHandlers(schema: Schema): {
if (n.hardBreak) {
nodeHandlers.hardBreak = (pmNode: PmNode) => ({
type: 'break' as const,
- data: { sourceStyle: pmNode.attrs.hardBreakStyle },
+ data: {
+ sourceStyle: pmNode.attrs.hardBreakStyle,
+ ...(typeof pmNode.attrs.sourceRaw === 'string' && pmNode.attrs.sourceRaw.length > 0
+ ? { sourceRaw: pmNode.attrs.sourceRaw }
+ : {}),
+ },
});
}
@@ -1184,21 +1191,25 @@ function buildPmToMdastHandlers(schema: Schema): {
};
}
if (n.tableRow) nodeHandlers.tableRow = fromPmNode('tableRow');
- const cellToMdast = (pmNode: PmNode) => {
+ const buildCellNode = (pmNode: PmNode, children: MdastNodes[]): TableCell => {
+ const cell: TableCell = { type: 'tableCell', children: flattenCellBlocks(children) };
const padding = pmNode.attrs.sourcePadding;
- const data: { sourcePadding?: { left: number; right: number } } = {};
if (
padding !== null &&
typeof padding === 'object' &&
typeof (padding as { left?: unknown }).left === 'number' &&
typeof (padding as { right?: unknown }).right === 'number'
) {
- data.sourcePadding = padding as { left: number; right: number };
+ cell.data = { sourcePadding: padding as { left: number; right: number } };
}
- return Object.keys(data).length > 0 ? { data } : {};
+ return cell;
};
- if (n.tableCell) nodeHandlers.tableCell = fromPmNode('tableCell', cellToMdast);
- if (n.tableHeader) nodeHandlers.tableHeader = fromPmNode('tableCell', cellToMdast);
+ if (n.tableCell)
+ nodeHandlers.tableCell = (pmNode: PmNode, _parent, state) =>
+ buildCellNode(pmNode, state.all(pmNode));
+ if (n.tableHeader)
+ nodeHandlers.tableHeader = (pmNode: PmNode, _parent, state) =>
+ buildCellNode(pmNode, state.all(pmNode));
if (n.image) {
nodeHandlers.image = (pmNode: PmNode) => ({
diff --git a/packages/core/src/markdown/mdast-augmentation.ts b/packages/core/src/markdown/mdast-augmentation.ts
index 757424675..e04f30337 100644
--- a/packages/core/src/markdown/mdast-augmentation.ts
+++ b/packages/core/src/markdown/mdast-augmentation.ts
@@ -139,6 +139,7 @@ declare module 'mdast' {
}
interface BreakData {
sourceStyle?: string;
+ sourceRaw?: string;
}
interface HeadingData {
sourceStyle?: string;
diff --git a/packages/core/src/markdown/pipeline.ts b/packages/core/src/markdown/pipeline.ts
index 922ebca2a..5d6db7aa8 100644
--- a/packages/core/src/markdown/pipeline.ts
+++ b/packages/core/src/markdown/pipeline.ts
@@ -34,6 +34,7 @@ import { positionAwareBlankLineJoin } from './position-aware-join.ts';
import { remarkMdxAgnostic } from './remark-mdx-agnostic.ts';
import { singleDollarMathPromoterPlugin } from './single-dollar-math-promoter.ts';
import { remarkTags } from './tag-to-markdown.ts';
+import { voidBrPromoterPlugin } from './void-br-promoter.ts';
import { remarkWikiLink } from './wiki-link-micromark.ts';
interface PipelineOptions {
@@ -78,6 +79,7 @@ export const ACTIVE_MDAST_PLUGINS = [
},
{ name: 'callout-transformer', plugin: calloutTransformerPlugin },
{ name: 'restore-from-mdx', plugin: restoreFromMdx },
+ { name: 'void-br-promoter', plugin: voidBrPromoterPlugin },
{ name: 'restore-entity-refs', plugin: restoreEntityRefsPlugin },
{ name: 'restore-backslash-escapes', plugin: restoreBackslashEscapesPlugin },
{ name: 'details-accordion-promoter', plugin: detailsAccordionPromoterPlugin },
diff --git a/packages/core/src/markdown/position-slice.ts b/packages/core/src/markdown/position-slice.ts
index 90a412726..50a193a16 100644
--- a/packages/core/src/markdown/position-slice.ts
+++ b/packages/core/src/markdown/position-slice.ts
@@ -528,6 +528,7 @@ export function applyPositionSliceToNode(
}
case 'break': {
+ if (node.data.sourceRaw) break;
const slice = source.slice(startOff, endOff);
if (slice.includes('\\')) {
node.data.sourceStyle = 'backslash';
diff --git a/packages/core/src/markdown/table-cell-flatten.ts b/packages/core/src/markdown/table-cell-flatten.ts
new file mode 100644
index 000000000..d2915bfc1
--- /dev/null
+++ b/packages/core/src/markdown/table-cell-flatten.ts
@@ -0,0 +1,142 @@
+import type { Html, List, Nodes, PhrasingContent, Root, TableCell } from 'mdast';
+import { visit } from 'unist-util-visit';
+
+const CELL_PHRASING_TYPES: ReadonlySet<string> = new Set([
+ 'break',
+ 'delete',
+ 'emphasis',
+ 'footnoteReference',
+ 'html',
+ 'image',
+ 'imageReference',
+ 'inlineCode',
+ 'link',
+ 'linkReference',
+ 'strong',
+ 'text',
+]);
+
+function substituteBreaksWithBrHtml(node: { children?: Nodes[] }): void {
+ const children = node.children;
+ if (!Array.isArray(children)) return;
+ children.forEach((child, i) => {
+ if (child.type === 'break') {
+ children[i] = { type: 'html', value: '<br />' } satisfies Html;
+ return;
+ }
+ substituteBreaksWithBrHtml(child as { children?: Nodes[] });
+ });
+}
+
+function normalizeCellValueNodes(node: { children?: Nodes[] }): void {
+ const children = node.children;
+ if (!Array.isArray(children)) return;
+ for (const child of children) {
+ if (child.type === 'inlineCode' || child.type === 'html') {
+ if (child.value.includes('\n')) {
+ child.value = child.value.replace(/[ \t]*\n[ \t]*/g, ' ');
+ }
+ if (child.type === 'html' && child.value.includes('|')) {
+ child.value = child.value.replace(/\|/g, '\\|');
+ }
+ }
+ normalizeCellValueNodes(child as { children?: Nodes[] });
+ }
+}
+
+export function flattenTableCellsInTree(tree: Root): void {
+ visit(tree, 'tableCell', (cell: TableCell) => {
+ const hasBlockChild = cell.children.some((child) => !CELL_PHRASING_TYPES.has(child.type));
+ if (hasBlockChild) {
+ cell.children = flattenCellBlocks(cell.children);
+ }
+ substituteBreaksWithBrHtml(cell);
+ normalizeCellValueNodes(cell);
+ });
+}
+
+export function flattenCellBlocks(children: readonly Nodes[]): PhrasingContent[] {
+ const out: PhrasingContent[] = [];
+ let pendingBreaks = 0;
+
+ const pushBreak = (): void => {
+ if (out.length > 0) pendingBreaks += 1;
+ };
+
+ const emit = (node: PhrasingContent): void => {
+ while (pendingBreaks > 0) {
+ out.push({ type: 'break' });
+ pendingBreaks -= 1;
+ }
+ out.push(node);
+ };
+
+ const pushLines = (value: string): void => {
+ value.split('\n').forEach((line, i) => {
+ if (i > 0) pushBreak();
+ if (line.length > 0) emit({ type: 'text', value: line });
+ });
+ };
+
+ const pushList = (list: List): void => {
+ const ordered = list.ordered === true;
+ const start = typeof list.start === 'number' ? list.start : 1;
+ const data = list.data as { bulletMarker?: string; listMarkerDelimiter?: string } | undefined;
+ const bullet = typeof data?.bulletMarker === 'string' ? data.bulletMarker : '-';
+ const delimiter =
+ typeof data?.listMarkerDelimiter === 'string' ? data.listMarkerDelimiter : '.';
+ list.children.forEach((item, i) => {
+ if (i > 0) pushBreak();
+ const marker = ordered ? `${start + i}${delimiter} ` : `${bullet} `;
+ emit({ type: 'text', value: marker });
+ item.children.forEach((block, j) => {
+ if (j > 0) pushBreak();
+ pushBlock(block);
+ });
+ });
+ };
+
+ const pushBlock = (node: Nodes): void => {
+ switch (node.type) {
+ case 'paragraph':
+ case 'heading':
+ node.children.forEach(emit);
+ return;
+ case 'list':
+ pushList(node);
+ return;
+ case 'code':
+ case 'html':
+ pushLines(node.value);
+ return;
+ case 'thematicBreak':
+ emit({ type: 'text', value: node.data?.sourceRaw ?? '---' });
+ return;
+ default: {
+ const withChildren = node as { children?: Nodes[] };
+ if (Array.isArray(withChildren.children)) {
+ withChildren.children.forEach((child, i) => {
+ if (i > 0) pushBreak();
+ pushBlock(child);
+ });
+ return;
+ }
+ const withValue = node as { value?: unknown };
+ if (typeof withValue.value === 'string') {
+ pushLines(withValue.value);
+ return;
+ }
+ console.warn(
+ JSON.stringify({ event: 'table-cell-flatten-dropped-block', nodeType: node.type }),
+ );
+ }
+ }
+ };
+
+ children.forEach((child, i) => {
+ if (i > 0) pushBreak();
+ pushBlock(child);
+ });
+
+ return out;
+}
diff --git a/packages/core/src/markdown/to-markdown-handlers.ts b/packages/core/src/markdown/to-markdown-handlers.ts
index f622bdd21..fc8d29987 100644
--- a/packages/core/src/markdown/to-markdown-handlers.ts
+++ b/packages/core/src/markdown/to-markdown-handlers.ts
@@ -11,6 +11,7 @@ declare module 'mdast-util-to-markdown' {
comment: 'comment';
commentBlock: 'commentBlock';
strikethrough: 'strikethrough';
+ okAtxHeading: 'okAtxHeading';
}
}
@@ -242,7 +243,12 @@ export const toMarkdownHandlers = {
return node.data?.sourceRaw ?? '---';
},
- break(node) {
+ break(node, _parent, state) {
+ const raw = node.data?.sourceRaw;
+ if (typeof raw === 'string' && raw.length > 0) return raw;
+ if (state.stack.includes('tableCell') || state.stack.includes('okAtxHeading')) {
+ return '<br />';
+ }
if (node.data?.sourceStyle === 'backslash') return '\\\n';
return ' \n';
},
@@ -345,11 +351,13 @@ export const toMarkdownHandlers = {
const interior =
typeof interiorCount === 'number' && interiorCount >= 2 ? ' '.repeat(interiorCount) : ' ';
const hashes = '#'.repeat(depth);
+ const atxExit = state.enter('okAtxHeading');
const content = state.containerPhrasing(node, {
...info,
before: `${hashes} `,
after: '\n',
});
+ atxExit();
const trailingCount = node.data?.sourceTrailingHashes;
if (typeof trailingCount === 'number' && trailingCount > 0) {
const trailing = '#'.repeat(trailingCount);
diff --git a/packages/core/src/markdown/void-br-promoter.ts b/packages/core/src/markdown/void-br-promoter.ts
new file mode 100644
index 000000000..a0aecd430
--- /dev/null
+++ b/packages/core/src/markdown/void-br-promoter.ts
@@ -0,0 +1,85 @@
+import type { Break, Parent, Root, Text } from 'mdast';
+import { SKIP, visit } from 'unist-util-visit';
+import type { VFile } from 'vfile';
+import { deriveFragmentPosition } from './promoter-position.ts';
+
+const VOID_BR_IN_TEXT_RE = /(?<!\uE102)<br[ \t]*\/?>/g;
+
+const SKIP_SUBTREE_TYPES = new Set(['mdxJsxTextElement', 'mdxJsxFlowElement']);
+
+function promoteVoidBrInParent(parent: Parent, source: string): void {
+ const newChildren: Parent['children'] = [];
+ let changed = false;
+
+ for (const child of parent.children) {
+ if (child.type !== 'text') {
+ newChildren.push(child);
+ continue;
+ }
+
+ const text = (child as Text).value;
+ VOID_BR_IN_TEXT_RE.lastIndex = 0;
+
+ const segments: Parent['children'] = [];
+ let lastIndex = 0;
+
+ for (;;) {
+ const match = VOID_BR_IN_TEXT_RE.exec(text);
+ if (match === null) break;
+ const matchStart = match.index;
+
+ if (matchStart > lastIndex) {
+ const lead: Text = { type: 'text', value: text.slice(lastIndex, matchStart) };
+ const pos = deriveFragmentPosition(source, child as Text, lastIndex, matchStart);
+ if (pos) lead.position = pos;
+ segments.push(lead);
+ }
+
+ const brNode: Break = {
+ type: 'break',
+ data: { sourceStyle: 'html', sourceRaw: match[0] },
+ };
+ const brPos = deriveFragmentPosition(
+ source,
+ child as Text,
+ matchStart,
+ matchStart + match[0].length,
+ );
+ if (brPos) brNode.position = brPos;
+ segments.push(brNode);
+
+ lastIndex = matchStart + match[0].length;
+ changed = true;
+ }
+
+ if (segments.length === 0) {
+ newChildren.push(child);
+ } else {
+ if (lastIndex < text.length) {
+ const tail: Text = { type: 'text', value: text.slice(lastIndex) };
+ const pos = deriveFragmentPosition(source, child as Text, lastIndex, text.length);
+ if (pos) tail.position = pos;
+ segments.push(tail);
+ }
+ newChildren.push(...segments);
+ }
+ }
+
+ if (changed) {
+ parent.children = newChildren;
+ }
+}
+
+export function voidBrPromoterPlugin() {
+ return (tree: Root, file: VFile) => {
+ const source = typeof file.value === 'string' ? file.value : '';
+ visit(tree, (node) => {
+ if (SKIP_SUBTREE_TYPES.has(node.type)) return SKIP;
+ const maybeParent = node as Partial<Parent>;
+ if (Array.isArray(maybeParent.children)) {
+ promoteVoidBrInParent(node as Parent, source);
+ }
+ return undefined;
+ });
+ };
+}
diff --git a/packages/core/src/schema-snapshot.json b/packages/core/src/schema-snapshot.json
index 4c0619bbe..8b6ff0b22 100644
--- a/packages/core/src/schema-snapshot.json
+++ b/packages/core/src/schema-snapshot.json
@@ -332,6 +332,9 @@
"attrs": {
"hardBreakStyle": {
"hasDefault": true
+ },
+ "sourceRaw": {
+ "hasDefault": true
}
},
"content": "",
diff --git a/packages/server/src/bridge-watchdog.test.ts b/packages/server/src/bridge-watchdog.test.ts
index 0eebd9c40..2be88f3f1 100644
--- a/packages/server/src/bridge-watchdog.test.ts
+++ b/packages/server/src/bridge-watchdog.test.ts
@@ -19,7 +19,10 @@
import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
import {
BridgeInvariantViolationError,
+ MarkdownManager,
+ normalizeBridge,
setToleranceTelemetryHook,
+ sharedExtensions,
type ToleranceFireRecord,
} from '@inkeep/open-knowledge-core';
import {
@@ -1063,3 +1066,95 @@ describe('assertBridgeInvariant — return value reflects normalize-equality', (
}
});
});
+
+describe('assertBridgeInvariant — parse-equivalence fallback (canonicalizeBody opt)', () => {
+ // CommonMark lazy continuations serialize non-identically beyond the
+ // normalizeBridge tolerance set (step 7f keeps them divergent for the
+ // router's byte-preserving residual merge), yet parse identically — the
+ // fragment IS parse(ytext), so the health check must not fire. The
+ // `canonicalizeBody` opt lets a call site holding a MarkdownManager
+ // supply the authoritative equivalence: canonicalize(left body) equals
+ // the (already-canonical) right body ⇒ tolerated, not a violation.
+ const mgr = new MarkdownManager({ extensions: sharedExtensions });
+ const canonicalizeBody = (body: string): string => mgr.serialize(mgr.parseWithFallback(body));
+
+ const LAZY_RAW = '- item one continues here,\nlazily on the next line.\n';
+
+ test('normalize-divergent but parse-equivalent inputs are tolerated (returns true, no throw)', () => {
+ const canonical = canonicalizeBody(LAZY_RAW);
+ // Class trigger: byte-divergent AND beyond normalizeBridge tolerance.
+ expect(canonical).not.toBe(LAZY_RAW);
+ expect(normalizeBridge(canonical)).not.toBe(normalizeBridge(LAZY_RAW));
+
+ const result = assertBridgeInvariant(LAZY_RAW, canonical, {
+ site: 'persistence',
+ docName: 'lazy-doc',
+ canonicalizeBody,
+ });
+ expect(result).toBe(true);
+ expect(getMetrics().bridgeInvariantViolations).toBe(0);
+ expect(getMetrics().bridgeInvariantViolationsSuppressed).toBe(0);
+ });
+
+ test('tolerated parse-equivalent pair emits bridge-tolerance-applied with the parse-equivalence class', () => {
+ const canonical = canonicalizeBody(LAZY_RAW);
+ const originalWarn = console.warn;
+ const warnings: string[] = [];
+ console.warn = (...args: unknown[]) => {
+ warnings.push(args.map(String).join(' '));
+ };
+ try {
+ assertBridgeInvariant(LAZY_RAW, canonical, {
+ site: 'persistence',
+ docName: 'lazy-doc',
+ canonicalizeBody,
+ });
+ } finally {
+ console.warn = originalWarn;
+ }
+ const toleranceEvents = warnings
+ .map((w) => {
+ try {
+ return JSON.parse(w) as Record<string, unknown>;
+ } catch {
+ return null;
+ }
+ })
+ .filter((e): e is Record<string, unknown> => e?.event === 'bridge-tolerance-applied');
+ expect(toleranceEvents.map((e) => e.class)).toContain('parse-equivalence');
+ });
+
+ test('genuinely divergent inputs still throw with canonicalizeBody provided', () => {
+ // Right side lacks content the left carries — canonicalizing the left
+ // cannot bridge that gap, so the loud-fail contract must hold.
+ expect(() => {
+ assertBridgeInvariant(LAZY_RAW, '# Completely different fragment\n', {
+ site: 'observer-b',
+ docName: 'diverged-doc',
+ canonicalizeBody,
+ });
+ }).toThrow(BridgeInvariantViolationError);
+ });
+
+ test('frontmatter divergence is not bridged by body parse-equivalence', () => {
+ const left = `---\ntitle: A\n---\n\n${LAZY_RAW}`;
+ const right = `---\ntitle: B\n---\n\n${canonicalizeBody(LAZY_RAW)}`;
+ expect(() => {
+ assertBridgeInvariant(left, right, {
+ site: 'observer-b',
+ docName: 'fm-diverged-doc',
+ canonicalizeBody,
+ });
+ }).toThrow(BridgeInvariantViolationError);
+ });
+
+ test('without canonicalizeBody the strict normalize-only behavior is preserved', () => {
+ const canonical = canonicalizeBody(LAZY_RAW);
+ expect(() => {
+ assertBridgeInvariant(LAZY_RAW, canonical, {
+ site: 'observer-b',
+ docName: 'lazy-doc-strict',
+ });
+ }).toThrow(BridgeInvariantViolationError);
+ });
+});
diff --git a/packages/server/src/bridge-watchdog.ts b/packages/server/src/bridge-watchdog.ts
index 5820a3423..729880f39 100644
--- a/packages/server/src/bridge-watchdog.ts
+++ b/packages/server/src/bridge-watchdog.ts
@@ -28,8 +28,9 @@
*
* Telemetry payload is bounded-cardinality and content-redacted by default:
* site, docName-or-null, the tolerance-class label (`'untracked'` for
- * unknown classes — the comparator tolerates known classes, so a violation
- * outside tolerance is by definition untracked), and FNV-1a digests of the
+ * unknown classes — the comparator stack tolerates known byte classes plus
+ * the parse-equivalence fallback, so a violation past ALL of them is by
+ * definition untracked), and FNV-1a digests of the
* ytext + fragment snapshots for cross-event correlation. The truncated
* unifiedDiff is included as `diff` ONLY when `OK_TELEMETRY_VERBOSE=1`
* (mirrors the sibling `bridge-merge-content-loss` opt-in pattern). Full
@@ -39,14 +40,17 @@
* @see packages/core/src/bridge/bridge-invariant.ts (error type)
*/
+import type { MarkdownManager } from '@inkeep/open-knowledge-core';
import {
type BridgeInvariantSite,
type BridgeInvariantViolation,
BridgeInvariantViolationError,
- type BridgeToleranceClass,
+ type BridgeToleranceSignal,
detectAppliedToleranceClasses,
emitToleranceFire,
+ isParseEquivalentBridge,
normalizeBridge,
+ PARSE_EQUIVALENCE_TOLERANCE,
toBridgeInvariantLog,
} from '@inkeep/open-knowledge-core';
import {
@@ -111,12 +115,13 @@ const lastEmitMs = new Map<string, number>();
const MAX_VIOLATION_RATE_TUPLES = 1024;
/** Map<rateKey, last-emit-Unix-ms> for the bridge-tolerance-applied event.
- * rateKey = `${site}::${class}`. Bounded cardinality: 16 classes × 3 sites =
- * 48 entries max globally. Per-(site, class) windows let operators see how
+ * rateKey = `${site}::${class}`. Bounded cardinality: 17 signals (16
+ * normalizeBridge classes + the parse-equivalence fallback) × 3 sites =
+ * 51 entries max globally. Per-(site, class) windows let operators see how
* often each site relies on each tolerance class — observer-b CRLF rates
* vs persistence CRLF rates surface separately.
*
- * WARN: same module-level state caveat as `lastEmitMs` above. The 48-entry
+ * WARN: same module-level state caveat as `lastEmitMs` above. The 51-entry
* bound is global; under multi-server-per-process, a single server's
* tolerance event would suppress another server's same-class event in
* the same window. Less concerning than the violation rate-limiter
@@ -146,14 +151,14 @@ export type BridgeSplitBrainSite = 'identity-gate' | 'post-merge' | 'error-recov
/** Map<rateKey, last-emit-Unix-ms> for the bridge-split-brain-rederive
* event. rateKey = `${site}::${docName ?? '__nodoc__'}` — per-(site, doc)
- * so a chatty doc can't suppress signal from quieter docs and the two
- * detection sites surface independently. Bounded: 2 sites × docs, with
+ * so a chatty doc can't suppress signal from quieter docs and the three
+ * detection sites surface independently. Bounded: 3 sites × docs, with
* the same lazy prune as `lastEmitMs`.
*
* WARN: same module-level state caveat as `lastEmitMs` above. */
const lastSplitBrainEmitMs = new Map<string, number>();
-function toleranceRateKey(site: BridgeInvariantSite, cls: BridgeToleranceClass): string {
+function toleranceRateKey(site: BridgeInvariantSite, cls: BridgeToleranceSignal): string {
return `${site}::${cls}`;
}
@@ -225,13 +230,13 @@ export function shouldEmitBridgeInvariantViolation(
*
* Keying by (site, class) — not just class — so observer-b's CRLF reliance
* doesn't suppress persistence-site CRLF reliance within the same window.
- * Bounded: 16 classes × 3 sites = 48 entries.
+ * Bounded: 17 signals × 3 sites = 51 entries.
*
* Test seam: pass `nowMs` to deterministically advance the clock.
*/
export function shouldEmitBridgeToleranceApplied(
site: BridgeInvariantSite,
- toleranceClass: BridgeToleranceClass,
+ toleranceClass: BridgeToleranceSignal,
nowMs: number = Date.now(),
): boolean {
const key = toleranceRateKey(site, toleranceClass);
@@ -409,6 +414,43 @@ export function shouldThrowOnBridgeInvariantViolation(
return env.NODE_ENV === 'test' || env.OK_BRIDGE_THROW_ON_VIOLATION === '1';
}
+// ─────────────────────────────────────────────────────────────
+// Doc canonicalizer
+// ─────────────────────────────────────────────────────────────
+
+/** The per-doc parse surface a canonicalizer must replicate — the embed
+ * option pair every fragment derivation threads into `parseWithFallback`,
+ * keyed to the doc that owns the comparison. */
+type DocParseSurface = Pick<
+ NonNullable<Parameters<MarkdownManager['parseWithFallback']>[1]>,
+ 'resolveEmbed' | 'resolveSize'
+> & { docName?: string };
+
+/**
+ * Bind a body canonicalizer to a doc's own parse surface. Every
+ * parse-equivalence decision is only as good as the promise that the
+ * canonicalizer parses EXACTLY like the fragment derivation it is compared
+ * against — hand-replicating the option triple at each call site lets the
+ * copies drift apart silently, and a drifted canonicalizer can mask (or
+ * spuriously alert on) embed-bearing docs. Constructing the closure here
+ * makes the same-surface invariant hold by construction for every consumer.
+ */
+export function createDocCanonicalizer(
+ mdManager: MarkdownManager,
+ opts: DocParseSurface,
+): (body: string) => string {
+ const parseOpts =
+ opts.resolveEmbed && opts.docName
+ ? {
+ resolveEmbed: opts.resolveEmbed,
+ resolveSize: opts.resolveSize,
+ sourcePath: opts.docName,
+ }
+ : undefined;
+ return (body: string): string =>
+ mdManager.serialize(mdManager.parseWithFallback(body, parseOpts));
+}
+
// ─────────────────────────────────────────────────────────────
// Watchdog assertion
// ─────────────────────────────────────────────────────────────
@@ -442,6 +484,23 @@ interface AssertBridgeInvariantOpts {
* thrown errors. Observer B still throws by default.
*/
suppressDevThrow?: boolean;
+ /**
+ * Parse-equivalence fallback (`isParseEquivalentBridge`). When the inputs
+ * diverge beyond every `normalizeBridge` byte class, canonicalize the
+ * ytext body through the caller's own parse→serialize pipeline and accept
+ * the pair when the canonical forms match — the fragment then IS
+ * `parse(ytext)` (precedent #38), so a resting serializer canonicalization
+ * (CommonMark lazy continuations: an unindented wrapped list line, a
+ * paragraph glued under a list, a `> `-less blockquote continuation) is a
+ * tolerated equivalence, not a violation. Reported through the
+ * `bridge-tolerance-applied` channel as `parse-equivalence`.
+ *
+ * Callers MUST bind the same parse options the doc's fragment derivation
+ * uses (embed resolution, source path) — a mismatched pipeline degrades
+ * safely toward alerting, never masking. Omitting the callback preserves
+ * the strict normalize-only behavior.
+ */
+ canonicalizeBody?: (body: string) => string;
}
/**
@@ -475,42 +534,64 @@ export function assertBridgeInvariant(
fragmentMdSnapshot: string,
opts: AssertBridgeInvariantOpts,
): boolean {
+ // Tolerance-applied reporting, shared by both tolerated paths (normalize-
+ // equal and parse-equivalent). Two consumers, two policies. The metric
+ // counter + console.warn loop are noise/cardinality controls, so they
+ // share ONE rate-limit decision per (site, class). The JSONL file hook is
+ // evidence collection for the aggregator CLI: it needs every fire to
+ // measure frequency, and the RotatingAppender already bounds disk (~16MB
+ // across two generations), so it intentionally gets the full
+ // un-rate-limited class list. Bounded cardinality: 17 signals × 3 sites
+ // = 51 series globally; rate-limited per (site, class).
+ const reportTolerated = (classes: readonly BridgeToleranceSignal[]): void => {
+ const emittedClasses = classes.filter((cls) =>
+ shouldEmitBridgeToleranceApplied(opts.site, cls, opts.nowMs),
+ );
+ if (classes.length > 0) {
+ emitToleranceFire(classes, ytextSnapshot, fragmentMdSnapshot, opts.docName);
+ }
+ for (const cls of emittedClasses) {
+ incrementBridgeToleranceApplied(cls);
+ console.warn(
+ JSON.stringify({
+ event: 'bridge-tolerance-applied',
+ site: opts.site,
+ class: cls,
+ }),
+ );
+ }
+ };
+
const ytextNorm = normalizeBridge(ytextSnapshot);
const fragNorm = normalizeBridge(fragmentMdSnapshot);
if (ytextNorm === fragNorm) {
- // Tolerance-applied path: bytes are normalize-equal but may differ
- // pre-normalization. Emit one bridge-tolerance-applied event per
- // detected (site, class) tuple so operators can prioritize closing
- // tolerance gaps per-site. Bounded cardinality: 16 classes × 3 sites
- // = 48 series globally; rate-limited per (site, class).
+ // Bytes are normalize-equal but may differ pre-normalization — emit one
+ // bridge-tolerance-applied event per detected (site, class) tuple so
+ // operators can prioritize closing tolerance gaps per-site.
if (ytextSnapshot !== fragmentMdSnapshot) {
- const classes = detectAppliedToleranceClasses(ytextSnapshot, fragmentMdSnapshot);
- // Two consumers, two policies. The metric counter + console.warn loop
- // are noise/cardinality controls, so they share ONE rate-limit decision
- // per (site, class). The JSONL file hook is evidence collection for the
- // aggregator CLI: it needs every fire to measure frequency, and the
- // RotatingAppender already bounds disk (~16MB across two generations),
- // so it intentionally gets the full un-rate-limited class list.
- const emittedClasses = classes.filter((cls) =>
- shouldEmitBridgeToleranceApplied(opts.site, cls, opts.nowMs),
- );
- if (classes.length > 0) {
- emitToleranceFire(classes, ytextSnapshot, fragmentMdSnapshot, opts.docName);
- }
- for (const cls of emittedClasses) {
- incrementBridgeToleranceApplied(cls);
- console.warn(
- JSON.stringify({
- event: 'bridge-tolerance-applied',
- site: opts.site,
- class: cls,
- }),
- );
- }
+ reportTolerated(detectAppliedToleranceClasses(ytextSnapshot, fragmentMdSnapshot));
}
return true;
}
+ // Parse-equivalence fallback — runs only after every byte class failed,
+ // i.e. exactly the inputs that would otherwise alert. When the caller's
+ // canonicalize pipeline maps the ytext body onto the fragment side, the
+ // divergence is a resting serializer canonicalization of organic input
+ // (fragment ≡ parse(ytext) holds), not a broken bridge. The heuristic
+ // byte-class labels ride along for diagnostic color (e.g. an unindented
+ // wrapped list line also registers paragraph-continuation-indent).
+ if (
+ opts.canonicalizeBody &&
+ isParseEquivalentBridge(ytextSnapshot, fragmentMdSnapshot, opts.canonicalizeBody)
+ ) {
+ reportTolerated([
+ ...detectAppliedToleranceClasses(ytextSnapshot, fragmentMdSnapshot),
+ PARSE_EQUIVALENCE_TOLERANCE,
+ ]);
+ return true;
+ }
+
const violation: BridgeInvariantViolation = {
site: opts.site,
origin: opts.origin,
diff --git a/packages/server/src/metrics.ts b/packages/server/src/metrics.ts
index 179019c10..4431c0dcb 100644
--- a/packages/server/src/metrics.ts
+++ b/packages/server/src/metrics.ts
@@ -4,12 +4,12 @@
* Exposed via GET /api/metrics/reconciliation.
*/
-import type { BridgeToleranceClass, CC1Channel } from '@inkeep/open-knowledge-core';
+import type { BridgeToleranceSignal, CC1Channel } from '@inkeep/open-knowledge-core';
/**
* Why an Observer A drain could not be served by the map-driven splice.
* Closed union — keys a bounded-cardinality counter map, so call sites are
- * compile-time checked the same way `BridgeToleranceClass` keys are.
+ * compile-time checked the same way `BridgeToleranceSignal` keys are.
*/
export type MapDrivenSpliceFallbackReason =
| 'text-mismatch'
@@ -152,12 +152,13 @@ export interface ReconciliationMetrics {
* (architectural-floor cases like CRLF/leading-newline are normal); growth
* highlights which classes are most worth closing via fidelity attrs.
*
- * Keys are bounded to `BridgeToleranceClass` (the enumerated labels in
- * `BRIDGE_TOLERANCE_CLASSES`). Lazy keys: a class only appears once the
+ * Keys are bounded to `BridgeToleranceSignal` (the enumerated labels in
+ * `BRIDGE_TOLERANCE_CLASSES` plus the check-layer parse-equivalence
+ * fallback). Lazy keys: a class only appears once the
* first event for it has fired. Tightening from `string` enforces typo-
* resistance at compile time so no caller can silently pollute the map
* with arbitrary labels and drop those counters from operator visibility. */
- bridgeToleranceApplied: Partial<Record<BridgeToleranceClass, number>>;
+ bridgeToleranceApplied: Partial<Record<BridgeToleranceSignal, number>>;
/** Y.Text-is-truth contract — count of Observer A Path B
* fires (mergeThreeWay slow path triggered by ytext divergence from the
* baseline) that escaped the per-doc rate-limiter and emitted a
@@ -521,12 +522,13 @@ export function incrementAgentPatchFindMismatches(): void {
/**
* Increment the bridge-tolerance-applied counter for a specific tolerance
* class. Bounded cardinality: the enumerated labels in
- * `BRIDGE_TOLERANCE_CLASSES` from `@inkeep/open-knowledge-core`. Tightening
- * from `string` to `BridgeToleranceClass` is a compile-time guard against
+ * `BRIDGE_TOLERANCE_CLASSES` plus the check-layer parse-equivalence
+ * fallback (`BridgeToleranceSignal` in `@inkeep/open-knowledge-core`).
+ * Tightening from `string` is a compile-time guard against
* typos at call sites that would otherwise silently pollute the metrics map
* with arbitrary keys and drop those counters from operator dashboards.
*/
-export function incrementBridgeToleranceApplied(toleranceClass: BridgeToleranceClass): void {
+export function incrementBridgeToleranceApplied(toleranceClass: BridgeToleranceSignal): void {
counters.bridgeToleranceApplied[toleranceClass] =
(counters.bridgeToleranceApplied[toleranceClass] ?? 0) + 1;
}
diff --git a/packages/server/src/persistence.ts b/packages/server/src/persistence.ts
index 7be7b0f3e..7de02cbf5 100644
--- a/packages/server/src/persistence.ts
+++ b/packages/server/src/persistence.ts
@@ -44,7 +44,7 @@ import * as Y from 'yjs';
import { LINEAGE_EPOCH_KEY } from './auth-token-schema.ts';
import type { BacklinkIndex } from './backlink-index.ts';
import { getMsSinceLastUserTx, isDocQuiescent } from './bridge-quiescence.ts';
-import { assertBridgeInvariant } from './bridge-watchdog.ts';
+import { assertBridgeInvariant, createDocCanonicalizer } from './bridge-watchdog.ts';
import { isConfigDoc, isManagedArtifactDoc, isSystemDoc } from './cc1-broadcast.ts';
import { type ConfigPersistenceCtx, loadConfigDoc, storeConfigDoc } from './config-persistence.ts';
import type { ContributorEntry } from './contributor-tracker.ts';
@@ -1372,6 +1372,17 @@ export function createPersistenceExtension(options?: PersistenceOptions): Persis
site: 'persistence',
docName: documentName,
suppressDevThrow: true,
+ // Parse-equivalence fallback: a doc resting on a serializer
+ // canonicalization (CommonMark lazy continuations et al.) is
+ // NOT a divergence — without this, every persist of such a doc
+ // would warn AND run the synchronous reconcileFragmentNow below
+ // for a fragment that already equals parse(ytext). Same `mgr` +
+ // parse surface as the fragment derivation, by construction.
+ canonicalizeBody: createDocCanonicalizer(mgr, {
+ resolveEmbed: options?.resolveEmbed,
+ resolveSize: options?.resolveSize,
+ docName: documentName,
+ }),
});
} catch (err) {
// Counter + structured event give the serialize-throw failure
diff --git a/packages/server/src/server-observers.lazy-continuation.test.ts b/packages/server/src/server-observers.lazy-continuation.test.ts
new file mode 100644
index 000000000..f11563ed1
--- /dev/null
+++ b/packages/server/src/server-observers.lazy-continuation.test.ts
@@ -0,0 +1,225 @@
+/**
+ * Bridge health checks on CommonMark lazy-continuation docs.
+ *
+ * A doc whose source carries a lazy continuation (an unindented wrapped line
+ * inside a list item, a paragraph glued to a list's last line, a blockquote
+ * continuation without the `> ` prefix) parses identically to its canonical
+ * form (CommonMark §5.2), but serializes differently — `serialize(parse(md))
+ * !== md`, and the difference sits deliberately OUTSIDE the normalizeBridge
+ * tolerance set (step 7f keeps list/blockquote continuations divergent so the
+ * router's residual-merge keeps protecting the raw bytes; the step-7f
+ * pins live next to normalizeBridge).
+ *
+ * The health checks layered on top of that router must NOT treat this
+ * resting canonicalization as a broken bridge: the fragment IS
+ * `parse(ytext)` (Y.Text-is-truth, precedent #38), so neither the
+ * observer-b watchdog throw/warn nor the split-brain rederive may fire on
+ * organic lazy-continuation input. Genuine fragment↔Y.Text divergence
+ * (content one side lacks) must keep firing — pinned by the control test.
+ *
+ * Uses a synthetic Y.Doc (no Hocuspocus), production-order seeding
+ * (paired-write intake first, observer attach second) — the same rig as
+ * `server-observers.test.ts`.
+ */
+import { beforeEach, describe, expect, test } from 'bun:test';
+import {
+ MarkdownManager,
+ normalizeBridge,
+ prependFrontmatter,
+ sharedExtensions,
+ stripFrontmatter,
+} from '@inkeep/open-knowledge-core';
+import { getSchema } from '@tiptap/core';
+import { updateYFragment, yXmlFragmentToProseMirrorRootNode } from '@tiptap/y-tiptap';
+import * as Y from 'yjs';
+import { composeAndWriteRawBody } from './bridge-intake.ts';
+import { __resetBridgeWatchdogForTests } from './bridge-watchdog.ts';
+import { FILE_WATCHER_ORIGIN } from './external-change.ts';
+import { getMetrics, resetMetrics } from './metrics.ts';
+import { OBSERVER_SYNC_ORIGIN, setupServerObservers } from './server-observers.ts';
+
+const mdManager = new MarkdownManager({ extensions: sharedExtensions });
+const schema = getSchema(sharedExtensions);
+
+function canonicalOf(raw: string): string {
+ const { frontmatter, body } = stripFrontmatter(raw);
+ return prependFrontmatter(frontmatter, mdManager.serialize(mdManager.parseWithFallback(body)));
+}
+
+/** Populate XmlFragment with markdown content via updateYFragment. */
+function populateFragment(doc: Y.Doc, xmlFragment: Y.XmlFragment, md: string): void {
+ const json = mdManager.parse(md);
+ const pmNode = schema.nodeFromJSON(json);
+ const meta = { mapping: new Map(), isOMark: new Map() };
+ updateYFragment(doc, xmlFragment, pmNode, meta);
+}
+
+/** Seed a doc production-order: paired-write intake first, attach second. */
+function seedThenAttach(raw: string, docName: string) {
+ const doc = new Y.Doc();
+ const xmlFragment = doc.getXmlFragment('default');
+ const ytext = doc.getText('source');
+ doc.transact(() => {
+ composeAndWriteRawBody(doc, raw, 'file-watcher');
+ }, FILE_WATCHER_ORIGIN);
+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema, docName });
+ return { doc, xmlFragment, ytext, cleanup };
+}
+
+function serializeFragmentBody(xmlFragment: Y.XmlFragment): string {
+ return mdManager.serialize(yXmlFragmentToProseMirrorRootNode(xmlFragment, schema).toJSON());
+}
+
+/** Run `fn` capturing structured console.warn events with the given names. */
+function captureEvents(fn: () => void, ...eventNames: string[]): Record<string, unknown>[] {
+ const originalWarn = console.warn;
+ const warnings: string[] = [];
+ console.warn = (...args: unknown[]) => {
+ warnings.push(args.map(String).join(' '));
+ };
+ try {
+ fn();
+ } finally {
+ console.warn = originalWarn;
+ }
+ return warnings
+ .map((w) => {
+ try {
+ return JSON.parse(w);
+ } catch {
+ return null;
+ }
+ })
+ .filter((e): e is Record<string, unknown> => e !== null)
+ .filter((e) => eventNames.includes(e.event as string));
+}
+
+/** Total split-brain rederive detections — emit-gated counter + suppressed
+ * counter covers every fire even when the per-(site, doc) rate-limiter
+ * closes. */
+const totalSplitBrainRederives = (): number =>
+ getMetrics().bridgeSplitBrainRederives + getMetrics().bridgeSplitBrainRederivesSuppressed;
+
+// ─── Fixtures: the lazy-continuation family ──────────────────
+
+const FIXTURES = [
+ {
+ label: 'list in-item continuation',
+ raw: '---\ntitle: List continuation fixture\n---\n\n- Also read at session start: operating rules,\nproject status, and input source paths.\n\nBody text stays.\n',
+ // The unindented wrapped line must survive verbatim (storage never
+ // sanitizes) — canonical form would indent it under the marker.
+ preservedSlice: 'operating rules,\nproject status',
+ },
+ {
+ label: 'trailing paragraph absorbed into the last bullet',
+ raw: '---\ntitle: Trailing label fixture\n---\n\n**Why not now:**\n- First bullet\n- Second bullet\n- Third bullet\n**Trigger to revisit:** revisit after launch.\n',
+ // Canonical form gains a blank separator and indents the absorbed
+ // paragraph under the third bullet.
+ preservedSlice: '- Third bullet\n**Trigger to revisit:**',
+ },
+ {
+ label: 'blockquote lazy continuation',
+ raw: '---\ntitle: Blockquote continuation fixture\n---\n\n# Hello\n\n> Lazy quote\nstays lazy.\n\nBody text stays.\n',
+ // Canonical form gains the `> ` prefix on the continuation line.
+ preservedSlice: '> Lazy quote\nstays lazy.',
+ },
+] as const;
+
+beforeEach(() => {
+ __resetBridgeWatchdogForTests();
+ resetMetrics();
+});
+
+describe('lazy-continuation docs — bridge health checks', () => {
+ test('routing precondition: every fixture rests beyond normalizeBridge tolerance', () => {
+ // Guards the fix SHAPE, not just the fix: these byte forms must STAY
+ // beyond normalizeBridge tolerance so fragment edits keep routing the
+ // byte-preserving residual merge (step 7f's deliberate exclusion —
+ // widening the normalizer would flip routing toward canonical rewrites
+ // and sanitize user bytes on unrelated edits).
+ for (const { raw } of FIXTURES) {
+ expect(canonicalOf(raw)).not.toBe(raw);
+ expect(normalizeBridge(canonicalOf(raw))).not.toBe(normalizeBridge(raw));
+ }
+ });
+
+ for (const { label, raw, preservedSlice } of FIXTURES) {
+ test(`source-mode edit on a ${label} doc absorbs cleanly without a bridge-invariant violation`, () => {
+ const { doc, xmlFragment, ytext, cleanup } = seedThenAttach(raw, `lazy-src-${label}`);
+ expect(ytext.toString()).toBe(raw);
+
+ const violations = captureEvents(() => {
+ // A real Y.Text edit → Observer B full-fires (parse + fragment
+ // re-derive + watchdog). Under NODE_ENV=test a watchdog violation
+ // THROWS out of this transact — the defect this test pins.
+ doc.transact(() => {
+ ytext.insert(ytext.length, '\nAppended from source mode.\n');
+ });
+ }, 'bridge-invariant-violation');
+
+ expect(violations).toHaveLength(0);
+ // Observer B absorbed the edit into the fragment…
+ expect(serializeFragmentBody(xmlFragment)).toContain('Appended from source mode.');
+ // …and Y.Text keeps the user's source form verbatim.
+ const finalText = ytext.toString();
+ expect(finalText).toContain(preservedSlice);
+ expect(finalText).toContain('Appended from source mode.');
+
+ cleanup();
+ });
+
+ test(`WYSIWYG edit on a ${label} doc settles without split-brain rederive churn`, () => {
+ const { doc, xmlFragment, ytext, cleanup } = seedThenAttach(raw, `lazy-wysiwyg-${label}`);
+
+ const redsBefore = totalSplitBrainRederives();
+ const events = captureEvents(() => {
+ populateFragment(
+ doc,
+ xmlFragment,
+ `${serializeFragmentBody(xmlFragment)}\nWysiwyg paragraph.\n`,
+ );
+ }, 'bridge-split-brain-rederive');
+
+ expect(events).toHaveLength(0);
+ expect(totalSplitBrainRederives()).toBe(redsBefore);
+
+ // The residual merge spliced the fragment delta into the raw bytes —
+ // the lazy continuation survives verbatim alongside the new content.
+ const finalText = ytext.toString();
+ expect(finalText).toContain(preservedSlice);
+ expect(finalText).toContain('Wysiwyg paragraph.');
+
+ cleanup();
+ });
+ }
+
+ test('control: genuine fragment↔Y.Text divergence still fires the split-brain rederive', () => {
+ const { doc, xmlFragment, ytext, cleanup } = seedThenAttach(
+ FIXTURES[0].raw,
+ 'lazy-genuine-divergence',
+ );
+
+ // Divergence the observers never saw: a self-origin write skips dispatch
+ // and leaves every witness stale, so the fragment now LACKS real Y.Text
+ // content — not a serializer canonicalization.
+ doc.transact(() => {
+ ytext.insert(ytext.length, '\nSmuggled paragraph the fragment lacks.\n');
+ }, OBSERVER_SYNC_ORIGIN);
+
+ const redsBefore = totalSplitBrainRederives();
+ populateFragment(
+ doc,
+ xmlFragment,
+ `${serializeFragmentBody(xmlFragment)}\nWysiwyg paragraph.\n`,
+ );
+
+ // The settlement check must still classify this as split-brain.
+ expect(totalSplitBrainRederives()).toBeGreaterThan(redsBefore);
+ // Both contents survive in Y.Text (the merge preserves, never drops).
+ const finalText = ytext.toString();
+ expect(finalText).toContain('Smuggled paragraph the fragment lacks.');
+ expect(finalText).toContain('Wysiwyg paragraph.');
+
+ cleanup();
+ });
+});
diff --git a/packages/server/src/server-observers.ts b/packages/server/src/server-observers.ts
index d8d774207..af2f4b517 100644
--- a/packages/server/src/server-observers.ts
+++ b/packages/server/src/server-observers.ts
@@ -29,6 +29,7 @@ import {
applyIncrementalDiff,
BridgeInvariantViolationError,
BridgeMergeContentLossError,
+ isParseEquivalentBridge,
mergeThreeWay,
normalizeBridge,
prependFrontmatter,
@@ -44,6 +45,7 @@ import { attachQuiescenceTracker } from './bridge-quiescence.ts';
import {
assertBridgeInvariant,
type BridgeSplitBrainSite,
+ createDocCanonicalizer,
emitBridgeSplitBrainRederive,
emitObserverAPathBFired,
} from './bridge-watchdog.ts';
@@ -104,7 +106,7 @@ export const OBSERVER_SYNC_ORIGIN = {
* still match; `satisfies PairedWriteOrigin` is the authoring-site gate,
* not a runtime `instanceof` narrowing.
*
- * Today's paired origin count: 4. When adding a 5th, the ONLY required
+ * Today's paired origin count: 5. When adding a 6th, the ONLY required
* change is `satisfies PairedWriteOrigin` at the literal. No registry
* update. No Observer A/B wiring. No `BRIDGE_ENFORCING_ORIGINS` change
* (that set is unrelated — it enforces the bridge-invariant watcher's
@@ -458,10 +460,13 @@ export function setupServerObservers(opts: SetupServerObserversOpts): () => void
};
/**
- * Telemetry for the split-brain settlement check. No organic input
- * produces this divergence at HEAD (producers were narrowed to
- * dependency/plugin drift), so this firing in production is itself the
- * drift alert — and the operator's only handle on a doc stuck re-deriving
+ * Telemetry for the split-brain settlement check. The settlement predicate
+ * tolerates resting serializer canonicalizations of organic input via the
+ * parse-equivalence fallback (`settlesSplitBrainChecked` — fragment ≡
+ * parse(ytext) verified through the doc's own parse pipeline), so a fire
+ * means the fragment genuinely does not derive from Y.Text: dependency/
+ * plugin drift or a degraded fragment. That makes this event the drift
+ * alert — and the operator's only handle on a doc stuck re-deriving
* its fragment on every drain. Rate-limited per (site, doc) through
* `emitBridgeSplitBrainRederive` (mirroring `emitObserverAPathBFired`);
* the counter increments only on emit, the suppressed counter inside the
@@ -603,6 +608,70 @@ export function setupServerObservers(opts: SetupServerObserversOpts): () => void
*/
const readCurrentFm = (): string => stripFrontmatter(ytext.toString()).frontmatter;
+ /** Parse options for THIS doc's text→tree derivations. One shape shared by
+ * Observer B's full fire, the attach-time settlement check, and the
+ * parse-equivalence canonicalizer, so every parse of this doc resolves
+ * embeds identically — a mismatched pipeline would read the same bytes
+ * into different trees. */
+ const observerParseOpts =
+ opts.resolveEmbed && opts.docName
+ ? {
+ resolveEmbed: opts.resolveEmbed,
+ resolveSize: opts.resolveSize,
+ sourcePath: opts.docName,
+ }
+ : undefined;
+
+ /** Canonicalize a body through this doc's own parse pipeline — the
+ * parse-equivalence fallback's callback (`isParseEquivalentBridge`). */
+ const canonicalizeBody = createDocCanonicalizer(mdManager, {
+ resolveEmbed: opts.resolveEmbed,
+ resolveSize: opts.resolveSize,
+ docName: opts.docName,
+ });
+
+ // Positive-result memo for the parse-equivalence fallback. A doc resting
+ // on a serializer canonicalization (lazy continuations et al.) hits the
+ // settlement checks on every drain with the SAME byte pair; the memo
+ // caps that at one parse per distinct pair (string compares are 10-100×
+ // cheaper than a parse on the per-drain hot path). Negative results are
+ // deliberately NOT cached — genuine divergence must keep re-evaluating
+ // (and alerting) as the doc changes.
+ let memoParseEquivalentLeft = '';
+ let memoParseEquivalentRight = '';
+ let hasParseEquivalentMemo = false;
+ const isRestingParseEquivalent = (left: string, right: string): boolean => {
+ if (
+ hasParseEquivalentMemo &&
+ left === memoParseEquivalentLeft &&
+ right === memoParseEquivalentRight
+ ) {
+ return true;
+ }
+ const equivalent = isParseEquivalentBridge(left, right, canonicalizeBody);
+ if (equivalent) {
+ memoParseEquivalentLeft = left;
+ memoParseEquivalentRight = right;
+ hasParseEquivalentMemo = true;
+ }
+ return equivalent;
+ };
+
+ /**
+ * Health-check refinement of `settlesSplitBrain`: a drain settles
+ * split-brain only when the byte comparison fails AND the pair is not
+ * parse-equivalent. Beyond-tolerance bytes whose parse matches the
+ * fragment (organic resting canonicalizations — CommonMark lazy
+ * continuations and kin) are a healthy steady state: the router still
+ * classifies them as residual-bearing (normalizeBridge untouched, so
+ * fragment edits keep the byte-preserving residual merge), but no
+ * re-derive is enqueued and no split-brain telemetry fires. The parse
+ * runs only after byte + normalize inequality — the drains that would
+ * otherwise settle split-brain and pay a full Observer B re-derive.
+ */
+ const settlesSplitBrainChecked = (settledText: string, md: string, normMdPre?: string): boolean =>
+ settlesSplitBrain(settledText, md, normMdPre) && !isRestingParseEquivalent(settledText, md);
+
/** Initialize Observer A baseline from current XmlFragment state. */
try {
const initialJson = yXmlFragmentToProseMirrorRootNode(xmlFragment, schema).toJSON();
@@ -614,30 +683,13 @@ export function setupServerObservers(opts: SetupServerObserversOpts): () => void
// witness then captures the seed bytes so the first fragment change on a
// residual-bearing doc routes Path A instead of a spurious Path B merge.
// But that is an assumption, not a given: a partially-failed paired write
- // can leave the fragment behind Y.Text at attach. Verify by comparing the
- // fragment's canonical body against the canonical body of parse(ytext) —
- // canonical-vs-canonical, so the check is tolerance-independent and
- // residual bytes never flip it.
- const initParseOpts =
- opts.resolveEmbed && opts.docName
- ? {
- resolveEmbed: opts.resolveEmbed,
- resolveSize: opts.resolveSize,
- sourcePath: opts.docName,
- }
- : undefined;
- const ytextCanonicalBody = mdManager.serialize(
- mdManager.parseWithFallback(stripFrontmatter(ytext.toString()).body, initParseOpts),
- );
- // Doc-boundary-normalized compare: parse(ytext) re-captures the
- // sourceDocBoundary leading/trailing newline forms from the raw bytes
- // (e.g. the blank line after a stripped FM region becomes a leading
- // capture), while the live fragment structurally drops doc-node attrs
- // (pinned by doc-boundary-fragment-drop.test.ts) — so the two canonical
- // serializations legitimately differ by exactly the boundary newlines
- // on a perfectly settled doc. Everything else stays strict-byte.
- const stripDocBoundary = (b: string): string => b.replace(/^\n+/, '').replace(/\n+$/, '');
- if (stripDocBoundary(ytextCanonicalBody) === stripDocBoundary(initialBody)) {
+ // can leave the fragment behind Y.Text at attach. Verify parse-
+ // equivalence: canonical-vs-canonical through the doc's own parse
+ // pipeline (tolerance-independent, residual bytes never flip it), with
+ // the boundary-newline handling documented on `isParseEquivalentBridge`
+ // (parse(ytext) re-captures sourceDocBoundary forms the live fragment
+ // structurally drops — pinned by doc-boundary-fragment-drop.test.ts).
+ if (isRestingParseEquivalent(ytext.toString(), canonicalInit)) {
recordSettledBaselines(canonicalInit);
} else {
recordDivergedAttachBaselines(canonicalInit);
@@ -711,7 +763,7 @@ export function setupServerObservers(opts: SetupServerObserversOpts): () => void
//
// (2) Perf short-circuit. Absent split-brain, the fragment is at the
// witnessed settlement and Y.Text agrees — nothing to do.
- if (settlesSplitBrain(ytext.toString(), md)) {
+ if (settlesSplitBrainChecked(ytext.toString(), md)) {
// Force Observer B to re-derive in this same drain: move BOTH
// witnesses to the canonical form so B's early-exit comparand
// (`normalizeBridge(lastSyncedYTextBytes)`) no longer matches the
@@ -977,23 +1029,24 @@ export function setupServerObservers(opts: SetupServerObserversOpts): () => void
// Split-brain settlement guard (Y.Text-is-truth, precedent #38). After
// the write, the raw witness is at the post-write ytext and the canonical
- // witness is at md. The `settlesSplitBrain` predicate fires on ANY
- // beyond-tolerance residual — which covers BOTH a legitimate NG-class doc
- // (un-padded tables, multi-blank lines: storage never sanitizes) and a
- // degraded-fallback divergence. The two-witness router already
- // DISCRIMINATES and protects them identically: the next fragment-change
- // drain sees the beyond-tolerance residual and routes the byte-preserving
- // residual-merge (row 2), never a wholesale Path A rewrite. We still
- // enqueue a same-drain Observer B to ATTEMPT convergence (Y.Text-is-truth
- // re-derive). Deliberately WITHOUT moving the witnesses to the canonical
- // form: for an NG-class doc the fragment and Y.Text agree within
- // serialization (B early-exits — its raw-witness comparand equals the
- // current ytext — leaving the row-2 steady state intact), and only a
- // genuinely irreducible fallback divergence keeps B re-deriving. Forcing
- // the diverged-attach witness shape here would wrongly re-derive every
- // NG-class doc on every WYSIWYG edit (regressing the residual-merge
+ // witness is at md. The checked predicate fires on a beyond-tolerance
+ // residual whose parse does NOT match the fragment — a genuinely
+ // degraded divergence. A parse-equivalent resting canonicalization
+ // (lazy continuations and other organic constructs the serializer
+ // re-shapes: storage never sanitizes) is a healthy steady state and is
+ // excluded here, while the two-witness router still classifies it as
+ // residual-bearing: the next fragment-change drain sees the
+ // beyond-tolerance residual and routes the byte-preserving
+ // residual-merge (row 2), never a wholesale Path A rewrite. On a
+ // genuine fire we enqueue a same-drain Observer B to ATTEMPT
+ // convergence (Y.Text-is-truth re-derive), deliberately WITHOUT moving
+ // the witnesses to the canonical form: only a genuinely irreducible
+ // fallback divergence keeps B re-deriving (B early-exits when its
+ // raw-witness comparand equals the current ytext). Forcing the
+ // diverged-attach witness shape here would wrongly re-derive every
+ // residual doc on every WYSIWYG edit (regressing the residual-merge
// steady state); the bytes are already safe either way.
- if (settlesSplitBrain(ytext.toString(), md, normMd)) {
+ if (settlesSplitBrainChecked(ytext.toString(), md, normMd)) {
textDirty = true;
recordSplitBrainRederive('post-merge');
}
@@ -1031,7 +1084,7 @@ export function setupServerObservers(opts: SetupServerObserversOpts): () => void
const recoveryJson = yXmlFragmentToProseMirrorRootNode(xmlFragment, schema).toJSON();
const recoveryBody = mdManager.serialize(recoveryJson);
const recoveryMd = prependFrontmatter(readCurrentFm(), recoveryBody);
- if (settlesSplitBrain(ytext.toString(), recoveryMd)) {
+ if (settlesSplitBrainChecked(ytext.toString(), recoveryMd)) {
recordSplitBrainRecoveryBaselines(recoveryMd);
textDirty = true;
recordSplitBrainRederive('error-recovery');
@@ -1210,21 +1263,13 @@ export function setupServerObservers(opts: SetupServerObserversOpts): () => void
// Bridge always-live: parseWithFallback never throws — it always
// produces a valid JSONContent tree, falling back to rawMdxFallback
- // for unparseable spans. Threads `resolveEmbed` + `sourcePath` so
- // `![[photo.png]]` mdast nodes resolve to disk paths before PM
- // dispatch. Under server-authoritative architecture (precedent #14),
- // this observer is the sole writer for XmlFragment — the "always-
- // live" contract here means no client sees frozen WYSIWYG when
- // another peer is mid-typing a broken MDX tag.
- const parseOpts =
- opts.resolveEmbed && opts.docName
- ? {
- resolveEmbed: opts.resolveEmbed,
- resolveSize: opts.resolveSize,
- sourcePath: opts.docName,
- }
- : undefined;
- const parsedJson = mdManager.parseWithFallback(body, parseOpts);
+ // for unparseable spans. `observerParseOpts` threads `resolveEmbed` +
+ // `sourcePath` so `![[photo.png]]` mdast nodes resolve to disk paths
+ // before PM dispatch. Under server-authoritative architecture
+ // (precedent #14), this observer is the sole writer for XmlFragment —
+ // the "always-live" contract here means no client sees frozen WYSIWYG
+ // when another peer is mid-typing a broken MDX tag.
+ const parsedJson = mdManager.parseWithFallback(body, observerParseOpts);
const pmNode = opts.schema.nodeFromJSON(parsedJson);
@@ -1261,6 +1306,11 @@ export function setupServerObservers(opts: SetupServerObserversOpts): () => void
assertBridgeInvariant(ytext.toString(), canonicalYText, {
site: 'observer-b',
docName: opts.docName,
+ // One-shot reuse of the canonicalization this fire just computed:
+ // the watchdog's fallback canonicalizes the SAME body B parsed
+ // above, and re-running parse+serialize per fire is exactly the
+ // extra O(N) pass the parsedJson reuse note above exists to avoid.
+ canonicalizeBody: (b) => (b === body ? canonicalBody : canonicalizeBody(b)),
});
// Maintain Observer A's witnesses — B just absorbed Y.Text into the
// fragment, a true settlement point. The canonical witness records
diff --git a/patches/@handlewithcare%[email protected] b/patches/@handlewithcare%[email protected]
index 8cfdac101..fbbcdd94d 100644
--- a/patches/@handlewithcare%[email protected]
+++ b/patches/@handlewithcare%[email protected]
@@ -1,11 +1,20 @@
+diff --git a/node_modules/@handlewithcare/remark-prosemirror/.bun-tag-5a1ee271d50e78cc b/.bun-tag-5a1ee271d50e78cc
+new file mode 100644
+index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
+diff --git a/node_modules/@handlewithcare/remark-prosemirror/.bun-tag-8534289bc1b3df99 b/.bun-tag-8534289bc1b3df99
+new file mode 100644
+index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
diff --git a/node_modules/@handlewithcare/remark-prosemirror/.bun-tag-c30c30ad62176876 b/.bun-tag-c30c30ad62176876
new file mode 100644
index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
+diff --git a/node_modules/@handlewithcare/remark-prosemirror/.bun-tag-fa6b3e9bc7828de2 b/.bun-tag-fa6b3e9bc7828de2
+new file mode 100644
+index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
diff --git a/lib/mdast-util-from-prosemirror.js b/lib/mdast-util-from-prosemirror.js
-index 8b1d0173e7a95e5dc57d9b85b2b0a2360190fcb3..7b4c210f2117efb04a49e8e07a19338c7b56637c 100644
+index 8b1d0173e7a95e5dc57d9b85b2b0a2360190fcb3..b4b68ed8ec7d7b41e5e71560359119c00fa4a067 100644
--- a/lib/mdast-util-from-prosemirror.js
+++ b/lib/mdast-util-from-prosemirror.js
-@@ -32,55 +32,148 @@ function createState(nodeHandlers, markHandlers) {
+@@ -32,55 +32,155 @@ function createState(nodeHandlers, markHandlers) {
}
return null;
}
@@ -174,6 +183,13 @@ index 8b1d0173e7a95e5dc57d9b85b2b0a2360190fcb3..7b4c210f2117efb04a49e8e07a19338c
+ // on the same shape without termination guarantee). This branch is
+ // not expected in practice — kept as safety net.
+ if (groups.length === 1 && spans.length > 1) {
++ // Silent activation would round-trip subtly wrong mark nesting
++ // with no trace, so make any real-world hit diagnosable.
++ console.warn('[mdast-util-from-prosemirror] degenerate mark partition; falling back to per-span rendering', {
++ spanCount: spans.length,
++ firstSpanMarks: spans[0]?.marks.map((m)=>m.type.name),
++ lastSpanMarks: spans[spans.length - 1]?.marks.map((m)=>m.type.name)
++ });
+ const result = [];
+ for (const s of spans) {
+ for (const r of hydrateMarks([
@@ -199,10 +215,48 @@ index 8b1d0173e7a95e5dc57d9b85b2b0a2360190fcb3..7b4c210f2117efb04a49e8e07a19338c
function all(pmNode) {
return hydrateMarks(pmNode.children.map((child)=>({
diff --git a/lib/mdast-util-to-prosemirror.js b/lib/mdast-util-to-prosemirror.js
-index 8433f31fff4e82e5fbb0fae2bb8ba1f68cc30f35..9adb6cd9be3462cbca81f4b6a0dca30571e3d05a 100644
+index 8433f31fff4e82e5fbb0fae2bb8ba1f68cc30f35..617679158b4d2bb7dab2505e5a2ade7c8266e9c5 100644
--- a/lib/mdast-util-to-prosemirror.js
+++ b/lib/mdast-util-to-prosemirror.js
-@@ -167,7 +167,9 @@ function handle(schema, handlers, htmlHandlers, node, parent, state) {
+@@ -45,19 +45,30 @@ export function createState(schema, handlers, htmlHandlers, tree) {
+ let result = state.one(nodes[index], parent);
+ // To do: see if we van clean this? Can we merge texts?
+ if (result) {
+- if (index && nodes[index - 1]?.type === "break") {
++ // Post-break leading-space trimming implements CommonMark's
++ // "spaces at the beginning of the next line are ignored".
++ // Inline-HTML breaks (data.sourceStyle === 'html', e.g. a
++ // promoted `<br />`) create no source line boundary, so the
++ // following space is ordinary content — skip the trim to
++ // keep it byte-identical. And a whitespace-only text child
++ // must drop entirely rather than construct an empty PM text
++ // node (schema.text('') throws RangeError).
++ const prevChild = index ? nodes[index - 1] : undefined;
++ if (prevChild?.type === "break" && prevChild.data?.sourceStyle !== "html") {
+ if (!Array.isArray(result) && result.type === schema.nodes["text"]) {
+- // eslint-disable-next-line @typescript-eslint/no-non-null-assertion
+- result = schema.text(trimMarkdownSpaceStart(result.text));
+- }
+- if (!Array.isArray(result) && result.type !== schema.nodes["text"]) {
++ const trimmed = trimMarkdownSpaceStart(result.text);
++ result = trimmed.length > 0 ? schema.text(trimmed) : undefined;
++ } else if (!Array.isArray(result) && result.type !== schema.nodes["text"]) {
+ const head = result.firstChild;
+ if (head && head.type === schema.nodes["text"]) {
+- result = result.copy(Fragment.from(result.content).replaceChild(0, // eslint-disable-next-line @typescript-eslint/no-non-null-assertion
+- schema.text(trimMarkdownSpaceStart(head.text))));
++ const trimmed = trimMarkdownSpaceStart(head.text);
++ const fragment = Fragment.from(result.content);
++ result = result.copy(trimmed.length > 0 ? fragment.replaceChild(0, schema.text(trimmed)) : fragment.cut(head.nodeSize));
+ }
+ }
+ }
++ }
++ if (result) {
+ if (Array.isArray(result)) {
+ values.push(...result);
+ } else {
+@@ -167,7 +178,9 @@ function handle(schema, handlers, htmlHandlers, node, parent, state) {
return schema.text(node.value);
},
text (node) {