Beautiful, AI-native markdown IDE and LLM wiki
TypeScript 2nd-brainagent-skillsclaudecodexcompany-braindocsknowledge-knowledge-baseknowledge-managementllmllm-wikillm-wiki-karpathymarkdownmarkdown-editormdmdxnotespkmsecond-brainskills
feat(open-knowledge): markdown byte-fidelity engine + conformance suite (#1775)
Nick Gomez5b5eda4893b1
diff --git a/.changeset/markdown-fidelity-e2e.md b/.changeset/markdown-fidelity-e2e.mdnew file mode 100644index 000000000..43f1a8473--- /dev/null+++ b/.changeset/markdown-fidelity-e2e.md@@ -0,0 +1,14 @@+---+"@inkeep/open-knowledge-core": minor+"@inkeep/open-knowledge-server": minor+---++feat(markdown): end-to-end byte fidelity — markdown round-trips byte-exact, agents' untouched bytes survive concurrent edits++The markdown engine no longer canonicalizes source formatting on the WYSIWYG/cross-surface edit path. Every producer-fixable byte-residual axis now round-trips byte-exact through `serialize(parse(x))`: table outer pipes, delimiter-row dashes and alignment padding, ordered-list numbering (`1. 1. 1.` stays), list marker spacing and continuation indents, fence length and info-string padding, strikethrough delimiter (`~` vs `~~`), task-checkbox case, wiki-link inner padding, opaque-extension embeds (`![[notes.foo]]` no longer downcasts to a plain link), inline math (`$x$` stays `$x$` instead of rewriting to `$$x$$`, with currency safety preserved), doc-start `---` thematic breaks (no longer rewritten to `***`, and a doc-start `---` no longer corrupts adjacent blocks), BOM, leading/trailing blank lines, inter-block blank-line counts, and tab-indented code. One visible consequence: a literal frontmatter-shaped block (`---`/`key: val`/`---`) ingested directly into the parse engine now renders as visible markdown (thematic break + setext heading) instead of being silently dropped.++Silent data-loss fixes: boundary-whitespace emphasis (`<em>foo </em>`) now serializes to a re-parseable form instead of degrading to plain text; entity references in a fenced-code info string survive instead of being stripped; `[link](</my uri>)` parses as a link instead of literal text; inline code containing `|` inside a table cell keeps its escape instead of corrupting the table. The serializer also stops escaping bare `*`/`_`/`~` and line-start `=` where CommonMark does not require it.++Agent find-fidelity (PRD-6654): a concurrent human WYSIWYG edit no longer rewrites an agent's untouched blocks in `Y.Text` — the map-driven Observer A splice is now the default path and a parse-equivalent `row-no-trailing-pipe` divergence no longer forces a canonicalizing write-back — so exact-match find-and-replace on previously written content keeps succeeding.++Clipboard/export rendering: `img`/`video`/`audio` compat components render as real HTML elements instead of placeholders, and entity references and `<>` destinations in link/image URLs and titles decode correctly (with URL sanitization preserved).diff --git a/biome-plugins/README.md b/biome-plugins/README.mdindex e4068bf77..7406ea599 100644--- a/biome-plugins/README.md+++ b/biome-plugins/README.md@@ -99,6 +99,54 @@ The rule does NOT catch:Plugin: [`biome-plugins/playwright-topass-budget.grit`](playwright-topass-budget.grit). Fixture: [`biome-plugins/__fixtures__/playwright-topass-budget.fixture.tsx`](__fixtures__/playwright-topass-budget.fixture.tsx). Test: [`packages/desktop/tests/integration/playwright-topass-budget.test.ts`](../packages/desktop/tests/integration/playwright-topass-budget.test.ts). See [PRECEDENTS.md #42](../PRECEDENTS.md#custom-lint-enforcement-precedent-42) for the GritQL-plugin convention.+### `path-conditional-map-driven-origin.grit`++Observer A origin discipline. Inside `packages/server/src/server-observers.ts`, every `Y.Doc.transact()` call MUST pass the sanctioned origin `OBSERVER_SYNC_ORIGIN` as its second argument (`doc.transact(fn, OBSERVER_SYNC_ORIGIN)`). Bare `doc.transact(fn)` - or a wrong origin - routes the write to `openknowledge-service` and breaks per-session UndoManager attribution (the `trackedOrigins` Set-identity match skips the transaction).++**Scoped via `overrides[].plugins`** to `packages/server/src/server-observers.ts`. Other server files routing through `session.dc.document.transact(fn, session.origin)` are out of scope - that contract is enforced by `paired-write-enforcement.test.ts`.++The rule checks the **argument position** (the second argument node), not whether the call subtree contains the identifier somewhere. It matches every `transact(...)` call, then excludes the two sanctioned shapes (`transact($_, OBSERVER_SYNC_ORIGIN)` and the 3-arg `transact($_, OBSERVER_SYNC_ORIGIN, $...)`). A callback body that mentions `OBSERVER_SYNC_ORIGIN` for an unrelated reason therefore neither clears a bare/wrong call nor trips a correct one - the prior `contains`-based shape was falsely cleared by such a mention.++The rule does NOT catch:+- Transact calls outside `server-observers.ts` (scoped to the one observer-dispatch file)+- An origin passed in a position other than the second argument (the contract is second-argument placement; no real call site does otherwise)++Plugin: [`biome-plugins/path-conditional-map-driven-origin.grit`](path-conditional-map-driven-origin.grit). Fixture: [`biome-plugins/__fixtures__/path-conditional-map-driven-origin.fixture.tsx`](__fixtures__/path-conditional-map-driven-origin.fixture.tsx). Test: [`packages/server/src/path-conditional-map-driven-origin.test.ts`](../packages/server/src/path-conditional-map-driven-origin.test.ts).++### `cst-pm-handler-todo-stub.grit`++Codemod handler TODO-stub grep. Flags handler files under `packages/md-conformance/src/substrates/*/handlers/` that still contain the codemod's stub body (`throw new Error("TODO: implement <substrate>:<dir>/<key>")`), so a codemod stub that survives to lint time is caught.++This is a TODO-stub grep, NOT an exhaustiveness check. Whether a substrate covers every PM node type (the node-set traversal over `packages/core/schema-snapshot.json`) is a separate concern owned by a suite-self-consistency gate. The compile-time backstop for missing methods is the TypeScript handlers-table type check against `ICstEngine`.++**Scoped via `overrides[].plugins`** to `packages/md-conformance/src/substrates/*/handlers/**/*.ts`. The codemod implementation, harness tests, and oracles are out of scope (they legitimately produce strings containing "TODO: implement" in their own contexts).++The rule does NOT catch:+- Missing handler FILES (file-presence is out of GritQL's scope; TypeScript + codemod coverage handle this)+- Whether the handler set is complete per the PM schema (a separate node-set traversal, deferred to the suite-self-consistency gate)+- Error subclasses (only matches `new Error(...)`)+- Non-anchored "TODO" mentions (regex requires the message to start with `TODO: implement`)++Plugin: [`biome-plugins/cst-pm-handler-todo-stub.grit`](cst-pm-handler-todo-stub.grit). Fixture: [`biome-plugins/__fixtures__/cst-pm-handler-todo-stub.fixture.tsx`](__fixtures__/cst-pm-handler-todo-stub.fixture.tsx). Test: [`packages/md-conformance/src/lint-plugins/cst-pm-handler-todo-stub.test.ts`](../packages/md-conformance/src/lint-plugins/cst-pm-handler-todo-stub.test.ts).++### `class-proof-registration-discipline.grit`++Class-proof DSL contract enforcement. Two patterns (GritQL `or` — short-circuit, Pattern A wins when both apply):++- **Pattern A (missing args):** flags `defineClassProof(name, opts)` when `opts` doesn't contain both `predicate:` and `proof:` property assignments.+- **Pattern B (outside canonical dir):** flags any `defineClassProof(...)` call. The override scope excludes `packages/md-conformance/src/class-proofs/proofs/**`, so this fires only on registrations outside the sanctioned location.++Inside the canonical proofs/ dir, neither pattern fires — TypeScript's `ClassProofOptions<M>` signature backstops the missing-args check, and any `defineClassProof` call there is in the sanctioned location.++**Scoped via `overrides[].plugins`** to all `.ts`/`.tsx` files EXCEPT `packages/md-conformance/src/class-proofs/proofs/**` and `*.test.ts`/`*.test.tsx` (test files that exercise the DSL are exempt — `dsl.test.ts` legitimately calls `defineClassProof` outside the canonical dir).++The rule does NOT catch:+- Missing-args violations inside the canonical proofs/ dir (TypeScript catches those)+- Functions with similar names (`myDefineClassProofVariant`) — pattern matches the exact function name+- Type references to `defineClassProof` — pattern scopes to call expressions++Plugin: [`biome-plugins/class-proof-registration-discipline.grit`](class-proof-registration-discipline.grit). Fixture: [`biome-plugins/__fixtures__/class-proof-registration-discipline.fixture.tsx`](__fixtures__/class-proof-registration-discipline.fixture.tsx). Test: [`packages/md-conformance/src/lint-plugins/class-proof-registration-discipline.test.ts`](../packages/md-conformance/src/lint-plugins/class-proof-registration-discipline.test.ts).+### `playwright-prefer-to-have-count.grit`Flags `expect(await locator.count())` — the one-shot count snapshot that never retries. Under CI CPU contention the DOM settles a few frames after the read, so the assertion flakes while the auto-retrying web-first form `await expect(locator).toHaveCount(n)` passes deterministically (the no-retry read was one of the hidden-flake shapes in the 2026-06 e2e CI audit). The pattern matches the probe sub-expression regardless of the matcher that follows (`.toBe`, `.toEqual`, `.toBeGreaterThanOrEqual`, ...). Upstream precedent: eslint-plugin-playwright `prefer-to-have-count`.diff --git a/biome-plugins/__fixtures__/class-proof-registration-discipline.fixture.tsx b/biome-plugins/__fixtures__/class-proof-registration-discipline.fixture.tsxnew file mode 100644index 000000000..54246c886--- /dev/null+++ b/biome-plugins/__fixtures__/class-proof-registration-discipline.fixture.tsx@@ -0,0 +1,73 @@+// FIXTURE — drives `class-proof-registration-discipline.test.ts` via+// shell-out to `bunx biome check`. Not part of the main lint (override+// scope keeps the rule off this fixture's path under normal `bun run lint`;+// the test explicitly invokes biome on this file path).+//+// GritQL's `or { ... }` is short-circuit: when a call matches the first+// branch (Pattern A: missing predicate/proof), the second branch (Pattern+// B: any defineClassProof) is NOT also considered. Missing-args calls+// therefore get the more-specific Pattern A diagnostic; well-formed calls+// outside the canonical dir get Pattern B's diagnostic.+//+// 3 expected diagnostic fires (one per violating call):+// - Positive A1: missing predicate (1 fire — Pattern A diagnostic)+// - Positive A2: missing proof (1 fire — Pattern A diagnostic)+// - Positive B1: well-formed but outside canonical dir+// (1 fire — Pattern B diagnostic)+// - Negative cases: 0 fires (must NOT fire)+//+// Exact-equality (`toBe(3)`) catches both false-negative regressions (drop+// below 3) and false-positive widenings (above 3).++// biome-ignore lint/correctness/noExplicitAny: fixture-only — production types unimportant here+declare const defineClassProof: any;+// biome-ignore lint/correctness/noExplicitAny: fixture-only — production types unimportant here+declare const enumerateThings: any;+// biome-ignore lint/correctness/noExplicitAny: fixture-only — production types unimportant here+declare const someOtherFunction: any;++// === Positive cases — must fire ===++// (A1) Missing `predicate` field — Pattern A's missing-args diagnostic+// wins over Pattern B's outside-canonical diagnostic via `or`'s+// short-circuit. Total: 1 diagnostic.+export const badProofMissingPredicate = defineClassProof('cp-bad-1', {+ enumerate: enumerateThings,+ proof: () => ({ pass: true }),+});++// (A2) Missing `proof` field — same as A1: Pattern A fires. 1 diagnostic.+export const badProofMissingProof = defineClassProof('cp-bad-2', {+ enumerate: enumerateThings,+ predicate: () => true,+});++// (B1) All three fields present, but the file is outside the canonical+// dir — Pattern A doesn't match (args complete), so Pattern B fires.+// Total: 1 diagnostic.+export const wellFormedButWrongLocation = defineClassProof('cp-bad-3', {+ enumerate: enumerateThings,+ predicate: () => true,+ proof: () => ({ pass: true }),+});++// === Negative cases — must NOT fire ===++// (1) An unrelated function call with the same arg shape — must NOT fire.+// The rule's pattern matches the literal function name `defineClassProof`.+export const unrelated = someOtherFunction('cp-1', {+ enumerate: enumerateThings,+ predicate: () => true,+});++// (2) An identifier whose name contains `defineClassProof` substring but is+// not exactly that — must NOT fire. The rule matches the bare+// identifier, not arbitrary affixes.+export const customDef = (() => {+ const myDefineClassProofVariant = (_n: string, _o: unknown) => ({ ok: true });+ return myDefineClassProofVariant('cp-2', { enumerate: enumerateThings });+})();++// (3) A type reference that mentions `defineClassProof` (e.g. for an alias)+// — must NOT fire. The rule scopes to call expressions.+export type DefineClassProofAlias = typeof defineClassProof;diff --git a/biome-plugins/__fixtures__/cst-pm-handler-todo-stub.fixture.tsx b/biome-plugins/__fixtures__/cst-pm-handler-todo-stub.fixture.tsxnew file mode 100644index 000000000..1c261d335--- /dev/null+++ b/biome-plugins/__fixtures__/cst-pm-handler-todo-stub.fixture.tsx@@ -0,0 +1,71 @@+// FIXTURE - drives `cst-pm-handler-todo-stub.test.ts` via shell-out+// to `bunx biome check`. Not part of the main lint (override scope keeps+// the rule off this fixture's path under normal `bun run lint`; the test+// explicitly invokes biome on this file path).+//+// 4 positive cases (deliberate codemod-stub bodies — plugin must fire) ++// 4 negative cases (filled-in bodies + non-handler error throws that must+// NOT fire). Exact-equality (`toBe(4)`) in the test catches both+// false-negative regressions (drop below 4) and false-positive widenings+// (above 4).+//+// Parameters are prefixed with `_` per the TS / biome convention for+// intentionally unused — the rule scopes to the throw-statement shape,+// not parameter usage. The codemod's emitted stubs name params `node`+// without the underscore; in real substrate adapters those names will+// be referenced once the body is filled in.++// === Positive cases — codemod-emitted stubs that must fire ===++// (1) Single-quoted message - the codemod's `JSON.stringify(...)` output+// renders as a double-quoted literal in TS source. Single-quoted here+// covers the prettier-rewritten / hand-edited variant.+export function handleParagraph(_node: unknown): unknown {+ throw new Error('TODO: implement micromark-event-fold:mdast-to-pm/paragraph');+}++// (2) Double-quoted message — the codemod's emitted shape verbatim.+// biome-ignore lint/style/useSingleQuote: fixture covers both quote styles+export function handleHeading(_node: unknown): unknown {+ throw new Error("TODO: implement micromark-event-fold:mdast-to-pm/heading");+}++// (3) Template-literal message — the third quote style; still a TODO stub.+export function handleStrong(_node: unknown): unknown {+ throw new Error(`TODO: implement micromark-event-fold:pm-to-mdast-mark/strong`);+}++// (4) Stub for a different substrate id — pattern is substrate-agnostic.+export function handleEmphasis(_node: unknown): unknown {+ throw new Error('TODO: implement lezer-markdown-derived:pm-to-mdast-mark/emphasis');+}++// === Negative cases — implemented handlers + unrelated throws ===++// (1) A filled-in handler that does real work — must NOT fire.+export function handleBlockquote(node: unknown): unknown {+ return { type: 'blockquote', content: [node] };+}++// (2) An error thrown for a legitimate runtime contract violation — must+// NOT fire. Message doesn't start with the codemod's `TODO: implement`+// marker.+export function handleCodeBlock(node: unknown): unknown {+ if (typeof node !== 'object' || node === null) {+ throw new Error('handleCodeBlock: expected mdast code node, got non-object');+ }+ return { type: 'codeBlock', content: [] };+}++// (3) An error thrown via a different constructor — must NOT fire.+// The rule only matches `Error` (the JS built-in) by name.+class HandlerSubstrateError extends Error {}+export function handleListItem(_node: unknown): unknown {+ throw new HandlerSubstrateError('TODO: implement list item handler');+}++// (4) An error whose message MENTIONS "TODO" inline but doesn't start with+// `TODO: implement` — the rule's regex is anchored. Must NOT fire.+export function handleTable(_node: unknown): unknown {+ throw new Error('Failed to handle table (TODO: revisit after column-align support lands)');+}diff --git a/biome-plugins/__fixtures__/path-conditional-map-driven-origin.fixture.tsx b/biome-plugins/__fixtures__/path-conditional-map-driven-origin.fixture.tsxnew file mode 100644index 000000000..ee862aa96--- /dev/null+++ b/biome-plugins/__fixtures__/path-conditional-map-driven-origin.fixture.tsx@@ -0,0 +1,97 @@+// FIXTURE — drives `path-conditional-map-driven-origin.test.ts` via shell-out+// to `biome check`. Not part of the main lint (override scope keeps the rule+// off this fixture's path under normal `bun run lint`; the test explicitly+// invokes biome on this file path).+//+// 7 positive cases (deliberate violations — plugin must fire) + 3 negative+// cases (sanctioned shapes that must NOT fire). Exact-equality (`toBe(7)`) in+// the test catches both false-negative regressions (drop below 7) and+// false-positive widenings (above 7).+//+// The rule checks the ORIGIN ARGUMENT POSITION (the second argument), not+// "the call subtree contains the identifier anywhere". Positive7 + Negative3+// are the load-bearing pair: a callback body that mentions OBSERVER_SYNC_ORIGIN+// must NOT clear a call whose origin argument is missing/wrong (Positive7), and+// must NOT trip a call whose origin argument is correct (Negative3).++// biome-ignore lint/correctness/noExplicitAny: fixture-only — production types unimportant here+declare const doc: any;+// biome-ignore lint/correctness/noExplicitAny: fixture-only — production types unimportant here+declare const session: any;+// biome-ignore lint/correctness/noExplicitAny: fixture-only — production types unimportant here+declare const OBSERVER_SYNC_ORIGIN: any;+// biome-ignore lint/correctness/noExplicitAny: fixture-only — production types unimportant here+declare const SOME_OTHER_ORIGIN: any;+// biome-ignore lint/correctness/noExplicitAny: fixture-only — production types unimportant here+declare const current: any;++const noop = (): void => undefined;++// === Positive cases — must fire ===++// (1) Bare `doc.transact(fn)` — no origin at all.+export function Positive1(): void {+ doc.transact(noop);+}++// (2) Bare `session.doc.transact(fn)` via a different receiver — still bare.+export function Positive2(): void {+ session.doc.transact(noop);+}++// (3) 2-arg with a non-sanctioned origin identifier.+export function Positive3(): void {+ doc.transact(noop, SOME_OTHER_ORIGIN);+}++// (4) 2-arg with a string-literal origin (a common shortcut that bypasses the+// typed allowlist).+export function Positive4(): void {+ doc.transact(noop, 'made-up-origin');+}++// (5) 2-arg using `session.origin` — the per-session origin used by non-observer+// paths. Inside observer dispatch only `OBSERVER_SYNC_ORIGIN` is sanctioned,+// so `session.origin` is a violation in observer scope.+export function Positive5(): void {+ doc.transact(noop, session.origin);+}++// (6) 3-arg form `(fn, origin, local)` with a wrong origin in the middle slot.+export function Positive6(): void {+ doc.transact(noop, SOME_OTHER_ORIGIN, true);+}++// (7) Nested-identifier: the callback body mentions OBSERVER_SYNC_ORIGIN but the+// call itself is bare — no origin argument. The old `contains`-based rule+// was falsely cleared by the nested mention; the argument-position rule+// fires because the second argument is absent.+export function Positive7(): void {+ doc.transact(() => {+ if (current === OBSERVER_SYNC_ORIGIN) return;+ noop();+ });+}++// === Negative cases — must NOT fire ===++// (1) Sanctioned origin in the second position.+export function Negative1(): void {+ doc.transact(noop, OBSERVER_SYNC_ORIGIN);+}++// (2) 3-arg form with the sanctioned origin in the second position.+export function Negative2(): void {+ doc.transact(noop, OBSERVER_SYNC_ORIGIN, true);+}++// (3) Nested-identifier WITH the correct origin — mirrors the real+// `server-observers.ts` drain shape: the callback references+// OBSERVER_SYNC_ORIGIN and the second argument is OBSERVER_SYNC_ORIGIN.+// Must NOT fire.+export function Negative3(): void {+ doc.transact(() => {+ if (current === OBSERVER_SYNC_ORIGIN) return;+ noop();+ }, OBSERVER_SYNC_ORIGIN);+}diff --git a/biome-plugins/class-proof-registration-discipline.grit b/biome-plugins/class-proof-registration-discipline.gritnew file mode 100644index 000000000..de27b7735--- /dev/null+++ b/biome-plugins/class-proof-registration-discipline.grit@@ -0,0 +1,53 @@+// class-proof-registration-discipline — class-proof DSL contract enforcement.+//+// `defineClassProof()` (from `packages/md-conformance/src/class-proofs/dsl.ts`)+// registers a class predicate + per-class proof obligation against ts-morph+// reflective enumeration. Two discipline rules:+//+// (a) Registrations live ONLY in+// `packages/md-conformance/src/class-proofs/proofs/<name>.ts`. Authoring+// proofs elsewhere splinters the run-all manifest and breaks+// `runAllClassProofs(engine)`'s single-import contract.+// (b) Every call must pass both `predicate` and `proof` options. TypeScript+// enforces this at compile time via `ClassProofOptions<M>`; this rule+// is the lint-time backstop for the (rare) case where a registration+// is written outside the canonical dir AND escapes type-checking (e.g.+// a `.test.ts` file demoed before the test runs).+//+// Scoped via `overrides[].plugins` to ALL .ts/.tsx files EXCEPT the canonical+// `packages/md-conformance/src/class-proofs/proofs/` directory. Inside the+// canonical dir, neither pattern fires — TypeScript's+// `ClassProofOptions<M>` signature backstops the missing-args check. Outside+// the canonical dir, ANY `defineClassProof` call fires (= "wrong location"),+// and a missing-args call additionally fires a second diagnostic with a+// dedicated message so the author sees both reasons.+//+// Suppress legitimate uses (e.g. a test file that constructs a class-proof+// inline as part of a DSL-machinery test) with:+// // biome-ignore lint/plugin/class-proof-registration-discipline: <reason>++language js++or {+ // Pattern A: missing predicate or proof option. `$opts` is the options+ // bag; `contains` traverses into the object literal to find each key.+ `defineClassProof($_, $opts)` as $call where {+ or {+ not $opts <: contains `predicate: $_`,+ not $opts <: contains `proof: $_`+ },+ register_diagnostic(+ span = $call,+ message = "Class-proof registration missing `predicate` or `proof` option — both are required by `ClassProofOptions<M>`. Provide all three of `{ enumerate, predicate, proof }` to `defineClassProof()`. See https://github.com/inkeep/open-knowledge-legacy/blob/main/biome-plugins/README.md#class-proof-registration-disciplinegrit"+ )+ },+ // Pattern B: any `defineClassProof` call at all. The override scope+ // excludes the canonical proofs/ dir, so this fires only on registrations+ // outside the sanctioned location.+ `defineClassProof($_, $_)` as $call where {+ register_diagnostic(+ span = $call,+ message = "Class-proof registered outside the canonical dir. Move the `defineClassProof(...)` call to `packages/md-conformance/src/class-proofs/proofs/<name>.ts` and export it from `run-all.ts` so it participates in the orchestrator manifest. See https://github.com/inkeep/open-knowledge-legacy/blob/main/biome-plugins/README.md#class-proof-registration-disciplinegrit"+ )+ }+}diff --git a/biome-plugins/cst-pm-handler-todo-stub.grit b/biome-plugins/cst-pm-handler-todo-stub.gritnew file mode 100644index 000000000..17ba7184b--- /dev/null+++ b/biome-plugins/cst-pm-handler-todo-stub.grit@@ -0,0 +1,34 @@+// cst-pm-handler-todo-stub - codemod handler TODO-stub grep.+//+// The substrate-adapter codemod seeds a per-substrate handler scaffold by+// emitting one stub file per PM node type, each whose body is+// `throw new Error("TODO: implement <substrate>:<dir>/<key>")`. This rule is a+// TODO-stub grep: it flags any `throw new Error` whose message starts with+// `TODO: implement ` inside a handler-directory file, so a codemod stub that+// survives to lint time is caught.+//+// It is NOT an exhaustiveness check. Whether a substrate covers every PM node+// type (the node-set traversal over `packages/core/schema-snapshot.json`) is a+// separate concern: the TypeScript handlers-table type check against+// `ICstEngine` is the compile-time backstop for missing methods, and the+// per-substrate node-set traversal belongs to a suite-self-consistency gate.+//+// Scoped via `overrides[].plugins` to+// `packages/md-conformance/src/substrates/*/handlers/**/*.ts` (and the fixture).+// Out of scope: the codemod implementation itself, harness tests, oracles - all+// of which legitimately throw error tags that include the literal string+// `TODO: implement` (e.g. an oracle producing a synthetic failure reason).+//+// Suppress legitimate uses (a handler that legitimately throws "TODO" pending+// upstream work) with:+// // biome-ignore lint/plugin/cst-pm-handler-todo-stub: <reason>++language js++`throw new Error($msg)` as $stmt where {+ $msg <: r"^['\"`]TODO: implement.*",+ register_diagnostic(+ span = $stmt,+ message = "Codemod-emitted handler stub still throws TODO marker - fill in the substrate-specific body so the handler is implemented per the ICstEngine contract. See https://github.com/inkeep/open-knowledge-legacy/blob/main/biome-plugins/README.md#cst-pm-handler-todo-stubgrit"+ )+}diff --git a/biome-plugins/path-conditional-map-driven-origin.grit b/biome-plugins/path-conditional-map-driven-origin.gritnew file mode 100644index 000000000..054f7ea5e--- /dev/null+++ b/biome-plugins/path-conditional-map-driven-origin.grit@@ -0,0 +1,37 @@+// path-conditional-map-driven-origin — Observer A origin discipline.+//+// Inside the server-side observer module, every `Y.Doc.transact()` call MUST+// pass the sanctioned origin `OBSERVER_SYNC_ORIGIN` as its SECOND argument:+//+// doc.transact(() => { ... }, OBSERVER_SYNC_ORIGIN);+//+// Bare `doc.transact(fn)` — no origin at all — silently classifies the write as+// `openknowledge-service` (precedent #25) and breaks per-session UndoManager+// attribution (the `trackedOrigins` Set-identity match skips the txn). A wrong+// origin in the second position is the same failure with a different cause.+//+// The rule checks the ARGUMENT POSITION (the second argument), not whether the+// call subtree contains the identifier somewhere. A callback body that mentions+// OBSERVER_SYNC_ORIGIN for an unrelated reason therefore neither clears a+// bare/wrong call nor is required to — only the second-argument node decides.+//+// Scoped via `overrides[].plugins` to `packages/server/src/server-observers.ts`+// (and the plugin fixture). The file's transact-call surface is the entire+// observer dispatch spine — every call site there is observer-cross-CRDT and+// must honor the discipline. Other server files routing through+// `session.dc.document.transact(fn, session.origin)` are out of scope (that+// contract is enforced by `paired-write-enforcement.test.ts`).+//+// Suppress legitimate exceptions (none expected today) with:+// // biome-ignore lint/plugin/path-conditional-map-driven-origin: <reason>++language js++`$_.transact($...)` as $call where {+ not $call <: `$_.transact($_, OBSERVER_SYNC_ORIGIN)`,+ not $call <: `$_.transact($_, OBSERVER_SYNC_ORIGIN, $...)`,+ register_diagnostic(+ span = $call,+ message = "Observer-side transact call missing sanctioned origin. Pass `OBSERVER_SYNC_ORIGIN` as the second argument: `doc.transact(fn, OBSERVER_SYNC_ORIGIN)`. Bare `doc.transact(fn)` (or a wrong origin) routes the write to `openknowledge-service` and breaks per-session UndoManager attribution. See https://github.com/inkeep/open-knowledge-legacy/blob/main/biome-plugins/README.md#path-conditional-map-driven-origingrit"+ )+}diff --git a/biome.jsonc b/biome.jsoncindex 09ff7b6ca..04258ddf4 100644--- a/biome.jsonc+++ b/biome.jsonc@@ -54,6 +54,63 @@}},"overrides": [+ {+ // Observer A origin discipline. Scoped to the one file that owns the+ // observer-cross-CRDT transact spine. Inside this file every+ // `doc.transact(fn, origin)` must pass `OBSERVER_SYNC_ORIGIN` as the+ // second argument; bare `doc.transact(fn)` (or a wrong origin) routes the+ // write to `openknowledge-service` and breaks per-session UndoManager+ // attribution. Other server files routing through+ // `session.dc.document.transact(fn, session.origin)` are out of scope -+ // that contract is enforced by `paired-write-enforcement.test.ts`.+ "includes": [+ "packages/server/src/server-observers.ts",+ "biome-plugins/__fixtures__/path-conditional-map-driven-origin.fixture.tsx"+ ],+ "plugins": ["./biome-plugins/path-conditional-map-driven-origin.grit"]+ },+ {+ // Codemod handler TODO-stub grep. The substrate-adapter codemod seeds+ // per-substrate handler scaffolds under+ // packages/md-conformance/src/substrates/<id>/handlers/{mdast-to-pm,+ // pm-to-mdast-node,pm-to-mdast-mark}/<key>.ts, each containing+ // `throw new Error("TODO: implement <substrate>:<dir>/<key>")`. This rule+ // greps for stubs that survive to lint time so an unfinished handler+ // can't pass review. It is NOT an exhaustiveness check: the per-substrate+ // node-set traversal belongs to a suite-self-consistency gate, and the+ // handlers-table type check against ICstEngine is the compile-time+ // backstop for missing methods. Scoped narrowly to the handlers/ subtree+ // under any substrate dir, so the codemod implementation + harness tests ++ // oracles (which legitimately produce strings containing "TODO: implement"+ // in their own contexts) are out of scope.+ "includes": [+ "packages/md-conformance/src/substrates/*/handlers/**/*.ts",+ "biome-plugins/__fixtures__/cst-pm-handler-todo-stub.fixture.tsx"+ ],+ "plugins": ["./biome-plugins/cst-pm-handler-todo-stub.grit"]+ },+ {+ // Class-proof DSL registration discipline. Per the dsl.ts contract+ // (`defineClassProof` from packages/md-conformance/src/class-proofs/dsl.ts),+ // class-proof registrations MUST live in the canonical proofs/ dir so+ // run-all.ts's manifest stays the single source of truth, and EVERY+ // registration MUST pass `predicate` + `proof` options. The plugin's+ // override scope EXCLUDES the canonical proofs/ dir — inside the dir,+ // TypeScript's `ClassProofOptions<M>` signature is the missing-args+ // backstop. Outside the dir, any `defineClassProof(...)` call fires+ // (= wrong location), and missing-args fires an additional diagnostic.+ "includes": [+ "**/*.ts",+ "**/*.tsx",+ "!**/node_modules/**",+ "!**/dist/**",+ "!**/*.test.ts",+ "!**/*.test.tsx",+ "!packages/md-conformance/src/class-proofs/proofs/**",+ "biome-plugins/__fixtures__/class-proof-registration-discipline.fixture.tsx"+ ],+ "plugins": ["./biome-plugins/class-proof-registration-discipline.grit"]+ },{// Apple-Event / IPC roundtrip toPass budget enforcement is scoped to// the desktop-smoke files (and the plugin's own fixture for testing).@@ -191,7 +248,26 @@"!**/tests/perf/results","!**/tests/perf/playwright-results","!packages/desktop/out",- "!packages/desktop/dist-desktop"+ "!packages/desktop/dist-desktop",+ // The md-audit framework was folded under packages/md-conformance/md-audit/+ // but is a self-contained unit with its own tsconfig + conventions; it was+ // never OK-biome-formatted (it lived outside the lint scan targets before+ // the fold) and ships a generated docs/byte-preservation-audit.json. Its+ // own typecheck + tests run via `bun --cwd packages/md-conformance/md-audit`+ // and the md-conformance test task; keep it out of OK's biome format scan.+ "!packages/md-conformance/md-audit",+ // Mechanically generated by scripts/enumerate-conformance.ts (JSON.stringify);+ // its byte format is owned by the generator, not biome (biome would collapse+ // the short per-entry testTier arrays the generator emits expanded). Content+ // correctness is enforced by check-conformance-fresh.ts (sha256 pin), so the+ // format scan adds nothing. Path is copybara-excluded, like the md-audit line.+ "!packages/md-conformance/conformance-catalog.json",+ // Mechanically extracted by src/corpora/authoring/extract-gfm-extensions.ts+ // (JSON.stringify); its byte format is owned by the extractor, not biome+ // (biome would collapse the single-element `extensions` arrays the extractor+ // emits expanded). Content correctness is enforced by the re-extraction+ // freshness gate (gfm-extensions-freshness.test.ts). Path is copybara-excluded.+ "!packages/md-conformance/corpora/gfm-extensions.json"]}}diff --git a/bun.lock b/bun.lockindex ad102179e..8ca39f0e0 100644--- a/bun.lock+++ b/bun.lock@@ -249,15 +249,20 @@"@tiptap/extension-image": "^3.22.3","@tiptap/extension-italic": "^3.22.3","@tiptap/extension-link": "^3.22.3",+ "@tiptap/extension-strike": "^3.22.3","@tiptap/extension-table": "^3.22.3","@tiptap/pm": "^3.22.3","@tiptap/starter-kit": "^3.22.3",+ "decode-named-character-reference": "^1.3.0","diff-match-patch": "^1.0.5","mdast-util-from-markdown": "^2.0.3","mdast-util-math": "^3.0.0","mdast-util-mdx": "3.0.0","mdast-util-to-markdown": "^2.1.2","micromark-extension-mdx": "2.1.0",+ "micromark-util-classify-character": "^2.0.1",+ "micromark-util-decode-numeric-character-reference": "^2.0.2",+ "micromark-util-sanitize-uri": "^2.0.1","micromark-util-types": "^2.0.2","node-diff3": "^3.2.0","rehype-parse": "^9.0.1",@@ -295,6 +300,8 @@"culori": "^4.0.2","fast-check": "^4.6.0","mdast-util-to-hast": "^13.2.1",+ "micromark": "^4.0.2",+ "micromark-util-symbol": "^2.0.1","tsdown": "^0.21.7","typescript": "^5.9.3",},@@ -373,6 +380,7 @@"zod": "^4.3.6",},"devDependencies": {+ "@types/mdast": "^4.0.4","@types/shell-quote": "^1.7.5","@types/ws": "^8.18.1","@types/yazl": "^2.4.6",@@ -1734,7 +1742,7 @@"@types/ms": ["@types/[email protected]", "", {}, "sha512-GsCCIZDE/p3i96vtEqx+7dBUGXrc7zeSK3wwPHIaRThS+9OhWIXRqzs4d6k1SVU8g91DrNRWxWUGhp5KXQb2VA=="],- "@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],+ "@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],"@types/node-fetch": ["@types/[email protected]", "", { "dependencies": { "@types/node": "*", "form-data": "^4.0.4" } }, "sha512-QGpRVpzSaUs30JBSGPjOg4Uveu384erbHBoT1zeONvyCfwQxIkUshLAOqN/k9EjGviPRmWTTe6aH2qySWKTVSw=="],@@ -3954,7 +3962,7 @@"undici": ["[email protected]", "", {}, "sha512-xXnp4kTyor2Zq+J1FfPI6Eq3ew5h6Vl0F/8d9XU5zZQf1tX9s2Su1/3PiMmUANFULpmksxkClamIZcaUqryHsQ=="],- "undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],+ "undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],"unicorn-magic": ["[email protected]", "", {}, "sha512-+QBBXBCvifc56fsbuxZQ6Sic3wqqc3WWaqxs58gvJrcOuN83HGTCwz3oS5phzU9LthRNE9VrJCFCLUgHeeFnfA=="],@@ -4204,7 +4212,11 @@"@inkeep/cxkit-styled/tailwind-merge": ["[email protected]", "", {}, "sha512-P+Vu1qXfzediirmHOC3xKGAYeZtPcV9g76X+xg2FD4tYgR71ewMA35Y3sCz3zhiN/dwefRpJX0yBcgwi1fXNQA=="],- "@inkeep/open-knowledge-core/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],+ "@inkeep/open-knowledge/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],++ "@inkeep/open-knowledge-desktop/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],++ "@inkeep/open-knowledge-docs/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],"@inkeep/open-knowledge-docs/lucide-react": ["[email protected]", "", { "peerDependencies": { "react": "^16.5.1 || ^17.0.0 || ^18.0.0 || ^19.0.0" } }, "sha512-HGGkdlPWQ0vTF8jJ5TdIqhQXZi6uh3LnNgfZ8MHiuxFfX3RZeA79r2MW2tHAZKlAVfoNE8esm3p+O6VkIvpj6w=="],@@ -4230,8 +4242,6 @@"@isaacs/cliui/wrap-ansi": ["[email protected]", "", { "dependencies": { "ansi-styles": "^6.1.0", "string-width": "^5.0.1", "strip-ansi": "^7.0.1" } }, "sha512-si7QWI6zUMq56bESFvagtmzMdGOtoxfR+Sez11Mobfc7tm+VkUckk9bW2UeffTGVUbOksxmSw0AA2gs8g71NCQ=="],- "@jest/types/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],-"@lingui/cli/jiti": ["[email protected]", "", { "bin": { "jiti": "lib/jiti-cli.mjs" } }, "sha512-ekilCSN1jwRvIbgeg/57YFh8qQDNbwDb9xT/qu2DAHbFFZUicIl4ygVaAvzveMhMVr3LnpSKTNnwt8PoOfmKhQ=="],"@lingui/conf/jiti": ["[email protected]", "", { "bin": { "jiti": "lib/jiti-cli.mjs" } }, "sha512-ekilCSN1jwRvIbgeg/57YFh8qQDNbwDb9xT/qu2DAHbFFZUicIl4ygVaAvzveMhMVr3LnpSKTNnwt8PoOfmKhQ=="],@@ -4318,13 +4328,23 @@"@tiptap/starter-kit/@tiptap/pm": ["@tiptap/[email protected]", "", { "dependencies": { "prosemirror-changeset": "^2.3.0", "prosemirror-collab": "^1.3.1", "prosemirror-commands": "^1.6.2", "prosemirror-dropcursor": "^1.8.1", "prosemirror-gapcursor": "^1.3.2", "prosemirror-history": "^1.4.1", "prosemirror-inputrules": "^1.4.0", "prosemirror-keymap": "^1.2.2", "prosemirror-markdown": "^1.13.1", "prosemirror-menu": "^1.2.4", "prosemirror-model": "^1.24.1", "prosemirror-schema-basic": "^1.2.3", "prosemirror-schema-list": "^1.5.0", "prosemirror-state": "^1.4.3", "prosemirror-tables": "^1.6.4", "prosemirror-trailing-node": "^3.0.0", "prosemirror-transform": "^1.10.2", "prosemirror-view": "^1.38.1" } }, "sha512-NjfWjZuvrqmpICT+GZWNIjtOdhPyqFKDMtQy7tsQ5rErM9L2ZQdy/+T/BKSO1JdTeBhdg9OP+0yfsqoYp2aT6A=="],- "@types/jsdom/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],+ "@types/busboy/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],++ "@types/cacheable-request/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],++ "@types/fs-extra/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],"@types/jsdom/undici-types": ["[email protected]", "", {}, "sha512-AXNgS1Byr27fTI+2bsPEkV9CxkT8H6xNyRI68b3TatlZo3RkzlqQBLL+w7SmGPVpokjHbcuNVQUWE7FRTg+LRA=="],- "@types/node-fetch/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],+ "@types/keyv/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],- "@types/yazl/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],+ "@types/plist/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],++ "@types/responselike/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],++ "@types/ws/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],++ "@types/yauzl/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],"ajv-keywords/ajv": ["[email protected]", "", { "dependencies": { "fast-deep-equal": "^3.1.1", "fast-json-stable-stringify": "^2.0.0", "json-schema-traverse": "^0.4.1", "uri-js": "^4.2.2" } }, "sha512-IWrosm/yrn43eiKqkfkHis7QioDleaXQHdDVPKg0FSwwd/DuvyX79TZnFOnYpB7dcsFAMmtFztZuXPDvSePkFw=="],@@ -4348,8 +4368,6 @@"builder-util/fs-extra": ["[email protected]", "", { "dependencies": { "graceful-fs": "^4.2.0", "jsonfile": "^6.0.1", "universalify": "^2.0.0" } }, "sha512-oRXApq54ETRj4eMiFzGnHWGy+zo5raudjuxN0b8H7s/RU2oW0Wvsx9O0ACRN/kRq9E8Vu/ReskGB5o3ji+FzHQ=="],- "bun-types/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],-"cacache/glob": ["[email protected]", "", { "dependencies": { "foreground-child": "^3.1.0", "jackspeak": "^3.1.2", "minimatch": "^9.0.4", "minipass": "^7.1.2", "package-json-from-dist": "^1.0.0", "path-scurry": "^1.11.1" }, "bin": { "glob": "dist/esm/bin.mjs" } }, "sha512-DfXN8DfhJ7NH3Oe7cFmu3NCu1wKbkReJ8TorzSAFbSKrlNaQSKfIzqYqVY8zlbs2NLBbWpRiU52GX2PbaBVNkg=="],"cacache/lru-cache": ["[email protected]", "", {}, "sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ=="],@@ -4388,6 +4406,8 @@"dotenv-expand/dotenv": ["[email protected]", "", {}, "sha512-uBq4egWHTcTt33a72vpSG0z3HnPuIl6NqYcTrKEg2azoEyl2hpW0zqlxysq2pK9HlDIHyHyakeYaYnSAwd8bow=="],+ "electron/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],+"electron-builder/fs-extra": ["[email protected]", "", { "dependencies": { "graceful-fs": "^4.2.0", "jsonfile": "^6.0.1", "universalify": "^2.0.0" } }, "sha512-oRXApq54ETRj4eMiFzGnHWGy+zo5raudjuxN0b8H7s/RU2oW0Wvsx9O0ACRN/kRq9E8Vu/ReskGB5o3ji+FzHQ=="],"electron-publish/fs-extra": ["[email protected]", "", { "dependencies": { "graceful-fs": "^4.2.0", "jsonfile": "^6.0.1", "universalify": "^2.0.0" } }, "sha512-oRXApq54ETRj4eMiFzGnHWGy+zo5raudjuxN0b8H7s/RU2oW0Wvsx9O0ACRN/kRq9E8Vu/ReskGB5o3ji+FzHQ=="],@@ -4512,8 +4532,6 @@"prosemirror-trailing-node/escape-string-regexp": ["[email protected]", "", {}, "sha512-TtpcNJ3XAzx3Gq8sWRzJaVajRs0uVxA2YAkdb1jm2YkPz4G6egUFAyA3n5vtEIZefPk5Wa4UXbKuS5fKkJWdgA=="],- "protobufjs/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],-"proxy-addr/ipaddr.js": ["[email protected]", "", {}, "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g=="],"proxy-agent/lru-cache": ["[email protected]", "", {}, "sha512-jumlc0BIUrS3qJGgIkWZsyfAM7NCWiBcCDhnd+3NNM5KbBmLTgHVfWBcg6W+rLUsIpzpERPsvwUP7CckAQSOoA=="],@@ -4728,9 +4746,11 @@"@inkeep/cxkit-primitives/@radix-ui/react-tooltip/@radix-ui/react-visually-hidden": ["@radix-ui/[email protected]", "", { "dependencies": { "@radix-ui/react-primitive": "2.0.1" }, "peerDependencies": { "@types/react": "*", "@types/react-dom": "*", "react": "^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc", "react-dom": "^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc" }, "optionalPeers": ["@types/react", "@types/react-dom"] }, "sha512-vVfA2IZ9q/J+gEamvj761Oq1FpWgCDaNOOIfbPVp2MVPLEomUr5+Vf7kJGwQ24YxZSlQVar7Bes8kyTo5Dshpg=="],- "@inkeep/open-knowledge-core/@types/node/undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],+ "@inkeep/open-knowledge-desktop/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],- "@inkeep/open-knowledge-server/@types/yazl/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],+ "@inkeep/open-knowledge-docs/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],++ "@inkeep/open-knowledge/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],"@inquirer/confirm/@inquirer/core/@inquirer/ansi": ["@inquirer/[email protected]", "", {}, "sha512-S8qNSZiYzFd0wAcyG5AXCvUHC5Sr7xpZ9wZ2py9XR88jUz8wooStVx5M6dRzczbBWjic9NP7+rY0Xi7qqK/aMQ=="],@@ -4748,8 +4768,6 @@"@isaacs/cliui/wrap-ansi/ansi-styles": ["[email protected]", "", {}, "sha512-4Dj6M28JB+oAH8kFkTLUo+a2jwOFkuqb3yucU0CANcRRUbxS0cP0nZYCGjcc3BNXwRIsUVmDGgzawme7zvJHvg=="],- "@jest/types/@types/node/undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],-"@malept/flatpak-bundler/fs-extra/jsonfile": ["[email protected]", "", { "dependencies": { "universalify": "^2.0.0" }, "optionalDependencies": { "graceful-fs": "^4.1.6" } }, "sha512-FGuPw30AdOIUTRMC2OMRtQV+jkVj2cfPqSeWXv1NEAJ1qZ5zb1X6z1mFhbfOB/iy3ssJCD+3KuZ8r8C3uVFlAg=="],"@malept/flatpak-bundler/fs-extra/universalify": ["[email protected]", "", {}, "sha512-gptHNQghINnc/vTGIk0SOFGFNXw7JVrlRUtConJRlvaw6DuX0wO5Jeko9sWrMBhh+PsYAZ7oXAiOnf/UKogyiw=="],@@ -4760,11 +4778,21 @@"@opentelemetry/instrumentation-fetch/@opentelemetry/sdk-trace-web/@opentelemetry/sdk-trace-base": ["@opentelemetry/[email protected]", "", { "dependencies": { "@opentelemetry/core": "2.1.0", "@opentelemetry/resources": "2.1.0", "@opentelemetry/semantic-conventions": "^1.29.0" }, "peerDependencies": { "@opentelemetry/api": ">=1.3.0 <1.10.0" } }, "sha512-uTX9FBlVQm4S2gVQO1sb5qyBLq/FPjbp+tmGoxu4tIgtYGmBYB44+KX/725RFDe30yBSaA9Ml9fqphe1hbUyLQ=="],- "@types/jsdom/@types/node/undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],+ "@types/busboy/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],++ "@types/cacheable-request/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],++ "@types/fs-extra/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],++ "@types/keyv/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],++ "@types/plist/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],- "@types/node-fetch/@types/node/undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],+ "@types/responselike/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],- "@types/yazl/@types/node/undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],+ "@types/ws/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],++ "@types/yauzl/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],"ajv-keywords/ajv/json-schema-traverse": ["[email protected]", "", {}, "sha512-xbbCH5dCYU5T8LcEhhuh7HJ88HXuW3qsI3Y0zOZFKfZEHcpWiHU/Jxzk629Brsab/mMiHQti9wMP+845RPe3Vg=="],@@ -4788,8 +4816,6 @@"builder-util/fs-extra/universalify": ["[email protected]", "", {}, "sha512-gptHNQghINnc/vTGIk0SOFGFNXw7JVrlRUtConJRlvaw6DuX0wO5Jeko9sWrMBhh+PsYAZ7oXAiOnf/UKogyiw=="],- "bun-types/@types/node/undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],-"cacache/glob/minimatch": ["[email protected]", "", { "dependencies": { "brace-expansion": "^2.0.2" } }, "sha512-OBwBN9AL4dqmETlpS2zasx+vTeWclWzkblfZk7KTA5j3jeOONz/tRCnZomUyvNg83wL5Zv9Ss6HMJXAgL8R2Yg=="],"cli-truncate/string-width/strip-ansi": ["[email protected]", "", { "dependencies": { "ansi-regex": "^6.2.2" } }, "sha512-yDPMNjp4WyfYBkHnjIRLfca1i6KMyGCtsVgoKe/z1+6vukgaENdgGBZt+ZmKPc4gavvEZ5OgHfHdrazhgNyG7w=="],@@ -4818,6 +4844,8 @@"electron-updater/fs-extra/universalify": ["[email protected]", "", {}, "sha512-gptHNQghINnc/vTGIk0SOFGFNXw7JVrlRUtConJRlvaw6DuX0wO5Jeko9sWrMBhh+PsYAZ7oXAiOnf/UKogyiw=="],+ "electron/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],+"filelist/minimatch/brace-expansion": ["[email protected]", "", { "dependencies": { "balanced-match": "^1.0.0" } }, "sha512-TN1kCZAgdgweJhWWpgKYrQaMNHcDULHkWwQIspdtjV4Y5aurRdZpjAqn6yX3FPqTA9ngHCc4hJxMAMgGfve85w=="],"form-data/mime-types/mime-db": ["[email protected]", "", {}, "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg=="],@@ -4914,8 +4942,6 @@"prebuild-install/tar-fs/tar-stream": ["[email protected]", "", { "dependencies": { "bl": "^4.0.3", "end-of-stream": "^1.4.1", "fs-constants": "^1.0.0", "inherits": "^2.0.3", "readable-stream": "^3.1.1" } }, "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ=="],- "protobufjs/@types/node/undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],-"puppeteer-core/chromium-bidi/zod": ["[email protected]", "", {}, "sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ=="],"puppeteer/puppeteer-core/webdriver-bidi-protocol": ["[email protected]", "", {}, "sha512-ARrjNjtWRRs2w4Tk7nqrf2gBI0QXWuOmMCx2hU+1jUt6d00MjMxURrhxhGbrsoiZKJrhTSTzbIrc554iKI10qw=="],@@ -5104,8 +5130,6 @@"@inkeep/cxkit-primitives/@radix-ui/react-tooltip/@radix-ui/react-use-controllable-state/@radix-ui/react-use-callback-ref": ["@radix-ui/[email protected]", "", { "peerDependencies": { "@types/react": "*", "react": "^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc" }, "optionalPeers": ["@types/react"] }, "sha512-CasTfvsy+frcFkbXtSJ2Zu9JHpN8TYKxkgJGWbjiZhFivxaeW7rMeZt7QELGVLaYVfFMsKHjb7Ak0nMEe+2Vfw=="],- "@inkeep/open-knowledge-server/@types/yazl/@types/node/undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],-"@opentelemetry/instrumentation-fetch/@opentelemetry/sdk-trace-web/@opentelemetry/sdk-trace-base/@opentelemetry/resources": ["@opentelemetry/[email protected]", "", { "dependencies": { "@opentelemetry/core": "2.1.0", "@opentelemetry/semantic-conventions": "^1.29.0" }, "peerDependencies": { "@opentelemetry/api": ">=1.3.0 <1.10.0" } }, "sha512-1CJjf3LCvoefUOgegxi8h6r4B/wLSzInyhGP2UmIBYNlo4Qk5CZ73e1eEyWmfXvFtm1ybkmfb2DqWvspsYLrWw=="],"app-builder-lib/@electron/fuses/fs-extra/jsonfile": ["[email protected]", "", { "dependencies": { "universalify": "^2.0.0" }, "optionalDependencies": { "graceful-fs": "^4.1.6" } }, "sha512-FGuPw30AdOIUTRMC2OMRtQV+jkVj2cfPqSeWXv1NEAJ1qZ5zb1X6z1mFhbfOB/iy3ssJCD+3KuZ8r8C3uVFlAg=="],diff --git a/knip.config.ts b/knip.config.tsindex 676e4b77a..58556b66b 100644--- a/knip.config.ts+++ b/knip.config.ts@@ -7,6 +7,7 @@ export default {'husky','@lingui/babel-plugin-lingui-macro','@lingui/format-po',+ 'micromark',],ignoreBinaries: ['printf'],ignoreIssues: {@@ -15,7 +16,7 @@ export default {'docs/source.config.ts': ['exports'],'packages/app/src/editor/extensions/internal-link.ts': ['exports', 'types'],'packages/app/src/editor/clipboard/serialize.ts': ['types'],- '{tech-probes,reports,specs}/**': ['files'],+ '{tech-probes,reports,specs}/**': ['files', 'exports', 'types'],'tests/integration/**': ['files'],'packages/core/src/desktop-bridge.ts': ['files', 'types'],'packages/desktop/src/shared/ipc-events.ts': ['files'],diff --git a/package.json b/package.jsonindex e6b288412..c459fbace 100644--- a/package.json+++ b/package.json@@ -16,7 +16,6 @@"knip": "turbo run build --filter=@inkeep/open-knowledge-server --filter=@inkeep/open-knowledge-core --output-logs=errors-only && knip","notices": "bun scripts/generate-third-party-notices.mjs","schema:dump": "bun packages/core/scripts/dump-schema.ts",- "md-audit:measure-comment-discipline": "bun run --cwd md-audit measure-comment-discipline","build": "turbo run build","build:desktop": "turbo run build:desktop","build:desktop:dir": "turbo run build:desktop:dir",diff --git a/packages/app/tests/fidelity/doc-boundary-fragment-drop.test.ts b/packages/app/tests/fidelity/doc-boundary-fragment-drop.test.tsnew file mode 100644index 000000000..db5fe1228--- /dev/null+++ b/packages/app/tests/fidelity/doc-boundary-fragment-drop.test.ts@@ -0,0 +1,25 @@+import { describe, expect, test } from 'bun:test';+import { MarkdownManager, sharedExtensions } from '@inkeep/open-knowledge-core';+import { getSchema } from '@tiptap/core';+import { updateYFragment, yXmlFragmentToProseMirrorRootNode } from '@tiptap/y-tiptap';+import * as Y from 'yjs';++const mdManager = new MarkdownManager({ extensions: sharedExtensions });+const schema = getSchema(sharedExtensions);++describe('sourceDocBoundary never survives the Y.XmlFragment path', () => {+ test('a captured boundary snapshot resets to the schema default through the fragment round-trip', () => {+ const json = mdManager.parse('\u{FEFF}\n\nHello\n\n\nWorld\n');+ expect(json.attrs?.sourceDocBoundary ?? null).not.toBeNull();++ const doc = new Y.Doc();+ const fragment = doc.getXmlFragment('default');+ const meta = { mapping: new Map(), isOMark: new Map() };+ updateYFragment(doc, fragment, schema.nodeFromJSON(json), meta);++ const back = yXmlFragmentToProseMirrorRootNode(fragment, schema).toJSON();+ expect(back.attrs?.sourceDocBoundary ?? null).toBeNull();+ expect(mdManager.serialize(back)).toBe('Hello\n\nWorld\n');+ doc.destroy();+ });+});diff --git a/packages/app/tests/fidelity/invariant-i5.test.ts b/packages/app/tests/fidelity/invariant-i5.test.tsindex 62d212c29..4f6f391a0 100644--- a/packages/app/tests/fidelity/invariant-i5.test.ts+++ b/packages/app/tests/fidelity/invariant-i5.test.ts@@ -1,5 +1,6 @@import { describe, expect, test } from 'bun:test';import { MarkdownManager, sharedExtensions } from '@inkeep/open-knowledge-core';+import type { JSONContent } from '@tiptap/core';import { getSchema } from '@tiptap/core';import { updateYFragment, yXmlFragmentToProseMirrorRootNode } from '@tiptap/y-tiptap';import * as fc from 'fast-check';@@ -10,14 +11,21 @@ import { NUM_RUNS, normalize } from './helpers';const mdManager = new MarkdownManager({ extensions: sharedExtensions });const schema = getSchema(sharedExtensions);+/** Drop the doc-boundary snapshot so both layers consume the identical+ * fragment-representable doc (the Y path cannot carry doc attrs). */+function stripDocBoundary(json: JSONContent): JSONContent {+ const { sourceDocBoundary: _dropped, ...attrs } = (json.attrs ?? {}) as Record<string, unknown>;+ return { ...json, attrs };+}+function layerA(md: string): string {- return mdManager.serialize(mdManager.parse(md));+ return mdManager.serialize(stripDocBoundary(mdManager.parse(md)));}function layerB(md: string): string {const doc = new Y.Doc();const fragment = doc.getXmlFragment('default');- const json = mdManager.parse(md);+ const json = stripDocBoundary(mdManager.parse(md));const pmNode = schema.nodeFromJSON(json);const meta = { mapping: new Map(), isOMark: new Map() };updateYFragment(doc, fragment, pmNode, meta);diff --git a/packages/app/tests/fidelity/invariant-i8.test.ts b/packages/app/tests/fidelity/invariant-i8.test.tsdeleted file mode 100644index d2e6233a5..000000000--- a/packages/app/tests/fidelity/invariant-i8.test.ts+++ /dev/null@@ -1,134 +0,0 @@-import { describe, test } from 'bun:test';-import { VFileMessage } from '@inkeep/open-knowledge-core';-import * as fc from 'fast-check';-import { block } from './arbitraries';-import { mdManager, NUM_RUNS, PBT_TIMEOUT_MS } from './helpers';--function isExpectedParseError(err: unknown): boolean {- if (err instanceof SyntaxError) return true;- if (err instanceof VFileMessage) return true;- if (err instanceof RangeError && err.message.includes('Invalid content for node')) return true;- return false;-}--function assertNoCrash(input: string): void {- try {- mdManager.parse(input);- } catch (err) {- if (!isExpectedParseError(err)) {- throw err; // Unexpected error type — test fails- }- }-}--const dangerousText = fc- .array(- fc.oneof(- fc.constantFrom(...'abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.,;?!\n'),- fc.constantFrom(...'<>&[]{}:@`#\\*_$~!/()'),- ),- { maxLength: 150 },- )- .map((chars) => chars.join(''));--describe('I8 — crash resistance: parse() never throws unexpected errors', () => {- test(- 'arbitrary prose with dangerous characters',- () => {- fc.assert(- fc.property(dangerousText, (s) => {- assertNoCrash(s);- }),- { numRuns: NUM_RUNS, seed: 42 },- );- },- PBT_TIMEOUT_MS,- );-- test(- 'multi-block documents with dangerous chars adjacent to valid blocks',- () => {- fc.assert(- fc.property(- fc- .tuple(dangerousText, block, dangerousText)- .map(([pre, b, post]) => `${pre}\n\n${b}\n\n${post}`),- (md) => {- assertNoCrash(md);- },- ),- { numRuns: NUM_RUNS, seed: 42 },- );- },- PBT_TIMEOUT_MS,- );-- test('hardcoded regression cases — known-crashworthy inputs', () => {- const mustNotCrash = [- 'text <foo bar',- '<Component',- '<',- 'a<b',- 'a<B>c<D',- '<$special>',- '<_private>',- '<$',- '<_',- '</',- '</foo',- '</Callout',- '<<<merge>>>',- '<<<<<<< HEAD',- '>>>>>>> main',- '<script>alert(1)</script>',- ':::note\nopen directive',- '::leafDirective',- '{',- '{ ',- 'text {',- '{ unclosed',- 'a{b',- '{a',- '{{',- '{{{',- '{a{b',- '{a:b}',- '{a b}',- '{a;b}',- '{if(x)y}',- '{a {b}}',- '{<>}',- '{&}',- '{*}',- '{#}',- '{expression}',- '{/* comment */}',- '{}',- '~:a~',- '*:a*',- '---\n\n---',- '---\n---',- '< < < <',- 'a<b<c<d',- 'end of line <',- '< start of line',- 'mid < dle',- '<div',- '<div class="x"',- '<span style=',- '**bold** and <foo unclosed',- 'if (x < y) { z > w }',- 'a < b && c > d',- '<foo and {bar',- '<Callout>{content}</Callout>',- '',- ' ',- '\n',- '\n\n\n',- ];-- for (const input of mustNotCrash) {- assertNoCrash(input);- }- });-});diff --git a/packages/app/tests/fidelity/invariant-wiki-link-embed-conversion.test.ts b/packages/app/tests/fidelity/invariant-wiki-link-embed-conversion.test.tsindex bd2ce37cb..aae22ae26 100644--- a/packages/app/tests/fidelity/invariant-wiki-link-embed-conversion.test.ts+++ b/packages/app/tests/fidelity/invariant-wiki-link-embed-conversion.test.ts@@ -10,7 +10,6 @@ const nonImageExt = fc.constantFrom('mp4', 'webm', 'mov', 'mp3', 'wav', 'ogg', 'const fileAttachmentExt = fc.constantFrom('pdf', 'zip', 'docx', 'csv');const opaqueExt = fc.constantFrom('xyz', 'qux', 'wat');-const renderableExt = fc.oneof(imageExt, nonImageExt, fileAttachmentExt);const allExts = fc.oneof(imageExt, nonImageExt, fileAttachmentExt, opaqueExt);const anchor = fc.option(fc.stringMatching(/^[a-zA-Z0-9_=-]{1,12}$/), { nil: null });@@ -23,19 +22,16 @@ function buildEmbed(s: string, e: string, a: string | null, l: string | null): sreturn `${out}]]`;}-const renderableEmbedMd = fc- .tuple(stem, renderableExt, anchor, alias)- .map(([s, e, a, l]) => buildEmbed(s, e, a, l));const anyEmbedMd = fc.tuple(stem, allExts, anchor, alias).map(([s, e, a, l]) => buildEmbed(s, e, a, l));describe('wiki-embed conversion invariants — mdManager path (US-010)', () => {test(- 'I1 — parse → serialize is byte-identical for renderable extensions (image + non-image wikiembed)',+ 'I1 — parse → serialize is byte-identical for every extension class (renderable + opaque)',() => {assertAcrossSeeds(- fc.property(renderableEmbedMd, (md) => {+ fc.property(anyEmbedMd, (md) => {const out = normalize(mdRoundTrip(md));expect(out).toBe(md);}),@@ -58,9 +54,9 @@ describe('wiki-embed conversion invariants — mdManager path (US-010)', () => {PBT_TIMEOUT_MS,);- test('truly-opaque wikiembed round-trip normalizes to plain markdown link', () => {- expect(normalize(mdRoundTrip('![[archive.xyz]]'))).toBe('[archive.xyz](archive.xyz)');- expect(normalize(mdRoundTrip('![[archive.xyz|Download]]'))).toBe('[Download](archive.xyz)');+ test('truly-opaque wikiembed round-trips byte-identical (no downcast to plain link)', () => {+ expect(normalize(mdRoundTrip('![[archive.xyz]]'))).toBe('![[archive.xyz]]');+ expect(normalize(mdRoundTrip('![[archive.xyz|Download]]'))).toBe('![[archive.xyz|Download]]');});test('block-context image-extension embed dispatches to jsxComponent(WikiEmbedImage)', () => {@@ -86,14 +82,15 @@ describe('wiki-embed conversion invariants — mdManager path (US-010)', () => {expect(props?.alias).toBe('Draft');});- test('US-013 — truly-opaque extensions dispatch to plain link (no sourceForm)', () => {+ test('truly-opaque extensions dispatch to the wikiembed-tagged link chip, never a PM wikiLinkEmbed node', () => {const json = mdManager.parse('![[archive.xyz]]');const para = json.content?.[0];const text = para?.content?.[0];expect(text?.type).toBe('text');const linkMark = text?.marks?.find((mk) => mk.type === 'link');expect(linkMark).toBeDefined();- expect(linkMark?.attrs?.sourceForm).toBeNull();+ expect(linkMark?.attrs?.sourceForm).toBe('wikiembed');+ expect(linkMark?.attrs?.target).toBe('archive.xyz');expect(linkMark?.attrs?.href).toBe('archive.xyz');});diff --git a/packages/app/tests/fidelity/mark-rename-verification.test.ts b/packages/app/tests/fidelity/mark-rename-verification.test.tsindex 238874169..ea0e239bd 100644--- a/packages/app/tests/fidelity/mark-rename-verification.test.ts+++ b/packages/app/tests/fidelity/mark-rename-verification.test.ts@@ -72,8 +72,8 @@ describe('renamed marks round-trip markdown correctly', () => {expect(normalize(mdRoundTrip('*em*\n'))).toBe(normalize('*em*\n'));});- test('doc-start --- normalizes to *** (NG10)', () => {- expect(normalize(mdRoundTrip('---\n'))).toBe(normalize('***\n'));+ test('doc-start --- round-trips byte-identical (NG10 retired)', () => {+ expect(normalize(mdRoundTrip('---\n'))).toBe(normalize('---\n'));});test('non-doc-start --- preserves authoring form', () => {diff --git a/packages/app/tests/fidelity/ng-pinned.test.ts b/packages/app/tests/fidelity/ng-pinned.test.tsindex 7e34b44dd..a427c62bc 100644--- a/packages/app/tests/fidelity/ng-pinned.test.ts+++ b/packages/app/tests/fidelity/ng-pinned.test.ts@@ -4,18 +4,17 @@ import { loadNgPinnedCases } from '../../../core/src/markdown/fixtures/index.ts'const mdManager = new MarkdownManager({ extensions: sharedExtensions });-describe('NG1 — blank-line count normalization (byte-identity)', () => {- test('four newlines between blocks collapse to one blank line', () => {+describe('NG1 — blank-line count between blocks (byte-identity)', () => {+ test('four newlines between blocks round-trip byte-identical', () => {const input = '# H\n\n\n\nP\n';const output = mdManager.serialize(mdManager.parse(input));- expect(output).toBe('# H\n\nP\n');+ expect(output).toBe('# H\n\n\n\nP\n');});});-describe('NG11 — ignore-typed-only doc triggers ensureNonEmptyDoc synthesis (byte-identity)', () => {- test('yaml frontmatter alone → PM doc has one empty paragraph; serializes to empty string', () => {- const input = '---\ntitle: X\n---\n';- const pm = mdManager.parse(input);+describe('NG11 — empty doc triggers ensureNonEmptyDoc synthesis (byte-identity)', () => {+ test('empty input → PM doc has one empty paragraph; serializes to empty string', () => {+ const pm = mdManager.parse('');expect(pm).toMatchObject({type: 'doc',@@ -28,6 +27,12 @@ describe('NG11 — ignore-typed-only doc triggers ensureNonEmptyDoc synthesis (bconst output = mdManager.serialize(pm);expect(output).toBe('');});++ test('frontmatter-shaped input no longer empties the doc (content stays visible)', () => {+ const pm = mdManager.parse('---\ntitle: X\n---\n');+ const output = mdManager.serialize(pm);+ expect(output).toBe('---\n\ntitle: X\n---\n');+ });});describe('NG12 — edited-node quoting normalization (idempotence probe)', () => {diff --git a/packages/app/tests/integration/disk-authority-reconcile.test.ts b/packages/app/tests/integration/disk-authority-reconcile.test.tsindex 1aa90256d..fbbf276af 100644--- a/packages/app/tests/integration/disk-authority-reconcile.test.ts+++ b/packages/app/tests/integration/disk-authority-reconcile.test.ts@@ -5,9 +5,11 @@ import { join } from 'node:path';import {agentPatch,agentWriteMd,+ createTestClients,createTestServer,pollUntil,readTestDoc,+ type TestClient,type TestServer,} from './test-harness.ts';@@ -104,6 +106,51 @@ describe('PRD-6832 β L1: agent write reconciles a newer out-of-band disk edit',expect(after).toContain('native-body-line'); // out-of-band body edit preserved});+ test('concurrent un-flushed CRDT edit: the L1 wholesale ingest drops it (current behavior, recorded honestly)', async () => {+ server = await createTestServer({ debounce: 300_000, maxDebounce: 600_000 });+ const { port, contentDir } = server;+ const docName = `reconcile-concurrent-${randomUUID()}`;+ const filePath = join(contentDir, `${docName}.md`);++ writeFileSync(filePath, '# Doc\n\nseed-body\n', 'utf-8');++ let clients: TestClient[] = [];+ try {+ clients = await createTestClients(port, { count: 1, docName });+ const client = clients[0];+ if (!client) throw new Error('client setup failed');+ await pollUntil(() => client.ytext.toString().includes('seed-body'));++ client.doc.transact(() => {+ client.ytext.insert(client.ytext.length, '\ncrdt-unflushed-line\n');+ });+ const serverYtext = () =>+ server?.instance.hocuspocus.documents.get(docName)?.getText('source').toString() ?? '';+ await pollUntil(() => serverYtext().includes('crdt-unflushed-line'));+ expect(readTestDoc(contentDir, docName)).not.toContain('crdt-unflushed-line');++ writeFileSync(filePath, '# Doc\n\nseed-body\n\ndisk-oob-line\n', 'utf-8');++ const res = await fetch(`http://localhost:${port}/api/agent-write-md`, {+ method: 'POST',+ headers: { 'Content-Type': 'application/json' },+ body: JSON.stringify({ docName, markdown: 'agent-line\n', position: 'append' }),+ });+ expect(res.status).toBe(200);+ const body = (await res.json()) as { warning?: { kind?: string } };+ expect(body.warning?.kind).toBe('disk-edit-reconciled');++ await pollUntil(() => serverYtext().includes('agent-line'));+ expect(serverYtext()).toContain('disk-oob-line');++ expect(serverYtext()).not.toContain('crdt-unflushed-line');+ } finally {+ for (const c of clients) {+ await c.cleanup();+ }+ }+ });+test('rename: the renamed doc carries the newer out-of-band disk content', async () => {server = await createTestServer({ debounce: 50, maxDebounce: 200 });const { port, contentDir } = server;diff --git a/packages/app/tests/integration/init-load-byte-stable-corpus-coverage.test.ts b/packages/app/tests/integration/init-load-byte-stable-corpus-coverage.test.tsdeleted file mode 100644index 4f38b15f1..000000000--- a/packages/app/tests/integration/init-load-byte-stable-corpus-coverage.test.ts+++ /dev/null@@ -1,126 +0,0 @@-/**- * Mega-combo corpus coverage drift check.- *- * Asserts that the mega-combo entry's `ng[]` field, combined with the- * explicit MEGA_COMBO_EXCLUSIONS set defined here, exhaustively covers- * the canonical `NG_TAXONOMY` from `md-audit/src/data/ng-taxonomy.ts`.- *- * Failure modes this catches:- * - A new NG slug is added to the canonical taxonomy without being- * addressed in this corpus → drift assertion fails.- * - A typo in mega-combo's `ng[]` field references a slug that doesn't- * exist in the canonical taxonomy → second assertion fails.- * - A stale entry remains in MEGA_COMBO_EXCLUSIONS after the canonical- * taxonomy retired the corresponding slug → third assertion fails.- *- * If this test fails because a new canonical NG class was added, the fix is- * one of:- * (a) extend the mega-combo body (in- * `_fixtures/init-load-byte-stable-corpus.ts`) to include a snippet- * exercising the new class, AND add the new slug to mega-combo's- * `ng[]` field;- * (b) add the new slug to MEGA_COMBO_EXCLUSIONS below WITH a one-line- * inline rationale explaining why it cannot compose into a single .md- * (e.g., requires .mdx, structurally no-body, etc.);- * (c) decide the new class isn't relevant to load-without-mutate at all- * (rare — most NG classes are byte-shift behaviors that should be- * absorbed by the markdownSemanticallyUnchanged short-circuit).- *- * NOT a behavioral test of byte-stability — that's- * `init-load-byte-stable.test.ts`. This file is a metadata-consistency- * check between two artifacts.- */--import { describe, expect, test } from 'bun:test';-import { NG_TAXONOMY, type NgFloorId } from '../../../../md-audit/src/data/ng-taxonomy.ts';-import { CORPUS, type CorpusEntry } from './_fixtures/init-load-byte-stable-corpus.ts';--const MEGA_COMBO_EXCLUSIONS = new Set<NgFloorId>([- 'storage-no-html-sanitize', // Storage-policy structural property (HTML pass-through, no sanitization). Not a content shape exercisable via byte-stability — the storage layer accepts whatever bytes the user content emits, no normalization gate. mega-combo's HTML entity content covers entity-ref-preservation, not storage-no-html-sanitize.- 'mdx-jsx-thematic-break', // Requires .mdx — JSX-context `---` parsing is MDX-grammar-only. Covered at integration tier as standalone .mdx entry (ng5-mdx-yaml-in-jsx.mdx).- 'mdx-inline-gfm-block-flatten', // Requires .mdx — block-GFM-inside-inline-JSX flattening is MDX-grammar-only. Covered at integration tier as standalone .mdx entry (ng6-block-inside-jsx.mdx).- 'empty-doc-paragraph-synthesis', // Structurally "no body" — the synthesized-empty-paragraph behavior only fires when the parsed mdast body is empty. Adding body content to mega-combo destroys the property. Covered at integration tier as standalone entry (ng8-frontmatter-only.md).- 'code-fence-quoting-canonicalize', // Edited-node quoting normalization. Not yet reviewed for inclusion in this corpus.- 'jsx-component-single-anchor', // Requires .mdx — jsxComponent is the PM-schema anchor for user JSX components (Note/Card/Callout/...). MDX-grammar-only.- 'jsx-inline-atom', // Requires .mdx — jsxInline is the PM atom for inline-position MDX content (`<Icon />`, `<Mention />`). MDX-grammar-only.- 'setext-heading-contiguous-paragraph', // Taxonomy slot retained for cell-resolver matcher reference. Produces byte-identical output; no load-without-mutate coverage gap to fill.- 'backslash-byte-identity', // Taxonomy slot retained for cell-resolver matcher continuity (position-slice extends CommonMark §2.4 ESCAPABLE_CHARS coverage). Produces byte-identical output; no load-without-mutate coverage gap to fill.-]);--function findMegaCombo(): CorpusEntry {- const entry = CORPUS.find((c) => c.filename === 'mega-combo-8ng.md');- if (!entry) {- throw new Error('CORPUS invariant violated: mega-combo-8ng.md not found');- }- return entry;-}--describe('mega-combo corpus coverage drift check', () => {- test('every slug in the canonical taxonomy is either covered by mega-combo or explicitly excluded', () => {- const megaCombo = findMegaCombo();- const covered = new Set<string>(megaCombo.ng);- const taxonomyIds = NG_TAXONOMY.map((entry) => entry.id);-- const unaddressed = taxonomyIds.filter(- (id) => !covered.has(id) && !(MEGA_COMBO_EXCLUSIONS as Set<string>).has(id),- );-- if (unaddressed.length > 0) {- throw new Error(- `Canonical NG_TAXONOMY contains entries not addressed in mega-combo or MEGA_COMBO_EXCLUSIONS: ${unaddressed.join(', ')}. ` +- `Either extend the mega-combo body to exercise the new class(es) and add the slugs to mega-combo.ng[], ` +- `OR add the slugs to MEGA_COMBO_EXCLUSIONS in init-load-byte-stable-corpus-coverage.test.ts with an inline rationale.`,- );- }-- expect(unaddressed).toEqual([]);- });-- test('every slug in mega-combo.ng[] exists in the canonical taxonomy (no typos)', () => {- const megaCombo = findMegaCombo();- const taxonomyIds = new Set<string>(NG_TAXONOMY.map((entry) => entry.id));-- const unknown = megaCombo.ng.filter((id) => !taxonomyIds.has(id));-- if (unknown.length > 0) {- throw new Error(- `mega-combo.ng[] references slugs that do not exist in the canonical NG_TAXONOMY: ${unknown.join(', ')}. ` +- `Either fix the typo in the corpus fixture, or add the missing entries to md-audit/src/data/ng-taxonomy.ts.`,- );- }-- expect(unknown).toEqual([]);- });-- test('every slug in MEGA_COMBO_EXCLUSIONS exists in the canonical taxonomy (no stale exclusions)', () => {- const taxonomyIds = new Set<string>(NG_TAXONOMY.map((entry) => entry.id));-- const stale = [...MEGA_COMBO_EXCLUSIONS].filter((id) => !taxonomyIds.has(id));-- if (stale.length > 0) {- throw new Error(- `MEGA_COMBO_EXCLUSIONS references slugs that do not exist in the canonical NG_TAXONOMY: ${stale.join(', ')}. ` +- `Either remove the stale exclusion (the canonical taxonomy retired the entry), ` +- `or fix the typo in the exclusion slug.`,- );- }-- expect(stale).toEqual([]);- });-- test('mega-combo.ng[] and MEGA_COMBO_EXCLUSIONS are disjoint (no contradictions)', () => {- const megaCombo = findMegaCombo();- const covered = new Set<string>(megaCombo.ng);-- const overlap = [...MEGA_COMBO_EXCLUSIONS].filter((id) => covered.has(id));-- if (overlap.length > 0) {- throw new Error(- `Contradiction: slug(s) appear in BOTH mega-combo.ng[] AND MEGA_COMBO_EXCLUSIONS: ${overlap.join(', ')}. ` +- `An NG class is either covered by mega-combo or explicitly excluded from it; it cannot be both.`,- );- }-- expect(overlap).toEqual([]);- });-});diff --git a/packages/app/tests/integration/map-driven-observer-a-crdt.test.ts b/packages/app/tests/integration/map-driven-observer-a-crdt.test.tsnew file mode 100644index 000000000..a83f8ad53--- /dev/null+++ b/packages/app/tests/integration/map-driven-observer-a-crdt.test.ts@@ -0,0 +1,98 @@+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';+import { setTimeout as wait } from 'node:timers/promises';+import * as Y from 'yjs';+import {+ assertBridgeInvariant,+ createTestClients,+ createTestServer,+ pollUntil,+ type TestClient,+ type TestServer,+} from './test-harness';++let server: TestServer;++beforeAll(async () => {+ server = await createTestServer();+});++afterAll(async () => {+ await server.cleanup();+});++function appendParagraph(client: TestClient, text: string): void {+ const paragraph = new Y.XmlElement('paragraph');+ const ytext = new Y.XmlText();+ ytext.applyDelta([{ insert: text }]);+ paragraph.insert(0, [ytext]);+ client.fragment.push([paragraph]);+}++async function awaitConvergence(clients: TestClient[], markers: string[]): Promise<void> {+ for (const marker of markers) {+ for (const client of clients) {+ await pollUntil(() => client.ytext.toString().includes(marker), 5000);+ }+ }+ await wait(500);+}++describe('map-driven Observer A — cross-CRDT integration', () => {+ test('(a) two-client concurrent edits converge to byte-identical Y.Text', async () => {+ const clients = await createTestClients(server.port, {+ count: 2,+ perClientOptions: { skipInvariantWatcher: true },+ });+ try {+ appendParagraph(clients[0], 'MAP-DRIVEN-A-WYSIWYG');+ appendParagraph(clients[1], 'MAP-DRIVEN-B-WYSIWYG');++ await awaitConvergence(clients, ['MAP-DRIVEN-A-WYSIWYG', 'MAP-DRIVEN-B-WYSIWYG']);++ const ytexts = clients.map((c) => c.ytext.toString());+ expect(ytexts[1]).toBe(ytexts[0]);++ for (const client of clients) {+ assertBridgeInvariant(client.ytext, client.fragment);+ }+ } finally {+ for (const c of clients) await c.cleanup();+ }+ });++ test('(b) per-replica AC1 ≡ post-convergence AC1 — non-overlapping splices preserve each others bytes', async () => {+ const clients = await createTestClients(server.port, {+ count: 2,+ perClientOptions: { skipInvariantWatcher: true },+ });+ try {+ appendParagraph(clients[0], 'AC1-NONOVERLAP-CLIENT-A');+ await awaitConvergence(clients, ['AC1-NONOVERLAP-CLIENT-A']);++ appendParagraph(clients[0], 'AC1-NONOVERLAP-EDIT-A');+ appendParagraph(clients[1], 'AC1-NONOVERLAP-EDIT-B');++ await awaitConvergence(clients, [+ 'AC1-NONOVERLAP-CLIENT-A',+ 'AC1-NONOVERLAP-EDIT-A',+ 'AC1-NONOVERLAP-EDIT-B',+ ]);++ for (const client of clients) {+ const text = client.ytext.toString();+ expect(text).toContain('AC1-NONOVERLAP-CLIENT-A');+ expect(text).toContain('AC1-NONOVERLAP-EDIT-A');+ expect(text).toContain('AC1-NONOVERLAP-EDIT-B');+ }++ const ytexts = clients.map((c) => c.ytext.toString());+ expect(ytexts[1]).toBe(ytexts[0]);++ for (const client of clients) {+ assertBridgeInvariant(client.ytext, client.fragment);+ }+ } finally {+ for (const c of clients) await c.cleanup();+ }+ });+});diff --git a/packages/app/tests/integration/single-file-mode.test.ts b/packages/app/tests/integration/single-file-mode.test.tsindex 918df632c..1d74930d1 100644--- a/packages/app/tests/integration/single-file-mode.test.ts+++ b/packages/app/tests/integration/single-file-mode.test.ts@@ -10,6 +10,7 @@ import {} from 'node:fs';import { tmpdir } from 'node:os';import { dirname, join } from 'node:path';+import { MarkdownManager, sharedExtensions } from '@inkeep/open-knowledge-core';import * as Y from 'yjs';import { createTestClient, createTestServer, pollUntil, wait } from './test-harness';@@ -154,8 +155,20 @@ describe('single-file mode — write-back (FR3)', () => {});describe('single-file mode — no rewrite on open (FR4 / G8)', () => {- const RAW = '> q\nq2\n';- const CANONICAL = '> q\n> q2\n';+ const RAW = '\tfoo\n';+ const CANONICAL = ' foo\n';++ /** Parse-side tab dropper: reproduces the pre-capture engine's leading-tab+ * expansion (a parse-time loss) while serialize stays the real engine. */+ class TabDroppingManager extends MarkdownManager {+ override parseWithFallback(+ markdown: string,+ opts?: Parameters<MarkdownManager['parseWithFallback']>[1],+ ): ReturnType<MarkdownManager['parseWithFallback']> {+ return super.parseWithFallback(markdown.replace(/^\t/gm, ' '), opts);+ }+ }+ const tabDroppingManager = new TabDroppingManager({ extensions: sharedExtensions });test('a reconciliation to the file’s own canonical form is suppressed; a genuine edit persists', async () => {const contentDir = ephemeralContentDir({ 'notes.md': RAW });@@ -165,6 +178,7 @@ describe('single-file mode — no rewrite on open (FR4 / G8)', () => {keepContentDir: true,singleDocRelPath: 'notes.md',debounce: 100,+ mdManager: tabDroppingManager,});const client = await createTestClient(server.port, 'notes');try {@@ -186,7 +200,12 @@ describe('single-file mode — no rewrite on open (FR4 / G8)', () => {test('CONTRAST: a regular (non-ephemeral) project persists the same canonicalization on open', async () => {const contentDir = ephemeralContentDir({ 'notes.md': RAW });- const server = await createTestServer({ contentDir, keepContentDir: true, debounce: 100 });+ const server = await createTestServer({+ contentDir,+ keepContentDir: true,+ debounce: 100,+ mdManager: tabDroppingManager,+ });const client = await createTestClient(server.port, 'notes');try {setSource(client.ytext, CANONICAL);diff --git a/packages/app/tests/integration/test-harness.ts b/packages/app/tests/integration/test-harness.tsindex 5a2bc074c..31dee4441 100644--- a/packages/app/tests/integration/test-harness.ts+++ b/packages/app/tests/integration/test-harness.ts@@ -88,6 +88,7 @@ export interface CreateTestServerOptions {ephemeral?: boolean;projectDir?: string;singleDocRelPath?: string;+ mdManager?: MarkdownManager;}export async function createTestServer(options: CreateTestServerOptions = {}): Promise<TestServer> {@@ -128,6 +129,7 @@ export async function createTestServer(options: CreateTestServerOptions = {}): PcontentRoot: options.gitEnabled === true ? '.' : undefined,enableTestRoutes: true,localOpCliArgs: options.localOpCliArgs,+ mdManager: options.mdManager,...(ephemeral ? { ephemeral: true, singleDocRelPath: options.singleDocRelPath } : {}),skipStateManifestCheck: true,});diff --git a/packages/app/tests/stress/bridge-convergence.fuzz.test.ts b/packages/app/tests/stress/bridge-convergence.fuzz.test.tsindex feead0e7a..30090ce1c 100644--- a/packages/app/tests/stress/bridge-convergence.fuzz.test.ts+++ b/packages/app/tests/stress/bridge-convergence.fuzz.test.ts@@ -246,6 +246,7 @@ async function applyOp(}case 'external-change': {try {+ writeFileSync(join(server.contentDir, `${docName}.md`), op.newContent, 'utf-8');applyExternalChange(server.instance.hocuspocus, docName, op.newContent);} catch {}break;@@ -649,18 +650,27 @@ describe('bridge-convergence fuzzer (FR-17)', () => {acceptableForPrefix.set(prefix, accepts);}+ const chunkGlueRe = /^M\d+-chunked-/;+ const hasChunkedPaste = ops.some((o) => o.kind === 'chunked-source-paste');+const missingContent: Array<{ clientIdx: number; prefix: string }> = [];for (let ci = 0; ci < clients.length; ci++) {const client = clients[ci];if (!client) continue;- const gotLines = new Set(- client.ytext- .toString()- .split('\n')- .map((l) => l.trimEnd()),- );+ const gotLineList = client.ytext+ .toString()+ .split('\n')+ .map((l) => l.trimEnd());+ const gotLines = new Set(gotLineList);for (const [prefix, accepts] of acceptableForPrefix) {- const matched = [...accepts].some((l) => gotLines.has(l));+ const matched = [...accepts].some(+ (l) =>+ gotLines.has(l) ||+ (hasChunkedPaste &&+ gotLineList.some(+ (line) => line.startsWith(l) && chunkGlueRe.test(line.slice(l.length)),+ )),+ );if (!matched) {missingContent.push({ clientIdx: ci, prefix });}diff --git a/packages/cli/scripts/aggregate-tolerance-class-fires.ts b/packages/cli/scripts/aggregate-tolerance-class-fires.tsnew file mode 100644index 000000000..6acabdbd5--- /dev/null+++ b/packages/cli/scripts/aggregate-tolerance-class-fires.ts@@ -0,0 +1,104 @@+#!/usr/bin/env bun++import { readFileSync } from 'node:fs';++interface FireRecord {+ event: string;+ timestamp: string;+ class: string;+ document: string | null;+ codeUnitPosition: number;+ severity: string;+}++interface ClassStats {+ totalFires: number;+ uniqueDocuments: number;+ documents: Record<string, number>;+ severity: string;+ codeUnitPositions: { min: number; max: number; median: number };+}++function main(): void {+ const logPath = process.argv[2];+ if (!logPath) {+ console.error(+ 'Usage: bun run aggregate-tolerance-class-fires.ts <path-to-tolerance-telemetry.jsonl>',+ );+ process.exit(1);+ }++ const raw = readFileSync(logPath, 'utf-8');+ const lines = raw.split('\n').filter((l) => l.trim().length > 0);+ const records: FireRecord[] = [];++ for (const line of lines) {+ try {+ const parsed = JSON.parse(line) as FireRecord;+ if (parsed.event === 'bridge-tolerance-fire') {+ records.push(parsed);+ }+ } catch {}+ }++ if (records.length === 0) {+ console.log(JSON.stringify({ totalRecords: 0, classes: {}, bySeverity: {} }, null, 2));+ return;+ }++ const byClass = new Map<string, FireRecord[]>();+ const bySeverity = new Map<string, number>();++ for (const r of records) {+ const arr = byClass.get(r.class) ?? [];+ arr.push(r);+ byClass.set(r.class, arr);+ bySeverity.set(r.severity, (bySeverity.get(r.severity) ?? 0) + 1);+ }++ const classStats: Record<string, ClassStats> = {};+ for (const [cls, fires] of byClass) {+ const docCounts = new Map<string, number>();+ const positions: number[] = [];++ for (const f of fires) {+ const doc = f.document ?? '__unknown__';+ docCounts.set(doc, (docCounts.get(doc) ?? 0) + 1);+ if (f.codeUnitPosition >= 0) positions.push(f.codeUnitPosition);+ }++ positions.sort((a, b) => a - b);+ const median = positions.length > 0 ? (positions[Math.floor(positions.length / 2)] ?? -1) : -1;++ classStats[cls] = {+ totalFires: fires.length,+ uniqueDocuments: docCounts.size,+ documents: Object.fromEntries(docCounts),+ severity: fires[0]?.severity,+ codeUnitPositions: {+ min: positions.length > 0 ? (positions[0] ?? -1) : -1,+ max: positions.length > 0 ? (positions[positions.length - 1] ?? -1) : -1,+ median,+ },+ };+ }++ const timeRange =+ records.length > 0+ ? {+ earliest: records.reduce((a, b) => (a.timestamp < b.timestamp ? a : b)).timestamp,+ latest: records.reduce((a, b) => (a.timestamp > b.timestamp ? a : b)).timestamp,+ }+ : null;++ const report = {+ totalRecords: records.length,+ timeRange,+ bySeverity: Object.fromEntries(bySeverity),+ classes: classStats,+ };++ console.log(JSON.stringify(report, null, 2));+}++main();diff --git a/packages/core/package.json b/packages/core/package.jsonindex d508e8aaf..445d06abd 100644--- a/packages/core/package.json+++ b/packages/core/package.json@@ -51,15 +51,20 @@"@tiptap/extension-image": "^3.22.3","@tiptap/extension-italic": "^3.22.3","@tiptap/extension-link": "^3.22.3",+ "@tiptap/extension-strike": "^3.22.3","@tiptap/extension-table": "^3.22.3","@tiptap/pm": "^3.22.3","@tiptap/starter-kit": "^3.22.3",+ "decode-named-character-reference": "^1.3.0","diff-match-patch": "^1.0.5","mdast-util-from-markdown": "^2.0.3","mdast-util-math": "^3.0.0","mdast-util-mdx": "3.0.0","mdast-util-to-markdown": "^2.1.2","micromark-extension-mdx": "2.1.0",+ "micromark-util-classify-character": "^2.0.1",+ "micromark-util-decode-numeric-character-reference": "^2.0.2",+ "micromark-util-sanitize-uri": "^2.0.1","micromark-util-types": "^2.0.2","node-diff3": "^3.2.0","rehype-parse": "^9.0.1",@@ -91,7 +96,8 @@"test:perf:regression": "bun run tests/perf/run-regression-gate.ts","test:health:unit": "bun test tests/health/parse-health-gate.test.ts","test:health": "bun run tests/health/parse-health-gate.ts tests/health/baseline.json",- "regen-schema-snapshot": "bun run scripts/regen-schema-snapshot.ts"+ "regen-schema-snapshot": "bun run scripts/regen-schema-snapshot.ts",+ "gen:guard-flanking-matrix": "OK_UPDATE_GUARD_FLANKING_MATRIX=1 bun test src/markdown/guard-flanking-matrix.test.ts && biome check --write src/markdown/guard-flanking-matrix.json"},"devDependencies": {"@standard-schema/spec": "^1.1.0",@@ -110,6 +116,8 @@"culori": "^4.0.2","fast-check": "^4.6.0","mdast-util-to-hast": "^13.2.1",+ "micromark": "^4.0.2",+ "micromark-util-symbol": "^2.0.1","tsdown": "^0.21.7","typescript": "^5.9.3"}diff --git a/packages/core/schema-snapshot.json b/packages/core/schema-snapshot.jsonindex cfa4a0a80..ff2962da2 100644--- a/packages/core/schema-snapshot.json+++ b/packages/core/schema-snapshot.json@@ -35,6 +35,22 @@"hasDefault": true,"default": null},+ "sourceClosingFenceLength": {+ "hasDefault": true,+ "default": null+ },+ "sourceFenceIndent": {+ "hasDefault": true,+ "default": null+ },+ "sourceIndents": {+ "hasDefault": true,+ "default": null+ },+ "sourceInfoPadding": {+ "hasDefault": true,+ "default": null+ },"sourceStyle": {"hasDefault": true,"default": "fenced"@@ -68,7 +84,12 @@"isolating": false},"doc": {- "attrs": {},+ "attrs": {+ "sourceDocBoundary": {+ "hasDefault": true,+ "default": null+ }+ },"content": "block+","marks": "","group": "",@@ -144,6 +165,14 @@"hasDefault": true,"default": false},+ "sourceInteriorSpacing": {+ "hasDefault": true,+ "default": null+ },+ "sourceLeadingIndent": {+ "hasDefault": true,+ "default": null+ },"sourceTrailingHashes": {"hasDefault": true,"default": null@@ -351,6 +380,22 @@"hasDefault": true,"default": null},+ "sourceCheckboxChar": {+ "hasDefault": true,+ "default": null+ },+ "sourceContinuationIndent": {+ "hasDefault": true,+ "default": null+ },+ "sourceMarkerSpacing": {+ "hasDefault": true,+ "default": null+ },+ "sourceOrdinal": {+ "hasDefault": true,+ "default": null+ },"spread": {"hasDefault": true,"default": false@@ -377,6 +422,10 @@"language": {"hasDefault": true,"default": "latex"+ },+ "sourceDelimiter": {+ "hasDefault": true,+ "default": null}},"content": "",@@ -421,9 +470,17 @@},"table": {"attrs": {+ "sourceAlignmentPadding": {+ "hasDefault": true,+ "default": null+ },"sourceDashCounts": {"hasDefault": true,"default": null+ },+ "sourceOuterPipes": {+ "hasDefault": true,+ "default": null}},"content": "tableRow+",@@ -560,6 +617,18 @@"hasDefault": true,"default": false},+ "sourceAlias": {+ "hasDefault": true,+ "default": null+ },+ "sourceAnchor": {+ "hasDefault": true,+ "default": null+ },+ "sourceTarget": {+ "hasDefault": true,+ "default": null+ },"target": {"hasDefault": true,"default": ""@@ -611,6 +680,10 @@"sourceFenceLength": {"hasDefault": true,"default": 1+ },+ "sourcePadded": {+ "hasDefault": true,+ "default": false}},"excludes": "",@@ -726,7 +799,12 @@"spanning": true},"strike": {- "attrs": {},+ "attrs": {+ "sourceDelimiter": {+ "hasDefault": true,+ "default": "~~"+ }+ },"excludes": "strike","group": "","inclusive": true,@@ -758,11 +836,6 @@"version": "11.0.0","hasOptions": false},- {- "name": "remark-frontmatter",- "version": "5.0.0",- "hasOptions": true- },{"name": "remark-mdx-agnostic","version": null,diff --git a/packages/core/src/bridge/index.ts b/packages/core/src/bridge/index.tsindex 2315275e3..d96ecc3aa 100644--- a/packages/core/src/bridge/index.ts+++ b/packages/core/src/bridge/index.ts@@ -54,3 +54,12 @@ export {normalizeBridge,} from './normalize.ts';export { defaultScheduler, type Scheduler } from './scheduler.ts';+export {+ classifySeverity,+ emitToleranceFire,+ findFirstDivergenceIndex,+ setToleranceTelemetryHook,+ type ToleranceClassSeverity,+ type ToleranceFireRecord,+ type ToleranceTelemetryHook,+} from './tolerance-telemetry.ts';diff --git a/packages/core/src/bridge/normalize.test.ts b/packages/core/src/bridge/normalize.test.tsindex 80587e2f3..22970543d 100644--- a/packages/core/src/bridge/normalize.test.ts+++ b/packages/core/src/bridge/normalize.test.ts@@ -1,4 +1,6 @@import { describe, expect, test } from 'bun:test';+import { sharedExtensions } from '../extensions/shared.ts';+import { MarkdownManager } from '../markdown/index.ts';import {BRIDGE_TOLERANCE_CLASSES,detectAppliedToleranceClasses,@@ -23,7 +25,7 @@ describe('per-line trailing whitespace strip', () => {});});-describe('3+ newline collapse to 2 (NG1 floor)', () => {+describe('3+ newline collapse to 2 (NG1 comparison-only tolerance)', () => {test('three newlines collapse to two', () => {expect(normalizeBridge('P1\n\n\nP2')).toBe(normalizeBridge('P1\n\nP2'));});@@ -63,7 +65,7 @@ describe('trailing newline policy', () => {});});-describe('leading newline policy (NG1 architectural floor at doc-start)', () => {+describe('leading newline policy (NG1 comparison-only tolerance at doc-start)', () => {test('no leading newline', () => {expect(normalizeBridge('P')).toBe(normalizeBridge('P'));});@@ -507,6 +509,7 @@ describe('detectAppliedToleranceClasses (FR-41)', () => {'doc-start-thematic','block-separator-collapse','table-align-row-spacing',+ 'row-no-trailing-pipe','list-indent-canonical','ordered-list-marker-number','trailing-whitespace',@@ -679,3 +682,163 @@ describe('detectAppliedToleranceClasses (FR-41)', () => {}});});++describe('fence-opener tracking (CommonMark 4.5: closer must match opener char)', () => {+ const FENCED_A = '```ts\n~~~\n| a | b |\n| - | - |\n| 1 | 2\n```\n';+ const FENCED_B = '```ts\n~~~\n| a | b |\n| - | - |\n| 1 | 2 |\n```\n';++ test('does not absorb a trailing-pipe divergence inside an interleaved fence', () => {+ expect(normalizeBridge(FENCED_A)).not.toBe(normalizeBridge(FENCED_B));+ });++ test('still tolerates the divergence outside any fence (control)', () => {+ const a = '| a | b |\n| - | - |\n| 1 | 2\n';+ const b = '| a | b |\n| - | - |\n| 1 | 2 |\n';+ expect(normalizeBridge(a)).toBe(normalizeBridge(b));+ });++ test('matching-char closers still close the fence (control)', () => {+ const a = '```ts\ncode\n```\n\n| a | b |\n| - | - |\n| 1 | 2\n';+ const b = '```ts\ncode\n```\n\n| a | b |\n| - | - |\n| 1 | 2 |\n';+ expect(normalizeBridge(a)).toBe(normalizeBridge(b));+ });+});++describe('table-row trailing-pipe tolerance (row-no-trailing-pipe)', () => {+ const mm = new MarkdownManager({ extensions: sharedExtensions });++ const NON_UNIFORM = '# Notes\n\n| a | b |\n| - | - |\n| 1 | 2\n';+ const UNIFORM = '# Notes\n\n| a | b\n| - | -\n| 1 | 2\n';++ function semanticTree(md: string): unknown {+ const strip = (node: unknown): unknown => {+ if (Array.isArray(node)) return node.map(strip);+ if (!node || typeof node !== 'object') return node;+ const out: Record<string, unknown> = {};+ for (const [k, v] of Object.entries(node as Record<string, unknown>)) {+ if (k === 'position' || k === 'data') continue;+ out[k] = strip(v);+ }+ return out;+ };+ return strip(mm.parseToMdast(md));+ }++ function stripTableCaptureAttrs(json: unknown): void {+ if (!json || typeof json !== 'object') return;+ const node = json as { type?: string; attrs?: Record<string, unknown>; content?: unknown[] };+ if (node.type === 'table' && node.attrs) {+ node.attrs.sourceOuterPipes = null;+ }+ for (const child of node.content ?? []) stripTableCaptureAttrs(child);+ }++ test('non-uniform witness ≡ its real serialize(parse) twin (β-proof: bytes differ, comparator equal, parses structurally equal)', () => {+ const canon = mm.serialize(mm.parse(NON_UNIFORM));+ expect(canon).not.toBe(NON_UNIFORM);+ expect(normalizeBridge(NON_UNIFORM)).toBe(normalizeBridge(canon));+ expect(semanticTree(NON_UNIFORM)).toEqual(semanticTree(canon));+ });++ test('uniform witness ≡ the capture-attr-loss canonical twin (every row gains a bare trailing pipe)', () => {+ const stripped = mm.parse(UNIFORM) as unknown as Record<string, unknown>;+ stripTableCaptureAttrs(stripped);+ const canon = mm.serialize(stripped);+ expect(canon).toBe('# Notes\n\n| a | b|\n| - | -|\n| 1 | 2|\n');+ expect(normalizeBridge(UNIFORM)).toBe(normalizeBridge(canon));+ expect(semanticTree(UNIFORM)).toEqual(semanticTree(canon));+ });++ test('padded trailing pipe on a data row tolerated in table context', () => {+ expect(normalizeBridge('| a | b |\n| - | - |\n| 1 | 2 |')).toBe(+ normalizeBridge('| a | b |\n| - | - |\n| 1 | 2'),+ );+ });++ test('delimiter-row trailing pipe tolerated in table context', () => {+ expect(normalizeBridge('| a | b\n| - | -|\n| 1 | 2')).toBe(+ normalizeBridge('| a | b\n| - | -\n| 1 | 2'),+ );+ });++ test('fully piped table ≡ uniformly unpiped-trailing table (per-row class application)', () => {+ expect(normalizeBridge('| a | b |\n| - | - |\n| 1 | 2 |\n| 3 | 4 |')).toBe(+ normalizeBridge('| a | b\n| - | -\n| 1 | 2\n| 3 | 4'),+ );+ });++ test('normalizeBridge stays idempotent with the class active', () => {+ for (const input of [+ NON_UNIFORM,+ UNIFORM,+ mm.serialize(mm.parse(NON_UNIFORM)),+ '| a | b |\n| - | - |\n| 1 | 2 |',+ ]) {+ const once = normalizeBridge(input);+ expect(normalizeBridge(once)).toBe(once);+ }+ });++ test('detectAppliedToleranceClasses reports the class on the witness pair', () => {+ const canon = mm.serialize(mm.parse(NON_UNIFORM));+ expect(detectAppliedToleranceClasses(NON_UNIFORM, canon)).toContain('row-no-trailing-pipe');+ });++ test('detectAppliedToleranceClasses does not report the class on a tableless pair', () => {+ expect(detectAppliedToleranceClasses('plain paragraph', 'plain paragraph\n')).not.toContain(+ 'row-no-trailing-pipe',+ );+ });++ test('BRIDGE_TOLERANCE_CLASSES carries the class label', () => {+ expect(BRIDGE_TOLERANCE_CLASSES).toContain('row-no-trailing-pipe');+ });++ describe('context gating — pipe-leading lines outside a table are untouched', () => {+ test('paragraph with vs without trailing pipe is a genuine divergence', () => {+ expect(normalizeBridge('| foo')).not.toBe(normalizeBridge('| foo |'));+ });++ test('pipe-leading paragraph elsewhere in a table-bearing doc keeps its trailing pipe', () => {+ const withPipe = '| p | q |\n\n| a | b |\n| - | - |\n| 1 | 2 |';+ const withoutPipe = '| p | q\n\n| a | b |\n| - | - |\n| 1 | 2 |';+ expect(normalizeBridge(withPipe)).not.toBe(normalizeBridge(withoutPipe));+ });++ test('table-shaped lines inside a fenced code block are untouched', () => {+ const piped = '```\n| a | b |\n| - | - |\n| 1 | 2 |\n```';+ const unpiped = '```\n| a | b |\n| - | - |\n| 1 | 2\n```';+ expect(normalizeBridge(piped)).not.toBe(normalizeBridge(unpiped));+ });+ });++ describe('guards — genuine non-β-safe divergence is NOT absorbed by the tolerance set', () => {+ test('embed downcast is a genuine divergence', () => {+ expect(normalizeBridge('![[note]]')).not.toBe(normalizeBridge('[note](note)'));+ });++ test('touched-cell edit is a genuine divergence', () => {+ expect(normalizeBridge('| a | b |\n| - | - |\n| 1 | 2 |')).not.toBe(+ normalizeBridge('| a | b |\n| - | - |\n| 1 | 99 |'),+ );+ });++ test('trailing EMPTY cell is a genuine divergence (single strip, not fixpoint)', () => {+ expect(normalizeBridge('| a | b |\n| - | - |\n| 1 | 2 | |')).not.toBe(+ normalizeBridge('| a | b |\n| - | - |\n| 1 | 2 |'),+ );+ });++ test('unescaped content pipe at cell end is a genuine divergence', () => {+ expect(normalizeBridge('| a | b |\n| - | - |\n| 1 | 2| |')).not.toBe(+ normalizeBridge('| a | b |\n| - | - |\n| 1 | 2 |'),+ );+ });++ test('missing LEADING pipe is out of class (genuine divergence)', () => {+ expect(normalizeBridge('| a | b |\n| - | - |\n1 | 2')).not.toBe(+ normalizeBridge('| a | b |\n| - | - |\n| 1 | 2 |'),+ );+ });+ });+});diff --git a/packages/core/src/bridge/normalize.ts b/packages/core/src/bridge/normalize.tsindex 5dea59752..fe758e0d1 100644--- a/packages/core/src/bridge/normalize.ts+++ b/packages/core/src/bridge/normalize.ts@@ -13,8 +13,45 @@ const ORDERED_LIST_MARKER_CANONICAL = '1';const EMPHASIS_AROUND_CODE_RE = /\*\*\s*(`[^`]+`)\s*\*\*/g;+const FENCE_LINE_RE = /^(`{3,}|~{3,})/;++function findTableRowLines(lines: readonly string[]): Set<number> {+ const rows = new Set<number>();+ let fenceOpener: '`' | '~' | null = null;+ for (let i = 0; i < lines.length; i++) {+ const trimmed = lines[i].trimEnd();+ const fence = FENCE_LINE_RE.exec(trimmed);+ if (fence) {+ const char = fence[1][0] as '`' | '~';+ if (fenceOpener === null) {+ fenceOpener = char;+ continue;+ }+ if (fenceOpener === char) {+ fenceOpener = null;+ continue;+ }+ }+ if (fenceOpener !== null || i === 0) continue;+ if (!TABLE_ALIGN_ROW_RE.test(trimmed)) continue;+ const prev = lines[i - 1].trimEnd();+ if (!prev.startsWith('|') || TABLE_ALIGN_ROW_RE.test(prev)) continue;+ rows.add(i - 1);+ rows.add(i);+ for (let j = i + 1; j < lines.length; j++) {+ if (!lines[j].trimEnd().startsWith('|')) break;+ rows.add(j);+ }+ }+ return rows;+}++function stripTrailingPipe(line: string): string {+ return line.slice(0, -1).trimEnd();+}+export function normalizeBridge(s: string): string {- return s+ const lines = s.replace(/^/, '').replace(/\r/g, '').replace(COMMONMARK_ESCAPE_RE, '$1')@@ -23,16 +60,25 @@ export function normalizeBridge(s: string): string {.replace(/^[*-]{3,}(?=\n|$)/, '---').replace(/(\n)([#>+-]|\d+[.)]|`{3,}|~{3,})/g, '\n\n$2').replace(/^([#>+-].*|\d+[.)].*|`{3,}.*|~{3,}.*)\n([^\n])/gm, '$1\n\n$2')- .split('\n')- .map((l) => {+ .split('\n');+ const tableRowLines = findTableRowLines(lines);+ return lines+ .map((l, i) => {const trimmed = l.trimEnd();if (TABLE_ALIGN_ROW_RE.test(trimmed)) {- return trimmed.replace(/\s+/g, '');+ const collapsed = trimmed.replace(/\s+/g, '');+ return tableRowLines.has(i) &&+ collapsed.length > 1 &&+ collapsed.startsWith('|') &&+ collapsed.endsWith('|')+ ? stripTrailingPipe(collapsed)+ : collapsed;}- const deindented = LIST_ITEM_INDENT_RE.test(trimmed)- ? trimmed.replace(/^[ \t]+/, '')- : trimmed;- return deindented.replace(ORDERED_LIST_MARKER_RE, `${ORDERED_LIST_MARKER_CANONICAL}$1`);+ let line = LIST_ITEM_INDENT_RE.test(trimmed) ? trimmed.replace(/^[ \t]+/, '') : trimmed;+ if (tableRowLines.has(i) && line.length > 1 && line.startsWith('|') && line.endsWith('|')) {+ line = stripTrailingPipe(line);+ }+ return line.replace(ORDERED_LIST_MARKER_RE, `${ORDERED_LIST_MARKER_CANONICAL}$1`);}).join('\n').replace(/\n{3,}/g, '\n\n')@@ -48,6 +94,7 @@ export const BRIDGE_TOLERANCE_CLASSES = ['doc-start-thematic','block-separator-collapse','table-align-row-spacing',+ 'row-no-trailing-pipe','list-indent-canonical','ordered-list-marker-number','trailing-whitespace',@@ -112,6 +159,12 @@ export function detectAppliedToleranceClasses(left: string, right: string): Bridclasses.push('table-align-row-spacing');}+ const hasUnterminatedPipeRow = (s: string): boolean =>+ tableAlignRowMultiline.test(s) && /^\|[^\n]*[^|\s][ \t]*$/m.test(s);+ if (hasUnterminatedPipeRow(leftLf) !== hasUnterminatedPipeRow(rightLf)) {+ classes.push('row-no-trailing-pipe');+ }+const listIndentMultiline = /^[ \t]+([-+*]|\d+[.)]|[a-zA-Z][.)])\s/m;if (listIndentMultiline.test(leftLf) || listIndentMultiline.test(rightLf)) {classes.push('list-indent-canonical');diff --git a/packages/core/src/bridge/tolerance-telemetry.test.ts b/packages/core/src/bridge/tolerance-telemetry.test.tsnew file mode 100644index 000000000..99f61bef2--- /dev/null+++ b/packages/core/src/bridge/tolerance-telemetry.test.ts@@ -0,0 +1,149 @@+import { afterEach, describe, expect, test } from 'bun:test';++import { BRIDGE_TOLERANCE_CLASSES, type BridgeToleranceClass } from './normalize.ts';+import {+ classifySeverity,+ emitToleranceFire,+ findFirstDivergenceIndex,+ setToleranceTelemetryHook,+ type ToleranceFireRecord,+} from './tolerance-telemetry.ts';++afterEach(() => {+ setToleranceTelemetryHook(null);+});++describe('classifySeverity', () => {+ const pmModelCaused: BridgeToleranceClass[] = ['emphasis-around-code'];+ const parserCaused: BridgeToleranceClass[] = [+ 'bom',+ 'crlf',+ 'commonmark-escape',+ 'leading-newline',+ 'doc-start-thematic',+ ];+ const serializerCaused: BridgeToleranceClass[] = [+ 'block-separator-collapse',+ 'table-align-row-spacing',+ 'row-no-trailing-pipe',+ 'ordered-list-marker-number',+ 'list-indent-canonical',+ 'trailing-whitespace',+ 'blank-line-collapse',+ 'trailing-newline',+ ];++ test('pm-model-caused classes', () => {+ for (const cls of pmModelCaused) {+ expect(classifySeverity(cls)).toBe('pm-model-caused');+ }+ });++ test('parser-caused classes', () => {+ for (const cls of parserCaused) {+ expect(classifySeverity(cls)).toBe('parser-caused');+ }+ });++ test('serializer-caused classes', () => {+ for (const cls of serializerCaused) {+ expect(classifySeverity(cls)).toBe('serializer-caused');+ }+ });++ test('the severity buckets partition every bridge tolerance class', () => {+ const bucketed = [...pmModelCaused, ...parserCaused, ...serializerCaused];+ expect(new Set(bucketed).size).toBe(bucketed.length);+ expect([...bucketed].sort()).toEqual([...BRIDGE_TOLERANCE_CLASSES].sort());+ });+});++describe('findFirstDivergenceIndex', () => {+ test('identical strings return -1', () => {+ expect(findFirstDivergenceIndex('hello', 'hello')).toBe(-1);+ });++ test('first char differs returns 0', () => {+ expect(findFirstDivergenceIndex('abc', 'xyz')).toBe(0);+ });++ test('middle char differs', () => {+ expect(findFirstDivergenceIndex('abcdef', 'abcXef')).toBe(3);+ });++ test('length difference at end', () => {+ expect(findFirstDivergenceIndex('abc', 'abcdef')).toBe(3);+ });++ test('empty vs non-empty', () => {+ expect(findFirstDivergenceIndex('', 'a')).toBe(0);+ });++ test('both empty returns -1', () => {+ expect(findFirstDivergenceIndex('', '')).toBe(-1);+ });+});++describe('emitToleranceFire', () => {+ test('does nothing when no hook is set', () => {+ emitToleranceFire(['crlf'], 'a\r\n', 'a\n', 'test-doc');+ });++ test('fires hook once per class', () => {+ const records: ToleranceFireRecord[] = [];+ setToleranceTelemetryHook((r) => records.push(r));++ emitToleranceFire(['bom', 'crlf'], 'a\r\n', 'a\n', 'my-doc');++ expect(records).toHaveLength(2);+ expect(records[0]?.className).toBe('bom');+ expect(records[1]?.className).toBe('crlf');+ });++ test('record has all required fields', () => {+ const records: ToleranceFireRecord[] = [];+ setToleranceTelemetryHook((r) => records.push(r));++ emitToleranceFire(['trailing-newline'], 'hello\n', 'hello', 'notes/foo');++ expect(records).toHaveLength(1);+ const r = records[0];+ expect(r).toBeDefined();+ expect(r?.timestamp).toMatch(/^\d{4}-\d{2}-\d{2}T/);+ expect(r?.className).toBe('trailing-newline');+ expect(r?.documentName).toBe('notes/foo');+ expect(r?.codeUnitPosition).toBe(5);+ expect(r?.severity).toBe('serializer-caused');+ });++ test('empty classes array does not fire hook', () => {+ let called = false;+ setToleranceTelemetryHook(() => {+ called = true;+ });++ emitToleranceFire([], 'same', 'same', 'doc');++ expect(called).toBe(false);+ });++ test('undefined documentName passes through', () => {+ const records: ToleranceFireRecord[] = [];+ setToleranceTelemetryHook((r) => records.push(r));++ emitToleranceFire(['crlf'], 'a\r', 'a', undefined);++ expect(records[0]?.documentName).toBeUndefined();+ });++ test('clearing hook stops emissions', () => {+ const records: ToleranceFireRecord[] = [];+ setToleranceTelemetryHook((r) => records.push(r));+ emitToleranceFire(['crlf'], 'a\r', 'a', 'doc');+ expect(records).toHaveLength(1);++ setToleranceTelemetryHook(null);+ emitToleranceFire(['crlf'], 'a\r', 'a', 'doc');+ expect(records).toHaveLength(1);+ });+});diff --git a/packages/core/src/bridge/tolerance-telemetry.ts b/packages/core/src/bridge/tolerance-telemetry.tsnew file mode 100644index 000000000..3fbde92ee--- /dev/null+++ b/packages/core/src/bridge/tolerance-telemetry.ts@@ -0,0 +1,69 @@+import type { BridgeToleranceClass } from './normalize.ts';++export type ToleranceClassSeverity = 'pm-model-caused' | 'parser-caused' | 'serializer-caused';++export interface ToleranceFireRecord {+ timestamp: string;+ className: BridgeToleranceClass;+ documentName: string | undefined;+ codeUnitPosition: number;+ severity: ToleranceClassSeverity;+}++export type ToleranceTelemetryHook = (record: ToleranceFireRecord) => void;++const SEVERITY_BY_CLASS = {+ 'emphasis-around-code': 'pm-model-caused',+ bom: 'parser-caused',+ crlf: 'parser-caused',+ 'commonmark-escape': 'parser-caused',+ 'leading-newline': 'parser-caused',+ 'doc-start-thematic': 'parser-caused',+ 'block-separator-collapse': 'serializer-caused',+ 'table-align-row-spacing': 'serializer-caused',+ 'row-no-trailing-pipe': 'serializer-caused',+ 'ordered-list-marker-number': 'serializer-caused',+ 'list-indent-canonical': 'serializer-caused',+ 'trailing-whitespace': 'serializer-caused',+ 'blank-line-collapse': 'serializer-caused',+ 'trailing-newline': 'serializer-caused',+} as const satisfies Record<BridgeToleranceClass, ToleranceClassSeverity>;++export function classifySeverity(cls: BridgeToleranceClass): ToleranceClassSeverity {+ return SEVERITY_BY_CLASS[cls];+}++export function findFirstDivergenceIndex(left: string, right: string): number {+ const len = Math.min(left.length, right.length);+ for (let i = 0; i < len; i++) {+ if (left.charCodeAt(i) !== right.charCodeAt(i)) return i;+ }+ if (left.length !== right.length) return len;+ return -1;+}++let hook: ToleranceTelemetryHook | null = null;++export function setToleranceTelemetryHook(h: ToleranceTelemetryHook | null): void {+ hook = h;+}++export function emitToleranceFire(+ classes: readonly BridgeToleranceClass[],+ left: string,+ right: string,+ documentName: string | undefined,+): void {+ if (!hook || classes.length === 0) return;+ const codeUnitPosition = findFirstDivergenceIndex(left, right);+ const timestamp = new Date().toISOString();+ for (const cls of classes) {+ hook({+ timestamp,+ className: cls,+ documentName,+ codeUnitPosition,+ severity: classifySeverity(cls),+ });+ }+}diff --git a/packages/core/src/extensions/code-block-fidelity.ts b/packages/core/src/extensions/code-block-fidelity.tsindex 1442f0563..021509e54 100644--- a/packages/core/src/extensions/code-block-fidelity.ts+++ b/packages/core/src/extensions/code-block-fidelity.ts@@ -10,6 +10,10 @@ export const CodeBlockFidelity = CodeBlock.extend({fenceLength: { default: 3 },meta: { default: null },sourceStyle: { default: 'fenced' },+ sourceClosingFenceLength: { default: null, rendered: false },+ sourceFenceIndent: { default: null, rendered: false },+ sourceInfoPadding: { default: null, rendered: false },+ sourceIndents: { default: null, rendered: false },};},});diff --git a/packages/core/src/extensions/code-mark-fidelity.ts b/packages/core/src/extensions/code-mark-fidelity.tsindex 158debba0..21f82352b 100644--- a/packages/core/src/extensions/code-mark-fidelity.ts+++ b/packages/core/src/extensions/code-mark-fidelity.ts@@ -8,6 +8,7 @@ export const CodeMarkFidelity = Code.extend({...this.parent?.(),sourceFenceChar: { default: '`' },sourceFenceLength: { default: 1 },+ sourcePadded: { default: false, rendered: false },};},});diff --git a/packages/core/src/extensions/doc-fidelity.ts b/packages/core/src/extensions/doc-fidelity.tsnew file mode 100644index 000000000..9281ad11a--- /dev/null+++ b/packages/core/src/extensions/doc-fidelity.ts@@ -0,0 +1,16 @@+import { Extension } from '@tiptap/core';++export const DocFidelity = Extension.create({+ name: 'docFidelity',++ addGlobalAttributes() {+ return [+ {+ types: ['doc'],+ attributes: {+ sourceDocBoundary: { default: null, rendered: false },+ },+ },+ ];+ },+});diff --git a/packages/core/src/extensions/heading-fidelity.ts b/packages/core/src/extensions/heading-fidelity.tsindex 6f58479eb..d7efe9a8e 100644--- a/packages/core/src/extensions/heading-fidelity.ts+++ b/packages/core/src/extensions/heading-fidelity.ts@@ -10,6 +10,8 @@ export const HeadingFidelity = Heading.extend({sourceTrailingHashes: { default: null },sourceUnderlineLength: { default: null },sourceContiguousNext: { default: false },+ sourceLeadingIndent: { default: null, rendered: false },+ sourceInteriorSpacing: { default: null, rendered: false },};},});diff --git a/packages/core/src/extensions/list.ts b/packages/core/src/extensions/list.tsindex c892ef982..dca5a9bcc 100644--- a/packages/core/src/extensions/list.ts+++ b/packages/core/src/extensions/list.ts@@ -263,6 +263,10 @@ export const ListItemNode = Node.create({return {checked: { default: null },spread: { default: false },+ sourceMarkerSpacing: { default: null, rendered: false },+ sourceOrdinal: { default: null, rendered: false },+ sourceCheckboxChar: { default: null, rendered: false },+ sourceContinuationIndent: { default: null, rendered: false },};},diff --git a/packages/core/src/extensions/math-inline.ts b/packages/core/src/extensions/math-inline.tsindex d41afc663..66c5772c2 100644--- a/packages/core/src/extensions/math-inline.ts+++ b/packages/core/src/extensions/math-inline.ts@@ -21,6 +21,7 @@ export const MathInline = Node.create({formula: { default: '' },id: { default: null },language: { default: 'latex' },+ sourceDelimiter: { default: null, rendered: false },};},diff --git a/packages/core/src/extensions/shared.ts b/packages/core/src/extensions/shared.tsindex da806a319..0fcce76dd 100644--- a/packages/core/src/extensions/shared.ts+++ b/packages/core/src/extensions/shared.ts@@ -6,6 +6,7 @@ import { CodeBlockFidelity } from './code-block-fidelity.ts';import { CodeMarkFidelity } from './code-mark-fidelity.ts';import { CommentBlock } from './comment-block.ts';import { CommentMark } from './comment-mark.ts';+import { DocFidelity } from './doc-fidelity.ts';import { EmphasisFidelity, StrongFidelity } from './emphasis-fidelity.ts';import { EscapeMark } from './escape-mark.ts';import { FootnoteDefinition } from './footnote-definition.ts';@@ -23,6 +24,7 @@ import { List, ListItem } from './list.ts';import { MathInline } from './math-inline.ts';import { RawMdxFallback } from './raw-mdx-fallback.ts';import { SourceLiteralMark } from './source-literal-mark.ts';+import { StrikeFidelity } from './strike-fidelity.ts';import { TableCellFidelity, TableFidelity, TableHeaderFidelity } from './table-fidelity.ts';import { Tag } from './tag.ts';import { ThematicBreakFidelity } from './thematic-break-fidelity.ts';@@ -41,6 +43,7 @@ export const sharedExtensions = [ListItem,EmphasisFidelity,StrongFidelity,+ StrikeFidelity,CodeMarkFidelity,CodeBlockFidelity,BlockquoteFidelity,@@ -51,6 +54,7 @@ export const sharedExtensions = [LinkRefDefFidelity,HardBreakFidelity,EscapeMark,+ DocFidelity,SourceLiteralMark,StarterKit.configure({undoRedo: false,@@ -59,6 +63,7 @@ export const sharedExtensions = [listItem: false,italic: false,bold: false,+ strike: false,code: false,codeBlock: false,heading: false,diff --git a/packages/core/src/extensions/strike-fidelity.ts b/packages/core/src/extensions/strike-fidelity.tsnew file mode 100644index 000000000..296177bd6--- /dev/null+++ b/packages/core/src/extensions/strike-fidelity.ts@@ -0,0 +1,12 @@+import Strike from '@tiptap/extension-strike';++export const StrikeFidelity = Strike.extend({+ priority: 60,++ addAttributes() {+ return {+ ...this.parent?.(),+ sourceDelimiter: { default: '~~', rendered: false },+ };+ },+});diff --git a/packages/core/src/extensions/table-fidelity.ts b/packages/core/src/extensions/table-fidelity.tsindex 259c11444..f85c22edc 100644--- a/packages/core/src/extensions/table-fidelity.ts+++ b/packages/core/src/extensions/table-fidelity.ts@@ -7,6 +7,8 @@ export const TableFidelity = Table.extend({return {...this.parent?.(),sourceDashCounts: { default: null },+ sourceOuterPipes: { default: null, rendered: false },+ sourceAlignmentPadding: { default: null, rendered: false },};},});diff --git a/packages/core/src/extensions/wiki-link.ts b/packages/core/src/extensions/wiki-link.tsindex 45b8c84a1..ea96e40cd 100644--- a/packages/core/src/extensions/wiki-link.ts+++ b/packages/core/src/extensions/wiki-link.ts@@ -77,6 +77,18 @@ export const WikiLink = Node.create({resolved: {default: false,},+ sourceTarget: {+ default: null,+ rendered: false,+ },+ sourceAnchor: {+ default: null,+ rendered: false,+ },+ sourceAlias: {+ default: null,+ rendered: false,+ },};},diff --git a/packages/core/src/index.ts b/packages/core/src/index.tsindex 826fdc5d5..5d97a741b 100644--- a/packages/core/src/index.ts+++ b/packages/core/src/index.ts@@ -18,11 +18,13 @@ export {type BridgeMergeContentLossWhich,type BridgeToleranceClass,bindFrontmatterDoc,+ classifySeverity,type DiffChange,defaultScheduler,detectAppliedToleranceClasses,detectFmRegion,diffLinesFast,+ emitToleranceFire,type FmEditError,type FmEditResult,FORM_WRITE_ORIGIN,@@ -36,6 +38,7 @@ export {type FrontmatterBindingUnsubscribe,type FrontmatterDocProvider,type FrontmatterSnapshot,+ findFirstDivergenceIndex,fnv1aDigest,type InvariantViolation,MAX_FM_REGION_BYTES,@@ -48,6 +51,10 @@ export {readFmMap,readFmRegionWithError,type Scheduler,+ setToleranceTelemetryHook,+ type ToleranceClassSeverity,+ type ToleranceFireRecord,+ type ToleranceTelemetryHook,toBridgeInvariantLog,} from './bridge/index.ts';export {diff --git a/packages/core/src/markdown/autolink-promotion.ts b/packages/core/src/markdown/autolink-promotion.tsindex a2a9067db..7745bb11c 100644--- a/packages/core/src/markdown/autolink-promotion.ts+++ b/packages/core/src/markdown/autolink-promotion.ts@@ -1,8 +1,9 @@import type { Link, Parent, Text } from 'mdast';+import { deriveFragmentPosition } from './promoter-position.ts';const AUTOLINK_IN_TEXT_RE = /<([a-zA-Z][a-zA-Z0-9+.-]*:[^\s<>]+)>/g;-export function promoteInParent(parent: Parent): void {+export function promoteInParent(parent: Parent, source: string = ''): void {const newChildren: Parent['children'] = [];let changed = false;@@ -26,16 +27,34 @@ export function promoteInParent(parent: Parent): void {const matchStart = match.index;if (matchStart > lastIndex) {- segments.push({ type: 'text', value: text.slice(lastIndex, matchStart) } as Text);+ const lead: Text = { type: 'text', value: text.slice(lastIndex, matchStart) };+ const pos = deriveFragmentPosition(source, child as Text, lastIndex, matchStart);+ if (pos) lead.position = pos;+ segments.push(lead);}+ const innerText: Text = { type: 'text', value: uri };+ const innerPos = deriveFragmentPosition(+ source,+ child as Text,+ matchStart + 1,+ matchStart + 1 + uri.length,+ );+ if (innerPos) innerText.position = innerPos;const linkNode: Link & { data: { sourceStyle: string } } = {type: 'link',url: uri,title: null,- children: [{ type: 'text', value: uri } as Text],+ children: [innerText],data: { sourceStyle: 'autolink' },};+ const linkPos = deriveFragmentPosition(+ source,+ child as Text,+ matchStart,+ matchStart + fullMatch.length,+ );+ if (linkPos) linkNode.position = linkPos;segments.push(linkNode);lastIndex = matchStart + fullMatch.length;@@ -46,7 +65,10 @@ export function promoteInParent(parent: Parent): void {newChildren.push(child);} else {if (lastIndex < text.length) {- segments.push({ type: 'text', value: text.slice(lastIndex) } as Text);+ const tail: Text = { type: 'text', value: text.slice(lastIndex) };+ const pos = deriveFragmentPosition(source, child as Text, lastIndex, text.length);+ if (pos) tail.position = pos;+ segments.push(tail);}newChildren.push(...segments);}diff --git a/packages/core/src/markdown/autolink-void-html-guard.test.ts b/packages/core/src/markdown/autolink-void-html-guard.test.tsindex 08848b87e..66eed34af 100644--- a/packages/core/src/markdown/autolink-void-html-guard.test.ts+++ b/packages/core/src/markdown/autolink-void-html-guard.test.ts@@ -1,7 +1,9 @@import { describe, expect, test } from 'bun:test';import type { JSONContent } from '@tiptap/core';import { sharedExtensions } from '../extensions/shared.ts';+import { protectFromMdx } from './autolink-void-html-guard.ts';import { MarkdownManager } from './index.ts';+import { mdastToHtml } from './mdast-to-html.ts';const mdManager = new MarkdownManager({ extensions: sharedExtensions });@@ -274,3 +276,115 @@ describe('R6a: safeText idempotency for §2.4 ambiguous chars', () => {expect(() => mdManager.parse('\\\\{a}\n')).not.toThrow();});});++describe('R23 guard: angle-bracket link destinations', () => {+ function findNodes(node: unknown, type: string): Array<Record<string, unknown>> {+ const out: Array<Record<string, unknown>> = [];+ const rec = node as Record<string, unknown>;+ if (rec && typeof rec === 'object') {+ if (rec.type === type) out.push(rec);+ const children = rec.children;+ if (Array.isArray(children)) {+ for (const child of children) out.push(...findNodes(child, type));+ }+ }+ return out;+ }++ function expectByteStable(md: string): void {+ const r1 = roundTrip(md);+ expect(r1).toBe(md);+ expect(roundTrip(r1)).toBe(r1);+ }++ test('[link](</my uri>) parses to a link node with the space-bearing url', () => {+ const tree = mdManager.parseToMdast('[link](</my uri>)\n');+ const links = findNodes(tree, 'link');+ expect(links.length).toBe(1);+ expect(links[0].url).toBe('/my uri');+ });++ test('the destination < is not PUA-protected when the resource forms', () => {+ const src = '[link](</my uri>)';+ expect(protectFromMdx(src)).toBe(src);+ });++ test('angle destination link renders href /my%20uri', () => {+ const html = mdastToHtml(mdManager.parseToMdast('[link](</my uri>)\n'));+ expect(html).toContain('href="/my%20uri"');+ expect(html).toContain('>link</a>');+ });++ test('angle destination link round-trips byte-identically and idempotently', () => {+ expectByteStable('[link](</my uri>)\n');+ });++ test('image form  parses to an image and round-trips', () => {+ const tree = mdManager.parseToMdast('\n');+ const images = [...findNodes(tree, 'image'), ...findNodes(tree, 'mdxJsxFlowElement')];+ expect(images.length).toBeGreaterThanOrEqual(1);+ expectByteStable('\n');+ });++ test('uppercase-name space-bearing destination forms a link', () => {+ const tree = mdManager.parseToMdast('[x](<My uri>)\n');+ const links = findNodes(tree, 'link');+ expect(links.length).toBe(1);+ expect(links[0].url).toBe('My uri');+ expectByteStable('[x](<My uri>)\n');+ });++ const mustStayGuarded: Array<[string, string]> = [+ ['[link](</my uri', 'mid-typing: unclosed destination'],+ ['[link](</my uri>', 'mid-typing: destination closed, resource open'],+ ['[link](</my uri> junk)', 'resource fails after destination'],+ ['x](</a b>)', 'no label-start before ]('],+ ['link](</a b>)', 'label-start deleted mid-edit'],+ ['a](<', 'bare ](< at EOF'],+ ['a](</', 'bare ](</ at EOF'],+ ['[x] y](</a b>)', 'label already closed before ]('],+ ['[[page]](</a b>)', 'wikilink-consumed brackets before ]('],+ ['[a\\](</b c>)', 'escaped ] cannot end a label'],+ ['`a[b` ](</x y>)', 'code-span-consumed [ before ]('],+ ['[^fn](</a b>)', 'footnote-shaped label'],+ ['[a][b](</c d>)', 'reference-link-shaped label'],+ ['[a](</x \\> y>)', 'escaped > inside destination'],+ ['[a](</x <i:j> k>)', 'autolink inside destination'],+ ['[a](<x\ny>)', 'destination spans a line ending'],+ ['[link](</my uri> "t")', 'title after destination'],+ ['[link]( </my uri>)', 'whitespace between ( and <'],+ ['[a](</b c> )', 'whitespace between > and )'],+ [`[a](<${' '.repeat(2000)}x y>)`, 'destination exceeding the scan cap'],+ ];++ for (const [input, label] of mustStayGuarded) {+ test(`does not throw: ${label}`, () => {+ expect(() => mdManager.parse(`${input}\n`)).not.toThrow();+ });+ }++ test('conservative rejections form no link and stay byte-stable', () => {+ for (const src of [+ '[link](</my uri> "t")\n',+ '[link]( </my uri>)\n',+ '[a](</b c> )\n',+ 'x](</a b>)\n',+ 'link](</a b>)\n',+ ]) {+ const links = findNodes(mdManager.parseToMdast(src), 'link');+ expect(links.length).toBe(0);+ expectByteStable(src);+ }+ });++ test('rejected destination < stays PUA-protected', () => {+ expect(protectFromMdx('x](</a b>)')).toContain('\uE000');+ expect(protectFromMdx('[link](</my uri')).toContain('\uE000');+ });++ test('no-space angle destinations keep their current byte-stable behavior', () => {+ expectByteStable('[t](<u>)\n');+ expectByteStable('[link](<foo%20bar>)\n');+ expectByteStable('[link](</no-space>)\n');+ });+});diff --git a/packages/core/src/markdown/autolink-void-html-guard.ts b/packages/core/src/markdown/autolink-void-html-guard.tsindex 4f863ab10..101069790 100644--- a/packages/core/src/markdown/autolink-void-html-guard.ts+++ b/packages/core/src/markdown/autolink-void-html-guard.ts@@ -7,6 +7,27 @@ const GUARD_COLON = '\uE002';const GUARD_AT = '\uE003';const GUARD_OPEN_BRACE = '\uE004';+const LITERAL_SENTINEL_ESCAPES: ReadonlyArray<readonly [string, string]> = [+ [GUARD_OPEN, '\uE005'],+ [GUARD_CLOSE, '\uE006'],+ [GUARD_COLON, '\uE007'],+ [GUARD_AT, '\uE008'],+ [GUARD_OPEN_BRACE, '\uE009'],+];+const HAS_LITERAL_SENTINEL_RE = /[\uE000-\uE004]/;+const HAS_ESCAPED_LITERAL_SENTINEL_RE = /[\uE005-\uE009]/;++export const R23_GUARD_SUBSTITUTIONS: ReadonlyArray<{ from: string; to: string }> = [+ { from: '<', to: GUARD_OPEN },+ { from: '>', to: GUARD_CLOSE },+ { from: ':', to: GUARD_COLON },+ { from: '@', to: GUARD_AT },+ { from: '{', to: GUARD_OPEN_BRACE },+];++export const R23_SENTINEL_ESCAPE_SUBSTITUTIONS: ReadonlyArray<{ from: string; to: string }> =+ LITERAL_SENTINEL_ESCAPES.map(([from, to]) => ({ from, to }));+const AUTOLINK_RE = /<([a-zA-Z][a-zA-Z0-9+.-]*:[^\s<>]+)>/g;const HTML_COMMENT_RE = /<!--[\s\S]*?-->/g;@@ -76,6 +97,57 @@ function indexGreaterThan(source: string): number[] {return positions;}+const ANGLE_DEST_SCAN_CAP = 1024;++function isAngleBracketDestinationOpen(offset: number, result: string): boolean {+ if (result[offset - 1] !== '(' || result[offset - 2] !== ']') return false;+ const labelEnd = offset - 2;+ let bs = 0;+ for (let j = labelEnd - 1; j >= 0 && result[j] === '\\'; j--) bs++;+ if (bs % 2 === 1) return false;++ let foundLabelStart = false;+ const scanFloor = Math.max(0, labelEnd - ANGLE_DEST_SCAN_CAP);+ for (let j = labelEnd - 1; j >= scanFloor; j--) {+ const ch = result[j];+ if (ch === '\n' || ch === '\r' || ch === '`') return false;+ if (ch !== '[' && ch !== ']') continue;+ let k = 0;+ for (let m = j - 1; m >= 0 && result[m] === '\\'; m--) k++;+ if (k % 2 === 1) continue;+ if (ch === ']') return false;+ if (result[j + 1] === '^') return false;+ if (result[j - 1] === ']') return false;+ foundLabelStart = true;+ break;+ }+ if (!foundLabelStart) return false;++ let sawWhitespace = false;+ let destClose = -1;+ const scanCeil = Math.min(result.length, offset + 1 + ANGLE_DEST_SCAN_CAP);+ for (let j = offset + 1; j < scanCeil; j++) {+ const ch = result[j];+ if (ch === '>') {+ destClose = j;+ break;+ }+ if (ch === '<' || ch === '\\' || ch === '\n' || ch === '\r') return false;+ if (+ ch === GUARD_OPEN ||+ ch === GUARD_CLOSE ||+ ch === GUARD_COLON ||+ ch === GUARD_AT ||+ ch === GUARD_OPEN_BRACE+ ) {+ return false;+ }+ if (ch === ' ' || ch === '\t') sawWhitespace = true;+ }+ if (destClose === -1 || !sawWhitespace) return false;+ return result[destClose + 1] === ')';+}+function isSelfClosingTagAt(offset: number,result: string,@@ -94,6 +166,12 @@ function isSelfClosingTagAt(export function protectFromMdx(source: string): string {let result = source;+ if (HAS_LITERAL_SENTINEL_RE.test(result)) {+ for (const [sentinel, escapeChar] of LITERAL_SENTINEL_ESCAPES) {+ result = result.replaceAll(sentinel, escapeChar);+ }+ }+result = result.replace(HTML_COMMENT_RE, (match) => {return match.replace(/</g, GUARD_OPEN).replace(/>/g, GUARD_CLOSE);});@@ -129,6 +207,8 @@ export function protectFromMdx(source: string): string {const greaterThanOffsets = indexGreaterThan(result);result = result.replace(/</g, (match, offset) => {+ if (isAngleBracketDestinationOpen(offset, result)) return match;+const lookahead = result.slice(offset, offset + 256);if (lookahead[1] === '/') {@@ -229,13 +309,7 @@ export function protectFromMdx(source: string): string {}function hasSentinels(s: string): boolean {- return (- s.includes('\uE000') ||- s.includes('\uE001') ||- s.includes('\uE002') ||- s.includes('\uE003') ||- s.includes('\uE004')- );+ return HAS_LITERAL_SENTINEL_RE.test(s) || HAS_ESCAPED_LITERAL_SENTINEL_RE.test(s);}export function restoreFromMdx() {@@ -254,15 +328,27 @@ export function restoreFromMdx() {if (typeof rec.alt === 'string' && hasSentinels(rec.alt)) {rec.alt = restoreString(rec.alt);}+ if (typeof rec.lang === 'string' && hasSentinels(rec.lang)) {+ rec.lang = restoreString(rec.lang);+ }+ if (typeof rec.meta === 'string' && hasSentinels(rec.meta)) {+ rec.meta = restoreString(rec.meta);+ }});};}function restoreString(s: string): string {- return s+ let out = s.replaceAll(GUARD_OPEN, '<').replaceAll(GUARD_CLOSE, '>').replaceAll(GUARD_COLON, ':').replaceAll(GUARD_AT, '@').replaceAll(GUARD_OPEN_BRACE, '{');+ if (HAS_ESCAPED_LITERAL_SENTINEL_RE.test(out)) {+ for (const [sentinel, escapeChar] of LITERAL_SENTINEL_ESCAPES) {+ out = out.replaceAll(escapeChar, sentinel);+ }+ }+ return out;}diff --git a/packages/core/src/markdown/backslash-escape-guard.ts b/packages/core/src/markdown/backslash-escape-guard.tsindex b7f3b7faa..69f415f4b 100644--- a/packages/core/src/markdown/backslash-escape-guard.ts+++ b/packages/core/src/markdown/backslash-escape-guard.ts@@ -5,6 +5,10 @@ import { protectPattern } from './entity-ref-guard.ts';const BACKSLASH_ESCAPE_PUA_MARK = '';+export const BACKSLASH_GUARD_SUBSTITUTIONS: ReadonlyArray<{ from: string; to: string }> = [+ { from: '\\', to: BACKSLASH_ESCAPE_PUA_MARK },+];+const BACKSLASH_ESCAPE_RE = /\\</g;export function encodeBackslashEscapes(source: string): string {diff --git a/packages/core/src/markdown/comment-promoter.ts b/packages/core/src/markdown/comment-promoter.tsindex 11e9217d5..193e286b0 100644--- a/packages/core/src/markdown/comment-promoter.ts+++ b/packages/core/src/markdown/comment-promoter.ts@@ -1,13 +1,16 @@import type { Nodes, Paragraph, PhrasingContent, Root, RootContent, Text } from 'mdast';import { SKIP, visit } from 'unist-util-visit';+import type { VFile } from 'vfile';import type { CommentBlockMdast, CommentMdast } from './mdast-augmentation.ts';+import { deriveFragmentPosition } from './promoter-position.ts';const PERCENT_COMMENT_RE = /(?<!%)%%([^\n]*?[^\n%])%%(?!%)/g;const HTML_COMMENT_INLINE_RE = /<!--([\s\S]*?)-->/g;export function commentPromoterPlugin() {- return (tree: Root) => {+ return (tree: Root, file: VFile) => {+ const source = typeof file.value === 'string' ? file.value : '';handleBlockCommentsAtRoot(tree);visit(tree, 'text', (node: Text, index, parent) => {@@ -23,18 +26,27 @@ export function commentPromoterPlugin() {let cursor = 0;for (const match of matches) {if (match.start > cursor) {- replacements.push({ type: 'text', value: value.slice(cursor, match.start) });+ const lead: Text = { type: 'text', value: value.slice(cursor, match.start) };+ const pos = deriveFragmentPosition(source, node, cursor, match.start);+ if (pos) lead.position = pos;+ replacements.push(lead);}+ const innerText: Text = { type: 'text', value: match.body };const commentNode: CommentMdast = {type: 'comment',- children: [{ type: 'text', value: match.body }],+ children: [innerText],data: { sourceForm: match.sourceForm },};+ const commentPos = deriveFragmentPosition(source, node, match.start, match.end);+ if (commentPos) commentNode.position = commentPos;replacements.push(commentNode as unknown as PhrasingContent);cursor = match.end;}if (cursor < value.length) {- replacements.push({ type: 'text', value: value.slice(cursor) });+ const tail: Text = { type: 'text', value: value.slice(cursor) };+ const pos = deriveFragmentPosition(source, node, cursor, value.length);+ if (pos) tail.position = pos;+ replacements.push(tail);}const arr = (parent as { children: PhrasingContent[] }).children;diff --git a/packages/core/src/markdown/doc-start-thematic-fix.test.ts b/packages/core/src/markdown/doc-start-thematic-fix.test.tsdeleted file mode 100644index 3f5f6f9a3..000000000--- a/packages/core/src/markdown/doc-start-thematic-fix.test.ts+++ /dev/null@@ -1,63 +0,0 @@-import { describe, expect, test } from 'bun:test';-import { MarkdownManager, sharedExtensions } from '@inkeep/open-knowledge-core';-import type { JSONContent } from '@tiptap/core';--const mdManager = new MarkdownManager({ extensions: sharedExtensions });--function roundTrip(md: string): string {- return mdManager.serialize(mdManager.parse(md));-}--function findNodeType(json: JSONContent, type: string): boolean {- if (json.type === type) return true;- for (const child of json.content ?? []) {- if (findNodeType(child, type)) return true;- }- return false;-}--describe('doc-start-thematic-fix: empty yaml → thematicBreak', () => {- test('---\\n\\n--- (empty yaml) produces thematicBreak PM nodes', () => {- const json = mdManager.parse('---\n\n---');- expect(findNodeType(json, 'thematicBreak')).toBe(true);- });-- test('---\\n--- (empty yaml, no blank) produces thematicBreak PM nodes', () => {- const json = mdManager.parse('---\n---');- expect(findNodeType(json, 'thematicBreak')).toBe(true);- });-- test('empty yaml round-trip is idempotent', () => {- const r1 = roundTrip('---\n\n---');- const r2 = roundTrip(r1);- expect(r2).toBe(r1);- });-});--describe('doc-start-thematic-fix: real frontmatter is untouched', () => {- test('yaml with content stays as yaml (ignored by PM → empty doc with paragraph)', () => {- const json = mdManager.parse('---\ntitle: x\n---\n\nbody\n');- const hasBody = JSON.stringify(json).includes('body');- expect(hasBody).toBe(true);- });-- test('real frontmatter + body round-trips without frontmatter (stripped by PM layer)', () => {- const r = roundTrip('---\ntitle: x\n---\n\nbody\n');- expect(r.trim()).toBe('body');- });-});--describe('doc-start-thematic-fix: unclosed --- fallback', () => {- test('single --- (unclosed) is already thematicBreak by remark-frontmatter', () => {- const json = mdManager.parse('---\n');- expect(findNodeType(json, 'thematicBreak')).toBe(true);- });-- test('--- followed by content (no closing ---) round-trips with NG10 normalization', () => {- const r1 = roundTrip('---\n\ntext\n');- expect(r1).toContain('***');- expect(r1).toContain('text');- const r2 = roundTrip(r1);- expect(r2).toBe(r1);- });-});diff --git a/packages/core/src/markdown/doc-start-thematic-fix.ts b/packages/core/src/markdown/doc-start-thematic-fix.tsdeleted file mode 100644index a6c8dc0c3..000000000--- a/packages/core/src/markdown/doc-start-thematic-fix.ts+++ /dev/null@@ -1,44 +0,0 @@-import type { Root, ThematicBreak } from 'mdast';-import type { VFile } from 'vfile';--export function applyDocStartThematicFix(tree: Root, file: VFile): void {- if (tree.children.length === 0) return;-- const first = tree.children[0];- if (first.type !== 'yaml') return;-- const value = (first as { value?: string }).value ?? '';- if (value.trim() !== '') return;-- const startOff = first.position?.start?.offset;- if (startOff !== 0) return;-- const endOff = first.position?.end?.offset;- if (typeof endOff !== 'number') return;-- const source = typeof file.value === 'string' ? file.value : '';- if (!source) return;-- const slice = source.slice(startOff, endOff);- const lines = slice.split('\n');-- const hrPattern = /^-{3,}\s*$/;- const hrLines = lines.filter((l) => hrPattern.test(l));-- if (hrLines.length === 0) return;-- const replacements: ThematicBreak[] = hrLines.map((raw) => ({- type: 'thematicBreak' as const,- data: { sourceRaw: raw.replace(/\s+$/, '') },- }));-- if (replacements.length > 0) {- const firstLineLen = hrLines[0].length;- replacements[0].position = {- start: { line: 1, column: 1, offset: 0 },- end: { line: 1, column: firstLineLen + 1, offset: firstLineLen },- };- }-- tree.children.splice(0, 1, ...replacements);-}diff --git a/packages/core/src/markdown/entity-ref-guard.test.ts b/packages/core/src/markdown/entity-ref-guard.test.tsindex e505d04a0..732320661 100644--- a/packages/core/src/markdown/entity-ref-guard.test.ts+++ b/packages/core/src/markdown/entity-ref-guard.test.ts@@ -253,6 +253,32 @@ describe('restoreEntityRefsPlugin — mdast walk', () => {expect(image.alt).toBe('Tom & Jerry');});+ test('restores encoded entity in code-node `lang` field (fenced-code info string)', () => {+ const encoded = encodeEntityRefs('föö');+ const tree: Root = {+ type: 'root',+ children: [{ type: 'code', lang: encoded, meta: null, value: 'code' }],+ };+ runPlugin(tree);+ const code = tree.children[0];+ if (code?.type !== 'code') throw new Error('expected code');+ expect(code.lang).toBe('föö');+ expect((code as { data?: unknown }).data).toBeUndefined();+ });++ test('restores encoded entity in code-node `meta` field (fenced-code info string)', () => {+ const encoded = encodeEntityRefs('title="x"');+ const tree: Root = {+ type: 'root',+ children: [{ type: 'code', lang: 'js', meta: encoded, value: 'code' }],+ };+ runPlugin(tree);+ const code = tree.children[0];+ if (code?.type !== 'code') throw new Error('expected code');+ expect(code.meta).toBe('title="x"');+ expect(code.lang).toBe('js');+ });+test('does NOT tag entityRefSpans on non-text nodes (e.g. inlineCode value)', () => {const encoded = encodeEntityRefs('foo & bar');const tree: Root = {@@ -291,6 +317,47 @@ describe('restoreEntityRefsPlugin — mdast walk', () => {});});+describe('fenced-code info-string entity refs — real-engine round-trip', () => {+ const mgr = new MarkdownManager({ extensions: sharedExtensions });+ const roundTrip = (src: string) => mgr.serialize(mgr.parse(src));++ test('named entity in lang round-trips byte-exact', () => {+ const src = '```föö\ncode\n```\n';+ expect(roundTrip(src)).toBe(src);+ });++ test('round-trip is idempotent (second pass emits identical bytes)', () => {+ const src = '```föö\ncode\n```\n';+ const once = roundTrip(src);+ expect(roundTrip(once)).toBe(once);+ });++ test('named entity in meta round-trips byte-exact', () => {+ const src = '```js meta&stuff\ncode\n```\n';+ expect(roundTrip(src)).toBe(src);+ });++ test('entities in both lang and meta round-trip byte-exact', () => {+ const src = '```föö title="x"\ncode\n```\n';+ expect(roundTrip(src)).toBe(src);+ });++ test('numeric hex entity in lang round-trips byte-exact', () => {+ const src = '```föö\ncode\n```\n';+ expect(roundTrip(src)).toBe(src);+ });++ test('tilde-fence info-string entity round-trips byte-exact', () => {+ const src = '~~~föö\ncode\n~~~\n';+ expect(roundTrip(src)).toBe(src);+ });++ test('plain info strings stay byte-identical (no encode/restore detour)', () => {+ const src = '```typescript title="x"\ncode\n```\n';+ expect(roundTrip(src)).toBe(src);+ });+});+describe('escape + entity-ref marker interleaving on the same text node', () => {test('text node with BOTH escapedChars AND entityRefSpans round-trips through parse → serialize byte-equal', () => {const mgr = new MarkdownManager({ extensions: sharedExtensions });diff --git a/packages/core/src/markdown/entity-ref-guard.ts b/packages/core/src/markdown/entity-ref-guard.tsindex 3991b0e81..5a3996fbf 100644--- a/packages/core/src/markdown/entity-ref-guard.ts+++ b/packages/core/src/markdown/entity-ref-guard.ts@@ -1,10 +1,17 @@+import { decodeNamedCharacterReference } from 'decode-named-character-reference';import type { Nodes, Root } from 'mdast';+import { decodeNumericCharacterReference } from 'micromark-util-decode-numeric-character-reference';import type { Plugin } from 'unified';import { visit } from 'unist-util-visit';const ENTITY_REF_PUA_OPEN = '';const ENTITY_REF_PUA_CLOSE = '';+export const ENTITY_REF_GUARD_SUBSTITUTIONS: ReadonlyArray<{ from: string; to: string }> = [+ { from: '&', to: ENTITY_REF_PUA_OPEN },+ { from: ';', to: ENTITY_REF_PUA_CLOSE },+];+const ENTITY_REF_RE = /&(?:[A-Za-z][A-Za-z0-9]*|#[0-9]+|#[xX][0-9A-Fa-f]+);/g;function countPrecedingBackslashes(source: string, before: number): number {@@ -26,6 +33,19 @@ export function protectPattern(});}+export function decodeEntityRefs(value: string): string {+ ENTITY_REF_RE.lastIndex = 0;+ return value.replace(ENTITY_REF_RE, (match) => {+ const body = match.slice(1, -1);+ if (body.startsWith('#')) {+ const hex = body[1] === 'x' || body[1] === 'X';+ return decodeNumericCharacterReference(body.slice(hex ? 2 : 1), hex ? 16 : 10);+ }+ const decoded = decodeNamedCharacterReference(body);+ return decoded === false ? match : decoded;+ });+}+export function encodeEntityRefs(source: string): string {return protectPattern(source, ENTITY_REF_RE, (match) => {const body = match.slice(1, -1);@@ -84,7 +104,7 @@ export function restoreEntityRefsPlugin(): ReturnType<Plugin<[], Root>> {textNode.data.entityRefSpans = spans;}}- for (const key of ['url', 'title', 'alt'] as const) {+ for (const key of ['url', 'title', 'alt', 'lang', 'meta'] as const) {const v = rec[key];if (typeof v === 'string' && v.includes(ENTITY_REF_PUA_OPEN)) {rec[key] = restoreEntityRefsInString(v).value;diff --git a/packages/core/src/markdown/guard-flanking-matrix.json b/packages/core/src/markdown/guard-flanking-matrix.jsonnew file mode 100644index 000000000..c6f92c8c0--- /dev/null+++ b/packages/core/src/markdown/guard-flanking-matrix.json@@ -0,0 +1,471 @@+{+ "delimiters": ["*", "_", "**", "~~", "=="],+ "substitutions": [+ {+ "alphabet": "entity-ref",+ "from": "&",+ "toCodePoint": "U+E100",+ "fromClass": "punctuation",+ "toClass": "other",+ "classChanged": true+ },+ {+ "alphabet": "entity-ref",+ "from": ";",+ "toCodePoint": "U+E101",+ "fromClass": "punctuation",+ "toClass": "other",+ "classChanged": true+ },+ {+ "alphabet": "r23",+ "from": "<",+ "toCodePoint": "U+E000",+ "fromClass": "punctuation",+ "toClass": "other",+ "classChanged": true+ },+ {+ "alphabet": "r23",+ "from": ">",+ "toCodePoint": "U+E001",+ "fromClass": "punctuation",+ "toClass": "other",+ "classChanged": true+ },+ {+ "alphabet": "r23",+ "from": ":",+ "toCodePoint": "U+E002",+ "fromClass": "punctuation",+ "toClass": "other",+ "classChanged": true+ },+ {+ "alphabet": "r23",+ "from": "@",+ "toCodePoint": "U+E003",+ "fromClass": "punctuation",+ "toClass": "other",+ "classChanged": true+ },+ {+ "alphabet": "r23",+ "from": "{",+ "toCodePoint": "U+E004",+ "fromClass": "punctuation",+ "toClass": "other",+ "classChanged": true+ },+ {+ "alphabet": "r23-sentinel-escape",+ "from": "U+E000",+ "toCodePoint": "U+E005",+ "fromClass": "other",+ "toClass": "other",+ "classChanged": false+ },+ {+ "alphabet": "r23-sentinel-escape",+ "from": "U+E001",+ "toCodePoint": "U+E006",+ "fromClass": "other",+ "toClass": "other",+ "classChanged": false+ },+ {+ "alphabet": "r23-sentinel-escape",+ "from": "U+E002",+ "toCodePoint": "U+E007",+ "fromClass": "other",+ "toClass": "other",+ "classChanged": false+ },+ {+ "alphabet": "r23-sentinel-escape",+ "from": "U+E003",+ "toCodePoint": "U+E008",+ "fromClass": "other",+ "toClass": "other",+ "classChanged": false+ },+ {+ "alphabet": "r23-sentinel-escape",+ "from": "U+E004",+ "toCodePoint": "U+E009",+ "fromClass": "other",+ "toClass": "other",+ "classChanged": false+ },+ {+ "alphabet": "backslash-escape",+ "from": "\\",+ "toCodePoint": "U+E102",+ "fromClass": "punctuation",+ "toClass": "other",+ "classChanged": true+ }+ ],+ "cells": [+ {+ "alphabet": "entity-ref",+ "from": "&",+ "toCodePoint": "U+E100",+ "delimiter": "*",+ "witness": "*.y.*Ab",+ "side": "after-close",+ "roundTrip": "\\*.y.\\*Ab\n"+ },+ {+ "alphabet": "entity-ref",+ "from": "&",+ "toCodePoint": "U+E100",+ "delimiter": "_",+ "witness": "_.y._Ab",+ "side": "after-close",+ "roundTrip": "\\_.y.\\_Ab\n"+ },+ {+ "alphabet": "entity-ref",+ "from": "&",+ "toCodePoint": "U+E100",+ "delimiter": "**",+ "witness": "**.y.**Ab",+ "side": "after-close",+ "roundTrip": "\\*\\*.y.\\*\\*Ab\n"+ },+ {+ "alphabet": "entity-ref",+ "from": "&",+ "toCodePoint": "U+E100",+ "delimiter": "~~",+ "witness": "~~.y.~~Ab",+ "side": "after-close",+ "roundTrip": "\\~\\~.y.\\~\\~Ab\n"+ },+ {+ "alphabet": "entity-ref",+ "from": "&",+ "toCodePoint": "U+E100",+ "delimiter": "==",+ "witness": "==.y.==Ab",+ "side": "after-close",+ "roundTrip": "==.y.==\\Ab\n"+ },+ {+ "alphabet": "entity-ref",+ "from": ";",+ "toCodePoint": "U+E101",+ "delimiter": "*",+ "witness": "aA*.y.*",+ "side": "before-open",+ "roundTrip": "aA\\*.y.\\*\n"+ },+ {+ "alphabet": "entity-ref",+ "from": ";",+ "toCodePoint": "U+E101",+ "delimiter": "_",+ "witness": "aA_.y._",+ "side": "before-open",+ "roundTrip": "aA\\_.y.\\_\n"+ },+ {+ "alphabet": "entity-ref",+ "from": ";",+ "toCodePoint": "U+E101",+ "delimiter": "**",+ "witness": "aA**.y.**",+ "side": "before-open",+ "roundTrip": "aA\\*\\*.y.\\*\\*\n"+ },+ {+ "alphabet": "entity-ref",+ "from": ";",+ "toCodePoint": "U+E101",+ "delimiter": "~~",+ "witness": "aA~~.y.~~",+ "side": "before-open",+ "roundTrip": "aA\\~\\~.y.\\~\\~\n"+ },+ {+ "alphabet": "entity-ref",+ "from": ";",+ "toCodePoint": "U+E101",+ "delimiter": "==",+ "witness": "aA==.y.==",+ "side": "before-open",+ "roundTrip": "a\\A==.y.==\n"+ },+ {+ "alphabet": "r23",+ "from": "<",+ "toCodePoint": "U+E000",+ "delimiter": "*",+ "witness": "*.y.*<br>b",+ "side": "after-close",+ "roundTrip": "\\*.y.\\*<br>b\n"+ },+ {+ "alphabet": "r23",+ "from": "<",+ "toCodePoint": "U+E000",+ "delimiter": "_",+ "witness": "_.y._<br>b",+ "side": "after-close",+ "roundTrip": "\\_.y.\\_<br>b\n"+ },+ {+ "alphabet": "r23",+ "from": "<",+ "toCodePoint": "U+E000",+ "delimiter": "**",+ "witness": "**.y.**<br>b",+ "side": "after-close",+ "roundTrip": "\\*\\*.y.\\*\\*<br>b\n"+ },+ {+ "alphabet": "r23",+ "from": "<",+ "toCodePoint": "U+E000",+ "delimiter": "~~",+ "witness": "~~.y.~~<br>b",+ "side": "after-close",+ "roundTrip": "\\~\\~.y.\\~\\~<br>b\n"+ },+ {+ "alphabet": "r23",+ "from": "<",+ "toCodePoint": "U+E000",+ "delimiter": "==",+ "witness": "==.y.==<br>b",+ "side": "after-close",+ "roundTrip": "==.y.==<br>b\n"+ },+ {+ "alphabet": "r23",+ "from": ">",+ "toCodePoint": "U+E001",+ "delimiter": "*",+ "witness": "a<br>*.y.*",+ "side": "before-open",+ "roundTrip": "a<br>\\*.y.\\*\n"+ },+ {+ "alphabet": "r23",+ "from": ">",+ "toCodePoint": "U+E001",+ "delimiter": "_",+ "witness": "a<br>_.y._",+ "side": "before-open",+ "roundTrip": "a<br>\\_.y.\\_\n"+ },+ {+ "alphabet": "r23",+ "from": ">",+ "toCodePoint": "U+E001",+ "delimiter": "**",+ "witness": "a<br>**.y.**",+ "side": "before-open",+ "roundTrip": "a<br>\\*\\*.y.\\*\\*\n"+ },+ {+ "alphabet": "r23",+ "from": ">",+ "toCodePoint": "U+E001",+ "delimiter": "~~",+ "witness": "a<br>~~.y.~~",+ "side": "before-open",+ "roundTrip": "a<br>\\~\\~.y.\\~\\~\n"+ },+ {+ "alphabet": "r23",+ "from": ">",+ "toCodePoint": "U+E001",+ "delimiter": "==",+ "witness": "a<br>==.y.==",+ "side": "before-open",+ "roundTrip": "a<br>==.y.==\n"+ },+ {+ "alphabet": "r23",+ "from": ":",+ "toCodePoint": "U+E002",+ "delimiter": "*",+ "witness": "<a:*.y.*>",+ "side": "before-open",+ "roundTrip": "<a:*.y.*>\n"+ },+ {+ "alphabet": "r23",+ "from": ":",+ "toCodePoint": "U+E002",+ "delimiter": "_",+ "witness": "<a:_.y._>",+ "side": "before-open",+ "roundTrip": "<a:_.y._>\n"+ },+ {+ "alphabet": "r23",+ "from": ":",+ "toCodePoint": "U+E002",+ "delimiter": "**",+ "witness": "<a:**.y.**>",+ "side": "before-open",+ "roundTrip": "<a:**.y.**>\n"+ },+ {+ "alphabet": "r23",+ "from": ":",+ "toCodePoint": "U+E002",+ "delimiter": "~~",+ "witness": "<a:~~.y.~~>",+ "side": "before-open",+ "roundTrip": "<a:~~.y.~~>\n"+ },+ {+ "alphabet": "r23",+ "from": ":",+ "toCodePoint": "U+E002",+ "delimiter": "==",+ "witness": "<a:==.y.==>",+ "side": "before-open",+ "roundTrip": "<a:==.y.==>\n"+ },+ {+ "alphabet": "r23",+ "from": "@",+ "toCodePoint": "U+E003",+ "delimiter": "*",+ "witness": "<mailto:a@*.y.*>",+ "side": "before-open",+ "roundTrip": "<mailto:a@*.y.*>\n"+ },+ {+ "alphabet": "r23",+ "from": "@",+ "toCodePoint": "U+E003",+ "delimiter": "_",+ "witness": "<mailto:a@_.y._>",+ "side": "before-open",+ "roundTrip": "<mailto:a@_.y._>\n"+ },+ {+ "alphabet": "r23",+ "from": "@",+ "toCodePoint": "U+E003",+ "delimiter": "**",+ "witness": "<mailto:a@**.y.**>",+ "side": "before-open",+ "roundTrip": "<mailto:a@**.y.**>\n"+ },+ {+ "alphabet": "r23",+ "from": "@",+ "toCodePoint": "U+E003",+ "delimiter": "~~",+ "witness": "<mailto:a@~~.y.~~>",+ "side": "before-open",+ "roundTrip": "<mailto:a@~~.y.~~>\n"+ },+ {+ "alphabet": "r23",+ "from": "@",+ "toCodePoint": "U+E003",+ "delimiter": "==",+ "witness": "<mailto:a@==.y.==>",+ "side": "before-open",+ "roundTrip": "<mailto:a@==.y.==>\n"+ },+ {+ "alphabet": "r23",+ "from": "{",+ "toCodePoint": "U+E004",+ "delimiter": "*",+ "witness": "*.y.*{b",+ "side": "after-close",+ "roundTrip": "\\*.y.\\*\\{b\n"+ },+ {+ "alphabet": "r23",+ "from": "{",+ "toCodePoint": "U+E004",+ "delimiter": "_",+ "witness": "_.y._{b",+ "side": "after-close",+ "roundTrip": "\\_.y.\\_\\{b\n"+ },+ {+ "alphabet": "r23",+ "from": "{",+ "toCodePoint": "U+E004",+ "delimiter": "**",+ "witness": "**.y.**{b",+ "side": "after-close",+ "roundTrip": "\\*\\*.y.\\*\\*\\{b\n"+ },+ {+ "alphabet": "r23",+ "from": "{",+ "toCodePoint": "U+E004",+ "delimiter": "~~",+ "witness": "~~.y.~~{b",+ "side": "after-close",+ "roundTrip": "\\~\\~.y.\\~\\~\\{b\n"+ },+ {+ "alphabet": "r23",+ "from": "{",+ "toCodePoint": "U+E004",+ "delimiter": "==",+ "witness": "==.y.=={b",+ "side": "after-close",+ "roundTrip": "==.y.==\\{b\n"+ },+ {+ "alphabet": "backslash-escape",+ "from": "\\",+ "toCodePoint": "U+E102",+ "delimiter": "*",+ "witness": "*.y.*\\<b",+ "side": "after-close",+ "roundTrip": "\\*.y.\\*\\<b\n"+ },+ {+ "alphabet": "backslash-escape",+ "from": "\\",+ "toCodePoint": "U+E102",+ "delimiter": "_",+ "witness": "_.y._\\<b",+ "side": "after-close",+ "roundTrip": "\\_.y.\\_\\<b\n"+ },+ {+ "alphabet": "backslash-escape",+ "from": "\\",+ "toCodePoint": "U+E102",+ "delimiter": "**",+ "witness": "**.y.**\\<b",+ "side": "after-close",+ "roundTrip": "\\*\\*.y.\\*\\*\\<b\n"+ },+ {+ "alphabet": "backslash-escape",+ "from": "\\",+ "toCodePoint": "U+E102",+ "delimiter": "~~",+ "witness": "~~.y.~~\\<b",+ "side": "after-close",+ "roundTrip": "\\~\\~.y.\\~\\~\\<b\n"+ },+ {+ "alphabet": "backslash-escape",+ "from": "\\",+ "toCodePoint": "U+E102",+ "delimiter": "==",+ "witness": "==.y.==\\<b",+ "side": "after-close",+ "roundTrip": "==.y.==\\<b\n"+ }+ ]+}diff --git a/packages/core/src/markdown/guard-flanking-matrix.test.ts b/packages/core/src/markdown/guard-flanking-matrix.test.tsnew file mode 100644index 000000000..c7a185819--- /dev/null+++ b/packages/core/src/markdown/guard-flanking-matrix.test.ts@@ -0,0 +1,131 @@+import { describe, expect, test } from 'bun:test';+import { readFileSync, writeFileSync } from 'node:fs';+import path from 'node:path';+import { sharedExtensions } from '../extensions/shared.ts';+import {+ ATTENTION_DELIMITERS,+ buildGuardFlankingMatrix,+ computeGuardFlankingCells,+ computeGuardSubstitutionRows,+ diffGuardFlankingMatrix,+ type FlankClass,+ type GuardFlankingMatrix,+} from './guard-flanking-matrix.ts';+import { MarkdownManager } from './index.ts';++const md = new MarkdownManager({ extensions: sharedExtensions });+const roundTrip = (source: string) => md.serialize(md.parse(source));++function textContent(source: string): string {+ let out = '';+ const walk = (node: unknown): void => {+ if (node === null || typeof node !== 'object') return;+ const rec = node as { value?: unknown; children?: unknown[] };+ if (typeof rec.value === 'string') out += rec.value;+ if (Array.isArray(rec.children)) rec.children.forEach(walk);+ };+ walk(md.parseToMdast(source));+ return out;+}++const MATRIX_PATH = path.join(import.meta.dir, 'guard-flanking-matrix.json');++function readCommitted(): GuardFlankingMatrix {+ return JSON.parse(readFileSync(MATRIX_PATH, 'utf8')) as GuardFlankingMatrix;+}++if (process.env.OK_UPDATE_GUARD_FLANKING_MATRIX === '1') {+ writeFileSync(MATRIX_PATH, `${JSON.stringify(buildGuardFlankingMatrix(roundTrip), null, 2)}\n`);+}++describe('guard-flanking matrix - computation', () => {+ test('every guard substitution is classified; the punctuation->PUA rows are class-changing', () => {+ const rows = computeGuardSubstitutionRows();+ const changed = rows.filter((r) => r.classChanged);+ const preserved = rows.filter((r) => !r.classChanged);+ expect(changed.map((r) => `${r.alphabet}:${r.from}`).sort()).toEqual([+ 'backslash-escape:\\',+ 'entity-ref:&',+ 'entity-ref:;',+ 'r23::',+ 'r23:<',+ 'r23:>',+ 'r23:@',+ 'r23:{',+ ]);+ const expectedDelta: { from: FlankClass; to: FlankClass } = {+ from: 'punctuation',+ to: 'other',+ };+ for (const row of changed) {+ expect(row.fromClass).toBe(expectedDelta.from);+ expect(row.toClass).toBe(expectedDelta.to);+ }+ for (const row of preserved) {+ expect(row.alphabet).toBe('r23-sentinel-escape');+ expect(row.fromClass).toBe('other');+ expect(row.toClass).toBe('other');+ }+ });++ test('every class-changing substitution x delimiter carries a cell with protect-layer-verified adjacency', () => {+ const cells = computeGuardFlankingCells();+ expect(cells.length).toBe(8 * ATTENTION_DELIMITERS.length);+ const keys = new Set(cells.map((c) => `${c.alphabet}:${c.from} x ${c.delimiter}`));+ for (const row of computeGuardSubstitutionRows().filter((r) => r.classChanged)) {+ for (const delimiter of ATTENTION_DELIMITERS) {+ expect(keys.has(`${row.alphabet}:${row.from} x ${delimiter}`)).toBe(true);+ }+ }+ });+});++describe('guard-flanking matrix - freshness gate', () => {+ test('committed snapshot matches a live recompute (regen: bun run gen:guard-flanking-matrix)', () => {+ const live = buildGuardFlankingMatrix(roundTrip);+ const mismatches = diffGuardFlankingMatrix(live, readCommitted());+ expect(mismatches).toEqual([]);+ for (const [index, cell] of live.cells.entries()) {+ expect(cell.roundTrip).toBe(readCommitted().cells[index]?.roundTrip ?? '');+ }+ });++ test('every committed cell is lossless, idempotent, and preserves the guarded bytes verbatim', () => {+ for (const cell of readCommitted().cells) {+ const rt1 = roundTrip(cell.witness);+ expect(rt1).toBe(cell.roundTrip);+ expect(roundTrip(rt1)).toBe(rt1);+ expect(textContent(rt1)).toBe(textContent(cell.witness));+ if (cell.alphabet === 'entity-ref') {+ expect(rt1).toContain('A');+ }+ }+ });+});++describe('guard-flanking matrix - non-vacuity (tamper)', () => {+ test('a dropped committed cell fails the freshness diff', () => {+ const live = buildGuardFlankingMatrix(roundTrip);+ const tampered: GuardFlankingMatrix = { ...live, cells: live.cells.slice(1) };+ const mismatches = diffGuardFlankingMatrix(live, tampered);+ expect(mismatches.some((m) => m.includes('not committed'))).toBe(true);+ });++ test('a drifted round-trip pin fails the per-cell byte compare', () => {+ const committed = readCommitted();+ const first = committed.cells[0];+ if (!first) throw new Error('empty committed matrix');+ expect(roundTrip(first.witness)).not.toBe(`${first.roundTrip}tampered`);+ });++ test('a substitution removed from a guard alphabet fails the freshness diff', () => {+ const live = buildGuardFlankingMatrix(roundTrip);+ const tampered: GuardFlankingMatrix = {+ ...live,+ substitutions: live.substitutions.filter((r) => r.from !== '&'),+ cells: live.cells.filter((c) => c.from !== '&'),+ };+ const mismatches = diffGuardFlankingMatrix(live, tampered);+ expect(mismatches.some((m) => m.includes('entity-ref:&'))).toBe(true);+ });+});diff --git a/packages/core/src/markdown/guard-flanking-matrix.ts b/packages/core/src/markdown/guard-flanking-matrix.tsnew file mode 100644index 000000000..420dc0fe5--- /dev/null+++ b/packages/core/src/markdown/guard-flanking-matrix.ts@@ -0,0 +1,184 @@+import { classifyCharacter } from 'micromark-util-classify-character';+import {+ protectFromMdx,+ R23_GUARD_SUBSTITUTIONS,+ R23_SENTINEL_ESCAPE_SUBSTITUTIONS,+} from './autolink-void-html-guard.ts';+import { BACKSLASH_GUARD_SUBSTITUTIONS, encodeBackslashEscapes } from './backslash-escape-guard.ts';+import { ENTITY_REF_GUARD_SUBSTITUTIONS, encodeEntityRefs } from './entity-ref-guard.ts';++export const ATTENTION_DELIMITERS = ['*', '_', '**', '~~', '=='] as const;++export type FlankClass = 'whitespace' | 'punctuation' | 'other';++export interface GuardSubstitutionRow {+ alphabet: string;+ from: string;+ toCodePoint: string;+ fromClass: FlankClass;+ toClass: FlankClass;+ classChanged: boolean;+}++export interface GuardFlankingCell {+ alphabet: string;+ from: string;+ toCodePoint: string;+ delimiter: string;+ witness: string;+ side: 'before-open' | 'after-close';+}++export interface GuardFlankingMatrix {+ delimiters: readonly string[];+ substitutions: GuardSubstitutionRow[];+ cells: Array<GuardFlankingCell & { roundTrip: string }>;+}++const ALPHABETS: ReadonlyArray<{+ alphabet: string;+ substitutions: ReadonlyArray<{ from: string; to: string }>;+}> = [+ { alphabet: 'entity-ref', substitutions: ENTITY_REF_GUARD_SUBSTITUTIONS },+ { alphabet: 'r23', substitutions: R23_GUARD_SUBSTITUTIONS },+ { alphabet: 'r23-sentinel-escape', substitutions: R23_SENTINEL_ESCAPE_SUBSTITUTIONS },+ { alphabet: 'backslash-escape', substitutions: BACKSLASH_GUARD_SUBSTITUTIONS },+];++const WITNESS_TEMPLATES: Record<+ string,+ { side: GuardFlankingCell['side']; make: (delim: string) => string }+> = {+ 'entity-ref:&': { side: 'after-close', make: (d) => `${d}.y.${d}Ab` },+ 'entity-ref:;': { side: 'before-open', make: (d) => `aA${d}.y.${d}` },+ 'r23:<': { side: 'after-close', make: (d) => `${d}.y.${d}<br>b` },+ 'r23:>': { side: 'before-open', make: (d) => `a<br>${d}.y.${d}` },+ 'r23::': { side: 'before-open', make: (d) => `<a:${d}.y.${d}>` },+ 'r23:@': { side: 'before-open', make: (d) => `<mailto:a@${d}.y.${d}>` },+ 'r23:{': { side: 'after-close', make: (d) => `${d}.y.${d}{b` },+ 'backslash-escape:\\': { side: 'after-close', make: (d) => `${d}.y.${d}\\<b` },+};++function classify(char: string): FlankClass {+ const code = char.codePointAt(0);+ if (code === undefined) throw new Error('empty substitution char');+ const group = classifyCharacter(code);+ if (group === 1) return 'whitespace';+ if (group === 2) return 'punctuation';+ return 'other';+}++function formatCodePoint(char: string): string {+ const code = char.codePointAt(0);+ if (code === undefined) throw new Error('empty substitution char');+ return `U+${code.toString(16).toUpperCase().padStart(4, '0')}`;+}++function displayChar(char: string): string {+ const code = char.codePointAt(0);+ if (code === undefined) throw new Error('empty substitution char');+ return code >= 0x20 && code <= 0x7e ? char : formatCodePoint(char);+}++function runProtectChain(source: string): string {+ return encodeEntityRefs(protectFromMdx(encodeBackslashEscapes(source)));+}++export function computeGuardSubstitutionRows(): GuardSubstitutionRow[] {+ const rows: GuardSubstitutionRow[] = [];+ for (const { alphabet, substitutions } of ALPHABETS) {+ for (const { from, to } of substitutions) {+ const fromClass = classify(from);+ const toClass = classify(to);+ rows.push({+ alphabet,+ from: displayChar(from),+ toCodePoint: formatCodePoint(to),+ fromClass,+ toClass,+ classChanged: fromClass !== toClass,+ });+ }+ }+ return rows;+}++export function computeGuardFlankingCells(): GuardFlankingCell[] {+ const cells: GuardFlankingCell[] = [];+ for (const { alphabet, substitutions } of ALPHABETS) {+ for (const { from, to } of substitutions) {+ if (classify(from) === classify(to)) continue;+ const template = WITNESS_TEMPLATES[`${alphabet}:${from}`];+ if (!template) {+ throw new Error(`no witness template for class-changing substitution ${alphabet}:${from}`);+ }+ for (const delimiter of ATTENTION_DELIMITERS) {+ const witness = template.make(delimiter);+ const protectedSource = runProtectChain(witness);+ const adjacencyProbe =+ template.side === 'before-open' ? `${to}${delimiter}` : `${delimiter}${to}`;+ if (!protectedSource.includes(adjacencyProbe)) {+ throw new Error(+ `witness for ${alphabet}:${from} x ${delimiter} lost sentinel-delimiter adjacency at the protect layer`,+ );+ }+ cells.push({+ alphabet,+ from,+ toCodePoint: formatCodePoint(to),+ delimiter,+ witness,+ side: template.side,+ });+ }+ }+ }+ return cells;+}++export function buildGuardFlankingMatrix(roundTrip: (md: string) => string): GuardFlankingMatrix {+ return {+ delimiters: [...ATTENTION_DELIMITERS],+ substitutions: computeGuardSubstitutionRows(),+ cells: computeGuardFlankingCells().map((cell) => ({+ ...cell,+ roundTrip: roundTrip(cell.witness),+ })),+ };+}++export function diffGuardFlankingMatrix(+ live: GuardFlankingMatrix,+ committed: GuardFlankingMatrix,+): string[] {+ const mismatches: string[] = [];+ if (JSON.stringify(live.delimiters) !== JSON.stringify(committed.delimiters)) {+ mismatches.push('delimiter set drifted');+ }+ const rowKey = (r: GuardSubstitutionRow) => `${r.alphabet}:${r.from}`;+ const liveRows = new Map(live.substitutions.map((r) => [rowKey(r), r]));+ const committedRows = new Map(committed.substitutions.map((r) => [rowKey(r), r]));+ for (const [key, row] of liveRows) {+ const prior = committedRows.get(key);+ if (!prior) mismatches.push(`substitution ${key} is live but not committed`);+ else if (JSON.stringify(row) !== JSON.stringify(prior)) {+ mismatches.push(`substitution ${key} drifted`);+ }+ }+ for (const key of committedRows.keys()) {+ if (!liveRows.has(key)) mismatches.push(`substitution ${key} is committed but no longer live`);+ }+ const cellKey = (c: GuardFlankingCell) =>+ `${c.alphabet}:${c.from} x ${JSON.stringify(c.delimiter)}`;+ const liveCells = new Map(live.cells.map((c) => [cellKey(c), c]));+ const committedCells = new Map(committed.cells.map((c) => [cellKey(c), c]));+ for (const [key, cell] of liveCells) {+ const prior = committedCells.get(key);+ if (!prior) mismatches.push(`cell ${key} is live but not committed`);+ else if (JSON.stringify(cell) !== JSON.stringify(prior)) mismatches.push(`cell ${key} drifted`);+ }+ for (const key of committedCells.keys()) {+ if (!liveCells.has(key)) mismatches.push(`cell ${key} is committed but no longer live`);+ }+ return mismatches;+}diff --git a/packages/core/src/markdown/handler-shadow-audit.test.ts b/packages/core/src/markdown/handler-shadow-audit.test.tsnew file mode 100644index 000000000..b717ef4d0--- /dev/null+++ b/packages/core/src/markdown/handler-shadow-audit.test.ts@@ -0,0 +1,99 @@+import { describe, expect, test } from 'bun:test';+import { sharedExtensions } from '../extensions/shared.ts';+import {+ collectExtensionHandlerKeys,+ computeShadowedHandlerKeys,+ HANDLER_SHADOW_ADJUDICATIONS,+ type HandlerShadowWitness,+} from './handler-shadow-audit.ts';+import { MarkdownManager } from './index.ts';+import { createSerializeProcessor } from './pipeline.ts';+import { toMarkdownHandlers } from './to-markdown-handlers.ts';++const md = new MarkdownManager({ extensions: sharedExtensions });+const roundTrip = (source: string) => md.serialize(md.parse(source));++function computeLiveShadowedKeys(): string[] {+ const processor = createSerializeProcessor({});+ const extensions = processor.data('toMarkdownExtensions');+ if (!Array.isArray(extensions) || extensions.length === 0) {+ throw new Error('serialize processor registered no toMarkdown extensions');+ }+ return computeShadowedHandlerKeys(+ collectExtensionHandlerKeys(extensions),+ Object.keys(toMarkdownHandlers),+ );+}++describe('handler-shadow audit - closure', () => {+ test('every shadowed handler key carries an adjudication and none is stale', () => {+ const shadowed = computeLiveShadowedKeys();+ expect(shadowed.length).toBeGreaterThan(0);+ const adjudicated = Object.keys(HANDLER_SHADOW_ADJUDICATIONS).sort();+ expect(shadowed).toEqual(adjudicated);+ });++ test('the known shadow surface is exactly the six adjudicated keys', () => {+ expect(computeLiveShadowedKeys()).toEqual([+ 'delete',+ 'inlineCode',+ 'inlineMath',+ 'mdxJsxFlowElement',+ 'mdxJsxTextElement',+ 'table',+ ]);+ });+});++describe('handler-shadow audit - live witnesses', () => {+ test('every adjudication witness round-trips byte-exactly through the real engine', () => {+ const runWitness = (key: string, witness: HandlerShadowWitness): void => {+ const out = roundTrip(witness.input);+ expect(+ out === witness.input || out === `${witness.input}\n`,+ `${key} witness drifted: ${JSON.stringify(witness.input)} -> ${JSON.stringify(out)}`,+ ).toBe(true);+ if (witness.expect === 'byte-and-reparse-type') {+ const reparsed = md.parseToMdast(out) as { children?: Array<{ type: string }> };+ expect(reparsed.children?.[0]?.type).toBe(witness.reparseType ?? '');+ }+ };+ for (const [key, adjudication] of Object.entries(HANDLER_SHADOW_ADJUDICATIONS)) {+ expect(adjudication.witnesses.length).toBeGreaterThan(0);+ for (const witness of adjudication.witnesses) runWitness(key, witness);+ }+ });+});++describe('handler-shadow audit - non-vacuity (tamper)', () => {+ test('removing the inlineCode (table-cell pipe re-escape) adjudication fails the closure', () => {+ const shadowed = computeLiveShadowedKeys();+ const { inlineCode: _omitted, ...rest } = HANDLER_SHADOW_ADJUDICATIONS;+ expect(shadowed).not.toEqual(Object.keys(rest).sort());+ const missing = shadowed.filter((key) => !(key in rest));+ expect(missing).toEqual(['inlineCode']);+ });++ test('a planted adjudication for an unshadowed key is detected as stale', () => {+ const shadowed = new Set(computeLiveShadowedKeys());+ const planted = {+ ...HANDLER_SHADOW_ADJUDICATIONS,+ paragraph: HANDLER_SHADOW_ADJUDICATIONS.delete,+ };+ const stale = Object.keys(planted).filter((key) => !shadowed.has(key));+ expect(stale).toEqual(['paragraph']);+ });++ test('the key collector sees through nested extension arrays', () => {+ const keys = collectExtensionHandlerKeys([+ {+ extensions: [+ { handlers: { alpha: () => '' } },+ { extensions: [{ handlers: { beta: () => '' } }] },+ ],+ },+ { handlers: { gamma: () => '' } },+ ]);+ expect(keys).toEqual(['alpha', 'beta', 'gamma']);+ });+});diff --git a/packages/core/src/markdown/handler-shadow-audit.ts b/packages/core/src/markdown/handler-shadow-audit.tsnew file mode 100644index 000000000..2653473da--- /dev/null+++ b/packages/core/src/markdown/handler-shadow-audit.ts@@ -0,0 +1,102 @@+export interface HandlerShadowWitness {+ input: string;+ expect: 'byte' | 'byte-and-reparse-type';+ reparseType?: string;+}++export interface HandlerShadowAdjudication {+ shadows: string;+ obligation: string;+ witnesses: ReadonlyArray<HandlerShadowWitness>;+}++export const HANDLER_SHADOW_ADJUDICATIONS: Record<string, HandlerShadowAdjudication> = {+ delete: {+ shadows: 'mdast-util-gfm-strikethrough handlers.delete (via remark-gfm)',+ obligation:+ 'emit tilde-delimited strikethrough; OK additionally re-emits the captured source ' ++ 'delimiter so single-tilde forms survive',+ witnesses: [+ { input: 'a ~~b~~ c', expect: 'byte' },+ { input: 'a ~b~ c', expect: 'byte' },+ ],+ },+ inlineCode: {+ shadows: 'mdast-util-gfm-table inlineCodeWithTable (via remark-gfm)',+ obligation:+ 'inside a tableCell, re-escape | in the emitted code span (GFM: an unescaped pipe splits ' ++ 'the cell, so dropping the escape re-parses the table as a paragraph); outside tables, ' ++ 'preserve fence char/length and padding',+ witnesses: [+ {+ input: '| `a\\|b` |\n| - |',+ expect: 'byte-and-reparse-type',+ reparseType: 'table',+ },+ { input: 'x `a|b` y', expect: 'byte' },+ ],+ },+ inlineMath: {+ shadows: 'mdast-util-math handlers.inlineMath (via remark-math)',+ obligation:+ 'emit dollar-delimited inline math with upstream sizing/padding; OK additionally re-emits ' ++ 'the captured single-dollar source form when the currency-safe promoter re-accepts it',+ witnesses: [+ { input: '$$x$$', expect: 'byte' },+ { input: '$x$', expect: 'byte' },+ ],+ },+ mdxJsxFlowElement: {+ shadows: 'mdast-util-mdx-jsx handlers.mdxJsxFlowElement (via the mdx utilities)',+ obligation:+ 'serialize JSX flow elements (name, attributes, children) as parseable MDX; OK re-emits ' ++ 'captured source for byte fidelity',+ witnesses: [{ input: '<Callout type="info">\nbody\n</Callout>', expect: 'byte' }],+ },+ mdxJsxTextElement: {+ shadows: 'mdast-util-mdx-jsx handlers.mdxJsxTextElement (via the mdx utilities)',+ obligation:+ 'serialize inline JSX elements as parseable MDX; OK re-emits captured source for byte fidelity',+ witnesses: [{ input: 'a <File name="x" /> b', expect: 'byte' }],+ },+ table: {+ shadows: 'mdast-util-gfm-table handlers.table (via remark-gfm)',+ obligation:+ 'emit the GFM table grid (header, delimiter row, body; cell |-escaping; alignment ' ++ 'markers); OK additionally threads captured padding/dash/outer-pipe source forms',+ witnesses: [+ {+ input: '| a | b |\n| --- | --- |\n| 1 | 2 |',+ expect: 'byte-and-reparse-type',+ reparseType: 'table',+ },+ {+ input: '| l | r |\n| :-- | --: |\n| 1 | 2 |',+ expect: 'byte-and-reparse-type',+ reparseType: 'table',+ },+ ],+ },+};++export function collectExtensionHandlerKeys(extensions: readonly unknown[]): string[] {+ const keys = new Set<string>();+ const visit = (ext: unknown): void => {+ if (ext === null || typeof ext !== 'object') return;+ const rec = ext as { handlers?: unknown; extensions?: unknown };+ if (rec.handlers !== null && typeof rec.handlers === 'object') {+ for (const key of Object.keys(rec.handlers as Record<string, unknown>)) keys.add(key);+ }+ if (Array.isArray(rec.extensions)) rec.extensions.forEach(visit);+ };+ extensions.forEach(visit);+ return [...keys].sort();+}++export function computeShadowedHandlerKeys(+ extensionHandlerKeys: readonly string[],+ overrideKeys: readonly string[],+): string[] {+ const overrides = new Set(overrideKeys);+ return extensionHandlerKeys.filter((key) => overrides.has(key)).sort();+}diff --git a/packages/core/src/markdown/highlight-promoter.ts b/packages/core/src/markdown/highlight-promoter.tsindex 16e39cb1f..f758cf737 100644--- a/packages/core/src/markdown/highlight-promoter.ts+++ b/packages/core/src/markdown/highlight-promoter.ts@@ -1,7 +1,9 @@import type { Nodes, Parent, PhrasingContent, Root, Text } from 'mdast';import type { MdxJsxTextElement } from 'mdast-util-mdx';import { SKIP, visit } from 'unist-util-visit';+import type { VFile } from 'vfile';import type { MarkMdast } from './mdast-augmentation.ts';+import { deriveFragmentPosition } from './promoter-position.ts';const HIGHLIGHT_RE = /(?<!=)==(?=\S)([^\n]*?[^\s=])==(?!=)/g;@@ -178,7 +180,8 @@ function walkPromote(node: Nodes): void {}export function highlightPromoterPlugin() {- return (tree: Root) => {+ return (tree: Root, file: VFile) => {+ const source = typeof file.value === 'string' ? file.value : '';visit(tree, 'text', (node: Text, index, parent) => {if (parent === undefined || index === undefined || index === null) return;@@ -201,17 +204,26 @@ export function highlightPromoterPlugin() {const end = start + match[0].length;if (start > cursor) {const lead: Text = { type: 'text', value: value.slice(cursor, start) };+ const pos = deriveFragmentPosition(source, node, cursor, start);+ if (pos) lead.position = pos;replacements.push(lead);}+ const innerText: Text = { type: 'text', value: match[1] };+ const innerPos = deriveFragmentPosition(source, node, start + 2, end - 2);+ if (innerPos) innerText.position = innerPos;const markNode: MarkMdast = {type: 'mark',- children: [{ type: 'text', value: match[1] }],+ children: [innerText],};+ const markPos = deriveFragmentPosition(source, node, start, end);+ if (markPos) markNode.position = markPos;replacements.push(markNode as unknown as PhrasingContent);cursor = end;}if (cursor < value.length) {const tail: Text = { type: 'text', value: value.slice(cursor) };+ const pos = deriveFragmentPosition(source, node, cursor, value.length);+ if (pos) tail.position = pos;replacements.push(tail);}diff --git a/packages/core/src/markdown/index.ts b/packages/core/src/markdown/index.tsindex 248a9e766..05203d59b 100644--- a/packages/core/src/markdown/index.ts+++ b/packages/core/src/markdown/index.ts@@ -15,6 +15,7 @@ import type {Break,Code,Definition,+ Delete,Emphasis,FootnoteDefinition,FootnoteReference,@@ -109,13 +110,7 @@ export class MarkdownManager {pmNodeHandlers: this.pmNodeHandlers,pmMarkHandlers: this.pmMarkHandlers,});- this.serializeProcessor = createSerializeProcessor({- schema: this.schema,- handlers: this.handlers,- pmNodeHandlers: this.pmNodeHandlers,- pmMarkHandlers: this.pmMarkHandlers,- toMarkdownHandlers,- });+ this.serializeProcessor = createSerializeProcessor({ toMarkdownHandlers });}parse(markdown: string, opts?: ParseContext): JSONContent {@@ -240,6 +235,7 @@ function isEmptyMdastParagraph(node: MdastNodes): boolean {interface InlineCodeFidelityData {sourceFenceChar?: string;sourceFenceLength?: number;+ sourcePadded?: boolean;}function withInlineCodeData(@@ -252,6 +248,9 @@ function withInlineCodeData(if (typeof data.sourceFenceLength === 'number') {dataEntry.sourceFenceLength = data.sourceFenceLength;}+ if (data.sourcePadded === true) {+ dataEntry.sourcePadded = true;+ }if (Object.keys(dataEntry).length === 0) return node as unknown as MdastNodes;return { ...node, data: dataEntry } as unknown as MdastNodes;}@@ -299,7 +298,6 @@ import {FILE_ATTACHMENT_EXTENSIONS,IMAGE_EXTENSIONS,VIDEO_EXTENSIONS,- WIKI_EMBED_EXTENSIONS,} from '../constants/upload.ts';const WIKI_EMBED_IMAGE_EXTS = IMAGE_EXTENSIONS;@@ -386,7 +384,12 @@ function buildMdastToPmHandlers(if (rowNode) pmRows.push(rowNode);}const sourceDashCounts = node.data?.sourceDashCounts ?? null;- return n.table.createAndFill({ sourceDashCounts }, pmRows.length > 0 ? pmRows : null);+ const sourceOuterPipes = node.data?.sourceOuterPipes ?? null;+ const sourceAlignmentPadding = node.data?.sourceAlignmentPadding ?? null;+ return n.table.createAndFill(+ { sourceDashCounts, sourceOuterPipes, sourceAlignmentPadding },+ pmRows.length > 0 ? pmRows : null,+ );};}@@ -489,12 +492,17 @@ function buildMdastToPmHandlers(sourceFenceChar: node.data?.sourceFenceChar ?? '`',sourceFenceLength:typeof node.data?.sourceFenceLength === 'number' ? node.data.sourceFenceLength : 1,+ sourcePadded: node.data?.sourcePadded === true,}),]);}const strikeMark = m.strike ?? m.delete;- if (strikeMark) handlers.delete = toPmMark(strikeMark);+ if (strikeMark) {+ handlers.delete = toPmMark(strikeMark, (node: Delete) => ({+ sourceDelimiter: node.data?.sourceDelimiter === '~' ? '~' : '~~',+ }));+ }if (m.highlight) handlers.mark = toPmMark(m.highlight);@@ -530,6 +538,8 @@ function buildMdastToPmHandlers(sourceTrailingHashes: node.data?.sourceTrailingHashes ?? null,sourceUnderlineLength: node.data?.sourceUnderlineLength ?? null,sourceContiguousNext: node.data?.sourceContiguousNext ?? false,+ sourceLeadingIndent: node.data?.sourceLeadingIndent ?? null,+ sourceInteriorSpacing: node.data?.sourceInteriorSpacing ?? null,}));}@@ -543,6 +553,10 @@ function buildMdastToPmHandlers(fenceDelimiter: node.data?.sourceFenceChar ?? '`',fenceLength: node.data?.sourceFenceLength ?? 3,sourceStyle: node.data?.sourceStyle ?? 'fenced',+ sourceClosingFenceLength: node.data?.sourceClosingFenceLength ?? null,+ sourceFenceIndent: node.data?.sourceFenceIndent ?? null,+ sourceInfoPadding: node.data?.sourceInfoPadding ?? null,+ sourceIndents: node.data?.sourceIndents ?? null,},textContent,);@@ -576,6 +590,10 @@ function buildMdastToPmHandlers(handlers.listItem = toPmNode(n.listItem, (node: ListItem) => ({checked: node.checked ?? null,spread: !!node.spread,+ sourceMarkerSpacing: node.data?.sourceMarkerSpacing ?? null,+ sourceOrdinal: node.data?.sourceOrdinal ?? null,+ sourceCheckboxChar: node.data?.sourceCheckboxChar ?? null,+ sourceContinuationIndent: node.data?.sourceContinuationIndent ?? null,}));}@@ -742,6 +760,9 @@ function buildMdastToPmHandlers(target: node.data?.target ?? '',alias: node.data?.alias ?? null,anchor: node.data?.anchor ?? null,+ sourceTarget: node.data?.sourceTarget ?? null,+ sourceAnchor: node.data?.sourceAnchor ?? null,+ sourceAlias: node.data?.sourceAlias ?? null,});}@@ -833,7 +854,7 @@ function buildMdastToPmHandlers(}const label = alias || (anchor ? `${target}#${anchor}` : target);- if (WIKI_EMBED_EXTENSIONS.has(ext) && m.link) {+ if (m.link) {const linkMark = m.link.create({href: srcOrTarget,title: null,@@ -847,16 +868,6 @@ function buildMdastToPmHandlers(return schema.text(label, [linkMark]);}- if (m.link) {- const linkMark = m.link.create({- href: target,- title: null,- linkStyle: 'inline',- refLabel: null,- });- return schema.text(label, [linkMark]);- }-throw new Error('[wikiLinkEmbed handler] schema lacks `link` mark — cannot dispatch ' +'without violating the STOP rule against emitting PM wikiLinkEmbed server-side',@@ -898,8 +909,15 @@ function buildMdastToPmHandlers(};if (n.mathInline) {- handlers.inlineMath = (node: { type: 'inlineMath'; value?: string }) =>- n.mathInline.create({ formula: node.value ?? '' });+ handlers.inlineMath = (node: {+ type: 'inlineMath';+ value?: string;+ data?: { sourceDelimiter?: string };+ }) =>+ n.mathInline.create({+ formula: node.value ?? '',+ sourceDelimiter: node.data?.sourceDelimiter ?? null,+ });}if (!handlers.math) handlers.math = blockUnknownHandler;@@ -995,7 +1013,7 @@ function buildPmToMdastHandlers(schema: Schema): {if (n.blockquote) {nodeHandlers.blockquote = fromPmNode('blockquote', (pmNode: PmNode) => {const spacings = pmNode.attrs.sourceMarkerSpacings;- const data: { sourceMarkerSpacings?: Array<'single' | 'none'> } = {};+ const data: { sourceMarkerSpacings?: Array<number | 'single' | 'none'> } = {};if (Array.isArray(spacings) && spacings.length > 0) {data.sourceMarkerSpacings = spacings;}@@ -1011,6 +1029,8 @@ function buildPmToMdastHandlers(schema: Schema): {sourceTrailingHashes: pmNode.attrs.sourceTrailingHashes ?? undefined,sourceUnderlineLength: pmNode.attrs.sourceUnderlineLength ?? undefined,sourceContiguousNext: pmNode.attrs.sourceContiguousNext === true ? true : undefined,+ sourceLeadingIndent: pmNode.attrs.sourceLeadingIndent ?? undefined,+ sourceInteriorSpacing: pmNode.attrs.sourceInteriorSpacing ?? undefined,},}));}@@ -1030,6 +1050,10 @@ function buildPmToMdastHandlers(schema: Schema): {sourceFenceChar: pmNode.attrs.fenceDelimiter,sourceFenceLength: pmNode.attrs.fenceLength,sourceStyle,+ sourceClosingFenceLength: pmNode.attrs.sourceClosingFenceLength ?? undefined,+ sourceFenceIndent: pmNode.attrs.sourceFenceIndent ?? undefined,+ sourceInfoPadding: pmNode.attrs.sourceInfoPadding ?? undefined,+ sourceIndents: pmNode.attrs.sourceIndents ?? undefined,},};};@@ -1066,10 +1090,24 @@ function buildPmToMdastHandlers(schema: Schema): {const children = state.all(pmNode);const stripped =children.length > 1 && isEmptyMdastParagraph(children[0]) ? children.slice(1) : children;+ const itemData: NonNullable<ListItem['data']> = {};+ if (typeof pmNode.attrs.sourceMarkerSpacing === 'number') {+ itemData.sourceMarkerSpacing = pmNode.attrs.sourceMarkerSpacing;+ }+ if (typeof pmNode.attrs.sourceOrdinal === 'number') {+ itemData.sourceOrdinal = pmNode.attrs.sourceOrdinal;+ }+ if (pmNode.attrs.sourceCheckboxChar === 'X') {+ itemData.sourceCheckboxChar = 'X';+ }+ if (typeof pmNode.attrs.sourceContinuationIndent === 'number') {+ itemData.sourceContinuationIndent = pmNode.attrs.sourceContinuationIndent;+ }return {type: 'listItem' as const,checked: pmNode.attrs.checked ?? null,spread: pmNode.attrs.spread ?? false,+ ...(Object.keys(itemData).length > 0 ? { data: itemData } : {}),children: stripped,} as ListItem;};@@ -1090,10 +1128,25 @@ function buildPmToMdastHandlers(schema: Schema): {(c): c is TableRow => (c as MdastNodes).type === 'tableRow',);const sourceDashCounts = pmNode.attrs.sourceDashCounts;+ const tableData: NonNullable<Table['data']> = {};+ if (Array.isArray(sourceDashCounts) && sourceDashCounts.length > 0) {+ tableData.sourceDashCounts = sourceDashCounts as number[];+ }+ const outerPipes = pmNode.attrs.sourceOuterPipes;+ if (+ outerPipes !== null &&+ typeof outerPipes === 'object' &&+ typeof (outerPipes as { leading?: unknown }).leading === 'boolean' &&+ typeof (outerPipes as { trailing?: unknown }).trailing === 'boolean'+ ) {+ tableData.sourceOuterPipes = outerPipes as { leading: boolean; trailing: boolean };+ }+ const alignPadding = pmNode.attrs.sourceAlignmentPadding;+ if (Array.isArray(alignPadding) && alignPadding.length > 0) {+ tableData.sourceAlignmentPadding = alignPadding as Array<{ left: number; right: number }>;+ }const data: Table['data'] | undefined =- Array.isArray(sourceDashCounts) && sourceDashCounts.length > 0- ? { sourceDashCounts: sourceDashCounts as number[] }- : undefined;+ Object.keys(tableData).length > 0 ? tableData : undefined;const result: Table = {type: 'table' as const,align,@@ -1262,9 +1315,13 @@ function buildPmToMdastHandlers(schema: Schema): {children: [],} as unknown as MdastNodes;}+ const sourceDelimiter = pmNode.attrs.sourceDelimiter;return {type: 'inlineMath' as const,value: formula,+ ...(typeof sourceDelimiter === 'string' && sourceDelimiter.length > 0+ ? { data: { sourceDelimiter } }+ : {}),} as unknown as MdastNodes;};}@@ -1275,10 +1332,16 @@ function buildPmToMdastHandlers(schema: Schema): {const anchor: string | null = pmNode.attrs.anchor ?? null;const alias: string | null = pmNode.attrs.alias ?? null;const label = alias ? alias : anchor ? `${target}#${anchor}` : target;+ const sourceTarget: string | null =+ typeof pmNode.attrs.sourceTarget === 'string' ? pmNode.attrs.sourceTarget : null;+ const sourceAnchor: string | null =+ typeof pmNode.attrs.sourceAnchor === 'string' ? pmNode.attrs.sourceAnchor : null;+ const sourceAlias: string | null =+ typeof pmNode.attrs.sourceAlias === 'string' ? pmNode.attrs.sourceAlias : null;return {type: 'wikiLink' as const,value: label,- data: { target, anchor, alias },+ data: { target, anchor, alias, sourceTarget, sourceAnchor, sourceAlias },children: [{ type: 'text' as const, value: label }],} as unknown as MdastNodes;};@@ -1323,14 +1386,17 @@ function buildPmToMdastHandlers(schema: Schema): {markHandlers.code = (mark: PmMark, _parent: PmNode, children: MdastNodes[]) => {const sourceFenceChar = mark.attrs.sourceFenceChar as string | undefined;const sourceFenceLength = mark.attrs.sourceFenceLength as number | undefined;- return wrapAsInlineCode(children, { sourceFenceChar, sourceFenceLength });+ const sourcePadded = mark.attrs.sourcePadded === true ? true : undefined;+ return wrapAsInlineCode(children, { sourceFenceChar, sourceFenceLength, sourcePadded });};}const strikeMark = m.strike ?? m.delete;if (strikeMark) {const name = m.strike ? 'strike' : 'delete';- markHandlers[name] = fromPmMark('delete');+ markHandlers[name] = fromPmMark('delete', (mark: PmMark) => ({+ data: { sourceDelimiter: mark.attrs.sourceDelimiter === '~' ? '~' : '~~' },+ }));}if (m.highlight) {@@ -1380,11 +1446,18 @@ function buildPmToMdastHandlers(schema: Schema): {typeof mark.attrs.anchor === 'string' && mark.attrs.anchor.length > 0? mark.attrs.anchor: null;- const alias: string | null =+ let alias: string | null =typeof mark.attrs.alias === 'string' && mark.attrs.alias.length > 0? mark.attrs.alias: null;- const label = alias ? alias : anchor ? `${target}#${anchor}` : target;+ let label = alias ? alias : anchor ? `${target}#${anchor}` : target;+ const visibleText = children+ .map((child) => ('value' in child && typeof child.value === 'string' ? child.value : ''))+ .join('');+ if (visibleText !== label && visibleText.trim().length > 0 && !visibleText.includes(']]')) {+ alias = visibleText;+ label = visibleText;+ }return {type: 'wikiLinkEmbed' as const,value: label,diff --git a/packages/core/src/markdown/inline-promoter-position-preservation.test.ts b/packages/core/src/markdown/inline-promoter-position-preservation.test.tsnew file mode 100644index 000000000..e99ac4d24--- /dev/null+++ b/packages/core/src/markdown/inline-promoter-position-preservation.test.ts@@ -0,0 +1,219 @@+import { describe, expect, test } from 'bun:test';+import type { Root as MdastRoot, Nodes } from 'mdast';+import { visit } from 'unist-util-visit';++import { sharedExtensions } from '../extensions/shared.ts';+import { createParseProcessor, parseMdToMdast } from './pipeline.ts';++const proc = createParseProcessor({ extensions: sharedExtensions });++function parse(source: string): MdastRoot {+ return parseMdToMdast(source, proc);+}++function findNodesByType(tree: MdastRoot, type: string): Nodes[] {+ const found: Nodes[] = [];+ visit(tree, type, (node) => {+ found.push(node as Nodes);+ });+ return found;+}++function assertHasPosition(node: Nodes) {+ expect(node.position).toBeDefined();+ expect(typeof node.position?.start.offset).toBe('number');+ expect(typeof node.position?.end?.offset).toBe('number');+}++function getStartOffset(node: Nodes): number {+ return node.position?.start?.offset ?? -1;+}++function getEndOffset(node: Nodes): number {+ return node.position?.end?.offset ?? -1;+}++function slicePosition(source: string, node: Nodes): string {+ const s = getStartOffset(node);+ const e = getEndOffset(node);+ if (s < 0 || e < 0) return '';+ return source.slice(s, e);+}++describe('inline-promoter position preservation', () => {+ describe('single-dollar-math ($x$)', () => {+ test('inlineMath node has position pointing to $', () => {+ const source = 'Hello $x$ world';+ const tree = parse(source);+ const nodes = findNodesByType(tree, 'inlineMath');+ expect(nodes.length).toBe(1);+ assertHasPosition(nodes[0]);+ expect(source[getStartOffset(nodes[0])]).toBe('$');+ });++ test('text fragments around $a$ have position', () => {+ const source = 'Hello $a$ world';+ const tree = parse(source);+ const paragraph = tree.children[0] as Nodes & { children: Nodes[] };+ for (const child of paragraph.children) {+ assertHasPosition(child);+ }+ });++ test('multiple math nodes preserve ordered positions', () => {+ const source = 'See $a$ and $b$ here';+ const tree = parse(source);+ const nodes = findNodesByType(tree, 'inlineMath');+ expect(nodes.length).toBe(2);+ for (const n of nodes) assertHasPosition(n);+ expect(getStartOffset(nodes[0])).toBeLessThan(getStartOffset(nodes[1]));+ });+ });++ describe('highlight (==text==)', () => {+ test('mark node has position starting at ==', () => {+ const source = 'Hello ==word== world';+ const tree = parse(source);+ const nodes = findNodesByType(tree, 'mark');+ expect(nodes.length).toBe(1);+ assertHasPosition(nodes[0]);+ expect(source.slice(getStartOffset(nodes[0]), getStartOffset(nodes[0]) + 2)).toBe('==');+ });++ test('text fragments around ==a== have position', () => {+ const source = 'Hello ==a== world';+ const tree = parse(source);+ const paragraph = tree.children[0] as Nodes & { children: Nodes[] };+ for (const child of paragraph.children) {+ assertHasPosition(child);+ }+ });+ });++ describe('comment (%%text%%)', () => {+ test('%%note%% comment has position starting at %%', () => {+ const source = 'Hello %%note%% world';+ const tree = parse(source);+ const nodes = findNodesByType(tree, 'comment');+ expect(nodes.length).toBe(1);+ assertHasPosition(nodes[0]);+ expect(source.slice(getStartOffset(nodes[0]), getStartOffset(nodes[0]) + 2)).toBe('%%');+ });++ test('<!-- html --> comment has position', () => {+ const source = 'Hello <!-- note --> world';+ const tree = parse(source);+ const nodes = findNodesByType(tree, 'comment');+ expect(nodes.length).toBe(1);+ assertHasPosition(nodes[0]);+ });+ });++ describe('tag (#tagname)', () => {+ test('#hello tag has position pointing to #', () => {+ const source = 'See #hello there';+ const tree = parse(source);+ const nodes = findNodesByType(tree, 'tag');+ expect(nodes.length).toBe(1);+ assertHasPosition(nodes[0]);+ expect(source[getStartOffset(nodes[0])]).toBe('#');+ });++ test('multiple tags preserve position', () => {+ const source = '#a and #b';+ const tree = parse(source);+ const nodes = findNodesByType(tree, 'tag');+ expect(nodes.length).toBe(2);+ for (const n of nodes) assertHasPosition(n);+ });+ });++ describe('autolink (<scheme:uri>)', () => {+ test('autolink link has position', () => {+ const source = 'Visit <https://x.com> now';+ const tree = parse(source);+ const links: Nodes[] = [];+ visit(tree, 'link', (node) => {+ const data = (node as Nodes & { data?: Record<string, unknown> }).data;+ if (data?.sourceStyle === 'autolink') links.push(node as Nodes);+ });+ expect(links.length).toBe(1);+ assertHasPosition(links[0]);+ });+ });++ describe('chain: mixed promoters in one paragraph', () => {+ test('$x$ + ==highlight== both have position', () => {+ const source = 'Math $x$ and ==hi== text';+ const tree = parse(source);+ const math = findNodesByType(tree, 'inlineMath');+ const marks = findNodesByType(tree, 'mark');+ expect(math.length).toBe(1);+ expect(marks.length).toBe(1);+ assertHasPosition(math[0]);+ assertHasPosition(marks[0]);+ });++ test('#tag + %%comment%% both have position', () => {+ const source = 'See #tag %%note%% end';+ const tree = parse(source);+ const tags = findNodesByType(tree, 'tag');+ const comments = findNodesByType(tree, 'comment');+ expect(tags.length).toBe(1);+ expect(comments.length).toBe(1);+ assertHasPosition(tags[0]);+ assertHasPosition(comments[0]);+ });+ });++ describe('escape-aware offset accuracy', () => {+ test('\\* before $x$ — math offset points to $', () => {+ const source = '\\*text\\* and $x$ here';+ const tree = parse(source);+ const nodes = findNodesByType(tree, 'inlineMath');+ expect(nodes.length).toBeGreaterThanOrEqual(1);+ expect(source[getStartOffset(nodes[0])]).toBe('$');+ });++ test('\\# is not promoted to tag', () => {+ const source = 'See \\#notag here';+ const tree = parse(source);+ const tags = findNodesByType(tree, 'tag');+ expect(tags.length).toBe(0);+ });+ });++ describe('position byte accuracy', () => {+ test('mark position spans ==world==', () => {+ const source = 'Hello ==world== there';+ const tree = parse(source);+ const marks = findNodesByType(tree, 'mark');+ expect(marks.length).toBe(1);+ expect(slicePosition(source, marks[0])).toBe('==world==');+ });++ test('inlineMath position spans $abc$', () => {+ const source = 'Hello $abc$ there';+ const tree = parse(source);+ const nodes = findNodesByType(tree, 'inlineMath');+ expect(nodes.length).toBe(1);+ expect(slicePosition(source, nodes[0])).toBe('$abc$');+ });++ test('tag position spans #hello', () => {+ const source = 'See #hello there';+ const tree = parse(source);+ const tags = findNodesByType(tree, 'tag');+ expect(tags.length).toBe(1);+ expect(slicePosition(source, tags[0])).toBe('#hello');+ });++ test('comment position spans %%note%%', () => {+ const source = 'See %%note%% there';+ const tree = parse(source);+ const comments = findNodesByType(tree, 'comment');+ expect(comments.length).toBe(1);+ expect(slicePosition(source, comments[0])).toBe('%%note%%');+ });+ });+});diff --git a/packages/core/src/markdown/math-block-guard.precision.test.ts b/packages/core/src/markdown/math-block-guard.precision.test.tsindex 9a16dbdb7..af928eac7 100644--- a/packages/core/src/markdown/math-block-guard.precision.test.ts+++ b/packages/core/src/markdown/math-block-guard.precision.test.ts@@ -77,11 +77,11 @@ describe('inline math (single-line `$$x$$` only) → mathInline atom', () => {expect(out).toBe(source);});- test('single-dollar `$x^2$` normalizes to `$$x^2$$` on serialize (D-M5b normalization)', () => {+ test('single-dollar `$x^2$` round-trips byte-identical on serialize', () => {const source = 'Result: $x^2$.\n';const json = mdManager.parse(source);const out = mdManager.serialize(json);- expect(out).toBe('Result: $$x^2$$.\n');+ expect(out).toBe('Result: $x^2$.\n');});test('`<InlineMath formula="x" id="eq-1" />` round-trips byte-identical (id-bearing JSX form)', () => {diff --git a/packages/core/src/markdown/mdast-augmentation.ts b/packages/core/src/markdown/mdast-augmentation.tsindex 04b84e4d6..757424675 100644--- a/packages/core/src/markdown/mdast-augmentation.ts+++ b/packages/core/src/markdown/mdast-augmentation.ts@@ -23,6 +23,9 @@ export interface WikiLinkMdast {target: string;alias: string | null;anchor: string | null;+ sourceTarget?: string | null;+ sourceAnchor?: string | null;+ sourceAlias?: string | null;[key: string]: unknown;};children: Array<{ type: 'text'; value: string }>;@@ -36,6 +39,12 @@ export interface WikiLinkEmbedMdast {target: string;alias: string | null;anchor: string | null;+ /** Untrimmed source segments — see WikiLinkMdast. Captured by the+ * shared micromark exits; the embed serialization path currently+ * ignores them (embed fidelity is tracked separately). */+ sourceTarget?: string | null;+ sourceAnchor?: string | null;+ sourceAlias?: string | null;[key: string]: unknown;};children: Array<{ type: 'text'; value: string }>;@@ -91,7 +100,20 @@ export interface TagMdast {position?: Position;}+export interface SourceDocBoundary {+ bom?: true;+ leading?: string;+ trailing?: string;+ gapBlankLines?: ReadonlyArray<number | null>;+}+declare module 'mdast' {+ interface Data {+ sourcePrecedingBlankLines?: number;+ }+ interface RootData {+ sourceDocBoundary?: SourceDocBoundary;+ }interface TextData {escapedChars?: Array<{ offset: number; char: string }>;sourceRaw?: string;@@ -121,27 +143,45 @@ declare module 'mdast' {interface HeadingData {sourceStyle?: string;sourceTrailingHashes?: number;+ sourceLeadingIndent?: number;+ sourceInteriorSpacing?: number;sourceUnderlineLength?: number;sourceContiguousNext?: boolean;}interface CodeData {sourceFenceChar?: string;sourceFenceLength?: number;+ sourceClosingFenceLength?: number;+ sourceFenceIndent?: number;+ sourceInfoPadding?: number;sourceStyle?: 'indented' | 'fenced';+ sourceIndents?: string[];}interface InlineCodeData {sourceFenceChar?: string;sourceFenceLength?: number;+ sourcePadded?: boolean;}interface ListData {bulletMarker?: string;listMarkerDelimiter?: string;}+ interface ListItemData {+ sourceMarkerSpacing?: number;+ sourceOrdinal?: number;+ sourceCheckboxChar?: 'X';+ sourceContinuationIndent?: number;+ }+ interface DeleteData {+ sourceDelimiter?: '~' | '~~';+ }interface BlockquoteData {- sourceMarkerSpacings?: Array<'single' | 'none'>;+ sourceMarkerSpacings?: Array<number | 'single' | 'none'>;}interface TableData {sourceDashCounts?: number[];+ sourceOuterPipes?: { leading: boolean; trailing: boolean };+ sourceAlignmentPadding?: Array<{ left: number; right: number }>;}interface TableCellData {sourcePadding?: { left: number; right: number };@@ -179,3 +219,9 @@ declare module 'mdast-util-mdx-expression' {sourceRaw?: string;}}++declare module 'mdast-util-math' {+ interface InlineMathData {+ sourceDelimiter?: string;+ }+}diff --git a/packages/core/src/markdown/mdast-to-hast-handlers.ts b/packages/core/src/markdown/mdast-to-hast-handlers.tsindex 97f9c528c..679c72ef7 100644--- a/packages/core/src/markdown/mdast-to-hast-handlers.ts+++ b/packages/core/src/markdown/mdast-to-hast-handlers.ts@@ -2,6 +2,8 @@ import type { Comment, Element, ElementContent, Properties } from 'hast';import type { FootnoteDefinition, FootnoteReference } from 'mdast';import type { MdxJsxFlowElement, MdxJsxTextElement } from 'mdast-util-mdx';import type { Handler, Handlers } from 'mdast-util-to-hast';+import { builtInComponents } from '../registry/built-ins.ts';+import type { CompatMeta } from '../registry/types.ts';import { wikiLinkHref } from '../utils/slug.ts';import type {MarkMdast,@@ -14,25 +16,58 @@ import type {const HTML_PRIMITIVE_TAGS = new Set(['img', 'video', 'audio', 'mark']);-function tryNativeHtmlPrimitive(node: MdxJsxFlowElement | MdxJsxTextElement): Element | null {- const name = node.name;- if (!name || !HTML_PRIMITIVE_TAGS.has(name)) return null;- const properties: Properties = {};+const compatPrimitiveByName: ReadonlyMap<string, CompatMeta> = new Map(+ builtInComponents.flatMap((meta) =>+ meta.surface === 'compat' && HTML_PRIMITIVE_TAGS.has(meta.rendersAs)+ ? [[meta.name, meta] as const]+ : [],+ ),+);++function collectStaticJsxAttributes(+ node: MdxJsxFlowElement | MdxJsxTextElement,+): Record<string, string | true> | null {+ const bag: Record<string, string | true> = {};for (const attr of node.attributes) {if (attr.type !== 'mdxJsxAttribute') return null;const lowerName = attr.name.toLowerCase();if (lowerName.length >= 3 && lowerName.startsWith('on')) continue;if (attr.value === null) {- properties[attr.name] = true;+ bag[attr.name] = true;} else if (typeof attr.value === 'string') {- properties[attr.name] = attr.value;+ bag[attr.name] = attr.value;} else {return null;}}+ return bag;+}++function tryNativeHtmlPrimitive(node: MdxJsxFlowElement | MdxJsxTextElement): Element | null {+ const name = node.name;+ if (!name || !HTML_PRIMITIVE_TAGS.has(name)) return null;+ const properties = collectStaticJsxAttributes(node);+ if (properties === null) return null;return { type: 'element', tagName: name, properties, children: [] };}+function tryCompatPrimitive(node: MdxJsxFlowElement | MdxJsxTextElement): Element | null {+ const name = node.name;+ if (!name) return null;+ const meta = compatPrimitiveByName.get(name);+ if (!meta) return null;+ const bag = collectStaticJsxAttributes(node);+ if (bag === null) return null;+ const translated = meta.translateProps(bag);+ const properties: Properties = {};+ for (const [key, value] of Object.entries(translated)) {+ if (typeof value === 'string' || typeof value === 'boolean') {+ properties[key] = value;+ }+ }+ return { type: 'element', tagName: meta.rendersAs, properties, children: [] };+}+const wikiLinkHandler: Handler = (state, node) => {const wiki = node as WikiLinkMdast;const { target, anchor, alias } = wiki.data;@@ -81,6 +116,15 @@ const mdxJsxFlowHandler: Handler = (state, node) => {state.patch(node, native);return state.applyData(node, native);}+ const compat = tryCompatPrimitive(jsx);+ if (compat) {+ state.patch(node, compat);+ const resolved = state.applyData(node, compat) as Element;+ if (resolved.tagName === 'img') {+ return { type: 'element', tagName: 'p', properties: {}, children: [resolved] };+ }+ return resolved;+ }const raw = typeof jsx.data?.sourceRaw === 'string' ? jsx.data.sourceRaw : '';const code: Element = {type: 'element',@@ -108,6 +152,11 @@ const mdxJsxTextHandler: Handler = (state, node) => {state.patch(node, native);return state.applyData(node, native);}+ const compat = tryCompatPrimitive(jsx);+ if (compat) {+ state.patch(node, compat);+ return state.applyData(node, compat);+ }const raw = typeof jsx.data?.sourceRaw === 'string' ? jsx.data.sourceRaw : '';const span: Element = {type: 'element',diff --git a/packages/core/src/markdown/mdast-to-html.test.ts b/packages/core/src/markdown/mdast-to-html.test.tsindex 90c34609a..732140210 100644--- a/packages/core/src/markdown/mdast-to-html.test.ts+++ b/packages/core/src/markdown/mdast-to-html.test.ts@@ -1,6 +1,14 @@import { describe, expect, test } from 'bun:test';+import { sharedExtensions } from '../extensions/shared.ts';+import { MarkdownManager } from './index.ts';import { markdownToHtml, mdastToHtml } from './mdast-to-html.ts';+const mdManager = new MarkdownManager({ extensions: sharedExtensions });++function renderFaithful(md: string): string {+ return mdastToHtml(mdManager.parseToMdast(md));+}+describe('markdownToHtml — markdown string → HTML', () => {test('paragraph', () => {expect(markdownToHtml('hello world')).toBe('<p>hello world</p>');@@ -258,3 +266,96 @@ describe('URL scheme filter — outbound clipboard HTML sanitization', () => {expect(html2).not.toMatch(/data:/i);});});++describe('faithful render path — compat media descriptors resolve to native elements', () => {+ test('CommonMarkImage emits <p><img></p>, not the mdx-component placeholder', () => {+ expect(renderFaithful('')).toBe(+ '<p><img src="/url" alt="foo" title="title"></p>',+ );+ });++ test('empty alt is preserved as alt="" (decorative opt-in), not dropped', () => {+ expect(renderFaithful('')).toBe('<p><img src="/url" alt=""></p>');+ });++ test('nested image description flattens to plain-text alt', () => {+ expect(renderFaithful('](/url2)')).toBe(+ '<p><img src="/url2" alt="foo bar"></p>',+ );+ });++ test('inline image inside prose stays a native inline <img>', () => {+ expect(renderFaithful('before  after')).toBe(+ '<p>before <img src="/i.png" alt="pic"> after</p>',+ );+ });++ test('lowercase MDX <img> keeps its bare flow shape (no paragraph wrap)', () => {+ expect(renderFaithful('<img src="/x.png" alt="y" />')).toBe('<img src="/x.png" alt="y">');+ });++ test('Callout component descriptor keeps the mdx-component placeholder', () => {+ const html = renderFaithful('> [!NOTE]\n> body');+ expect(html).toContain('mdx-component');+ expect(html).not.toContain('<img');+ });++ test('event-handler attributes are skipped on the compat path', () => {+ const html = renderFaithful('<CommonMarkImage src="/x.png" alt="y" onerror="alert(1)" />');+ expect(html).toContain('<img');+ expect(html).not.toContain('onerror');+ });+});++describe('faithful render path — source-preserved url/title attrs decode at render', () => {+ test('empty angle-bracket destination renders href=""', () => {+ expect(renderFaithful('[link](<>)')).toBe('<p><a href="">link</a></p>');+ });++ test('empty angle-bracket destination via reference definition renders href=""', () => {+ expect(renderFaithful('[foo]: <>\n\n[foo]')).toBe('<p><a href="">foo</a></p>');+ });++ test('entity refs in a link title decode to their characters', () => {+ expect(renderFaithful('[link](/url "title \\""")')).toBe(+ '<p><a href="/url" title="title """>link</a></p>',+ );+ });++ test('entity refs in a destination decode then percent-encode like the spec', () => {+ expect(renderFaithful('[foo](/föö "föö")')).toBe(+ '<p><a href="/f%C3%B6%C3%B6" title="föö">foo</a></p>',+ );+ });++ test('entity-SHAPED-but-invalid references stay literal', () => {+ const html = renderFaithful('[x](/a "t&hl;t")');+ expect(html).toContain('t&hl;t');+ });++ test('entity refs in a fenced-code info string decode in the language- class', () => {+ expect(renderFaithful('```föö\ncode\n```')).toBe(+ '<pre><code class="language-föö">code\n</code></pre>',+ );+ });++ test('entity-SHAPED-but-invalid info-string reference stays literal in the class', () => {+ const html = renderFaithful('```f&hl;\ncode\n```');+ expect(html).toContain('language-f&hl;');+ });++ test('body-text entity refs are NOT decoded (storage-fidelity dialect untouched)', () => {+ const html = renderFaithful('& in prose');+ expect(html).toContain('&amp; in prose');+ });++ test('entity-obfuscated javascript: scheme is still stripped', () => {+ const html = renderFaithful('[x](javascript:alert(1))');+ expect(html).not.toMatch(/javascript:/i);+ });++ test('entity-obfuscated scheme on an image src is still stripped', () => {+ const html = renderFaithful(')');+ expect(html).not.toMatch(/javascript:/i);+ });+});diff --git a/packages/core/src/markdown/mdast-to-html.ts b/packages/core/src/markdown/mdast-to-html.tsindex be6c79033..79072375c 100644--- a/packages/core/src/markdown/mdast-to-html.ts+++ b/packages/core/src/markdown/mdast-to-html.ts@@ -1,5 +1,6 @@import type { Element, Root as HastRoot } from 'hast';import type { Root as MdastRoot } from 'mdast';+import { normalizeUri } from 'micromark-util-sanitize-uri';import rehypeStringify from 'rehype-stringify';import remarkFrontmatter from 'remark-frontmatter';import remarkGfm from 'remark-gfm';@@ -7,6 +8,7 @@ import remarkParse from 'remark-parse';import remarkRehype from 'remark-rehype';import { type Plugin, unified } from 'unified';import { visit } from 'unist-util-visit';+import { decodeEntityRefs } from './entity-ref-guard.ts';import { customNodeHandlers } from './mdast-to-hast-handlers.ts';import { remarkMdxAgnostic } from './remark-mdx-agnostic.ts';import { isSafeUrl } from './safe-url.ts';@@ -33,6 +35,53 @@ function isSrcsetValueSafe(srcset: string): boolean {return true;}+const DECODE_ATTRS_BY_TAG: Record<string, readonly string[]> = {+ a: ['href', 'title'],+ img: ['src', 'alt', 'title'],+ video: ['src', 'title'],+ audio: ['src', 'title'],+ code: ['className'],+};++const URL_ATTRS = new Set(['href', 'src']);++const NORMALIZED_EMPTY_ANGLE_DEST = normalizeUri('<>');++const rehypeDecodeSourcePreservedAttrs: Plugin<[], HastRoot> = () => {+ return (tree) => {+ visit(tree, 'element', (node: Element) => {+ const attrs = DECODE_ATTRS_BY_TAG[node.tagName.toLowerCase()];+ const props = node.properties;+ if (!attrs || !props) return;+ for (const key of attrs) {+ const value = props[key];+ if (Array.isArray(value)) {+ const decodedItems = value.map((item) =>+ typeof item === 'string' ? decodeEntityRefs(item) : item,+ );+ if (decodedItems.some((item, i) => item !== value[i])) {+ props[key] = decodedItems;+ }+ continue;+ }+ if (typeof value !== 'string') continue;+ let decoded: string;+ if (URL_ATTRS.has(key)) {+ if (value === '<>' || value === NORMALIZED_EMPTY_ANGLE_DEST) {+ decoded = '';+ } else {+ const chars = decodeEntityRefs(value);+ decoded = chars === value ? value : normalizeUri(chars);+ }+ } else {+ decoded = decodeEntityRefs(value);+ }+ if (decoded !== value) props[key] = decoded;+ }+ });+ };+};+const rehypeSanitizeUrls: Plugin<[], HastRoot> = () => {return (tree) => {visit(tree, 'element', (node: Element) => {@@ -93,6 +142,7 @@ const rehypeSanitizeUrls: Plugin<[], HastRoot> = () => {export function mdastToHtml(tree: MdastRoot): string {const processor = unified().use(remarkRehype, { handlers: customNodeHandlers })+ .use(rehypeDecodeSourcePreservedAttrs).use(rehypeSanitizeUrls).use(rehypeStringify);const hast = processor.runSync(tree) as unknown as HastRoot;@@ -107,6 +157,7 @@ export function markdownToHtml(md: string): string {.use(remarkGfm).use(remarkWikiLink).use(remarkRehype, { handlers: customNodeHandlers })+ .use(rehypeDecodeSourcePreservedAttrs).use(rehypeSanitizeUrls).use(rehypeStringify);return String(processor.processSync(md));diff --git a/packages/core/src/markdown/merged-walker.test.ts b/packages/core/src/markdown/merged-walker.test.tsindex 7055a192b..a2611acd3 100644--- a/packages/core/src/markdown/merged-walker.test.ts+++ b/packages/core/src/markdown/merged-walker.test.ts@@ -34,7 +34,7 @@ describe('mergedPostParseWalkerPlugin — Phase B dispatch', () => {expect(para.children[0].data?.sourceStyle).toBe('autolink');});- test('pass 3: doc-start empty yaml → thematicBreak(s) with data.sourceRaw', () => {+ test('pass 3 retired: a root-position empty yaml node is left untouched', () => {const src = '---\n\n---\n';const tree: MdastRoot = {type: 'root',@@ -45,11 +45,8 @@ describe('mergedPostParseWalkerPlugin — Phase B dispatch', () => {};mergedPostParseWalkerPlugin()(tree, new VFile(src));- expect(tree.children.length).toBe(2);- expect(tree.children[0].type).toBe('thematicBreak');- expect(tree.children[1].type).toBe('thematicBreak');- expect(tree.children[0].data?.sourceRaw).toBe('---');- expect(tree.children[1].data?.sourceRaw).toBe('---');+ expect(tree.children.length).toBe(1);+ expect(tree.children[0].type).toBe('yaml');});test('pass 4: attaches sourceDelimiter to emphasis node based on source', () => {@@ -183,7 +180,7 @@ describe('mergedPostParseWalkerPlugin — Phase B dispatch', () => {expect(fallback.originalType).toBe('brandNew');});- test('non-root-first yaml is left untouched (pass 3 is root[0]-specific)', () => {+ test('non-root-first yaml is left untouched', () => {const tree = {type: 'root',children: [diff --git a/packages/core/src/markdown/merged-walker.ts b/packages/core/src/markdown/merged-walker.tsindex 873e3f2aa..35d74b85e 100644--- a/packages/core/src/markdown/merged-walker.ts+++ b/packages/core/src/markdown/merged-walker.ts@@ -2,7 +2,6 @@ import type { Nodes, Parent, Root } from 'mdast';import { SKIP, visit } from 'unist-util-visit';import type { VFile } from 'vfile';import { promoteInParent } from './autolink-promotion.ts';-import { applyDocStartThematicFix } from './doc-start-thematic-fix.ts';import { applyPositionSliceToNode } from './position-slice.ts';import { promoteTagsInParent } from './tag-promotion.ts';import { KNOWN_MDAST_TYPES, toRawMdxFallbackMdast } from './unknown-mdast-guard.ts';@@ -11,8 +10,6 @@ export function mergedPostParseWalkerPlugin() {return (tree: Root, file: VFile) => {const source = typeof file.value === 'string' ? file.value : '';- applyDocStartThematicFix(tree, file);-const debug = typeof process !== 'undefined' && process.env?.OK_DEBUG_POSITION_SLICE === '1';visit(tree, (node, index, parent) => {@@ -30,12 +27,17 @@ export function mergedPostParseWalkerPlugin() {if ('children' in node && Array.isArray((node as Parent).children)) {const parentLike = node as Parent;if (parentLike.children.some((c) => c.type === 'text')) {- promoteInParent(parentLike);+ promoteInParent(parentLike, source);promoteTagsInParent(parentLike, source);}}- applyPositionSliceToNode(node as Nodes, source, debug);+ applyPositionSliceToNode(+ node as Nodes,+ source,+ debug,+ parent as { type?: string } | undefined,+ );if (node.type === 'heading' &&diff --git a/packages/core/src/markdown/parser-drop-closure.ts b/packages/core/src/markdown/parser-drop-closure.tsnew file mode 100644index 000000000..e6ab4fd57--- /dev/null+++ b/packages/core/src/markdown/parser-drop-closure.ts@@ -0,0 +1,411 @@+export type DroppedTokenAdjudication =+ | {+ kind: 'format-dof-axis';+ axisIds: readonly string[];+ rationale: string;+ }+ | {+ kind: 'structural-only';+ rationale: string;+ }+ | {+ kind: 'retained-by-capture';+ witness: string;+ rationale: string;+ }+ | {+ kind: 'documented-residual';+ witnesses: ReadonlyArray<{ input: string; roundTrip: string }>;+ rationale: string;+ };++export const DROPPED_TOKEN_ADJUDICATIONS: Record<string, DroppedTokenAdjudication> = {+ atxHeadingText: {+ kind: 'structural-only',+ rationale: 'heading-text container; bytes ride inside its phrasing children',+ },+ autolinkMarker: {+ kind: 'format-dof-axis',+ axisIds: ['autolink:angle'],+ rationale: 'the < > wrapper of an autolink; the angle form is the witnessed axis',+ },+ blockQuoteMarker: {+ kind: 'format-dof-axis',+ axisIds: ['blockquote:marker-spacing'],+ rationale: 'the > marker byte; its spacing variation is the witnessed axis',+ },+ blockQuotePrefix: {+ kind: 'format-dof-axis',+ axisIds: ['blockquote:marker-spacing'],+ rationale: 'marker + following whitespace; covered by the marker-spacing axis',+ },+ blockQuotePrefixWhitespace: {+ kind: 'format-dof-axis',+ axisIds: ['blockquote:marker-spacing'],+ rationale:+ 'the whitespace after the > marker; the marker-spacing axis varies exactly these bytes',+ },+ characterReferenceMarker: {+ kind: 'format-dof-axis',+ axisIds: ['lexical:char-reference'],+ rationale:+ 'the & and ; of an entity reference; preserved by the entity-ref guard, witnessed by the lexical axis',+ },+ chunkContent: {+ kind: 'structural-only',+ rationale: 'tokenizer re-chunking artifact; spans only bytes owned by nested tokens',+ },+ chunkDocument: {+ kind: 'structural-only',+ rationale: 'tokenizer re-chunking artifact; spans only bytes owned by nested tokens',+ },+ chunkFlow: {+ kind: 'structural-only',+ rationale: 'tokenizer re-chunking artifact; spans only bytes owned by nested tokens',+ },+ chunkString: {+ kind: 'structural-only',+ rationale: 'tokenizer re-chunking artifact; spans only bytes owned by nested tokens',+ },+ chunkText: {+ kind: 'structural-only',+ rationale: 'tokenizer re-chunking artifact; spans only bytes owned by nested tokens',+ },+ codeFencedFenceSequence: {+ kind: 'format-dof-axis',+ axisIds: ['code-fence:char', 'code-fence:symmetric-length', 'code-fence:closing-length'],+ rationale:+ 'the fence run itself; char, length, and closing-length variation are the witnessed axes',+ },+ codeTextPadding: {+ kind: 'retained-by-capture',+ witness: '` x `',+ rationale: 'inline-code pad space; retained via the inlineCode sourcePadded capture',+ },+ codeTextSequence: {+ kind: 'retained-by-capture',+ witness: '``x``',+ rationale: 'inline-code fence run; retained via the inlineCode fence-length capture',+ },+ content: {+ kind: 'structural-only',+ rationale: 'flow-content container; spans only bytes owned by nested tokens',+ },+ definitionDestination: {+ kind: 'structural-only',+ rationale:+ 'destination container; bytes ride in its literal/raw child tokens into mdast definition.url',+ },+ definitionDestinationLiteral: {+ kind: 'retained-by-capture',+ witness: '[a]: </u v>',+ rationale: 'angle-bracketed definition destination; the angle form round-trips',+ },+ definitionDestinationLiteralMarker: {+ kind: 'retained-by-capture',+ witness: '[a]: </u v>',+ rationale: 'the < > of an angle-bracketed definition destination; round-trips with the literal',+ },+ definitionDestinationRaw: {+ kind: 'retained-by-capture',+ witness: '[a]: /u',+ rationale: 'raw definition destination; bytes land in mdast definition.url and re-emit',+ },+ definitionLabel: {+ kind: 'retained-by-capture',+ witness: '[A]: /u\n\n[a]',+ rationale: 'definition label with authored casing; the label byte form re-emits',+ },+ definitionLabelMarker: {+ kind: 'structural-only',+ rationale: 'fixed [ ] bytes around the definition label; no source-form freedom',+ },+ definitionMarker: {+ kind: 'structural-only',+ rationale: 'the fixed : byte after the definition label; no source-form freedom',+ },+ definitionTitle: {+ kind: 'retained-by-capture',+ witness: "[a]: /u 'ti'",+ rationale: 'definition title with authored quote style; the quote form re-emits',+ },+ definitionTitleMarker: {+ kind: 'retained-by-capture',+ witness: "[a]: /u 'ti'",+ rationale: 'the quote bytes of a definition title; round-trip with the title',+ },+ emphasisSequence: {+ kind: 'format-dof-axis',+ axisIds: ['emphasis:delimiter'],+ rationale: 'the * or _ run; delimiter choice is the witnessed axis',+ },+ emphasisText: {+ kind: 'structural-only',+ rationale: 'emphasis-content container; bytes ride inside its phrasing children',+ },+ escapeMarker: {+ kind: 'format-dof-axis',+ axisIds: ['lexical:backslash-escape'],+ rationale: 'the backslash of an escape; preserved end-to-end, witnessed by the lexical axis',+ },+ labelEnd: {+ kind: 'structural-only',+ rationale: 'fixed ] byte; the reference-form freedom is retained via mdast referenceType',+ },+ labelImage: {+ kind: 'structural-only',+ rationale: 'image-label container; fixed ![ ] bytes plus child phrasing',+ },+ labelImageMarker: {+ kind: 'structural-only',+ rationale: 'the fixed ! byte of an image label; no source-form freedom',+ },+ labelLink: {+ kind: 'structural-only',+ rationale: 'link-label container; fixed [ ] bytes plus child phrasing',+ },+ labelMarker: {+ kind: 'structural-only',+ rationale: 'fixed [ or ] byte; no source-form freedom',+ },+ lineEndingBlank: {+ kind: 'format-dof-axis',+ axisIds: [+ 'block-separator:blank-line-count',+ 'doc-boundary:doc-leading-blank',+ 'doc-boundary:doc-trailing-blank',+ ],+ rationale:+ 'blank-line newlines; leading/trailing/inter-block blank runs are the witnessed axes',+ },+ linePrefix: {+ kind: 'format-dof-axis',+ axisIds: ['lexical:tab-expansion', 'atx:leading-indent', 'code-fence:leading-indent'],+ rationale: 'leading line indent (tabs or spaces); the indent axes vary exactly these bytes',+ },+ lineSuffix: {+ kind: 'documented-residual',+ witnesses: [+ { input: 'a \nb', roundTrip: 'a\nb\n' },+ { input: 'a \nb', roundTrip: 'a \nb\n' },+ ],+ rationale:+ 'trailing intra-line whitespace: a single trailing space is stripped (CommonMark soft line ' ++ 'breaks treat it as presentational) and a hard-break space run beyond two normalizes to two. ' ++ 'Not an enumerated format-DOF axis and not captured today; pinned here so any shift is loud.',+ },+ listItemIndent: {+ kind: 'format-dof-axis',+ axisIds: ['list:nested-indent-width'],+ rationale: 'continuation indent of a list item; the nested-indent axis varies these bytes',+ },+ listItemMarker: {+ kind: 'format-dof-axis',+ axisIds: [+ 'list:bullet-plus',+ 'list:ordered-paren-delim',+ 'list:ordered-start-number',+ 'list:ordered-renumber',+ ],+ rationale: 'the bullet or ordinal marker; marker char and ordinal axes vary these bytes',+ },+ listItemPrefix: {+ kind: 'format-dof-axis',+ axisIds: ['list:item-marker-spacing'],+ rationale: 'marker + following whitespace; the marker-spacing axis varies these bytes',+ },+ listItemPrefixWhitespace: {+ kind: 'format-dof-axis',+ axisIds: ['list:item-marker-spacing'],+ rationale:+ 'the whitespace after a list marker; the marker-spacing axis varies exactly these bytes',+ },+ referenceMarker: {+ kind: 'structural-only',+ rationale:+ 'fixed [ ] bytes of a reference; the form freedom is retained via mdast referenceType',+ },+ resourceDestination: {+ kind: 'structural-only',+ rationale: 'destination container; bytes ride in its literal/raw child tokens into mdast url',+ },+ resourceDestinationLiteral: {+ kind: 'format-dof-axis',+ axisIds: ['link:angle-url'],+ rationale: 'angle-bracketed resource destination; the angle form is the witnessed axis',+ },+ resourceDestinationLiteralMarker: {+ kind: 'format-dof-axis',+ axisIds: ['link:angle-url'],+ rationale: 'the < > of an angle-bracketed destination; covered by the angle-url axis',+ },+ resourceDestinationRaw: {+ kind: 'retained-by-capture',+ witness: '[t](/u)',+ rationale: 'raw resource destination; bytes land in mdast url and re-emit via the link handler',+ },+ resourceMarker: {+ kind: 'structural-only',+ rationale: 'fixed ( ) bytes of a resource; no source-form freedom',+ },+ resourceTitle: {+ kind: 'structural-only',+ rationale:+ 'title container; the title value rides into mdast title, quote style via the title-quote axis',+ },+ resourceTitleMarker: {+ kind: 'format-dof-axis',+ axisIds: ['link:title-quote'],+ rationale: 'the quote bytes of a resource title; quote style is the witnessed axis',+ },+ setextHeadingLine: {+ kind: 'format-dof-axis',+ axisIds: ['setext:underline'],+ rationale: 'the setext underline run; the underline form is the witnessed axis',+ },+ strongSequence: {+ kind: 'format-dof-axis',+ axisIds: ['strong:delimiter'],+ rationale: 'the ** or __ run; delimiter choice is the witnessed axis',+ },+ strongText: {+ kind: 'structural-only',+ rationale: 'strong-content container; bytes ride inside its phrasing children',+ },+ thematicBreakSequence: {+ kind: 'format-dof-axis',+ axisIds: ['thematic:marker-mid-doc', 'thematic:doc-start-forcing'],+ rationale:+ 'the thematic-break marker run; marker form and doc-start forcing are the witnessed axes',+ },+ whitespace: {+ kind: 'documented-residual',+ witnesses: [{ input: '[a]: /u', roundTrip: '[a]: /u\n' }],+ rationale:+ 'inter-part whitespace inside definition/resource constructs collapses to a single space on ' ++ 'serialize. Not an enumerated format-DOF axis and not captured today; pinned here so any ' ++ 'shift is loud.',+ },+};++export type PreprocessTransformAdjudication =+ | { kind: 'format-dof-axis'; axisIds: readonly string[]; rationale: string }+ | { kind: 'structural-only'; rationale: string }+ | {+ kind: 'spec-mandated-replacement';+ witness: { input: string; roundTrip: string };+ rationale: string;+ };++export const PREPROCESS_TRANSFORM_ADJUDICATIONS: Record<string, PreprocessTransformAdjudication> = {+ '\0': {+ kind: 'spec-mandated-replacement',+ witness: { input: 'a\0b', roundTrip: 'a�b\n' },+ rationale:+ 'CommonMark 2.3 (insecure characters): U+0000 must be replaced with U+FFFD. A divergence no ' ++ 'fix may close; doubles as the instrument-alive control in the format-DOF gates.',+ },+ '\t': {+ kind: 'format-dof-axis',+ axisIds: ['lexical:tab-expansion'],+ rationale:+ 'tabs encode as virtual spaces in preprocess; the indent-tab byte form is the witnessed axis',+ },+ '\n': {+ kind: 'structural-only',+ rationale:+ 'LF chunk encoding; LF is the canonical line ending and round-trips byte-identically',+ },+ '\r': {+ kind: 'format-dof-axis',+ axisIds: ['doc-boundary:line-ending-normalize'],+ rationale:+ 'CR / CRLF encoding; line-ending form is the witnessed axis (OK retains interior CRLF)',+ },+ 'bom-head-check': {+ kind: 'format-dof-axis',+ axisIds: ['doc-boundary:bom-strip'],+ rationale:+ 'preprocess strips a document-head U+FEFF before any event exists; OK captures it with its ' ++ 'own pre-parse head check, witnessed by the bom-strip axis',+ },+};++export const PINNED_PREPROCESS_SOURCE_SHA256 =+ 'b582f16d9bc04d93d388004544be9c5293fda614eb86f8d02b97c3e57b9409e1';++export function extractFromMarkdownHandlerKeys(libSource: string): string[] {+ const keys = new Set<string>();+ for (const label of ['enter', 'exit']) {+ const start = libSource.indexOf(`${label}: {`);+ if (start < 0) throw new Error(`from-markdown source lost its "${label}" handler block`);+ let index = libSource.indexOf('{', start);+ let depth = 0;+ const lines: string[] = [];+ let lineStart = index + 1;+ for (; index < libSource.length; index++) {+ const ch = libSource[index];+ if (ch === '{') depth++;+ else if (ch === '}') {+ depth--;+ if (depth === 0) break;+ } else if (ch === '\n') {+ if (depth === 1) lines.push(libSource.slice(lineStart, index));+ lineStart = index + 1;+ }+ }+ if (depth !== 0) throw new Error(`unbalanced "${label}" handler block`);+ for (const line of lines) {+ const match = /^\s*([a-zA-Z][\w]*):/.exec(line);+ if (match?.[1]) keys.add(match[1]);+ }+ }+ if (keys.size === 0) throw new Error('extracted zero from-markdown handler keys');+ return [...keys].sort();+}++export function extractPreprocessTransforms(preprocessSource: string): {+ transformChars: string[];+ hasBomHeadCheck: boolean;+} {+ const match = /const search = \/\[(.+?)\]\/g/.exec(preprocessSource);+ if (!match?.[1]) throw new Error('preprocess source lost its search regex');+ const chars: string[] = [];+ const body = match[1];+ for (let i = 0; i < body.length; i++) {+ const ch = body[i];+ if (ch === '\\') {+ const next = body[i + 1];+ i++;+ if (next === '0') chars.push('\0');+ else if (next === 't') chars.push('\t');+ else if (next === 'n') chars.push('\n');+ else if (next === 'r') chars.push('\r');+ else if (next !== undefined) chars.push(next);+ } else if (ch !== undefined) {+ chars.push(ch);+ }+ }+ return {+ transformChars: chars,+ hasBomHeadCheck: /65279|byteOrderMarker/.test(preprocessSource),+ };+}++export interface ClosureCheckResult {+ unadjudicated: string[];+ stale: string[];+}++export function checkClosure(+ computed: readonly string[],+ adjudicated: Readonly<Record<string, unknown>>,+): ClosureCheckResult {+ const computedSet = new Set(computed);+ return {+ unadjudicated: computed.filter((key) => !(key in adjudicated)).sort(),+ stale: Object.keys(adjudicated)+ .filter((key) => !computedSet.has(key))+ .sort(),+ };+}diff --git a/packages/core/src/markdown/pipeline.doc-boundary.test.ts b/packages/core/src/markdown/pipeline.doc-boundary.test.tsnew file mode 100644index 000000000..59ce65552--- /dev/null+++ b/packages/core/src/markdown/pipeline.doc-boundary.test.ts@@ -0,0 +1,139 @@+import { describe, expect, test } from 'bun:test';+import type { JSONContent } from '@tiptap/core';+import { sharedExtensions } from '../extensions/shared.ts';+import { MarkdownManager } from './index.ts';++const md = new MarkdownManager({ extensions: sharedExtensions });++function rt(source: string): string {+ return md.serialize(md.parse(source));+}++function expectByteStable(out: string): void {+ expect(rt(out)).toBe(out);+}++describe('indented-code leading-indent capture (tab-expansion)', () => {+ test('a tab-indented code line round-trips byte-exact', () => {+ expect(rt('\tfoo\tbar')).toBe('\tfoo\tbar\n');+ expectByteStable('\tfoo\tbar\n');+ });++ test('multi-line tab indents round-trip per line', () => {+ expect(rt('\tfoo\n\tbar')).toBe('\tfoo\n\tbar\n');+ });++ test('mixed tab/space indents round-trip per line', () => {+ expect(rt('\tfoo\n bar')).toBe('\tfoo\n bar\n');+ });++ test('an interior blank line keeps its own (possibly partial) indent bytes', () => {+ expect(rt('\tfoo\n\n\tbar')).toBe('\tfoo\n\n\tbar\n');+ });++ test('canonical 4-space indents are unchanged (no capture noise)', () => {+ expect(rt(' foo')).toBe(' foo\n');+ expect(rt(' foo')).toBe(' foo\n');+ });++ test('a WYSIWYG-shaped edit that changes the line count falls back to the canonical indent', () => {+ const json = md.parse('\tfoo\n\tbar') as JSONContent;+ const code = json.content?.[0];+ expect(code?.type).toBe('codeBlock');+ if (code) code.content = [{ type: 'text', text: 'foo\nbar\nbaz' }];+ const out = md.serialize(json);+ expect(out).toBe(' foo\n bar\n baz\n');+ expectByteStable(out);+ });+});++describe('document-head BOM capture', () => {+ test('a BOM-led document round-trips the BOM', () => {+ expect(rt('\uFEFFhello')).toBe('\uFEFFhello\n');+ expectByteStable('\uFEFFhello\n');+ });++ test('the BOM composes with other captured boundary state', () => {+ expect(rt('\uFEFF\n\nhello')).toBe('\uFEFF\n\nhello\n');+ });+});++describe('BOM offset-rebase latent bug (first-node source-form mis-slice)', () => {+ test('a BOM-led spaced thematic break keeps its exact glyphs (non-doc-start)', () => {+ const out = rt('\uFEFFx\n\n- - -');+ expect(out).toBe('\uFEFFx\n\n- - -\n');+ expectByteStable(out);+ });++ test('a BOM-led ATX heading keeps its interior spacing capture', () => {+ expect(rt('\uFEFF## x')).toBe('\uFEFF## x\n');+ });++ test('a BOM-led doc-start dash break behaves like its non-BOM twin (glyphs preserved)', () => {+ expect(rt('\uFEFF- - -')).toBe('\uFEFF- - -\n');+ expect(rt('- - -')).toBe('- - -\n');+ expect(md.parseToMdast(rt('- - -')).children[0]?.type).toBe('thematicBreak');+ });++ test('non-BOM controls are byte-identical to their pre-capture behavior', () => {+ expect(rt('x\n\n- - -')).toBe('x\n\n- - -\n');+ expect(rt('## x')).toBe('## x\n');+ });+});++describe('doc-leading blank-line capture', () => {+ test('leading blank lines round-trip', () => {+ expect(rt('\n\nhello')).toBe('\n\nhello\n');+ expect(rt('\nhello')).toBe('\nhello\n');+ expectByteStable('\n\nhello\n');+ });+});++describe('doc-trailing blank-line capture', () => {+ test('trailing blank lines round-trip byte-exact', () => {+ expect(rt('hello\n\n')).toBe('hello\n\n');+ expect(rt('hello\n\n\n')).toBe('hello\n\n\n');+ });++ test('the canonical single final newline is unchanged', () => {+ expect(rt('hello')).toBe('hello\n');+ expect(rt('hello\n')).toBe('hello\n');+ });+});++describe('inter-block blank-line-count capture', () => {+ test('extra blank lines between blocks round-trip', () => {+ expect(rt('a\n\n\n\nb')).toBe('a\n\n\n\nb\n');+ expect(rt('a\n\n\nb')).toBe('a\n\n\nb\n');+ expectByteStable('a\n\n\n\nb\n');+ });++ test('per-boundary counts are independent across three blocks', () => {+ expect(rt('a\n\n\n\nb\n\n\nc')).toBe('a\n\n\n\nb\n\n\nc\n');+ });++ test('the canonical single blank line is unchanged', () => {+ expect(rt('a\n\nb')).toBe('a\n\nb\n');+ });++ test('a 0-blank adjacency still normalizes to the canonical blank line', () => {+ expect(rt('# H\nP')).toBe('# H\n\nP\n');+ });++ test('a WYSIWYG-shaped block deletion drops the stale gap capture', () => {+ const json = md.parse('a\n\n\n\nb\n\n\n\nc') as JSONContent;+ expect(json.content?.length).toBe(3);+ json.content = json.content?.slice(0, 2) ?? [];+ const out = md.serialize(json);+ expect(out).toBe('a\n\nb\n');+ expectByteStable(out);+ });+});++describe('composed doc-boundary state', () => {+ test('BOM + leading + gaps + trailing all round-trip together', () => {+ const doc = '\uFEFF\n\na\n\n\n\nb\n\n';+ expect(rt(doc)).toBe(doc);+ expectByteStable(doc);+ });+});diff --git a/packages/core/src/markdown/pipeline.doc-start.test.ts b/packages/core/src/markdown/pipeline.doc-start.test.tsnew file mode 100644index 000000000..9f4edc781--- /dev/null+++ b/packages/core/src/markdown/pipeline.doc-start.test.ts@@ -0,0 +1,92 @@+import { describe, expect, test } from 'bun:test';+import { stripFrontmatter } from '../extensions/frontmatter.ts';+import { sharedExtensions } from '../extensions/shared.ts';+import { MarkdownManager } from './index.ts';++const mdManager = new MarkdownManager({ extensions: sharedExtensions });++function rt(source: string): string {+ return mdManager.serialize(mdManager.parse(source));+}++function topLevelMdastTypes(source: string): string[] {+ return mdManager.parseToMdast(source).children.map((child) => child.type);+}++describe('content after a doc-start thematic break parses per CommonMark', () => {+ test('a list after a doc-start --- stays a list (no escaped-paragraph collapse)', () => {+ expect(topLevelMdastTypes('---\n\n- a\n- b')).toEqual(['thematicBreak', 'list']);+ expect(rt('---\n\n- a\n- b')).toBe('---\n\n- a\n- b\n');+ });++ test('the recovered-list output is byte-stable across a second round-trip', () => {+ const r1 = rt('---\n\n- a\n- b');+ expect(rt(r1)).toBe(r1);+ });+});++describe('frontmatter-shaped paste is visible content, never silently dropped', () => {+ const fmPaste = '---\nkey: val\n---\n\nbody';++ test('every content byte of a pasted frontmatter block stays visible', () => {+ const out = rt(fmPaste);+ expect(out).toContain('key: val');+ expect(out).toContain('body');+ expect(out).toBe('---\n\nkey: val\n---\n\nbody\n');+ });++ test('the pasted block parses as thematic break + setext heading + paragraph', () => {+ expect(topLevelMdastTypes(fmPaste)).toEqual(['thematicBreak', 'heading', 'paragraph']);+ const heading = mdManager.parseToMdast(fmPaste).children[1];+ expect(heading.type === 'heading' && heading.depth).toBe(2);+ });++ test('the visible-content output is byte-stable across a second round-trip', () => {+ const r1 = rt(fmPaste);+ expect(rt(r1)).toBe(r1);+ });++ test('a yaml node never enters the parse tree', () => {+ for (const input of [fmPaste, '---\ntitle: x\n---\n', '---\n---', '---\n\n---']) {+ expect(topLevelMdastTypes(input)).not.toContain('yaml');+ }+ });+});++describe('doc-start --- is preserved verbatim (the *** override is retired)', () => {+ test('a doc-start --- round-trips as ---', () => {+ expect(rt('---\n\nfoo')).toBe('---\n\nfoo\n');+ expect(rt('---\n')).toBe('---\n');+ });++ test('empty-fence shapes keep both --- lines', () => {+ expect(rt('---\n---')).toBe('---\n\n---\n');+ expect(rt('---\n\n---')).toBe('---\n\n---\n');+ });++ test('*** authored as *** still round-trips as ***', () => {+ expect(rt('***\n\nfoo')).toBe('***\n\nfoo\n');+ });++ test('mid-doc frontmatter-shaped content is untouched (change is doc-start-scoped)', () => {+ expect(rt('x\n\n---\nkey: val\n---')).toBe('x\n\n---\n\nkey: val\n---\n');+ });+});++describe('the pre-existing intake-strip edge (documented, not changed)', () => {+ test('a doc-start --- HR paired with a later --- line is claimed as frontmatter', () => {+ const doc = '---\n\nintro prose\n\n---\n\nbody';+ expect(stripFrontmatter(doc)).toEqual({+ frontmatter: '---\n\nintro prose\n\n---\n',+ body: '\nbody',+ });+ });++ test('a doc-start --- with no later --- line passes through to the parser intact', () => {+ expect(stripFrontmatter('---\n\nbody with no later dashes')).toEqual({+ frontmatter: '',+ body: '---\n\nbody with no later dashes',+ });+ expect(stripFrontmatter('---\n')).toEqual({ frontmatter: '', body: '---\n' });+ });+});diff --git a/packages/core/src/markdown/pipeline.test.ts b/packages/core/src/markdown/pipeline.test.tsindex 1516bd91c..a691bb6ea 100644--- a/packages/core/src/markdown/pipeline.test.ts+++ b/packages/core/src/markdown/pipeline.test.ts@@ -139,21 +139,31 @@ describe('position-aware blank-line Join (FR-12)', () => {});});- describe('PM round-trip path (positions absent → default 1 blank line)', () => {+ describe('PM round-trip path (doc-boundary snapshot carries the counts)', () => {test('1 blank line between paragraphs is the dominant case (byte-identical)', () => {const input = 'P1\n\nP2\n';const json = mdManager.parse(input);expect(mdManager.serialize(json)).toBe(input);});- test('2+ blank lines collapse to 1 (NG1 floor on PM route)', () => {+ test('2+ blank lines round-trip via the sourceDocBoundary attr', () => {const json = mdManager.parse('P1\n\n\nP2\n');- expect(mdManager.serialize(json)).toBe('P1\n\nP2\n');+ expect(mdManager.serialize(json)).toBe('P1\n\n\nP2\n');});- test('4 blank lines collapse to 1 on PM route (NG1 floor)', () => {+ test('4 blank lines round-trip via the sourceDocBoundary attr', () => {const json = mdManager.parse('P1\n\n\n\n\nP2\n');- expect(mdManager.serialize(json)).toBe('P1\n\nP2\n');+ expect(mdManager.serialize(json)).toBe('P1\n\n\n\n\nP2\n');+ });++ test('a doc without the attr (the CRDT/fragment shape) collapses to the canonical 1 blank line', () => {+ const json = mdManager.parse('P1\n\n\n\n\nP2\n');+ const { sourceDocBoundary: _dropped, ...attrs } = (json.attrs ?? {}) as Record<+ string,+ unknown+ >;+ const stripped = { ...json, attrs };+ expect(mdManager.serialize(stripped)).toBe('P1\n\nP2\n');});});@@ -165,7 +175,7 @@ describe('position-aware blank-line Join (FR-12)', () => {expect(round).toEqual(initial);});- test('PM identity holds even when source has 2+ blank lines (PM does not carry counts)', () => {+ test('PM identity holds even when source has 2+ blank lines', () => {const md = 'P1\n\n\n\nP2\n';const initial = mdManager.parse(md);const round = mdManager.parse(mdManager.serialize(initial));diff --git a/packages/core/src/markdown/pipeline.ts b/packages/core/src/markdown/pipeline.tsindex d9333f6ec..674e08091 100644--- a/packages/core/src/markdown/pipeline.ts+++ b/packages/core/src/markdown/pipeline.ts@@ -26,6 +26,7 @@ import { highlightPromoterPlugin } from './highlight-promoter.ts';import { imagePromoterPlugin } from './image-promoter.ts';import { indentedCodePromoterPlugin } from './indented-code-promoter.ts';import { mathPromoterPlugin } from './math-promoter.ts';+import type { SourceDocBoundary } from './mdast-augmentation.ts';import { mergedPostParseWalkerPlugin } from './merged-walker.ts';import { mermaidPromoterPlugin } from './mermaid-promoter.ts';import { positionAwareBlankLineJoin } from './position-aware-join.ts';@@ -65,7 +66,6 @@ function ensureNonEmptyDoc(tree: MdastRoot): MdastRoot {export const ACTIVE_MDAST_PLUGINS = [{ name: 'remark-parse', plugin: remarkParse },- { name: 'remark-frontmatter', plugin: remarkFrontmatter, options: ['yaml'] },{ name: 'remark-mdx-agnostic', plugin: remarkMdxAgnostic },{ name: 'remark-gfm', plugin: remarkGfm },{ name: 'remark-math', plugin: remarkMath, options: { singleDollarTextMath: false } },@@ -115,7 +115,9 @@ export function createParseProcessor(opts: PipelineOptions): Processor {return processor;}-export function createSerializeProcessor(opts: PipelineOptions): Processor {+export function createSerializeProcessor(+ opts: Pick<PipelineOptions, 'toMarkdownHandlers'>,+): Processor {const processor = unified().use(remarkFrontmatter, ['yaml']).use(remarkGfm, { tablePipeAlign: false })@@ -134,7 +136,90 @@ export function createSerializeProcessor(opts: PipelineOptions): Processor {return processor as unknown as Processor;}-export function parseMd(source: string, processor: Processor): PmNode {+function splitDocumentHeadBom(source: string): { source: string; hadBom: boolean } {+ return source.charCodeAt(0) === 0xfeff+ ? { source: source.slice(1), hadBom: true }+ : { source, hadBom: false };+}++function captureDocBoundary(+ root: MdastRoot,+ source: string,+ hadBom: boolean,+): SourceDocBoundary | undefined {+ let leading: string | undefined;+ let trailing: string | undefined;+ let gapBlankLines: Array<number | null> | undefined;++ const children = root.children;+ if (children.length > 0) {+ const firstStart = children[0]?.position?.start?.offset;+ if (typeof firstStart === 'number' && firstStart > 0) {+ const gap = source.slice(0, firstStart);+ if (/^\n+$/.test(gap)) leading = gap;+ }+ const lastEnd = children[children.length - 1]?.position?.end?.offset;+ if (typeof lastEnd === 'number' && lastEnd <= source.length) {+ const gap = source.slice(lastEnd);+ if (/^\n{2,}$/.test(gap)) trailing = gap;+ }+ const gaps: Array<number | null> = [];+ let anyGap = false;+ for (let i = 1; i < children.length; i++) {+ const prevEnd = children[i - 1]?.position?.end?.offset;+ const nextStart = children[i]?.position?.start?.offset;+ let blanks: number | null = null;+ if (typeof prevEnd === 'number' && typeof nextStart === 'number' && nextStart >= prevEnd) {+ const gap = source.slice(prevEnd, nextStart);+ if (/^\n{3,}$/.test(gap)) {+ blanks = gap.length - 1;+ anyGap = true;+ }+ }+ gaps.push(blanks);+ }+ if (anyGap) gapBlankLines = gaps;+ }++ if (!hadBom && leading === undefined && trailing === undefined && !gapBlankLines) {+ return undefined;+ }+ const boundary: SourceDocBoundary = {+ ...(hadBom ? { bom: true as const } : {}),+ ...(leading !== undefined ? { leading } : {}),+ ...(trailing !== undefined ? { trailing } : {}),+ ...(gapBlankLines ? { gapBlankLines } : {}),+ };+ root.data ??= {};+ root.data.sourceDocBoundary = boundary;+ return boundary;+}++function readDocBoundary(value: unknown): SourceDocBoundary | undefined {+ if (typeof value !== 'object' || value === null) return undefined;+ const v = value as Record<string, unknown>;+ const out: {+ bom?: true;+ leading?: string;+ trailing?: string;+ gapBlankLines?: Array<number | null>;+ } = {};+ if (v.bom === true) out.bom = true;+ if (typeof v.leading === 'string' && /^\n+$/.test(v.leading)) out.leading = v.leading;+ if (typeof v.trailing === 'string' && /^\n{2,}$/.test(v.trailing)) out.trailing = v.trailing;+ if (+ Array.isArray(v.gapBlankLines) &&+ v.gapBlankLines.every(+ (g) => g === null || (typeof g === 'number' && Number.isInteger(g) && g >= 2),+ )+ ) {+ out.gapBlankLines = v.gapBlankLines as Array<number | null>;+ }+ return out.bom || out.leading || out.trailing || out.gapBlankLines ? out : undefined;+}++export function parseMd(rawSource: string, processor: Processor): PmNode {+ const { source, hadBom } = splitDocumentHeadBom(rawSource);const protectedFr14 = encodeBackslashEscapes(source);const protectedR23 = protectFromMdx(protectedFr14);const protected_ = encodeEntityRefs(protectedR23);@@ -142,16 +227,22 @@ export function parseMd(source: string, processor: Processor): PmNode {const file = new VFile(protected_);const tree = processor.parse(file);file.value = source;- const transformed = processor.runSync(tree, file);- return (processor as unknown as { stringify(tree: unknown): PmNode }).stringify(transformed);+ const transformed = processor.runSync(tree, file) as MdastRoot;+ const boundary = captureDocBoundary(transformed, source, hadBom);+ const doc = (processor as unknown as { stringify(tree: unknown): PmNode }).stringify(transformed);+ if (!boundary) return doc;+ return doc.type.create({ ...doc.attrs, sourceDocBoundary: boundary }, doc.content, doc.marks);}-export function parseMdToMdast(source: string, processor: Processor): MdastRoot {+export function parseMdToMdast(rawSource: string, processor: Processor): MdastRoot {+ const { source, hadBom } = splitDocumentHeadBom(rawSource);const protected_ = encodeEntityRefs(protectFromMdx(encodeBackslashEscapes(source)));const file = new VFile(protected_);const tree = processor.parse(file);file.value = source;- return processor.runSync(tree, file) as MdastRoot;+ const transformed = processor.runSync(tree, file) as MdastRoot;+ captureDocBoundary(transformed, source, hadBom);+ return transformed;}export function serializeMd(doc: PmNode, processor: Processor, opts: SerializeMdOptions): string {@@ -161,5 +252,24 @@ export function serializeMd(doc: PmNode, processor: Processor, opts: SerializeMdmarkHandlers: opts.pmMarkHandlers,});- return String(processor.stringify(mdast));+ const boundary = readDocBoundary(doc.attrs?.sourceDocBoundary);+ const gaps = boundary?.gapBlankLines;+ if (gaps && gaps.length === mdast.children.length - 1) {+ for (let i = 1; i < mdast.children.length; i++) {+ const blanks = gaps[i - 1];+ if (typeof blanks === 'number') {+ const child = mdast.children[i];+ child.data ??= {};+ child.data.sourcePrecedingBlankLines = blanks;+ }+ }+ }++ let out = String(processor.stringify(mdast));+ if (boundary?.leading) out = boundary.leading + out;+ if (boundary?.trailing) {+ out = out.replace(/\n+$/, '') + boundary.trailing;+ }+ if (boundary?.bom) out = `\uFEFF${out}`;+ return out;}diff --git a/packages/core/src/markdown/position-aware-join.ts b/packages/core/src/markdown/position-aware-join.tsindex 524ac285e..248685cfe 100644--- a/packages/core/src/markdown/position-aware-join.ts+++ b/packages/core/src/markdown/position-aware-join.ts@@ -2,7 +2,10 @@ import type { Join } from 'mdast-util-to-markdown';import type { Position } from 'unist';type MaybePositioned = { position?: Position };-type MaybeTyped = { type?: string; data?: { sourceContiguousNext?: boolean } };+type MaybeTyped = {+ type?: string;+ data?: { sourceContiguousNext?: boolean; sourcePrecedingBlankLines?: number };+};type FlowNode = Parameters<Join>[0];function isContiguousSetextWithParagraph(left: FlowNode, right: FlowNode): boolean {@@ -13,6 +16,8 @@ function isContiguousSetextWithParagraph(left: FlowNode, right: FlowNode): booleexport const positionAwareBlankLineJoin: Join = (left, right) => {if (isContiguousSetextWithParagraph(left, right)) return 0;+ const dataGap = (right as MaybeTyped).data?.sourcePrecedingBlankLines;+ if (typeof dataGap === 'number' && dataGap >= 2) return dataGap;const lp = (left as MaybePositioned).position;const rp = (right as MaybePositioned).position;if (!lp || !rp) return undefined;diff --git a/packages/core/src/markdown/position-slice.test.ts b/packages/core/src/markdown/position-slice.test.tsindex f13c09d61..6a302a9b0 100644--- a/packages/core/src/markdown/position-slice.test.ts+++ b/packages/core/src/markdown/position-slice.test.ts@@ -643,74 +643,74 @@ describe('position-slice: inline-link title marker recovery (FR-20)', () => {});describe('position-slice: blockquote marker spacing recovery (FR-23)', () => {- test('single-space marker → sourceMarkerSpacings=["single"]', () => {+ test('single-space marker → sourceMarkerSpacings=[1]', () => {const tree = parseMdast('> foo\n');const bq = findNode(tree, 'blockquote');- expect(bq?.data?.sourceMarkerSpacings).toEqual(['single']);+ expect(bq?.data?.sourceMarkerSpacings).toEqual([1]);});- test('no-space marker → sourceMarkerSpacings=["none"]', () => {+ test('no-space marker → sourceMarkerSpacings=[0]', () => {const tree = parseMdast('>foo\n');const bq = findNode(tree, 'blockquote');- expect(bq?.data?.sourceMarkerSpacings).toEqual(['none']);+ expect(bq?.data?.sourceMarkerSpacings).toEqual([0]);});test('multi-line single-space → array of length 2', () => {const tree = parseMdast('> line 1\n> line 2\n');const bq = findNode(tree, 'blockquote');- expect(bq?.data?.sourceMarkerSpacings).toEqual(['single', 'single']);+ expect(bq?.data?.sourceMarkerSpacings).toEqual([1, 1]);});test('multi-line no-space → array of length 2', () => {const tree = parseMdast('>line 1\n>line 2\n');const bq = findNode(tree, 'blockquote');- expect(bq?.data?.sourceMarkerSpacings).toEqual(['none', 'none']);+ expect(bq?.data?.sourceMarkerSpacings).toEqual([0, 0]);});test('mixed single + none → per-line array', () => {const tree = parseMdast('> line 1\n>line 2\n');const bq = findNode(tree, 'blockquote');- expect(bq?.data?.sourceMarkerSpacings).toEqual(['single', 'none']);+ expect(bq?.data?.sourceMarkerSpacings).toEqual([1, 0]);});test('mixed none + single → per-line array', () => {const tree = parseMdast('>line 1\n> line 2\n');const bq = findNode(tree, 'blockquote');- expect(bq?.data?.sourceMarkerSpacings).toEqual(['none', 'single']);+ expect(bq?.data?.sourceMarkerSpacings).toEqual([0, 1]);});test('blank-line `>` continuation EXCLUDED from capture', () => {const tree = parseMdast('> p1\n>\n> p2\n');const bq = findNode(tree, 'blockquote');- expect(bq?.data?.sourceMarkerSpacings).toEqual(['single', 'single']);+ expect(bq?.data?.sourceMarkerSpacings).toEqual([1, 1]);});test('nested blockquote captures outer marker spacing on outer node', () => {const tree = parseMdast('> > nested\n');const allBqs = findNodes(tree, 'blockquote');expect(allBqs.length).toBe(2);- expect(allBqs[0]?.data?.sourceMarkerSpacings).toEqual(['single']);- expect(allBqs[1]?.data?.sourceMarkerSpacings).toEqual(['single']);+ expect(allBqs[0]?.data?.sourceMarkerSpacings).toEqual([1]);+ expect(allBqs[1]?.data?.sourceMarkerSpacings).toEqual([1]);});test('nested blockquote `>>` → outer none, inner single', () => {const tree = parseMdast('>> nested\n');const allBqs = findNodes(tree, 'blockquote');expect(allBqs.length).toBe(2);- expect(allBqs[0]?.data?.sourceMarkerSpacings).toEqual(['none']);- expect(allBqs[1]?.data?.sourceMarkerSpacings).toEqual(['single']);+ expect(allBqs[0]?.data?.sourceMarkerSpacings).toEqual([0]);+ expect(allBqs[1]?.data?.sourceMarkerSpacings).toEqual([1]);});test('tab-after-marker treated as single-spacing', () => {const tree = parseMdast('>\tfoo\n');const bq = findNode(tree, 'blockquote');- expect(bq?.data?.sourceMarkerSpacings).toEqual(['single']);+ expect(bq?.data?.sourceMarkerSpacings).toEqual([1]);});test('1-3 leading spaces of indent before `>` tolerated (CommonMark §5.1)', () => {const tree = parseMdast(' > foo\n');const bq = findNode(tree, 'blockquote');- expect(bq?.data?.sourceMarkerSpacings).toEqual(['single']);+ expect(bq?.data?.sourceMarkerSpacings).toEqual([1]);});});diff --git a/packages/core/src/markdown/position-slice.ts b/packages/core/src/markdown/position-slice.tsindex 7a49c4aa8..e6f7a26e2 100644--- a/packages/core/src/markdown/position-slice.ts+++ b/packages/core/src/markdown/position-slice.ts@@ -34,6 +34,7 @@ export function applyPositionSliceToNode(node: Nodes,source: string,debug: boolean = false,+ parent?: { type?: string },): void {if (!source) return;const pos = node.position;@@ -116,6 +117,18 @@ export function applyPositionSliceToNode(const segment = source.slice(startOff, endOff);if (prefix === '#') {node.data.sourceStyle = 'atx';+ if (parent?.type === 'root') {+ let lineStart = startOff;+ while (lineStart > 0 && source[lineStart - 1] !== '\n') lineStart--;+ const indentSlice = source.slice(lineStart, startOff);+ if (indentSlice.length >= 1 && indentSlice.length <= 3 && /^ +$/.test(indentSlice)) {+ node.data.sourceLeadingIndent = indentSlice.length;+ }+ }+ const interior = /^#+( {2,})[^ ]/.exec(segment);+ if (interior) {+ node.data.sourceInteriorSpacing = interior[1].length;+ }const trailing = /^.*?[ \t](#+)[ \t]*$/.exec(segment);if (trailing) {node.data.sourceTrailingHashes = trailing[1].length;@@ -147,7 +160,7 @@ export function applyPositionSliceToNode(const slice = source.slice(startOff, endOff);const lines = slice.split('\n');- const markerSpacings: Array<'single' | 'none'> = [];+ const markerSpacings: number[] = [];for (let lineIdx = 0; lineIdx < lines.length; lineIdx++) {const line = lines[lineIdx];if (line.length === 0) continue;@@ -156,11 +169,12 @@ export function applyPositionSliceToNode(if (stripped[0] !== '>') continue;const restAfterMarker = stripped.slice(1).trimEnd();if (restAfterMarker === '') continue;- const afterMarker = stripped[1];- if (afterMarker === ' ' || afterMarker === '\t') {- markerSpacings.push('single');+ if (stripped[1] === '\t') {+ markerSpacings.push(1);} else {- markerSpacings.push('none');+ let spaceRun = 0;+ while (stripped[1 + spaceRun] === ' ') spaceRun++;+ markerSpacings.push(spaceRun);}}if (markerSpacings.length > 0) {@@ -187,6 +201,58 @@ export function applyPositionSliceToNode(break;}+ case 'listItem': {+ const slice = source.slice(startOff, endOff);+ const m = /^(?:([-+*])|(\d{1,9})[.)])( *)/.exec(slice);+ if (!m) break;+ const markerLen = m[1] ? 1 : (m[2]?.length ?? 0) + 1;+ const spacing = m[3].length;+ if (spacing >= 2 && spacing <= 4) {+ node.data.sourceMarkerSpacing = spacing;+ }+ if (m[2]) {+ node.data.sourceOrdinal = Number.parseInt(m[2], 10);+ }+ if (node.checked != null) {+ const checkbox = /^\[([xX])\] /.exec(slice.slice(markerLen + Math.max(spacing, 1)));+ if (checkbox?.[1] === 'X') {+ node.data.sourceCheckboxChar = 'X';+ }+ }+ const effectiveSpacing = spacing >= 2 && spacing <= 4 ? spacing : 1;+ const contentCol = markerLen + effectiveSpacing;+ let itemLineStart = startOff;+ while (itemLineStart > 0 && source[itemLineStart - 1] !== '\n') itemLineStart--;+ const itemCol = pos.start.column;+ for (const child of node.children ?? []) {+ if (child.type !== 'list') continue;+ const childStart = child.position?.start?.offset;+ const childCol = child.position?.start?.column;+ if (typeof childStart !== 'number' || typeof childCol !== 'number') continue;+ let childLineStart = childStart;+ while (childLineStart > 0 && source[childLineStart - 1] !== '\n') childLineStart--;+ if (childLineStart === itemLineStart) continue; // same-line list — no pad applies+ const indent = childCol - itemCol;+ if (+ indent !== contentCol &&+ indent >= contentCol &&+ indent <= contentCol + 3 &&+ /^ +$/.test(source.slice(childLineStart, childStart))+ ) {+ node.data.sourceContinuationIndent = indent;+ }+ break;+ }+ break;+ }++ case 'delete': {+ if (source[startOff] === '~') {+ node.data.sourceDelimiter = source[startOff + 1] === '~' ? '~~' : '~';+ }+ break;+ }+case 'code': {const ch = source[startOff];if (ch === '`' || ch === '~') {@@ -199,8 +265,58 @@ export function applyPositionSliceToNode(if (count >= 3) {node.data.sourceFenceLength = count;}++ if (parent?.type === 'root') {+ let fenceLineStart = startOff;+ while (fenceLineStart > 0 && source[fenceLineStart - 1] !== '\n') fenceLineStart--;+ const fenceIndent = source.slice(fenceLineStart, startOff);+ if (fenceIndent.length >= 1 && fenceIndent.length <= 3 && /^ +$/.test(fenceIndent)) {+ node.data.sourceFenceIndent = fenceIndent.length;+ }+ }++ if (node.lang) {+ let gap = 0;+ while (source[startOff + count + gap] === ' ') gap++;+ if (gap >= 1) {+ node.data.sourceInfoPadding = gap;+ }+ }++ const fenceSlice = source.slice(startOff, endOff);+ const closing = new RegExp(`\\n[ \\t]*(\\${ch}+)[ \\t]*$`).exec(fenceSlice);+ if (closing && closing[1].length > count) {+ node.data.sourceClosingFenceLength = closing[1].length;+ }} else {node.data.sourceStyle = 'indented';+ if (parent?.type === 'root') {+ const sliceLines = source.slice(startOff, endOff).split('\n');+ const valueLines = (node.value ?? '').split('\n');+ if (sliceLines.length === valueLines.length) {+ const indents: string[] = [];+ let nonCanonical = false;+ let valid = true;+ for (let i = 0; i < sliceLines.length; i++) {+ const sliceLine = sliceLines[i];+ const valueLine = valueLines[i];+ if (!sliceLine.endsWith(valueLine)) {+ valid = false;+ break;+ }+ const indent = sliceLine.slice(0, sliceLine.length - valueLine.length);+ if (!/^[ \t]*$/.test(indent)) {+ valid = false;+ break;+ }+ indents.push(indent);+ if (indent !== (valueLine.length > 0 ? ' ' : '')) nonCanonical = true;+ }+ if (valid && nonCanonical) {+ node.data.sourceIndents = indents;+ }+ }+ }}break;}@@ -216,6 +332,22 @@ export function applyPositionSliceToNode(if (count >= 1) {node.data.sourceFenceLength = count;}+ const inner = source.slice(startOff + count, endOff - count);+ const value: string = node.value ?? '';+ if (inner !== value && inner === ` ${value} `) {+ node.data.sourcePadded = true;+ }+ }+ break;+ }++ case 'inlineMath': {+ if (source[startOff] === '$') {+ let count = 0;+ while (startOff + count < source.length && source[startOff + count] === '$') {+ count++;+ }+ node.data.sourceDelimiter = '$'.repeat(count);}break;}@@ -245,6 +377,50 @@ export function applyPositionSliceToNode(node.data.sourceDashCounts = dashCounts;}+ {+ const trimmedAlign = alignmentLine.trim();+ const segments = splitGfmCellSegments(trimmedAlign);+ if (segments.length > 0 && segments[0] === '' && trimmedAlign.startsWith('|')) {+ segments.shift();+ }+ if (+ segments.length > 0 &&+ segments[segments.length - 1] === '' &&+ trimmedAlign.endsWith('|')+ ) {+ segments.pop();+ }+ const allDelimiterCells = segments.every((seg) => /^[-: ]+$/.test(seg));+ const alignPadding: Array<{ left: number; right: number }> = [];+ for (const seg of segments) {+ let left = 0;+ while (left < seg.length && seg[left] === ' ') left++;+ let right = 0;+ while (right < seg.length - left && seg[seg.length - 1 - right] === ' ') right++;+ alignPadding.push({ left, right });+ }+ if (allDelimiterCells && alignPadding.length > 0) {+ node.data.sourceAlignmentPadding = alignPadding;+ }+ }++ {+ const tableLines = tableSrc.split('\n').filter((l) => l.trim().length > 0);+ if (tableLines.length >= 2) {+ const leadingStates = tableLines.map((l) => l.trimStart().startsWith('|'));+ const trailingStates = tableLines.map((l) => /(^|[^\\])\|$/.test(l.trimEnd()));+ const uniformLeading = leadingStates.every((s) => s === leadingStates[0]);+ const uniformTrailing = trailingStates.every((s) => s === trailingStates[0]);+ if (uniformLeading && uniformTrailing) {+ const leading = leadingStates[0];+ const trailing = trailingStates[0];+ if (!leading || !trailing) {+ node.data.sourceOuterPipes = { leading, trailing };+ }+ }+ }+ }+for (const row of node.children ?? []) {if (row.type !== 'tableRow' || !row.position) continue;const rowStart = row.position.start.offset;@@ -370,8 +546,8 @@ export function positionSlicePlugin() {const debug = typeof process !== 'undefined' && process.env?.OK_DEBUG_POSITION_SLICE === '1';- visit(tree, (node: Nodes) => {- applyPositionSliceToNode(node, source, debug);+ visit(tree, (node: Nodes, _index, parent) => {+ applyPositionSliceToNode(node, source, debug, parent ?? undefined);});};}diff --git a/packages/core/src/markdown/promoter-position.ts b/packages/core/src/markdown/promoter-position.tsnew file mode 100644index 000000000..ea7949ee1--- /dev/null+++ b/packages/core/src/markdown/promoter-position.ts@@ -0,0 +1,62 @@+import type { Text } from 'mdast';+import type { Point, Position } from 'unist';++function valueOffsetToSourceOffset(+ source: string,+ parentSourceStart: number,+ valueText: string,+ targetValueOffset: number,+): number {+ let srcCursor = parentSourceStart;+ let valCursor = 0;+ while (valCursor < targetValueOffset && srcCursor < source.length) {+ if (+ source[srcCursor] === '\\' &&+ srcCursor + 1 < source.length &&+ valueText[valCursor] === source[srcCursor + 1]+ ) {+ srcCursor += 2;+ valCursor += 1;+ } else {+ srcCursor += 1;+ valCursor += 1;+ }+ }+ return srcCursor;+}++function offsetToPoint(source: string, offset: number): Point {+ let line = 1;+ let column = 1;+ for (let i = 0; i < offset && i < source.length; i++) {+ if (source[i] === '\n') {+ line += 1;+ column = 1;+ } else {+ column += 1;+ }+ }+ return { line, column, offset };+}++function makePosition(source: string, startOffset: number, endOffset: number): Position {+ return {+ start: offsetToPoint(source, startOffset),+ end: offsetToPoint(source, endOffset),+ };+}++export function deriveFragmentPosition(+ source: string,+ parentNode: Text,+ valueStart: number,+ valueEnd: number,+): Position | undefined {+ if (!source || !parentNode.position || typeof parentNode.position.start?.offset !== 'number') {+ return undefined;+ }+ const parentOff = parentNode.position.start.offset;+ const srcStart = valueOffsetToSourceOffset(source, parentOff, parentNode.value, valueStart);+ const srcEnd = valueOffsetToSourceOffset(source, parentOff, parentNode.value, valueEnd);+ return makePosition(source, srcStart, srcEnd);+}diff --git a/packages/core/src/markdown/single-dollar-math-promoter.ts b/packages/core/src/markdown/single-dollar-math-promoter.tsindex a610c4663..be97d4d62 100644--- a/packages/core/src/markdown/single-dollar-math-promoter.ts+++ b/packages/core/src/markdown/single-dollar-math-promoter.ts@@ -1,11 +1,14 @@import type { PhrasingContent, Root, Text } from 'mdast';import type { InlineMath } from 'mdast-util-math';import { SKIP, visit } from 'unist-util-visit';+import type { VFile } from 'vfile';+import { deriveFragmentPosition } from './promoter-position.ts';const SINGLE_DOLLAR_MATH_RE = /(?<!\\)\$(?=\S)([^$\n]*?[^\s$])\$(?!\d)/g;export function singleDollarMathPromoterPlugin() {- return (tree: Root) => {+ return (tree: Root, file: VFile) => {+ const source = typeof file.value === 'string' ? file.value : '';visit(tree, 'text', (node: Text, index, parent) => {if (parent === undefined || index === undefined || index === null) return;@@ -28,14 +31,20 @@ export function singleDollarMathPromoterPlugin() {const end = start + match[0].length;if (start > cursor) {const lead: Text = { type: 'text', value: value.slice(cursor, start) };+ const pos = deriveFragmentPosition(source, node, cursor, start);+ if (pos) lead.position = pos;replacements.push(lead);}const mathNode: InlineMath = { type: 'inlineMath', value: match[1] };+ const fullPos = deriveFragmentPosition(source, node, start, end);+ if (fullPos) mathNode.position = fullPos;replacements.push(mathNode as unknown as PhrasingContent);cursor = end;}if (cursor < value.length) {const tail: Text = { type: 'text', value: value.slice(cursor) };+ const pos = deriveFragmentPosition(source, node, cursor, value.length);+ if (pos) tail.position = pos;replacements.push(tail);}diff --git a/packages/core/src/markdown/tag-promotion.ts b/packages/core/src/markdown/tag-promotion.tsindex aa0bb9efb..6d71623fa 100644--- a/packages/core/src/markdown/tag-promotion.ts+++ b/packages/core/src/markdown/tag-promotion.ts@@ -1,5 +1,6 @@import type { Parent, Text } from 'mdast';import type { TagMdast } from './mdast-augmentation.ts';+import { deriveFragmentPosition } from './promoter-position.ts';export const TAG_IN_TEXT_PATTERN_SOURCE = '(^|\\s)#([a-zA-Z][\\w/-]*)';export function createTagInTextRegex(): RegExp {@@ -71,10 +72,20 @@ export function promoteTagsInParent(parent: Parent, source: string = ''): void {}if (tagStart > lastIndex) {- segments.push({ type: 'text', value: text.slice(lastIndex, tagStart) } as Text);+ const lead: Text = { type: 'text', value: text.slice(lastIndex, tagStart) };+ const pos = deriveFragmentPosition(source, child as Text, lastIndex, tagStart);+ if (pos) lead.position = pos;+ segments.push(lead);}const tagNode: TagMdast = { type: 'tag', value: tagValue };+ const tagPos = deriveFragmentPosition(+ source,+ child as Text,+ tagStart,+ tagStart + 1 + tagValue.length,+ );+ if (tagPos) tagNode.position = tagPos;segments.push(tagNode as unknown as Parent['children'][number]);lastIndex = tagStart + 1 + tagValue.length; // 1 for `#`@@ -85,7 +96,10 @@ export function promoteTagsInParent(parent: Parent, source: string = ''): void {newChildren.push(child);} else {if (lastIndex < text.length) {- segments.push({ type: 'text', value: text.slice(lastIndex) } as Text);+ const tail: Text = { type: 'text', value: text.slice(lastIndex) };+ const pos = deriveFragmentPosition(source, child as Text, lastIndex, text.length);+ if (pos) tail.position = pos;+ segments.push(tail);}newChildren.push(...segments);}diff --git a/packages/core/src/markdown/to-markdown-handlers.attention.test.ts b/packages/core/src/markdown/to-markdown-handlers.attention.test.tsnew file mode 100644index 000000000..837c3e9d6--- /dev/null+++ b/packages/core/src/markdown/to-markdown-handlers.attention.test.ts@@ -0,0 +1,190 @@+import { describe, expect, test } from 'bun:test';+import type { JSONContent } from '@tiptap/core';+import type { Nodes } from 'mdast';+import { sharedExtensions } from '../extensions/shared.ts';+import { MarkdownManager } from './index.ts';++const md = new MarkdownManager({ extensions: sharedExtensions });++const em = (t: string, sourceDelimiter: '*' | '_' = '*'): JSONContent => ({+ type: 'text',+ marks: [{ type: 'emphasis', attrs: { sourceDelimiter } }],+ text: t,+});+const strong = (t: string, sourceDelimiter: '**' | '__' = '**'): JSONContent => ({+ type: 'text',+ marks: [{ type: 'strong', attrs: { sourceDelimiter } }],+ text: t,+});+const text = (t: string): JSONContent => ({ type: 'text', text: t });+const paragraph = (...inline: JSONContent[]): JSONContent => ({+ type: 'doc',+ content: [{ type: 'paragraph', content: inline }],+});++function allNodeTypes(markdown: string): string[] {+ const out: string[] = [];+ const walk = (node: Nodes): void => {+ out.push(node.type);+ if ('children' in node) {+ for (const child of node.children) {+ walk(child);+ }+ }+ };+ walk(md.parseToMdast(markdown));+ return out;+}++function expectByteStable(out: string): void {+ expect(md.serialize(md.parse(out))).toBe(out);+}++describe('boundary-whitespace attention encode (WYSIWYG-shaped marks)', () => {+ test('emphasis with trailing space re-parses as emphasis via char-ref', () => {+ const out = md.serialize(paragraph(em('foo ')));+ expect(out).toBe('*foo *\n');+ expect(allNodeTypes(out)).toContain('emphasis');+ expectByteStable(out);+ });++ test('emphasis with leading space does not degrade to a bullet list', () => {+ const out = md.serialize(paragraph(em(' foo')));+ expect(out).toBe('* foo*\n');+ const types = allNodeTypes(out);+ expect(types).toContain('emphasis');+ expect(types).not.toContain('list');+ expectByteStable(out);+ });++ test('strong with trailing space re-parses as strong', () => {+ const out = md.serialize(paragraph(strong('foo ')));+ expect(out).toBe('**foo **\n');+ expect(allNodeTypes(out)).toContain('strong');+ expectByteStable(out);+ });++ test('strong with leading space re-parses as strong', () => {+ const out = md.serialize(paragraph(strong(' foo')));+ expect(out).toBe('** foo**\n');+ expect(allNodeTypes(out)).toContain('strong');+ expectByteStable(out);+ });++ test('underscore emphasis with trailing space keeps its delimiter form', () => {+ const out = md.serialize(paragraph(em('foo ', '_')));+ expect(out).toBe('_foo _\n');+ expect(allNodeTypes(out)).toContain('emphasis');+ expectByteStable(out);+ });++ test('underscore strong with trailing space keeps its delimiter form', () => {+ const out = md.serialize(paragraph(strong('foo ', '__')));+ expect(out).toBe('__foo __\n');+ expect(allNodeTypes(out)).toContain('strong');+ expectByteStable(out);+ });++ test('NBSP inside an emphasis boundary downgrades to an encoded space', () => {+ const out = md.serialize(paragraph(em('a\u00A0')));+ expect(out).toBe('*a *\n');+ expect(allNodeTypes(out)).toContain('emphasis');+ expectByteStable(out);+ });++ test('whitespace-only emphasis survives', () => {+ const out = md.serialize(paragraph(em(' ')));+ expect(out).toBe('* *\n');+ expect(allNodeTypes(out)).toContain('emphasis');+ expectByteStable(out);+ });++ test('trailing space emphasis flush against following text survives', () => {+ const out = md.serialize(paragraph(em('foo '), text('bar')));+ expect(out).toBe('*foo *bar\n');+ expect(allNodeTypes(out)).toContain('emphasis');+ expectByteStable(out);+ });++ test('leading space emphasis flush against preceding text survives', () => {+ const out = md.serialize(paragraph(text('bar'), em(' foo')));+ expect(out).toBe('bar* foo*\n');+ expect(allNodeTypes(out)).toContain('emphasis');+ expectByteStable(out);+ });++ test('whitespace on both the run boundary and its neighbor needs no outside encode', () => {+ const out = md.serialize(paragraph(text('a '), em(' x')));+ expect(out).toBe('a * x*\n');+ expect(allNodeTypes(out)).toContain('emphasis');+ expectByteStable(out);+ });++ test('nested strong+emphasis with boundary space survives structurally', () => {+ const out = md.serialize(+ paragraph({+ type: 'text',+ marks: [+ { type: 'strong', attrs: { sourceDelimiter: '**' } },+ { type: 'emphasis', attrs: { sourceDelimiter: '*' } },+ ],+ text: 'x ',+ }),+ );+ expect(out).not.toMatch(/\\&/);+ const types = allNodeTypes(out);+ expect(types).toContain('strong');+ expect(types).toContain('emphasis');+ expectByteStable(out);+ });+});++describe('guard-irreducible adjacencies stay byte-identical to before', () => {+ test('letter before punctuation-fronted emphasis is left unencoded', () => {+ const out = md.serialize(paragraph(text('x'), em('.y.')));+ expect(out).toBe('x*.y.*\n');+ expect(out).not.toContain('&#');+ });++ test('letter after punctuation-tailed emphasis is left unencoded', () => {+ const out = md.serialize(paragraph(em('.y.'), text('z')));+ expect(out).toBe('*.y.*z\n');+ expect(out).not.toContain('&#');+ });++ test('intraword underscore emphasis is left unencoded', () => {+ const out = md.serialize(paragraph(text('a'), em('b', '_'), text('c')));+ expect(out).toBe('a_b_c\n');+ expect(out).not.toContain('&#');+ });++ test('underscore run with boundary space and letter neighbor is left unencoded', () => {+ const out = md.serialize(paragraph(em('foo ', '_'), text('bar')));+ expect(out).toBe('_foo _bar\n');+ expect(out).not.toContain('&#');+ });+});++describe('zero regression on already-valid boundaries', () => {+ test('plain emphasis and strong are byte-identical to before', () => {+ expect(md.serialize(paragraph(em('foo')))).toBe('*foo*\n');+ expect(md.serialize(paragraph(strong('foo')))).toBe('**foo**\n');+ expect(md.serialize(paragraph(em('foo', '_')))).toBe('_foo_\n');+ expect(md.serialize(paragraph(strong('foo', '__')))).toBe('__foo__\n');+ });++ test('parse-derived attention round-trips byte-identically', () => {+ const sources = [+ 'x *y* z\n',+ '**a** b\n',+ 'a _b_ c\n',+ '*.padded.* x\n',+ '.*foo*\n',+ 'intra*word*em\n',+ '*Afoo*\n',+ ];+ for (const src of sources) {+ expect(md.serialize(md.parse(src))).toBe(src);+ }+ });+});diff --git a/packages/core/src/markdown/to-markdown-handlers.charref-attention.test.ts b/packages/core/src/markdown/to-markdown-handlers.charref-attention.test.tsnew file mode 100644index 000000000..81a9914de--- /dev/null+++ b/packages/core/src/markdown/to-markdown-handlers.charref-attention.test.ts@@ -0,0 +1,83 @@+import { describe, expect, test } from 'bun:test';+import { sharedExtensions } from '../extensions/shared.ts';+import { MarkdownManager } from './index.ts';++const md = new MarkdownManager({ extensions: sharedExtensions });+const rt = (s: string): string => md.serialize(md.parse(s));++const expectStable = (src: string, out: string): void => {+ const once = rt(src);+ expect(once).toBe(out);+ expect(rt(once)).toBe(out);+};++const stripTree = (n: unknown): unknown => {+ if (Array.isArray(n)) return n.map(stripTree);+ if (n && typeof n === 'object') {+ const out: Record<string, unknown> = {};+ for (const [k, v] of Object.entries(n as Record<string, unknown>)) {+ if (k === 'position' || k === 'data') continue;+ out[k] = stripTree(v);+ }+ return out;+ }+ return n;+};++const expectContentPreserved = (src: string): void => {+ expect(stripTree(md.parseToMdast(rt(src)))).toEqual(stripTree(md.parseToMdast(src)));+};++describe('char-ref-inert escape churn (the entity is a bystander)', () => {+ test('punct-inner asterisk run after a numeric char ref gains escapes, stays lossless', () => {+ expectStable('aA*.y.*', 'aA\\*.y.\\*\n');+ expectContentPreserved('aA*.y.*');+ });++ test('entity bytes survive verbatim through the escape churn', () => {+ expect(rt('aA*.y.*')).toContain('aA');+ });++ test('the decoded twin escapes identically (the char ref does not alter the escape)', () => {+ expectStable('aA*.y.*', 'aA\\*.y.\\*\n');+ });++ test('the no-ref control produces the same escape', () => {+ expectStable('a*.y.*', 'a\\*.y.\\*\n');+ });+});++describe('char-ref-causal PUA-flanking divergence (the char ref alters escaping)', () => {+ test('underscore run after a numeric char ref escapes both delimiters, stays lossless', () => {+ expectStable('aA_foo_', 'aA\\_foo\\_\n');+ expectContentPreserved('aA_foo_');+ expect(rt('aA_foo_')).toContain('aA');+ });++ test('the decoded twin diverges: intraword opener stays bare, only the closer escapes', () => {+ expectStable('aA_foo_', 'aA_foo\\_\n');+ });++ test('punct-inner asterisk run after a named char ref escapes both delimiters, stays lossless', () => {+ expectStable('a&*.y.*', 'a&\\*.y.\\*\n');+ expectContentPreserved('a&*.y.*');+ expect(rt('a&*.y.*')).toContain('a&');+ });++ test('the decoded twin diverges: the bare ampersand form round-trips byte-exact with zero escapes', () => {+ expectStable('a&*.y.*', 'a&*.y.*\n');+ expect(rt('a&*.y.*')).not.toContain('\\');+ });++ test('the causal pairs differ in escape count, never in entity bytes or content', () => {+ const refUnderscore = rt('aA_foo_');+ const decodedUnderscore = rt('aA_foo_');+ expect((refUnderscore.match(/\\/g) ?? []).length).toBe(2);+ expect((decodedUnderscore.match(/\\/g) ?? []).length).toBe(1);++ const refAmp = rt('a&*.y.*');+ const decodedAmp = rt('a&*.y.*');+ expect((refAmp.match(/\\/g) ?? []).length).toBe(2);+ expect((decodedAmp.match(/\\/g) ?? []).length).toBe(0);+ });+});diff --git a/packages/core/src/markdown/to-markdown-handlers.inline-math.test.ts b/packages/core/src/markdown/to-markdown-handlers.inline-math.test.tsnew file mode 100644index 000000000..98b9d2e44--- /dev/null+++ b/packages/core/src/markdown/to-markdown-handlers.inline-math.test.ts@@ -0,0 +1,160 @@+import { describe, expect, test } from 'bun:test';+import type { JSONContent } from '@tiptap/core';+import { sharedExtensions } from '../extensions/shared.ts';+import { MarkdownManager } from './index.ts';++const md = new MarkdownManager({ extensions: sharedExtensions });++function findInJson(json: JSONContent, predicate: (n: JSONContent) => boolean): JSONContent | null {+ if (predicate(json)) return json;+ for (const child of json.content ?? []) {+ const found = findInJson(child, predicate);+ if (found) return found;+ }+ return null;+}++function countInJson(json: JSONContent, predicate: (n: JSONContent) => boolean): number {+ let count = predicate(json) ? 1 : 0;+ for (const child of json.content ?? []) {+ count += countInJson(child, predicate);+ }+ return count;+}++const isMath = (n: JSONContent) => n.type === 'mathInline';++function expectByteStable(source: string) {+ const once = md.serialize(md.parse(source));+ expect(once).toBe(source);+ expect(md.serialize(md.parse(once))).toBe(once);+}++function expectReparsesAsMath(emitted: string, formula: string) {+ const json = md.parse(emitted);+ expect(countInJson(json, isMath)).toBe(1);+ expect(findInJson(json, isMath)?.attrs?.formula).toBe(formula);+}++describe('single-dollar capture: `$x$` round-trips byte-identical', () => {+ test('`$x$` mid-prose round-trips byte-identical', () => {+ expectByteStable('A formula $x$ in prose.\n');+ });++ test('`$x^2$` round-trips byte-identical', () => {+ expectByteStable('Result: $x^2$.\n');+ });++ test('the captured delimiter lands on the mathInline atom', () => {+ const json = md.parse('A formula $x$ in prose.\n');+ const node = findInJson(json, isMath);+ expect(node?.attrs?.formula).toBe('x');+ expect(node?.attrs?.sourceDelimiter).toBe('$');+ });++ test('`$$x$$` keeps the paired-double form', () => {+ expectByteStable('Result: $$x^2$$.\n');+ const node = findInJson(md.parse('Result: $$x^2$$.\n'), isMath);+ expect(node?.attrs?.sourceDelimiter).toBe('$$');+ });++ test('mixed `$a$` and `$$b$$` in one paragraph each keep their form', () => {+ expectByteStable('Both $a$ and $$b$$ here.\n');+ });++ test('adjacent single-dollar maths `$y$$x$` round-trip byte-identical', () => {+ const source = '$y$$x$\n';+ const json = md.parse(source);+ expect(countInJson(json, isMath)).toBe(2);+ expect(md.serialize(json)).toBe(source);+ });++ test('LaTeX command body `$\\alpha$` round-trips byte-identical', () => {+ expectByteStable('Greek $\\alpha$ letter.\n');+ });+});++describe('currency precision: the promoter boundary is unchanged', () => {+ test('`Pay $5 to $10 dollars` stays prose, byte-identical', () => {+ const source = 'Pay $5 to $10 dollars.\n';+ expect(countInJson(md.parse(source), isMath)).toBe(0);+ expectByteStable(source);+ });++ test('`$PATH and $HOME` stays prose, byte-identical', () => {+ const source = '$PATH and $HOME\n';+ expect(countInJson(md.parse(source), isMath)).toBe(0);+ expectByteStable(source);+ });++ test('`$x$5` (digit after close) stays prose, byte-identical', () => {+ const source = 'value $x$5 here\n';+ expect(countInJson(md.parse(source), isMath)).toBe(0);+ expectByteStable(source);+ });++ test('`Cost $5$ each` parses as math (documented edge) and round-trips', () => {+ const source = 'Cost $5$ each\n';+ expect(countInJson(md.parse(source), isMath)).toBe(1);+ expectByteStable(source);+ });+});++describe('demotion boundary: a stale single-$ capture never emits bytes the promoter rejects', () => {+ function parsedCarrier(): JSONContent {+ const json = md.parse('A $x$ b\n');+ expect(findInJson(json, isMath)?.attrs?.sourceDelimiter).toBe('$');+ return json;+ }++ test('digit typed right after the atom demotes to `$$…$$`', () => {+ const json = parsedCarrier();+ const para = json.content?.[0];+ const after = para?.content?.[2];+ expect(after?.text).toBe(' b');+ if (after) after.text = '5 b';+ const out = md.serialize(json);+ expect(out).toBe('A $$x$$5 b\n');+ expectReparsesAsMath(out, 'x');+ });++ test('formula edited to contain `$` demotes and grows the fence', () => {+ const json = parsedCarrier();+ const node = findInJson(json, isMath);+ if (node?.attrs) node.attrs.formula = 'a$b';+ const out = md.serialize(json);+ expect(out).toBe('A $$a$b$$ b\n');+ expectReparsesAsMath(out, 'a$b');+ });++ test('formula edited to start with whitespace demotes', () => {+ const json = parsedCarrier();+ const node = findInJson(json, isMath);+ if (node?.attrs) node.attrs.formula = ' x';+ const out = md.serialize(json);+ expectReparsesAsMath(out, ' x');+ });++ test('formula edited to end with a backslash demotes (raw math text keeps it)', () => {+ const json = parsedCarrier();+ const node = findInJson(json, isMath);+ if (node?.attrs) node.attrs.formula = 'x\\';+ const out = md.serialize(json);+ expectReparsesAsMath(out, 'x\\');+ });++ test('formula with an escape-active backslash sequence demotes (prose re-parse would eat it)', () => {+ const json = parsedCarrier();+ const node = findInJson(json, isMath);+ if (node?.attrs) node.attrs.formula = 'a\\*b';+ const out = md.serialize(json);+ expectReparsesAsMath(out, 'a\\*b');+ });++ test('WYSIWYG-inserted math (no captured delimiter) keeps the `$$…$$` default', () => {+ const json = parsedCarrier();+ const node = findInJson(json, isMath);+ if (node?.attrs) node.attrs.sourceDelimiter = null;+ expect(md.serialize(json)).toBe('A $$x$$ b\n');+ });+});diff --git a/packages/core/src/markdown/to-markdown-handlers.overescape.test.ts b/packages/core/src/markdown/to-markdown-handlers.overescape.test.tsnew file mode 100644index 000000000..2caf792e8--- /dev/null+++ b/packages/core/src/markdown/to-markdown-handlers.overescape.test.ts@@ -0,0 +1,201 @@+import { describe, expect, test } from 'bun:test';+import { sharedExtensions } from '../extensions/shared.ts';+import { MarkdownManager } from './index.ts';++const md = new MarkdownManager({ extensions: sharedExtensions });+const rt = (s: string): string => md.serialize(md.parse(s));++const expectByteExact = (src: string): void => {+ const once = rt(src);+ expect(once).toBe(src);+ expect(rt(once)).toBe(src);+};++const expectStable = (src: string, out: string): void => {+ const once = rt(src);+ expect(once).toBe(out);+ expect(rt(once)).toBe(out);+};++describe('inert delimiters stay bare (over-escape cleared)', () => {+ test('whitespace-flanked asterisk round-trips byte-exact', () => {+ expectByteExact('a * b\n');+ });++ test('whitespace-flanked asterisk run round-trips byte-exact', () => {+ expectByteExact('a ** b\n');+ });++ test('intraword underscore round-trips byte-exact', () => {+ expectByteExact('a_b_c\n');+ });++ test('under_score round-trips without gaining a backslash', () => {+ expectByteExact('under_score\n');+ });++ test('snake_case identifiers in prose round-trip byte-exact', () => {+ expectByteExact('call snake_case_name with under_score args\n');+ });++ test('whitespace-flanked tilde round-trips byte-exact', () => {+ expectByteExact('word ~ word\n');+ });++ test('whitespace-flanked double tilde round-trips byte-exact', () => {+ expectByteExact('a ~~ b\n');+ });++ test('whitespace-flanked asterisk inside an ATX heading round-trips byte-exact', () => {+ expectByteExact('# a * b\n');+ });++ test('whitespace-flanked asterisk inside a blockquote round-trips byte-exact', () => {+ expectByteExact('> a * b\n');+ });++ test('inert asterisk keeps parsing as plain text after the bare round-trip', () => {+ const out = rt('a * b\n');+ const para = md.parseToMdast(out).children[0];+ expect(para?.type).toBe('paragraph');+ const children = (para as { children: Array<{ type: string; value?: string }> }).children;+ expect(children).toHaveLength(1);+ expect(children[0]?.type).toBe('text');+ expect(children[0]?.value).toBe('a * b');+ });++ test('intraword underscore inside an underscore-delimited emphasis stays bare', () => {+ expectByteExact('_x_y_\n');+ const para = md.parseToMdast('_x_y_\n').children[0];+ const em = (para as { children: Array<{ type: string; children?: Array<{ value?: string }> }> })+ .children[0];+ expect(em?.type).toBe('emphasis');+ expect(em?.children?.[0]?.value).toBe('x_y');+ });+});++describe('line-start = escapes only for the setext-underline shape', () => {+ test('padded highlight-like text does not gain a backslash', () => {+ expectByteExact('== foo ==\n');+ });++ test('mid-line double equals stays bare', () => {+ expectByteExact('x == y\n');+ });++ test('line-start = followed by prose stays bare', () => {+ expectByteExact('= line of prose\n');+ });++ test('non-contiguous equals line stays bare (not a setext-underline shape)', () => {+ expectByteExact('== ==\n');+ });++ test('lazy continuation line starting with = followed by prose stays bare', () => {+ expectByteExact('foo\n== bar\n');+ });++ test('pure = paragraph stays escaped (setext-underline shape)', () => {+ expectStable('==\n', '\\==\n');+ });++ test('escaped pure = line round-trips byte-exact', () => {+ expectByteExact('\\==\n');+ });++ test('escaped setext-underline shape after a soft break round-trips byte-exact', () => {+ expectByteExact('foo\n\\==\n');+ });++ test('escaped = before prose on a continuation line drops no bytes', () => {+ expectByteExact('foo\n\\= bar\n');+ });+});++describe('flanking-active delimiters still escape (precision sweep)', () => {+ test('escaped emphasis-able pair round-trips byte-exact', () => {+ expectByteExact('a \\*bc\\* d\n');+ });++ test('escaped full emphasis shape round-trips byte-exact', () => {+ expectByteExact('\\*foo\\*\n');+ });++ test('escaped left-flanking single asterisk round-trips byte-exact', () => {+ expectByteExact('a \\*b\n');+ });++ test('escaped underscore pair round-trips byte-exact', () => {+ expectByteExact('a \\_b\\_ c\n');+ });++ test('escaped tilde pairs round-trip byte-exact', () => {+ expectByteExact('a \\~\\~b\\~\\~ c\n');+ });++ test('line-start left-flanking asterisk still gains an escape', () => {+ expectStable('*x\n', '\\*x\n');+ });++ test('left-flanking asterisk mid-phrasing still gains an escape', () => {+ expectStable('a *b\n', 'a \\*b\n');+ });++ test('quote-flanked underscore still escapes (punctuation adjacency can open)', () => {+ expectStable('a "_" b\n', 'a "\\_" b\n');+ });++ test('text ending in bare asterisks before emphasis keeps the boundary escaped', () => {+ expectStable('foo***bar*\n', 'foo\\*\\**bar*\n');+ const para = md.parseToMdast('foo\\*\\**bar*\n').children[0];+ const children = (para as { children: Array<{ type: string; value?: string }> }).children;+ expect(children.map((c) => c.type)).toEqual(['text', 'emphasis']);+ expect(children[0]?.value).toBe('foo**');+ });++ test('text starting with bare asterisks after emphasis keeps the boundary escaped', () => {+ expectStable('*foo***bar\n', '*foo*\\*\\*bar\n');+ const para = md.parseToMdast('*foo*\\*\\*bar\n').children[0];+ const children = (para as { children: Array<{ type: string; value?: string }> }).children;+ expect(children.map((c) => c.type)).toEqual(['emphasis', 'text']);+ expect(children[1]?.value).toBe('**bar');+ });++ test('rule-of-three blocked shapes stay all-literal with every run escaped', () => {+ expectStable('foo**bar*\n', 'foo\\*\\*bar\\*\n');+ expectStable('*foo**bar\n', '\\*foo\\*\\*bar\n');+ });+});++describe('atBreak guards for * _ ~ are unchanged (list / thematic / fence threats)', () => {+ test('escaped list-marker shape at line start round-trips byte-exact', () => {+ expectByteExact('\\* foo\n');+ });++ test('line-start underscore before a space stays escaped (thematic-break guard)', () => {+ expectStable('_ foo\n', '\\_ foo\n');+ });++ test('line-start tilde before a space stays escaped (fence guard)', () => {+ expectStable('~ foo\n', '\\~ foo\n');+ });+});++describe('genuine formation cases are untouched', () => {+ test('emphasis, strong, strikethrough, and highlight round-trip byte-exact', () => {+ expectByteExact('*em*\n');+ expectByteExact('_em_\n');+ expectByteExact('**st**\n');+ expectByteExact('__st__\n');+ expectByteExact('~~del~~\n');+ expectByteExact('==hl==\n');+ });++ test('intraword strikethrough still parses and round-trips', () => {+ expectByteExact('a~~b~~c\n');+ });++ test('emphasis adjacent to inert delimiters round-trips byte-exact', () => {+ expectByteExact('a * b *em* c * d\n');+ });+});diff --git a/packages/core/src/markdown/to-markdown-handlers.residual-axes.test.ts b/packages/core/src/markdown/to-markdown-handlers.residual-axes.test.tsnew file mode 100644index 000000000..c2465b119--- /dev/null+++ b/packages/core/src/markdown/to-markdown-handlers.residual-axes.test.ts@@ -0,0 +1,323 @@+import { describe, expect, test } from 'bun:test';+import { sharedExtensions } from '../extensions/shared.ts';+import { MarkdownManager } from './index.ts';++const md = new MarkdownManager({ extensions: sharedExtensions });+const rt = (s: string): string => md.serialize(md.parse(s));++const expectByteExact = (src: string): void => {+ const expected = src.endsWith('\n') ? src : `${src}\n`;+ const once = rt(src);+ expect(once).toBe(expected);+ expect(rt(once)).toBe(expected);+};++describe('gfm-table: outer-pipe presence + dash-row padding', () => {+ test('pipe-less table round-trips byte-exact', () => {+ expectByteExact('col|val\n-|-\nalpha|99');+ });++ test('unpadded delimiter row keeps zero padding', () => {+ expectByteExact('|a|b|\n|-|-|\n|c|d|');+ });++ test('canonical padded table is unchanged', () => {+ expectByteExact('| a | b |\n| - | - |\n| c | d |');+ });++ test('leading-pipe-only style round-trips', () => {+ expectByteExact('|a|b\n|-|-\n|c|d');+ });++ test('alignment markers survive with zero padding', () => {+ expectByteExact('|a|b|\n|:-|-:|\n|c|d|');+ });++ test('mixed outer-pipe styles fall back to canonical (non-uniform: no capture)', () => {+ const out = rt('|a|b|\n-|-\nc|d');+ expect(out).toBe('|a|b|\n|-|-|\n|c|d|\n');+ expect(rt(out)).toBe(out);+ });++ test('stale outer-pipe capture with an emptied boundary cell falls back to piped form', () => {+ const json = md.parse('col|val\n-|-\nalpha|99');+ const firstRow = (+ json as unknown as {+ content: Array<{ type: string; content: Array<{ content: Array<unknown> }> }>;+ }+ ).content.find((n) => n.type === 'table');+ const firstCell = firstRow?.content[0]?.content[0] as { content?: unknown[] } | undefined;+ if (firstCell) firstCell.content = [];+ const out = md.serialize(json);+ expect(out.split('\n')[0]).toBe('||val|');+ expect(rt(out)).toBe(out);+ });+});++describe('list: marker spacing, ordinal replay, nested indent', () => {+ test('double-space marker spacing round-trips', () => {+ expectByteExact('- item');+ });++ test('quad-space marker spacing round-trips', () => {+ expectByteExact('- item');+ });++ test('all-ones ordered numbering is not force-incremented', () => {+ expectByteExact('1. a\n1. b');+ });++ test('descending-style numbering replays verbatim', () => {+ expectByteExact('3. a\n1. b\n1. c');+ });++ test('canonical incrementing numbering is unchanged', () => {+ expectByteExact('1. a\n2. b');+ });++ test('non-canonical nested-list indent round-trips', () => {+ expectByteExact('- a\n - b');+ });++ test('canonical nested-list indent is unchanged', () => {+ expectByteExact('- a\n - b');+ });++ test('three-level canonical nesting is idempotent (relative-indent capture)', () => {+ expectByteExact('- L1\n - L2\n - L3');+ });++ test('non-canonical indent on a doubly-nested list round-trips', () => {+ expectByteExact('- L1\n - L2\n - L3');+ });++ test('ordered marker spacing composes with ordinal replay', () => {+ expectByteExact('1. a\n1. b');+ });++ test('marker spacing of 5+ stays canonical (indented-code boundary)', () => {+ const out = rt('- item');+ expect(rt(out)).toBe(out);+ });+});++describe('gfm-task: checkbox case', () => {+ test('uppercase [X] round-trips', () => {+ expectByteExact('- [X] done');+ });++ test('lowercase [x] round-trips', () => {+ expectByteExact('- [x] done');+ });++ test('unchecked task round-trips', () => {+ expectByteExact('- [ ] todo');+ });++ test('mixed-case task list round-trips per item', () => {+ expectByteExact('- [X] a\n- [x] b\n- [ ] c');+ });+});++describe('blockquote: marker-spacing counts', () => {+ test('double-space marker round-trips', () => {+ expectByteExact('> quote');+ });++ test('triple-space marker round-trips', () => {+ expectByteExact('> quote');+ });++ test('indented code inside a blockquote does not double its indent', () => {+ expectByteExact('> code');+ });++ test('no-space marker still round-trips', () => {+ expectByteExact('>quote');+ });+});++describe('code-fence: closing length, leading indent, info padding', () => {+ test('longer closing fence round-trips', () => {+ expectByteExact('```\ncode\n````');+ });++ test('uniformly indented fence round-trips', () => {+ expectByteExact(' ```\n code\n ```');+ });++ test('info-string padding round-trips', () => {+ expectByteExact('``` js\ncode\n```');+ });++ test('single-space info padding round-trips', () => {+ expectByteExact('``` js\ncode\n```');+ });++ test('indent + info padding + closing length compose', () => {+ expectByteExact(' ``` ts\n code\n ````');+ });++ test('fence-recompute: synthesized value containing a closing run lengthens the fence', () => {+ const json = JSON.parse(JSON.stringify(md.parse('```\nplaceholder\n```')));+ const codeNode = (+ json as { content: Array<{ type: string; content?: Array<{ text?: string }> }> }+ ).content.find((n) => n.type === 'codeBlock');+ expect(codeNode).toBeDefined();+ if (codeNode?.content?.[0]) codeNode.content[0].text = 'a\n```\nb';+ const out = md.serialize(json as ReturnType<typeof md.parse>);+ expect(out).toBe('````\na\n```\nb\n````\n');+ expect(rt(out)).toBe(out);+ });+});++describe('atx heading: leading indent + interior space', () => {+ test('leading indent round-trips', () => {+ expectByteExact(' # Heading');+ });++ test('interior spacing round-trips', () => {+ expectByteExact('# Heading');+ });++ test('indent + interior compose', () => {+ expectByteExact(' # Heading');+ });++ test('interior spacing composes with a closing hash run', () => {+ expectByteExact('# Heading ##');+ });++ test('heading inside a list item does not capture the container pad', () => {+ expectByteExact('1. item\n\n # heading in item');+ });+});++describe('gfm-strikethrough: delimiter run', () => {+ test('single-tilde form round-trips', () => {+ expectByteExact('~x~ y');+ });++ test('double-tilde form round-trips', () => {+ expectByteExact('~~x~~ y');+ });++ test('both forms coexist per node', () => {+ expectByteExact('~a~ and ~~b~~');+ });+});++describe('wikiLink: authored padding survives the trim', () => {+ test('padded target round-trips', () => {+ expectByteExact('[[ Page ]]');+ });++ test('padded target + alias round-trip', () => {+ expectByteExact('[[Page | alias ]]');+ });++ test('padded anchor round-trips', () => {+ expectByteExact('[[Page# Anchor ]]');+ });++ test('unpadded forms are unchanged', () => {+ expectByteExact('[[Page]]');+ expectByteExact('[[Page|alias]]');+ expectByteExact('[[Page#Anchor|alias]]');+ });++ test('stale raw segment is dropped after a target edit', () => {+ const json = JSON.parse(JSON.stringify(md.parse('[[ Page ]]')));+ const para = (json as { content: Array<{ content?: Array<Record<string, unknown>> }> })+ .content[0];+ const wiki = para.content?.find((n) => n.type === 'wikiLink') as+ | { attrs: Record<string, unknown> }+ | undefined;+ expect(wiki).toBeDefined();+ if (wiki) wiki.attrs.target = 'Renamed';+ const out = md.serialize(json as ReturnType<typeof md.parse>);+ expect(out).toBe('[[Renamed]]\n');+ });+});++describe('inline-code: authored disambiguation pad', () => {+ test('padded inline code round-trips', () => {+ expectByteExact('a ` y ` b');+ });++ test('the intra-block multi-axis non-vacuity witness round-trips', () => {+ expectByteExact('one \ntwo with ` y ` pad');+ });++ test('unpadded inline code is unchanged', () => {+ expectByteExact('a `y` b');+ });++ test('structurally-required padding still applies without capture', () => {+ expectByteExact('a `` `tick `` b');+ });+});++describe('inline-code: pipe re-escape inside a table cell', () => {+ test('escaped pipe inside inline code in a cell round-trips byte-exact', () => {+ expectByteExact('| `a\\|b` |\n| - |');+ });++ test('the emitted bytes re-parse as a table, not a paragraph', () => {+ const out = rt('| `a\\|b` |\n| - |\n');+ expect(md.parseToMdast(out).children[0]?.type).toBe('table');+ });++ test('every pipe in a multi-pipe code span re-escapes', () => {+ expectByteExact('| `a\\|b\\|c` |\n| - |');+ });++ test('fence disambiguation composes with the cell escape', () => {+ expectByteExact('| ``a`\\|b`` |\n| - |');+ });++ test('inline code outside a table never gains the cell escape', () => {+ expectByteExact('`a|b`');+ expectByteExact('`a\\|b`');+ });++ test('plain-text cell pipe escape is unchanged', () => {+ expectByteExact('| a\\|b |\n| - |');+ });++ test('padded code with a pipe keeps the escape; the pad capture is a separate open residual', () => {+ expect(rt('| ` a\\|b ` |\n| - |\n')).toBe('| `a\\|b` |\n| - |\n');+ });+});++describe('guard-sentinel literal collision (escape alphabet)', () => {+ test('authored guard sentinels U+E000-E004 round-trip byte-exact', () => {+ expectByteExact('literal \uE000\uE001\uE002\uE003\uE004 chars');+ });++ test('sentinel adjacent to guarded syntax round-trips', () => {+ expectByteExact('x\uE000y and <https://example.com>');+ });++ test('sentinel inside a fence info string round-trips', () => {+ expectByteExact('```l\uE000ng\ncode\n```');+ });++ test('escape alphabet U+E005-E009 is the documented reserved range', () => {+ expect(rt('esc \uE005 char')).toBe('esc \uE000 char\n');+ });+});++describe('already-byte-identical constructs stay untouched', () => {+ test('%%-comment padding is unchanged', () => {+ expectByteExact('%% padded comment %%');+ });++ test('callout case is unchanged', () => {+ expectByteExact('> [!note]\n> body');+ });++ test('details-quote form is unchanged', () => {+ expectByteExact('<details>\n\n<summary>t</summary>\n\n</details>');+ });+});diff --git a/packages/core/src/markdown/to-markdown-handlers.test.ts b/packages/core/src/markdown/to-markdown-handlers.test.tsindex a7d0fa00a..aba70e1c8 100644--- a/packages/core/src/markdown/to-markdown-handlers.test.ts+++ b/packages/core/src/markdown/to-markdown-handlers.test.ts@@ -198,8 +198,8 @@ describe('to-markdown: lone backtick in text NOT escaped (FR-1d)', () => {});describe('to-markdown: thematic break preservation', () => {- test('doc-start --- normalizes to *** (NG10 serialize-side)', () => {- expect(roundTrip('---\n')).toBe('***\n');+ test('doc-start --- round-trips as --- (the *** override is retired)', () => {+ expect(roundTrip('---\n')).toBe('---\n');});test('*** round-trips as ***', () => {@@ -1406,7 +1406,7 @@ describe('to-markdown: blockquote marker spacing (FR-23)', () => {const doc = schema.nodeFromJSON(json);const blockquote = doc.firstChild;expect(blockquote?.type.name).toBe('blockquote');- expect(blockquote?.attrs.sourceMarkerSpacings).toEqual(['single', 'none']);+ expect(blockquote?.attrs.sourceMarkerSpacings).toEqual([1, 0]);});test('PM blockquote attr is null for WYSIWYG-authored blockquote (no source)', () => {diff --git a/packages/core/src/markdown/to-markdown-handlers.ts b/packages/core/src/markdown/to-markdown-handlers.tsindex 62542c87d..a5c985768 100644--- a/packages/core/src/markdown/to-markdown-handlers.ts+++ b/packages/core/src/markdown/to-markdown-handlers.ts@@ -1,6 +1,7 @@import type { Nodes, Parents } from 'mdast';import type { MdxJsxAttribute, MdxJsxExpressionAttribute, MdxJsxFlowElement } from 'mdast-util-mdx';import type { Info, State } from 'mdast-util-to-markdown';+import { classifyCharacter } from 'micromark-util-classify-character';import { isValidSourceLiteralRaw } from '../extensions/source-literal-mark.ts';declare module 'mdast-util-to-markdown' {@@ -8,6 +9,7 @@ declare module 'mdast-util-to-markdown' {mark: 'mark';comment: 'comment';commentBlock: 'commentBlock';+ strikethrough: 'strikethrough';}}@@ -24,6 +26,10 @@ type MdastToMarkdownHandlers = {mdxJsxFlowElement: (node: Nodes, parent: Parents | undefined, state: State, info: Info) => string;};+const inlineMathHandler = Object.assign(serializeInlineMath, { peek: () => '$' });++const deleteHandler = Object.assign(serializeDelete, { peek: () => '~' });+export const toMarkdownHandlers = {text(node, _parent, state, info) {if (typeof node.data?.sourceRaw === 'string') {@@ -39,13 +45,20 @@ export const toMarkdownHandlers = {let lastIdx = 0;for (const { offset, char } of escaped) {if (offset > lastIdx) {- result += safeText(state, value.slice(lastIdx, offset), info);+ result += safeText(state, value.slice(lastIdx, offset), {+ ...info,+ before: info.before + result,+ after: '\\',+ });}result += `\\${char}`;lastIdx = offset + 1;}if (lastIdx < value.length) {- result += safeText(state, value.slice(lastIdx), info);+ result += safeText(state, value.slice(lastIdx), {+ ...info,+ before: info.before + result,+ });}return result;}@@ -58,11 +71,15 @@ export const toMarkdownHandlers = {const tracker = state.createTracker(info);const exit = state.enter('emphasis');let value = tracker.move(delim);- value += state.containerPhrasing(node, {- before: value,- after: delim,- ...tracker.current(),- });+ value += encodeAttentionBoundaries(+ state.containerPhrasing(node, {+ before: value,+ after: delim,+ ...tracker.current(),+ }),+ info,+ delim,+ );value += tracker.move(delim);exit();return value;@@ -73,11 +90,15 @@ export const toMarkdownHandlers = {const tracker = state.createTracker(info);const exit = state.enter('strong');let value = tracker.move(delim);- value += state.containerPhrasing(node, {- before: value,- after: delim,- ...tracker.current(),- });+ value += encodeAttentionBoundaries(+ state.containerPhrasing(node, {+ before: value,+ after: delim,+ ...tracker.current(),+ }),+ info,+ delim,+ );value += tracker.move(delim);exit();return value;@@ -203,16 +224,8 @@ export const toMarkdownHandlers = {return value;},- thematicBreak(node, _parent, state) {- const sourceRaw = node.data?.sourceRaw;- const isDocStart =- Array.isArray(state?.indexStack) &&- state.indexStack.length === 1 &&- state.indexStack[0] === 0;- if (isDocStart && (!sourceRaw || /^-[-\s]*-\s*$/.test(sourceRaw))) {- return '***';- }- return sourceRaw ?? '---';+ thematicBreak(node) {+ return node.data?.sourceRaw ?? '---';},break(node) {@@ -223,22 +236,48 @@ export const toMarkdownHandlers = {code(node) {const value = node.value ?? '';if (node.data?.sourceStyle === 'indented') {- const indented = value- .split('\n')- .map((line) => (line.length > 0 ? ` ${line}` : line))- .join('\n');+ const lines = value.split('\n');+ const indents = node.data.sourceIndents;+ if (Array.isArray(indents) && indents.length === lines.length) {+ return lines+ .map((line, i) => `${indents[i] ?? (line.length > 0 ? ' ' : '')}${line}`)+ .join('\n');+ }+ const indented = lines.map((line) => (line.length > 0 ? ` ${line}` : line)).join('\n');return indented;}const fenceChar = node.data?.sourceFenceChar;const char = fenceChar === '~' ? '~' : '`';- const len = Math.max(3, node.data?.sourceFenceLength ?? 3);+ let len = Math.max(3, node.data?.sourceFenceLength ?? 3);+ const fenceRunRe = new RegExp(`^ {0,3}(\\${char}+)[ \\t]*$`);+ for (const line of value.split('\n')) {+ const run = fenceRunRe.exec(line);+ if (run && run[1].length >= len) {+ len = run[1].length + 1;+ }+ }+ const closingCaptured = node.data?.sourceClosingFenceLength;+ const closeLen =+ typeof closingCaptured === 'number' && closingCaptured > len ? closingCaptured : len;+ const indentCount = node.data?.sourceFenceIndent;+ const indent =+ typeof indentCount === 'number' && indentCount >= 1 && indentCount <= 3+ ? ' '.repeat(indentCount)+ : '';+ const infoGap = node.data?.sourceInfoPadding;const fence = char.repeat(len);const lang = node.lang ?? '';+ const langGap = lang && typeof infoGap === 'number' && infoGap >= 1 ? ' '.repeat(infoGap) : '';const meta = node.meta ? ` ${node.meta}` : '';- return `${fence}${lang}${meta}\n${value}\n${fence}`;+ const body = `${fence}${langGap}${lang}${meta}\n${value}\n${char.repeat(closeLen)}`;+ if (!indent) return body;+ return body+ .split('\n')+ .map((l) => (l.length > 0 ? `${indent}${l}` : l))+ .join('\n');},- inlineCode(node) {+ inlineCode(node, _parent, state) {let value = node.value ?? '';const requestedChar = node.data?.sourceFenceChar;const fenceChar = requestedChar === '~' ? '~' : '`';@@ -256,16 +295,22 @@ export const toMarkdownHandlers = {if (/[^ \r\n]/.test(value) &&- ((/^[ \r\n]/.test(value) && /[ \r\n]$/.test(value)) ||+ (node.data?.sourcePadded === true ||+ (/^[ \r\n]/.test(value) && /[ \r\n]$/.test(value)) ||value.startsWith(fenceChar) ||value.endsWith(fenceChar))) {value = ` ${value} `;}- return `${fence}${value}${fence}`;+ const span = `${fence}${value}${fence}`;+ return state.stack.includes('tableCell') ? span.replace(/\|/g, '\\$&') : span;},+ inlineMath: inlineMathHandler,++ delete: deleteHandler,+heading(node, _parent, state, info) {const style = node.data?.sourceStyle ?? 'atx';const depth = node.depth;@@ -277,6 +322,14 @@ export const toMarkdownHandlers = {const underline = (depth === 1 ? '=' : '-').repeat(underlineLength);return `${content}\n${underline}`;}+ const indentCount = node.data?.sourceLeadingIndent;+ const indent =+ typeof indentCount === 'number' && indentCount >= 1 && indentCount <= 3+ ? ' '.repeat(indentCount)+ : '';+ const interiorCount = node.data?.sourceInteriorSpacing;+ const interior =+ typeof interiorCount === 'number' && interiorCount >= 2 ? ' '.repeat(interiorCount) : ' ';const hashes = '#'.repeat(depth);const content = state.containerPhrasing(node, {...info,@@ -286,11 +339,11 @@ export const toMarkdownHandlers = {const trailingCount = node.data?.sourceTrailingHashes;if (typeof trailingCount === 'number' && trailingCount > 0) {const trailing = '#'.repeat(trailingCount);- if (!content) return `${hashes} ${trailing}`;- return `${hashes} ${content} ${trailing}`;+ if (!content) return `${indent}${hashes} ${trailing}`;+ return `${indent}${hashes}${interior}${content} ${trailing}`;}- if (!content) return hashes;- return `${hashes} ${content}`;+ if (!content) return `${indent}${hashes}`;+ return `${indent}${hashes}${interior}${content}`;},list(node, _parent, state, info) {@@ -313,15 +366,36 @@ export const toMarkdownHandlers = {for (let i = 0; i < children.length; i++) {const child = children[i];- const baseMarker = ordered ? `${counter + i}${delim}` : (node.data?.bulletMarker ?? bullet);+ const capturedOrdinal = child.data?.sourceOrdinal;+ const ordinal =+ typeof capturedOrdinal === 'number' && capturedOrdinal >= 0 ? capturedOrdinal : counter + i;+ const baseMarker = ordered ? `${ordinal}${delim}` : (node.data?.bulletMarker ?? bullet);+ const capturedSpacing = child.data?.sourceMarkerSpacing;+ const spacing =+ typeof capturedSpacing === 'number' && capturedSpacing >= 2 && capturedSpacing <= 4+ ? capturedSpacing+ : 1;+ const markerGap = ' '.repeat(spacing);let marker = baseMarker;- if (child.checked === true) marker += ' [x]';- else if (child.checked === false) marker += ' [ ]';- const pad = ' '.repeat(baseMarker.length + 1);+ if (child.checked === true) {+ marker += `${markerGap}[${child.data?.sourceCheckboxChar === 'X' ? 'X' : 'x'}]`;+ } else if (child.checked === false) {+ marker += `${markerGap}[ ]`;+ }+ const capturedIndent = child.data?.sourceContinuationIndent;+ const contentCol = baseMarker.length + spacing;+ const padWidth =+ typeof capturedIndent === 'number' &&+ capturedIndent >= contentCol &&+ capturedIndent <= contentCol + 3+ ? capturedIndent+ : contentCol;+ const pad = ' '.repeat(padWidth);+ const firstGap = child.checked === null || child.checked === undefined ? markerGap : ' ';const itemContent = state.containerFlow(child, info);const indented = itemContent.split('\n')- .map((l, idx) => (idx === 0 ? `${marker} ${l}` : l ? `${pad}${l}` : l))+ .map((l, idx) => (idx === 0 ? `${marker}${firstGap}${l}` : l ? `${pad}${l}` : l)).join('\n');out.push(indented);}@@ -403,14 +477,35 @@ export const toMarkdownHandlers = {const pad = paddings[i] ?? { left: 1, right: 1 };out.push(' '.repeat(pad.left) + content + ' '.repeat(pad.right));}- return `|${out.join('|')}|`;+ const leading = outerPipesSafe && !outerPipesSafe.leading ? '' : '|';+ const trailing = outerPipesSafe && !outerPipesSafe.trailing ? '' : '|';+ return `${leading}${out.join('|')}${trailing}`;};+ const capturedAlignPadding = node.data?.sourceAlignmentPadding;const alignmentPaddings: Array<{ left: number; right: number } | null> = Array.from({ length: mostCellsPerRow },- () => null,+ (_, col) => {+ const pad = capturedAlignPadding?.[col];+ return pad && typeof pad.left === 'number' && typeof pad.right === 'number' ? pad : null;+ },);+ const outerPipes = node.data?.sourceOuterPipes;+ const outerPipesSafe =+ outerPipes &&+ mostCellsPerRow >= 2 &&+ matrix.every((cells) => {+ const first = (cells[0] ?? '').trim();+ const last = (cells[mostCellsPerRow - 1] ?? '').trim();+ return (+ (outerPipes.leading || first.length > 0) &&+ (outerPipes.trailing || (last.length > 0 && !last.endsWith('\\')))+ );+ })+ ? outerPipes+ : null;+if (matrix.length === 0) {return formatRow(alignmentCells, alignmentPaddings);}@@ -553,7 +648,10 @@ export const toMarkdownHandlers = {return state.indentLines(inner, (line, _lineNumber, blank) => {if (blank) return '>';const spacing = spacings?.[nonBlankIdx++];- if (spacing === 'none') return `>${line}`;+ if (spacing === 'none' || spacing === 0) return `>${line}`;+ if (typeof spacing === 'number' && spacing >= 2 && !line.startsWith(' ')) {+ return `>${' '.repeat(spacing)}${line}`;+ }return `> ${line}`;});},@@ -649,6 +747,13 @@ function safeText(state: State, value: string, info: Info): string {if (u.character === '[') return false;if (u.character === '(') return false;if (u.character === '`') return false;+ if (+ (u.character === '*' || u.character === '_' || u.character === '~') &&+ u.inConstruct === 'phrasing'+ ) {+ return false;+ }+ if (u.character === '=' && u.atBreak === true) return false;return true;});let result: string;@@ -657,5 +762,178 @@ function safeText(state: State, value: string, info: Info): string {} finally {state.unsafe = originalUnsafe;}- return escapeEntityAmpersands(result);+ return escapeEntityAmpersands(escapeActiveDelimiterRuns(result, info));+}++const CHARACTER_GROUP_WHITESPACE = 1;++const CHARACTER_GROUP_PUNCTUATION = 2;++type FlankClass = 'ws' | 'punct' | 'other';++function classifyFlank(code: number): FlankClass {+ if (Number.isNaN(code)) return 'ws';+ const group = classifyCharacter(code);+ if (group === CHARACTER_GROUP_WHITESPACE) return 'ws';+ if (group === CHARACTER_GROUP_PUNCTUATION) return 'punct';+ return 'other';+}++function isActiveDelimiterRun(+ marker: '*' | '_' | '~',+ beforeCode: number,+ afterCode: number,+): boolean {+ const before = classifyFlank(beforeCode);+ const after = classifyFlank(afterCode);+ const leftFlanking = after !== 'ws' && (after === 'other' || before !== 'other');+ const rightFlanking = before !== 'ws' && (before === 'other' || after !== 'other');+ if (marker === '_') {+ const canOpen = leftFlanking && (!rightFlanking || before === 'punct');+ const canClose = rightFlanking && (!leftFlanking || after === 'punct');+ return canOpen || canClose;+ }+ return leftFlanking || rightFlanking;+}++function isSetextUnderlineShape(value: string, runEnd: number, info: Info): boolean {+ let k = runEnd;+ while (k < value.length && (value[k] === ' ' || value[k] === '\t')) k++;+ if (k < value.length) return value[k] === '\n' || value[k] === '\r';+ const next = info.after.charCodeAt(0);+ return Number.isNaN(next) || next === 0x0a || next === 0x0d;+}++function escapeActiveDelimiterRuns(value: string, info: Info): string {+ let result = '';+ let i = 0;+ while (i < value.length) {+ const ch = value.charAt(i);+ if (ch === '\\') {+ result += value.slice(i, i + 2);+ i += 2;+ continue;+ }+ if (ch === '*' || ch === '_' || ch === '~' || ch === '=') {+ let j = i + 1;+ while (j < value.length && value.charAt(j) === ch) j++;+ const beforeCode =+ i === 0 ? info.before.charCodeAt(info.before.length - 1) : value.charCodeAt(i - 1);+ if (ch === '=') {+ const atLineStart = Number.isNaN(beforeCode) || beforeCode === 0x0a || beforeCode === 0x0d;+ result +=+ atLineStart && isSetextUnderlineShape(value, j, info)+ ? `\\${value.slice(i, j)}`+ : value.slice(i, j);+ } else {+ const afterCode = j < value.length ? value.charCodeAt(j) : info.after.charCodeAt(0);+ result += isActiveDelimiterRun(ch, beforeCode, afterCode)+ ? `\\${ch}`.repeat(j - i)+ : value.slice(i, j);+ }+ i = j;+ continue;+ }+ result += ch;+ i += 1;+ }+ return result;+}++function encodeAttentionBoundaries(between: string, info: Info, delim: string): string {+ const marker = delim.startsWith('_') ? '_' : '*';+ let result = between;+ const head = result.charCodeAt(0);+ if (shouldEncodeAttentionBoundary(info.before.charCodeAt(info.before.length - 1), head, marker)) {+ result = encodeCharacterReference(head) + result.slice(1);+ }+ const tail = result.charCodeAt(result.length - 1);+ if (shouldEncodeAttentionBoundary(info.after.charCodeAt(0), tail, marker)) {+ result = result.slice(0, -1) + encodeCharacterReference(tail);+ }+ return result;+}++function shouldEncodeAttentionBoundary(+ outside: number,+ inside: number,+ marker: '*' | '_',+): boolean {+ if (classifyCharacter(inside) !== CHARACTER_GROUP_WHITESPACE) return false;+ if (marker === '_' && classifyCharacter(outside) === undefined) return false;+ return true;+}++function encodeCharacterReference(code: number): string {+ return `&#x${code.toString(16).toUpperCase()};`;+}++function serializeInlineMath(+ node: Extract<Nodes, { type: 'inlineMath' }>,+ _parent: Parents | undefined,+ state: State,+ info: Info,+): string {+ let value = node.value ?? '';+ const captured = node.data?.sourceDelimiter;+ let size = typeof captured === 'string' && /^\$+$/.test(captured) ? captured.length : 2;+ if (size === 1 && !singleDollarMathReparses(value, info)) {+ size = 2;+ }+ while (new RegExp(`(^|[^$])${'\\$'.repeat(size)}([^$]|$)`).test(value)) {+ size++;+ }+ const sequence = '$'.repeat(size);+ if (+ /[^ \r\n]/.test(value) &&+ ((/^[ \r\n]/.test(value) && /[ \r\n]$/.test(value)) || /^\$|\$$/.test(value))+ ) {+ value = ` ${value} `;+ }+ let index = -1;+ while (++index < state.unsafe.length) {+ const pattern = state.unsafe[index];+ if (!pattern.atBreak) continue;+ const expression = state.compilePattern(pattern);+ let match: RegExpExecArray | null;+ // biome-ignore lint/suspicious/noAssignInExpressions: idiomatic regex iteration+ while ((match = expression.exec(value)) !== null) {+ let position = match.index;+ if (value.codePointAt(position) === 10 && value.codePointAt(position - 1) === 13) {+ position--;+ }+ value = `${value.slice(0, position)} ${value.slice(match.index + 1)}`;+ }+ }+ return sequence + value + sequence;+}++function serializeDelete(+ node: Extract<Nodes, { type: 'delete' }>,+ _parent: Parents | undefined,+ state: State,+ info: Info,+): string {+ const delim = node.data?.sourceDelimiter === '~' ? '~' : '~~';+ const tracker = state.createTracker(info);+ const exit = state.enter('strikethrough');+ let value = tracker.move(delim);+ value += state.containerPhrasing(node, {+ ...tracker.current(),+ before: value,+ after: '~',+ });+ value += tracker.move(delim);+ exit();+ return value;+}++function singleDollarMathReparses(value: string, info: Info): boolean {+ if (value.length === 0) return false;+ if (!/^\S/.test(value) || !/\S$/.test(value)) return false;+ if (/[$\r\n]/.test(value)) return false;+ if (value.endsWith('\\') || /\\[!-/:-@[-`{-~]/.test(value)) return false;+ if (/^\d/.test(info.after)) return false;+ if (info.before.endsWith('\\')) return false;+ return true;}diff --git a/packages/core/src/markdown/wiki-embed-opaque.test.ts b/packages/core/src/markdown/wiki-embed-opaque.test.tsnew file mode 100644index 000000000..b5c4ef28a--- /dev/null+++ b/packages/core/src/markdown/wiki-embed-opaque.test.ts@@ -0,0 +1,181 @@+import { describe, expect, test } from 'bun:test';+import type { JSONContent } from '@tiptap/core';+import { sharedExtensions } from '../extensions/shared.ts';+import { MarkdownManager } from './index.ts';++const mdManager = new MarkdownManager({ extensions: sharedExtensions });++function roundTrip(md: string): string {+ return mdManager.serialize(mdManager.parse(md));+}++function expectByteStable(md: string): void {+ const once = roundTrip(md);+ expect(once).toBe(md);+ expect(roundTrip(once)).toBe(once);+}++function findNodes(json: JSONContent, type: string): JSONContent[] {+ const out: JSONContent[] = [];+ const visit = (n: JSONContent) => {+ if (n.type === type) out.push(n);+ for (const child of n.content ?? []) visit(child);+ };+ visit(json);+ return out;+}++describe('opaque-extension wiki-embed — byte-exact PM round-trip', () => {+ test('block-context opaque embed round-trips byte-identical', () => {+ expectByteStable('![[notes.foo]]\n');+ });++ test('inline mid-prose opaque embed round-trips byte-identical', () => {+ expectByteStable('see ![[notes.foo]] here\n');+ });++ test('extension-less embed round-trips byte-identical', () => {+ expectByteStable('![[plainname]]\n');+ });++ test('serveable-but-not-inline-renderable extension round-trips byte-identical', () => {+ expectByteStable('![[diagram.avi]]\n');+ });++ test('opaque embed with alias round-trips byte-identical', () => {+ expectByteStable('![[notes.foo|My Alias]]\n');+ });++ test('opaque embed with anchor round-trips byte-identical', () => {+ expectByteStable('![[notes.foo#section]]\n');+ });++ test('opaque embed with anchor and alias round-trips byte-identical', () => {+ expectByteStable('![[notes.foo#section|Custom]]\n');+ });++ test('opaque embed coexisting with wikiLink and allowlisted embed round-trips', () => {+ expectByteStable('See [[Index]], ![[diagram.png]] and ![[archive.xyz]] together.\n');+ });+});++describe('opaque-extension wiki-embed — WYSIWYG label edit is honored, never reverted', () => {+ function editEmbedLabel(json: JSONContent, newText: string): JSONContent {+ let edited = 0;+ const visit = (n: JSONContent) => {+ if (+ n.type === 'text' &&+ n.marks?.some((m) => m.type === 'link' && m.attrs?.sourceForm === 'wikiembed')+ ) {+ n.text = newText;+ edited++;+ }+ for (const child of n.content ?? []) visit(child);+ };+ visit(json);+ expect(edited).toBe(1);+ return json;+ }++ test('editing an aliased embed label replaces the alias', () => {+ const json = editEmbedLabel(mdManager.parse('![[notes.foo|My Label]]\n'), 'Edited Label');+ expect(mdManager.serialize(json)).toBe('![[notes.foo|Edited Label]]\n');+ });++ test('editing an alias-less embed label adds an alias (target preserved)', () => {+ const json = editEmbedLabel(mdManager.parse('![[notes.foo]]\n'), 'My New Text');+ expect(mdManager.serialize(json)).toBe('![[notes.foo|My New Text]]\n');+ });++ test('editing an anchored embed label adds an alias (target#anchor preserved)', () => {+ const json = editEmbedLabel(mdManager.parse('![[notes.foo#section]]\n'), 'Renamed');+ expect(mdManager.serialize(json)).toBe('![[notes.foo#section|Renamed]]\n');+ });++ test('the edited form is byte-stable on subsequent round-trips', () => {+ const json = editEmbedLabel(mdManager.parse('![[notes.foo|My Label]]\n'), 'Edited Label');+ const once = mdManager.serialize(json);+ expect(roundTrip(once)).toBe(once);+ });++ test('an unedited label still round-trips byte-identical through the attrs path', () => {+ const json = mdManager.parse('![[notes.foo|My Label]]\n');+ expect(mdManager.serialize(json)).toBe('![[notes.foo|My Label]]\n');+ });++ test('a whitespace-only label edit does not produce an empty alias', () => {+ const json = editEmbedLabel(mdManager.parse('![[notes.foo|My Label]]\n'), ' ');+ expect(mdManager.serialize(json)).toBe('![[notes.foo|My Label]]\n');+ });++ test('a label edit containing `]]` keeps the attrs label — the closing delimiter is unrepresentable in the alias slot', () => {+ const json = editEmbedLabel(mdManager.parse('![[notes.foo|My Label]]\n'), 'My ]] Label');+ const once = mdManager.serialize(json);+ expect(once).toBe('![[notes.foo|My Label]]\n');+ expect(roundTrip(once)).toBe(once);+ });+});++describe('opaque-extension wiki-embed — PM shape contract', () => {+ test('opaque embed lands on text+link-mark tagged wikiembed, never a PM wikiLinkEmbed node', () => {+ const json = mdManager.parse('![[notes.foo]]\n');+ expect(findNodes(json, 'wikiLinkEmbed')).toHaveLength(0);+ const texts = findNodes(json, 'text');+ expect(texts).toHaveLength(1);+ const linkMark = texts[0]?.marks?.find((mk) => mk.type === 'link');+ expect(linkMark).toBeDefined();+ expect(linkMark?.attrs?.sourceForm).toBe('wikiembed');+ expect(linkMark?.attrs?.target).toBe('notes.foo');+ expect(linkMark?.attrs?.anchor).toBeNull();+ expect(linkMark?.attrs?.alias).toBeNull();+ });++ test('alias and anchor are carried on the mark, label shows the alias', () => {+ const json = mdManager.parse('![[notes.foo#sec|My Alias]]\n');+ const texts = findNodes(json, 'text');+ expect(texts[0]?.text).toBe('My Alias');+ const linkMark = texts[0]?.marks?.find((mk) => mk.type === 'link');+ expect(linkMark?.attrs?.target).toBe('notes.foo');+ expect(linkMark?.attrs?.anchor).toBe('sec');+ expect(linkMark?.attrs?.alias).toBe('My Alias');+ });++ test('resolver-remapped href does not perturb the byte round-trip', () => {+ const json = mdManager.parse('![[notes.foo]]\n', {+ resolveEmbed: (target) => (target === 'notes.foo' ? 'attachments/notes.foo' : null),+ sourcePath: 'docs/meeting.md',+ });+ const texts = findNodes(json, 'text');+ const linkMark = texts[0]?.marks?.find((mk) => mk.type === 'link');+ expect(linkMark?.attrs?.href).toBe('/attachments/notes.foo');+ expect(linkMark?.attrs?.target).toBe('notes.foo');+ expect(mdManager.serialize(json)).toBe('![[notes.foo]]\n');+ });+});++describe('opaque-extension widening — no regression on neighbors', () => {+ test('plain markdown link keeps a null sourceForm and round-trips unchanged', () => {+ const json = mdManager.parse('[label](dest.foo)\n');+ const texts = findNodes(json, 'text');+ const linkMark = texts[0]?.marks?.find((mk) => mk.type === 'link');+ expect(linkMark).toBeDefined();+ expect(linkMark?.attrs?.sourceForm).toBeNull();+ expect(mdManager.serialize(json)).toBe('[label](dest.foo)\n');+ });++ test('allowlisted block-context embed still promotes to jsxComponent and round-trips', () => {+ const json = mdManager.parse('![[photo.png]]\n');+ const node = json.content?.[0];+ expect(node?.type).toBe('jsxComponent');+ expect(node?.attrs?.componentName).toBe('WikiEmbedImage');+ expect(mdManager.serialize(json)).toBe('![[photo.png]]\n');+ });++ test('allowlisted inline embed keeps the wikiembed chip and round-trips', () => {+ expectByteStable('text ![[photo.png]] more text\n');+ });++ test('wikiLink sibling is untouched', () => {+ expectByteStable('[[Page]]\n');+ });+});diff --git a/packages/core/src/markdown/wiki-link-micromark.ts b/packages/core/src/markdown/wiki-link-micromark.tsindex 17dc71614..7aaf1099c 100644--- a/packages/core/src/markdown/wiki-link-micromark.ts+++ b/packages/core/src/markdown/wiki-link-micromark.ts@@ -290,19 +290,31 @@ function topNode<T>(ctx: CompileContext): T {function exitTarget(this: CompileContext, token: Token) {const node = topNode<WikiLinkMdast | WikiLinkEmbedMdast>(this);- node.data.target = this.sliceSerialize(token).trim();+ const raw = this.sliceSerialize(token);+ node.data.target = raw.trim();+ if (raw !== node.data.target) {+ node.data.sourceTarget = raw;+ }}function exitAnchor(this: CompileContext, token: Token) {const node = topNode<WikiLinkMdast | WikiLinkEmbedMdast>(this);- const raw = this.sliceSerialize(token).trim();- node.data.anchor = raw.length ? raw : null;+ const raw = this.sliceSerialize(token);+ const trimmed = raw.trim();+ node.data.anchor = trimmed.length ? trimmed : null;+ if (node.data.anchor !== null && raw !== trimmed) {+ node.data.sourceAnchor = raw;+ }}function exitAlias(this: CompileContext, token: Token) {const node = topNode<WikiLinkMdast | WikiLinkEmbedMdast>(this);- const raw = this.sliceSerialize(token).trim();- node.data.alias = raw.length ? raw : null;+ const raw = this.sliceSerialize(token);+ const trimmed = raw.trim();+ node.data.alias = trimmed.length ? trimmed : null;+ if (node.data.alias !== null && raw !== trimmed) {+ node.data.sourceAlias = raw;+ }}function finalizeLabel(node: WikiLinkMdast | WikiLinkEmbedMdast): void {@@ -336,14 +348,18 @@ export const wikiLinkFromMarkdown: FromMarkdownExtension = {},};+function rawSegmentOr(raw: string | null | undefined, current: string): string {+ return typeof raw === 'string' && raw.trim() === current ? raw : current;+}+const wikiLinkHandler: ToMarkdownHandle = (node) => {const wiki = node as unknown as WikiLinkMdast;const target = wiki.data?.target ?? '';const anchor = wiki.data?.anchor;const alias = wiki.data?.alias;- let out = `[[${target}`;- if (anchor) out += `#${anchor}`;- if (alias) out += `|${alias}`;+ let out = `[[${rawSegmentOr(wiki.data?.sourceTarget, target)}`;+ if (anchor) out += `#${rawSegmentOr(wiki.data?.sourceAnchor, anchor)}`;+ if (alias) out += `|${rawSegmentOr(wiki.data?.sourceAlias, alias)}`;return `${out}]]`;};diff --git a/packages/core/src/schema-snapshot.json b/packages/core/src/schema-snapshot.jsonindex 3239a366f..4c0619bbe 100644--- a/packages/core/src/schema-snapshot.json+++ b/packages/core/src/schema-snapshot.json@@ -14,6 +14,18 @@},"spread": {"hasDefault": true+ },+ "sourceMarkerSpacing": {+ "hasDefault": true+ },+ "sourceOrdinal": {+ "hasDefault": true+ },+ "sourceCheckboxChar": {+ "hasDefault": true+ },+ "sourceContinuationIndent": {+ "hasDefault": true}},"content": "paragraph block*",@@ -22,7 +34,11 @@"atom": false},"doc": {- "attrs": {},+ "attrs": {+ "sourceDocBoundary": {+ "hasDefault": true+ }+ },"content": "block+","group": "","inline": false,@@ -99,6 +115,9 @@},"language": {"hasDefault": true+ },+ "sourceDelimiter": {+ "hasDefault": true}},"content": "",@@ -119,6 +138,15 @@},"resolved": {"hasDefault": true+ },+ "sourceTarget": {+ "hasDefault": true+ },+ "sourceAnchor": {+ "hasDefault": true+ },+ "sourceAlias": {+ "hasDefault": true}},"content": "",@@ -196,6 +224,18 @@},"sourceStyle": {"hasDefault": true+ },+ "sourceClosingFenceLength": {+ "hasDefault": true+ },+ "sourceFenceIndent": {+ "hasDefault": true+ },+ "sourceInfoPadding": {+ "hasDefault": true+ },+ "sourceIndents": {+ "hasDefault": true}},"content": "text*",@@ -230,6 +270,12 @@},"sourceContiguousNext": {"hasDefault": true+ },+ "sourceLeadingIndent": {+ "hasDefault": true+ },+ "sourceInteriorSpacing": {+ "hasDefault": true}},"content": "inline*",@@ -297,6 +343,12 @@"attrs": {"sourceDashCounts": {"hasDefault": true+ },+ "sourceOuterPipes": {+ "hasDefault": true+ },+ "sourceAlignmentPadding": {+ "hasDefault": true}},"content": "tableRow+",@@ -447,6 +499,9 @@},"sourceFenceLength": {"hasDefault": true+ },+ "sourcePadded": {+ "hasDefault": true}},"excludes": "",@@ -454,13 +509,6 @@"inclusive": true,"spanning": true},- "strike": {- "attrs": {},- "excludes": "strike",- "group": "",- "inclusive": true,- "spanning": true- },"underline": {"attrs": {},"excludes": "underline",@@ -497,6 +545,17 @@"inclusive": true,"spanning": true},+ "strike": {+ "attrs": {+ "sourceDelimiter": {+ "hasDefault": true+ }+ },+ "excludes": "strike",+ "group": "",+ "inclusive": true,+ "spanning": true+ },"link": {"attrs": {"href": {@@ -583,6 +642,7 @@"listItem","emphasis","strong",+ "strike","code","codeBlock","blockquote",@@ -593,6 +653,7 @@"linkRefDef","hardBreak","escapeMark",+ "docFidelity","sourceLiteral","starterKit","table",diff --git a/packages/server/package.json b/packages/server/package.jsonindex 0f153b79a..733354aba 100644--- a/packages/server/package.json+++ b/packages/server/package.json@@ -62,6 +62,7 @@"test": "bun test"},"devDependencies": {+ "@types/mdast": "^4.0.4","@types/shell-quote": "^1.7.5","lib0": "^0.2.117","@types/ws": "^8.18.1",diff --git a/packages/server/src/bridge-watchdog.test.ts b/packages/server/src/bridge-watchdog.test.tsindex 6305e5563..0dbec4a74 100644--- a/packages/server/src/bridge-watchdog.test.ts+++ b/packages/server/src/bridge-watchdog.test.ts@@ -101,6 +101,25 @@ describe('assertBridgeInvariant — no-op for tolerance-equivalent inputs', () =}).not.toThrow();expect(getMetrics().bridgeInvariantViolations).toBe(0);});++ test('table-row trailing-pipe divergence tolerated (row-no-trailing-pipe)', () => {+ expect(() => {+ assertBridgeInvariant('| a | b\n| - | -\n| 1 | 2\n', '| a | b|\n| - | -|\n| 1 | 2|\n', {+ site: 'observer-b',+ });+ }).not.toThrow();+ expect(getMetrics().bridgeInvariantViolations).toBe(0);+ });++ test('touched-cell table divergence is NOT absorbed by the trailing-pipe tolerance', () => {+ expect(() => {+ assertBridgeInvariant(+ '| a | b |\n| - | - |\n| 1 | 2 |\n',+ '| a | b |\n| - | - |\n| 1 | 99 |\n',+ { site: 'observer-b' },+ );+ }).toThrow(BridgeInvariantViolationError);+ });});describe('assertBridgeInvariant — throws under NODE_ENV=test (default for bun test)', () => {diff --git a/packages/server/src/bridge-watchdog.ts b/packages/server/src/bridge-watchdog.tsindex 74af45c29..6cf3483b4 100644--- a/packages/server/src/bridge-watchdog.ts+++ b/packages/server/src/bridge-watchdog.ts@@ -4,6 +4,7 @@ import {BridgeInvariantViolationError,type BridgeToleranceClass,detectAppliedToleranceClasses,+ emitToleranceFire,normalizeBridge,toBridgeInvariantLog,} from '@inkeep/open-knowledge-core';@@ -238,6 +239,7 @@ export function assertBridgeInvariant(if (ytextNorm === fragNorm) {if (ytextSnapshot !== fragmentMdSnapshot) {const classes = detectAppliedToleranceClasses(ytextSnapshot, fragmentMdSnapshot);+ emitToleranceFire(classes, ytextSnapshot, fragmentMdSnapshot, opts.docName);for (const cls of classes) {if (shouldEmitBridgeToleranceApplied(opts.site, cls, opts.nowMs)) {incrementBridgeToleranceApplied(cls);diff --git a/packages/server/src/index.ts b/packages/server/src/index.tsindex f99444644..a743dfc78 100644--- a/packages/server/src/index.ts+++ b/packages/server/src/index.ts@@ -481,6 +481,11 @@ export {spansCurrentPath,spansPreviousPath,} from './telemetry-file-sink.ts';+export {+ initToleranceTelemetryWriter,+ isToleranceTelemetryEnabled,+ teardownToleranceTelemetryWriter,+} from './tolerance-telemetry-writer.ts';export {acquireUiLock,readUiLock,diff --git a/packages/server/src/map-driven-observer-a.test.ts b/packages/server/src/map-driven-observer-a.test.tsnew file mode 100644index 000000000..22604edf9--- /dev/null+++ b/packages/server/src/map-driven-observer-a.test.ts@@ -0,0 +1,354 @@+import { describe, expect, test } from 'bun:test';+import { MarkdownManager, sharedExtensions } from '@inkeep/open-knowledge-core';+import { getSchema } from '@tiptap/core';+import { updateYFragment } from '@tiptap/y-tiptap';+import * as Y from 'yjs';+import { AGENT_WRITE_ORIGIN } from './agent-sessions.ts';+import { composeAndWriteRawBody } from './bridge-intake.ts';+import { computeMapDrivenBodySplice } from './map-driven-splice.ts';+import { getMetrics } from './metrics.ts';+import { OBSERVER_SYNC_ORIGIN, setupServerObservers } from './server-observers.ts';++const mdManager = new MarkdownManager({ extensions: sharedExtensions });+const schema = getSchema(sharedExtensions);++function createTestDoc() {+ const doc = new Y.Doc();+ const xmlFragment = doc.getXmlFragment('default');+ const ytext = doc.getText('source');+ return { doc, xmlFragment, ytext };+}++function populateFragment(doc: Y.Doc, xmlFragment: Y.XmlFragment, md: string): void {+ const json = mdManager.parse(md);+ const pmNode = schema.nodeFromJSON(json);+ const meta = { mapping: new Map(), isOMark: new Map() };+ updateYFragment(doc, xmlFragment, pmNode, meta);+}++interface CapturedDelta {+ readonly origin: unknown;+ readonly ops: ReadonlyArray<{ retain?: number; insert?: string | unknown[]; delete?: number }>;+}++function captureYTextDeltas(ytext: Y.Text): CapturedDelta[] {+ const captured: CapturedDelta[] = [];+ const handler = (event: Y.YTextEvent, transaction: Y.Transaction): void => {+ captured.push({ origin: transaction.origin, ops: event.changes.delta });+ };+ ytext.observe(handler);+ return captured;+}++describe('map-driven Observer A — default Path A behavior', () => {+ test('(a) single-block edit produces narrow splice covering only the edited block', () => {+ const { doc, xmlFragment, ytext } = createTestDoc();+ populateFragment(doc, xmlFragment, '# Heading\n\nFirst paragraph.\n\nSecond paragraph.\n');+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });++ const before = ytext.toString();+ const deltas = captureYTextDeltas(ytext);++ populateFragment(+ doc,+ xmlFragment,+ '# Heading\n\nFirst paragraph EDITED.\n\nSecond paragraph.\n',+ );++ const observerWrites = deltas.filter((d) => d.origin === OBSERVER_SYNC_ORIGIN);+ expect(observerWrites.length).toBeGreaterThanOrEqual(1);+ const mapDrivenWrite = observerWrites[observerWrites.length - 1];++ const retainOps = mapDrivenWrite.ops.filter((op) => op.retain !== undefined);+ const insertOps = mapDrivenWrite.ops.filter((op) => op.insert !== undefined);+ const deleteOps = mapDrivenWrite.ops.filter((op) => op.delete !== undefined);++ expect(retainOps.length).toBeLessThanOrEqual(2);+ expect(insertOps.length + deleteOps.length).toBeGreaterThanOrEqual(1);++ const headingEnd = before.indexOf('# Heading') + '# Heading'.length;+ const secondParaStart = before.indexOf('Second paragraph');++ const leadingRetain = mapDrivenWrite.ops[0]?.retain ?? 0;+ expect(leadingRetain).toBeGreaterThanOrEqual(headingEnd);+ expect(leadingRetain).toBeLessThanOrEqual(before.indexOf('First paragraph') + 1);++ let cursorAfterWrite = leadingRetain;+ for (const op of mapDrivenWrite.ops.slice(1)) {+ if (op.delete !== undefined) cursorAfterWrite += op.delete;+ }+ expect(cursorAfterWrite).toBeLessThanOrEqual(secondParaStart);++ cleanup();+ });++ test('(b) untouched bytes outside the splice are byte-identical pre→post (AC1)', () => {+ const { doc, xmlFragment, ytext } = createTestDoc();+ populateFragment(doc, xmlFragment, '# Heading\n\nFirst.\n\nUntouched bytes here.\n');+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });++ const before = ytext.toString();+ const untouchedSnippet = 'Untouched bytes here.';+ const untouchedStartBefore = before.indexOf(untouchedSnippet);+ expect(untouchedStartBefore).toBeGreaterThanOrEqual(0);+ const tailBefore = before.slice(untouchedStartBefore);++ populateFragment(doc, xmlFragment, '# Heading\n\nEDITED first.\n\nUntouched bytes here.\n');++ const after = ytext.toString();+ const untouchedStartAfter = after.indexOf(untouchedSnippet);+ expect(untouchedStartAfter).toBeGreaterThanOrEqual(0);+ expect(after.slice(untouchedStartAfter)).toBe(tailBefore);++ cleanup();+ });++ test('(c) contiguous multi-block edit produces splice union covering both edited blocks', () => {+ const { doc, xmlFragment, ytext } = createTestDoc();+ populateFragment(doc, xmlFragment, 'first.\n\nsecond.\n\nthird.\n');+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });++ const before = ytext.toString();+ const deltas = captureYTextDeltas(ytext);++ populateFragment(doc, xmlFragment, 'first EDITED.\n\nsecond EDITED.\n\nthird.\n');++ const observerWrites = deltas.filter((d) => d.origin === OBSERVER_SYNC_ORIGIN);+ const mapDrivenWrite = observerWrites[observerWrites.length - 1];++ const leadingRetain = mapDrivenWrite.ops[0]?.retain ?? 0;+ let cursorAfterDeletes = leadingRetain;+ for (const op of mapDrivenWrite.ops.slice(1)) {+ if (op.delete !== undefined) cursorAfterDeletes += op.delete;+ }+ const thirdStart = before.indexOf('third.');+ expect(cursorAfterDeletes).toBeLessThanOrEqual(thirdStart);++ const after = ytext.toString();+ expect(after).toContain('first EDITED.');+ expect(after).toContain('second EDITED.');+ expect(after.slice(after.indexOf('third.'))).toBe(before.slice(thirdStart));++ cleanup();+ });++ test('(d) synthetic-doc name short-circuits to fallback path (no map-driven splice attempted)', () => {+ const { doc, xmlFragment, ytext } = createTestDoc();+ populateFragment(doc, xmlFragment, 'A.\n\nB.\n');+ const cleanup = setupServerObservers({+ doc,+ xmlFragment,+ ytext,+ mdManager,+ schema,+ docName: '__system__',+ });++ populateFragment(doc, xmlFragment, 'A EDITED.\n\nB.\n');++ expect(ytext.toString()).toContain('A EDITED.');+ expect(ytext.toString()).toContain('B.');++ cleanup();+ });++ test('(e) edit in paragraph containing ==highlight== degrades to block granularity (documented sub-block limitation)', () => {+ const { doc, xmlFragment, ytext } = createTestDoc();+ populateFragment(doc, xmlFragment, 'Para with ==highlight== inside.\n\nUntouched after.\n');+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });++ const before = ytext.toString();+ const untouchedSnippet = 'Untouched after.';+ const tailBefore = before.slice(before.indexOf(untouchedSnippet));++ populateFragment(+ doc,+ xmlFragment,+ 'Para with ==highlight== inside EDITED.\n\nUntouched after.\n',+ );++ const after = ytext.toString();+ const untouchedStartAfter = after.indexOf(untouchedSnippet);+ expect(untouchedStartAfter).toBeGreaterThanOrEqual(0);+ expect(after.slice(untouchedStartAfter)).toBe(tailBefore);++ cleanup();+ });++ test('(f) map-driven splice is the default — active with no env configuration', () => {+ expect(process.env.OK_MAP_DRIVEN_OBSERVER_A).toBeUndefined();++ const raw = '# Notes\n\n| a | b |\n| - | - |\n| 1 | 2\n';+ const { doc, xmlFragment, ytext } = createTestDoc();+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });+ doc.transact(() => {+ composeAndWriteRawBody(doc, raw, 'agent');+ }, AGENT_WRITE_ORIGIN);+ expect(ytext.toString()).toBe(raw);++ populateFragment(doc, xmlFragment, raw.replace('# Notes', '# Notes EDITED'));++ expect(ytext.toString()).toContain('# Notes EDITED');+ expect(ytext.toString()).toContain('| 1 | 2\n');++ cleanup();+ });++ test('(g) fallback: an offset-less block (comment block) falls back to applyIncrementalDiff and still converges', () => {+ const { doc, xmlFragment, ytext } = createTestDoc();+ populateFragment(doc, xmlFragment, '<!-- note -->\n\nOriginal.\n');+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });++ populateFragment(doc, xmlFragment, '<!-- note -->\n\nOriginal.\n\nAdded.\n');++ expect(ytext.toString()).toContain('Original.');+ expect(ytext.toString()).toContain('Added.');++ cleanup();+ });++ describe('dash-count tripwire — concurrent source-form table edits are detected + spliced, never silently dropped', () => {+ const narrowDashBody = 'before\n\n| a | b |\n| - | - |\n| 1 | 2 |\n\nafter\n';+ const wideDashBody = 'before\n\n| a | b |\n| --- | --- |\n| 1 | 2 |\n\nafter\n';++ test('computeMapDrivenBodySplice detects a dash-count-only change and emits the splice', () => {+ const splice = computeMapDrivenBodySplice(+ narrowDashBody,+ mdManager.parse(wideDashBody),+ mdManager,+ );++ expect(splice).not.toBeNull();+ if (!splice) throw new Error('unreachable');+ const applied =+ narrowDashBody.slice(0, splice.spliceStart) ++ splice.newSlice ++ narrowDashBody.slice(splice.spliceEnd);+ expect(applied).toBe(wideDashBody);+ });++ test('Observer A applies a dash-count-only fragment change to Y.Text (detected + spliced, blocks outside untouched)', () => {+ const { doc, xmlFragment, ytext } = createTestDoc();+ populateFragment(doc, xmlFragment, narrowDashBody);+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });+ expect(ytext.toString()).toBe(narrowDashBody);++ populateFragment(doc, xmlFragment, wideDashBody);++ expect(ytext.toString()).toBe(wideDashBody);++ cleanup();+ });+ });++ describe('splice-path observability — applied vs fallback(reason) counters', () => {+ function fallbackTotal(m: ReturnType<typeof getMetrics>): number {+ return Object.values(m.mapDrivenSpliceFallback).reduce((a, b) => a + (b ?? 0), 0);+ }++ test('a successful map-driven splice increments mapDrivenSpliceApplied (no fallback)', () => {+ const raw = '# Heading\n\nFirst.\n\nSecond.\n';+ const { doc, xmlFragment, ytext } = createTestDoc();+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });+ doc.transact(() => {+ composeAndWriteRawBody(doc, raw, 'agent');+ }, AGENT_WRITE_ORIGIN);+ expect(ytext.toString()).toBe(raw);++ const before = getMetrics();+ populateFragment(doc, xmlFragment, raw.replace('First.', 'First EDITED.'));+ const after = getMetrics();++ expect(after.mapDrivenSpliceApplied - before.mapDrivenSpliceApplied).toBe(1);+ expect(fallbackTotal(after) - fallbackTotal(before)).toBe(0);++ cleanup();+ });++ test('a synthetic-doc drain increments fallback reason synthetic-doc, not applied', () => {+ const raw = 'A.\n\nB.\n';+ const { doc, xmlFragment, ytext } = createTestDoc();+ const cleanup = setupServerObservers({+ doc,+ xmlFragment,+ ytext,+ mdManager,+ schema,+ docName: '__system__',+ });+ doc.transact(() => {+ composeAndWriteRawBody(doc, raw, 'agent');+ }, AGENT_WRITE_ORIGIN);++ const before = getMetrics();+ populateFragment(doc, xmlFragment, 'A EDITED.\n\nB.\n');+ const after = getMetrics();++ expect(+ (after.mapDrivenSpliceFallback['synthetic-doc'] ?? 0) -+ (before.mapDrivenSpliceFallback['synthetic-doc'] ?? 0),+ ).toBe(1);+ expect(after.mapDrivenSpliceApplied - before.mapDrivenSpliceApplied).toBe(0);++ cleanup();+ });++ test('an offset-less block drain increments fallback reason missing-position', () => {+ const raw = '<!-- note -->\n\nOriginal.\n';+ const { doc, xmlFragment, ytext } = createTestDoc();+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });+ doc.transact(() => {+ composeAndWriteRawBody(doc, raw, 'agent');+ }, AGENT_WRITE_ORIGIN);++ const before = getMetrics();+ populateFragment(doc, xmlFragment, '<!-- note -->\n\nOriginal.\n\nAdded.\n');+ const after = getMetrics();++ expect(+ (after.mapDrivenSpliceFallback['missing-position'] ?? 0) -+ (before.mapDrivenSpliceFallback['missing-position'] ?? 0),+ ).toBe(1);+ expect(after.mapDrivenSpliceApplied - before.mapDrivenSpliceApplied).toBe(0);++ cleanup();+ });++ test('a parse/serialize throw inside the splice reports parse-error instead of vanishing', () => {+ const throwingManager = {+ parseToMdast: () => {+ throw new Error('synthetic parser regression');+ },+ serialize: () => '',+ } as unknown as MarkdownManager;+ const reasons: string[] = [];++ const splice = computeMapDrivenBodySplice(+ 'A.\n',+ mdManager.parse('A.\n'),+ throwingManager,+ (reason) => {+ reasons.push(reason);+ },+ );++ expect(splice).toBeNull();+ expect(reasons).toEqual(['parse-error']);+ });++ test('an offset-less block reports missing-position through the pure computer', () => {+ const reasons: string[] = [];+ const splice = computeMapDrivenBodySplice(+ '<!-- note -->\n',+ mdManager.parse('<!-- note -->\n\nX.\n'),+ mdManager,+ (reason) => {+ reasons.push(reason);+ },+ );++ expect(splice).toBeNull();+ expect(reasons).toEqual(['missing-position']);+ });+ });+});diff --git a/packages/server/src/map-driven-splice.test.ts b/packages/server/src/map-driven-splice.test.tsnew file mode 100644index 000000000..d674c46df--- /dev/null+++ b/packages/server/src/map-driven-splice.test.ts@@ -0,0 +1,206 @@+import { describe, expect, test } from 'bun:test';+import { MarkdownManager, sharedExtensions } from '@inkeep/open-knowledge-core';+import type { JSONContent } from '@tiptap/core';+import { computeMapDrivenBodySplice } from './map-driven-splice.ts';++const mdManager = new MarkdownManager({ extensions: sharedExtensions });++function applySplice(+ oldBody: string,+ splice: { spliceStart: number; spliceEnd: number; newSlice: string },+): string {+ return oldBody.slice(0, splice.spliceStart) + splice.newSlice + oldBody.slice(splice.spliceEnd);+}++function pmFromMd(md: string): JSONContent {+ return mdManager.parse(md);+}++describe('computeMapDrivenBodySplice', () => {+ describe('byte preservation outside the splice', () => {+ test('single-block edit produces splice covering only the edited block', () => {+ const oldBody = '# Heading\n\nFirst paragraph.\n\nSecond paragraph.\n';+ const newBody = '# Heading\n\nFirst paragraph EDITED.\n\nSecond paragraph.\n';++ const splice = computeMapDrivenBodySplice(oldBody, pmFromMd(newBody), mdManager);+ expect(splice).not.toBeNull();+ if (!splice) return;++ const headingEnd = oldBody.indexOf('# Heading') + '# Heading'.length;+ expect(splice.spliceStart).toBeGreaterThanOrEqual(headingEnd);+ const secondParaStart = oldBody.indexOf('Second paragraph');+ expect(splice.spliceEnd).toBeLessThanOrEqual(secondParaStart);++ expect(oldBody.slice(0, splice.spliceStart)).toBe(+ applySplice(oldBody, splice).slice(0, splice.spliceStart),+ );+ const reconstructed = applySplice(oldBody, splice);+ expect(reconstructed.slice(splice.spliceStart + splice.newSlice.length)).toBe(+ oldBody.slice(splice.spliceEnd),+ );+ });++ test('result of applying splice equals the canonical newBody serialization', () => {+ const oldBody = '# Heading\n\nFirst.\n\nSecond.\n';+ const newPm = pmFromMd('# Heading\n\nFirst CHANGED.\n\nSecond.\n');+ const splice = computeMapDrivenBodySplice(oldBody, newPm, mdManager);+ expect(splice).not.toBeNull();+ if (!splice) return;++ const reconstructed = applySplice(oldBody, splice);+ const canonicalNew = mdManager.serialize(newPm);+ const reconstructedMdast = mdManager.parseToMdast(reconstructed);+ const canonicalMdast = mdManager.parseToMdast(canonicalNew);+ expect(reconstructedMdast.children.length).toBe(canonicalMdast.children.length);+ });+ });++ describe('source-form preservation through structural equality', () => {+ test('an untouched block whose canonical form would canonicalize bytes is excluded from splice', () => {+ const oldBody = '*italic one*\n\nuntouched two\n';+ const newPmJson = pmFromMd('*italic one* EDIT\n\nuntouched two\n');+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);+ expect(splice).not.toBeNull();+ if (!splice) return;++ const result = applySplice(oldBody, splice);+ expect(result).toContain('untouched two');+ const oldUntouched = oldBody.slice(oldBody.indexOf('untouched two'));+ const newUntouched = result.slice(result.indexOf('untouched two'));+ expect(newUntouched).toBe(oldUntouched);+ });++ test('block matching the structural shape but canonicalized in newBody is NOT spliced', () => {+ const oldBody = '*italic*\n\nplain\n';+ const newPmJson = pmFromMd('*italic*\n\nplain CHANGED\n');+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);+ expect(splice).not.toBeNull();+ if (!splice) return;++ const result = applySplice(oldBody, splice);+ expect(result.startsWith('*italic*\n\n')).toBe(true);+ });+ });++ describe('insertions and deletions at boundaries', () => {+ test('append a new paragraph at end', () => {+ const oldBody = 'First.\n';+ const newPmJson = pmFromMd('First.\n\nSecond.\n');+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);+ expect(splice).not.toBeNull();+ if (!splice) return;++ const result = applySplice(oldBody, splice);+ expect(result).toContain('First.');+ expect(result).toContain('Second.');+ expect(result.indexOf('First.')).toBe(0);+ });++ test('prepend a new paragraph at start', () => {+ const oldBody = 'Second.\n';+ const newPmJson = pmFromMd('First.\n\nSecond.\n');+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);+ expect(splice).not.toBeNull();+ if (!splice) return;++ const result = applySplice(oldBody, splice);+ expect(result).toContain('First.');+ expect(result).toContain('Second.');+ expect(result.indexOf('First.')).toBeLessThan(result.indexOf('Second.'));+ });++ test('insert a paragraph in the middle preserves surrounding blocks byte-identically', () => {+ const oldBody = '*Pre*\n\nPost.\n';+ const newPmJson = pmFromMd('*Pre*\n\nMiddle.\n\nPost.\n');+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);+ expect(splice).not.toBeNull();+ if (!splice) return;++ const result = applySplice(oldBody, splice);+ expect(result.startsWith('*Pre*')).toBe(true);+ expect(result).toContain('Middle.');+ expect(result.endsWith('Post.\n')).toBe(true);+ });++ test('delete a middle block', () => {+ const oldBody = 'A.\n\nB.\n\nC.\n';+ const newPmJson = pmFromMd('A.\n\nC.\n');+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);+ expect(splice).not.toBeNull();+ if (!splice) return;++ const result = applySplice(oldBody, splice);+ expect(result).toContain('A.');+ expect(result).toContain('C.');+ expect(result).not.toContain('B.');+ });+ });++ describe('synthetic / empty inputs', () => {+ test('empty oldBody + new content produces splice that yields the new content', () => {+ const oldBody = '';+ const newPmJson = pmFromMd('A new paragraph.\n');+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);+ expect(splice).not.toBeNull();+ if (!splice) return;++ const result = applySplice(oldBody, splice);+ expect(result).toContain('A new paragraph.');+ });++ test('no-change input produces no-op splice', () => {+ const oldBody = 'A.\n\nB.\n';+ const newPmJson = pmFromMd(oldBody);+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);+ expect(splice).not.toBeNull();+ if (!splice) return;++ const result = applySplice(oldBody, splice);+ const oldChildren = mdManager.parseToMdast(oldBody).children;+ const resultChildren = mdManager.parseToMdast(result).children;+ expect(resultChildren.length).toBe(oldChildren.length);+ });+ });++ describe('contiguous multi-block edits', () => {+ test('editing two adjacent blocks unions their splice ranges', () => {+ const oldBody = 'first.\n\nsecond.\n\nthird.\n';+ const newPmJson = pmFromMd('first EDITED.\n\nsecond EDITED.\n\nthird.\n');+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);+ expect(splice).not.toBeNull();+ if (!splice) return;++ const result = applySplice(oldBody, splice);+ expect(result.endsWith('third.\n')).toBe(true);++ const thirdStartOld = oldBody.indexOf('third.');+ expect(splice.spliceEnd).toBeLessThanOrEqual(thirdStartOld);+ });++ test('non-contiguous multi-block edits collapse into one over-wide splice (documented AC2 degradation)', () => {+ const oldBody = 'first.\n\nmiddle.\n\nthird.\n';+ const newPmJson = pmFromMd('first EDITED.\n\nmiddle.\n\nthird EDITED.\n');+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);+ expect(splice).not.toBeNull();+ if (!splice) return;++ const middleStart = oldBody.indexOf('middle.');+ expect(splice.spliceStart).toBeLessThanOrEqual(middleStart);+ expect(splice.spliceEnd).toBeGreaterThanOrEqual(middleStart + 'middle.'.length);++ const result = applySplice(oldBody, splice);+ expect(result).toContain('first EDITED.');+ expect(result).toContain('middle.');+ expect(result).toContain('third EDITED.');+ });+ });++ describe('robustness to parse failure', () => {+ test('returns null when serialize throws on schema-rejected JSON', () => {+ const oldBody = 'A.\n';+ const malformed = { type: 'not-a-real-node-type' } as JSONContent;+ const splice = computeMapDrivenBodySplice(oldBody, malformed, mdManager);+ expect(splice).toBeNull();+ });+ });+});diff --git a/packages/server/src/map-driven-splice.ts b/packages/server/src/map-driven-splice.tsnew file mode 100644index 000000000..5c9d0ce41--- /dev/null+++ b/packages/server/src/map-driven-splice.ts@@ -0,0 +1,111 @@+import type { MarkdownManager } from '@inkeep/open-knowledge-core';+import type { JSONContent } from '@tiptap/core';+import type { RootContent } from 'mdast';++export interface MapDrivenSplice {+ readonly spliceStart: number;+ readonly spliceEnd: number;+ readonly newSlice: string;+}++export function computeMapDrivenBodySplice(+ oldBody: string,+ newPmJson: JSONContent,+ mdManager: MarkdownManager,+ onFallback?: (reason: 'parse-error' | 'missing-position') => void,+): MapDrivenSplice | null {+ let oldChildren: readonly RootContent[];+ let newBody: string;+ let newChildren: readonly RootContent[];+ try {+ oldChildren = mdManager.parseToMdast(oldBody).children;+ newBody = mdManager.serialize(newPmJson);+ newChildren = mdManager.parseToMdast(newBody).children;+ } catch {+ onFallback?.('parse-error');+ return null;+ }++ if (!allBlocksCarryPositions(oldChildren) || !allBlocksCarryPositions(newChildren)) {+ onFallback?.('missing-position');+ return null;+ }++ let prefixLen = 0;+ while (+ prefixLen < oldChildren.length &&+ prefixLen < newChildren.length &&+ structurallyEqual(oldChildren[prefixLen], newChildren[prefixLen])+ ) {+ prefixLen++;+ }++ let suffixLen = 0;+ while (+ suffixLen < oldChildren.length - prefixLen &&+ suffixLen < newChildren.length - prefixLen &&+ structurallyEqual(+ oldChildren[oldChildren.length - 1 - suffixLen],+ newChildren[newChildren.length - 1 - suffixLen],+ )+ ) {+ suffixLen++;+ }++ const spliceStart =+ prefixLen > 0+ ? blockEndOffset(oldChildren[prefixLen - 1])+ : oldChildren.length > 0+ ? blockStartOffset(oldChildren[0])+ : 0;+ const spliceEnd =+ suffixLen > 0 ? blockStartOffset(oldChildren[oldChildren.length - suffixLen]) : oldBody.length;++ const newSliceStart =+ prefixLen > 0+ ? blockEndOffset(newChildren[prefixLen - 1])+ : newChildren.length > 0+ ? blockStartOffset(newChildren[0])+ : 0;+ const newSliceEnd =+ suffixLen > 0 ? blockStartOffset(newChildren[newChildren.length - suffixLen]) : newBody.length;++ return {+ spliceStart,+ spliceEnd,+ newSlice: newBody.slice(newSliceStart, newSliceEnd),+ };+}++function allBlocksCarryPositions(children: readonly RootContent[]): boolean {+ for (const child of children) {+ const start = child.position?.start?.offset;+ const end = child.position?.end?.offset;+ if (typeof start !== 'number' || typeof end !== 'number') return false;+ }+ return true;+}++function blockStartOffset(node: RootContent): number {+ const offset = node.position?.start?.offset;+ if (typeof offset !== 'number') {+ throw new Error('mdast node missing position.start.offset');+ }+ return offset;+}++function blockEndOffset(node: RootContent): number {+ const offset = node.position?.end?.offset;+ if (typeof offset !== 'number') {+ throw new Error('mdast node missing position.end.offset');+ }+ return offset;+}++function structurallyEqual(a: RootContent, b: RootContent): boolean {+ return stringifyIgnorePosition(a) === stringifyIgnorePosition(b);+}++function stringifyIgnorePosition(node: unknown): string {+ return JSON.stringify(node, (key, value) => (key === 'position' ? undefined : value));+}diff --git a/packages/server/src/metrics.test.ts b/packages/server/src/metrics.test.tsindex ac6b809c2..4026bcfe5 100644--- a/packages/server/src/metrics.test.ts+++ b/packages/server/src/metrics.test.ts@@ -8,6 +8,8 @@ import {incrementBridgeMergeContentLoss,incrementCollabSocketFilteredError,incrementConflict,+ incrementMapDrivenSpliceApplied,+ incrementMapDrivenSpliceFallback,incrementPark,incrementReconcile,incrementRescueBuffer,@@ -66,6 +68,25 @@ describe('reconciliation metrics', () => {expect(m.parkCount).toBe(2);});+ test('map-driven splice counters: applied increments and fallback is keyed by reason', () => {+ resetMetrics();+ incrementMapDrivenSpliceApplied();+ incrementMapDrivenSpliceApplied();+ incrementMapDrivenSpliceFallback('parse-error');+ incrementMapDrivenSpliceFallback('parse-error');+ incrementMapDrivenSpliceFallback('text-mismatch');+ incrementMapDrivenSpliceFallback('synthetic-doc');+ incrementMapDrivenSpliceFallback('missing-position');+ const m = getMetrics();+ expect(m.mapDrivenSpliceApplied).toBe(2);+ expect(m.mapDrivenSpliceFallback).toEqual({+ 'parse-error': 2,+ 'text-mismatch': 1,+ 'synthetic-doc': 1,+ 'missing-position': 1,+ });+ });+test('getMetrics returns a snapshot (not a reference)', () => {resetMetrics();incrementReconcile();@@ -84,7 +105,11 @@ describe('reconciliation metrics', () => {resetMetrics();const m = getMetrics();for (const [key, value] of Object.entries(m)) {- if (key === 'cc1LastSeq' || key === 'bridgeToleranceApplied') {+ if (+ key === 'cc1LastSeq' ||+ key === 'bridgeToleranceApplied' ||+ key === 'mapDrivenSpliceFallback'+ ) {expect(value).toEqual({});} else {expect(value).toBe(0);diff --git a/packages/server/src/metrics.ts b/packages/server/src/metrics.tsindex 22ecca7e6..200916587 100644--- a/packages/server/src/metrics.ts+++ b/packages/server/src/metrics.ts@@ -1,5 +1,11 @@import type { BridgeToleranceClass, CC1Channel } from '@inkeep/open-knowledge-core';+export type MapDrivenSpliceFallbackReason =+ | 'text-mismatch'+ | 'synthetic-doc'+ | 'parse-error'+ | 'missing-position';+export interface ReconciliationMetrics {reconcileCount: number;conflictCount: number;@@ -159,6 +165,25 @@ export interface ReconciliationMetrics {* Path-B rate via `actual_rate = observerAPathBFires +* observerAPathBFiresSuppressed`. */observerAPathBFiresSuppressed: number;+ /** Count of Observer A drains where the map-driven block-aligned splice+ * computed and applied — the byte-preserving default path that keeps+ * untouched blocks byte-identical in Y.Text. The companion+ * `mapDrivenSpliceFallback` records every drain the splice could NOT+ * serve, keyed by reason; `splice_rate = applied / (applied + Σfallback)`+ * is the health signal for the default fidelity path. */+ mapDrivenSpliceApplied: number;+ /** Companion to `mapDrivenSpliceApplied` — count of Observer A drains+ * that fell back off the map-driven splice, keyed by reason:+ * `text-mismatch` (Y.Text diverged from baseline → Path B; expected+ * under concurrent edits), `synthetic-doc` (system/config docs never+ * use the splice; expected), `parse-error` (parse/serialize threw+ * inside the splice computation — a sustained non-zero here means a+ * serializer/parser regression is silently routing every edit through+ * the lossier incremental-diff fallback), and `missing-position` (a+ * top-level mdast block lacked byte offsets). Keys are a closed+ * 4-value union, mirroring `bridgeToleranceApplied`'s compile-time+ * bounded-cardinality guard. */+ mapDrivenSpliceFallback: Partial<Record<MapDrivenSpliceFallbackReason, number>>;/** Y.Text-is-truth contract — count of Observer A in-sync canonical-base* residual merges: the byte-preserving slow path on docs whose settled* bytes sit beyond `normalizeBridge` tolerance of the canonical witness.@@ -347,6 +372,8 @@ const counters: ReconciliationMetrics = {bridgeToleranceApplied: {},observerAPathBFires: 0,observerAPathBFiresSuppressed: 0,+ mapDrivenSpliceApplied: 0,+ mapDrivenSpliceFallback: {},observerAResidualMergeRuns: 0,bridgeSplitBrainRederives: 0,bridgeSplitBrainRederivesSuppressed: 0,@@ -479,6 +506,14 @@ export function incrementObserverAPathBFiresSuppressed(): void {counters.observerAPathBFiresSuppressed++;}+export function incrementMapDrivenSpliceApplied(): void {+ counters.mapDrivenSpliceApplied++;+}++export function incrementMapDrivenSpliceFallback(reason: MapDrivenSpliceFallbackReason): void {+ counters.mapDrivenSpliceFallback[reason] = (counters.mapDrivenSpliceFallback[reason] ?? 0) + 1;+}+export function incrementObserverAResidualMergeRuns(): void {counters.observerAResidualMergeRuns++;}@@ -577,6 +612,7 @@ export function getMetrics(): ReconciliationMetrics {...counters,cc1LastSeq: { ...counters.cc1LastSeq },bridgeToleranceApplied: { ...counters.bridgeToleranceApplied },+ mapDrivenSpliceFallback: { ...counters.mapDrivenSpliceFallback },};}@@ -618,6 +654,8 @@ export function resetMetrics(): void {counters.bridgeToleranceApplied = {};counters.observerAPathBFires = 0;counters.observerAPathBFiresSuppressed = 0;+ counters.mapDrivenSpliceApplied = 0;+ counters.mapDrivenSpliceFallback = {};counters.observerAResidualMergeRuns = 0;counters.bridgeSplitBrainRederives = 0;counters.bridgeSplitBrainRederivesSuppressed = 0;diff --git a/packages/server/src/path-conditional-map-driven-origin.test.ts b/packages/server/src/path-conditional-map-driven-origin.test.tsnew file mode 100644index 000000000..cdc11c727--- /dev/null+++ b/packages/server/src/path-conditional-map-driven-origin.test.ts@@ -0,0 +1,45 @@+import { describe, expect, test } from 'bun:test';+import { spawnSync } from 'node:child_process';+import { join } from 'node:path';++const REPO_ROOT = join(__dirname, '..', '..', '..');+const FIXTURE_REL = 'biome-plugins/__fixtures__/path-conditional-map-driven-origin.fixture.tsx';++describe('path-conditional-map-driven-origin GritQL plugin', () => {+ test('fires on exactly 7 positive cases (and on no negative case)', () => {+ const result = spawnSync('bunx', ['biome', 'check', FIXTURE_REL], {+ cwd: REPO_ROOT,+ encoding: 'utf-8',+ });+ expect(result.status).not.toBe(0);+ const output = `${result.stdout}\n${result.stderr}`;+ const fires = (output.match(/Observer-side transact call missing sanctioned origin/g) ?? [])+ .length;+ expect(fires).toBe(7);+ expect(output).toContain('Pass `OBSERVER_SYNC_ORIGIN` as the second argument');+ expect(output).toMatch(/https?:\/\/[^\s]+/);+ expect(output).toContain('biome-plugins/README.md#path-conditional-map-driven-origingrit');+ });++ test('plugin is registered in biome.jsonc via overrides (not root plugins)', () => {+ const config = require(join(REPO_ROOT, 'biome.jsonc')) as {+ plugins?: string[];+ overrides?: Array<{ includes?: string[]; plugins?: string[] }>;+ };++ const rootPlugins = config.plugins ?? [];+ expect(rootPlugins).not.toContain('./biome-plugins/path-conditional-map-driven-origin.grit');++ const overrides = config.overrides ?? [];+ const matchingOverride = overrides.find((entry) =>+ (entry.plugins ?? []).includes('./biome-plugins/path-conditional-map-driven-origin.grit'),+ );+ expect(matchingOverride).toBeDefined();++ const includes = matchingOverride?.includes ?? [];+ expect(includes).toContain('packages/server/src/server-observers.ts');+ expect(includes).toContain(+ 'biome-plugins/__fixtures__/path-conditional-map-driven-origin.fixture.tsx',+ );+ });+});diff --git a/packages/server/src/prd-6654-e2e-repro.test.ts b/packages/server/src/prd-6654-e2e-repro.test.tsnew file mode 100644index 000000000..417339fee--- /dev/null+++ b/packages/server/src/prd-6654-e2e-repro.test.ts@@ -0,0 +1,215 @@+import { describe, expect, test } from 'bun:test';+import { mkdirSync, mkdtempSync, rmSync } from 'node:fs';+import type { IncomingMessage, ServerResponse } from 'node:http';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import { Readable } from 'node:stream';+import { Hocuspocus } from '@hocuspocus/server';+import { MarkdownManager, sharedExtensions } from '@inkeep/open-knowledge-core';+import { getSchema } from '@tiptap/core';+import * as Y from 'yjs';+import { AGENT_WRITE_ORIGIN, AgentSessionManager } from './agent-sessions.ts';+import { createApiExtension } from './api-extension.ts';+import { composeAndWriteRawBody } from './bridge-intake.ts';+import { createServerObserverExtension } from './server-observer-extension.ts';++const mdManager = new MarkdownManager({ extensions: sharedExtensions });+const schema = getSchema(sharedExtensions);++interface CapturedResponse {+ status: number;+ body: string;+}++function makeJsonPostReq(body: unknown): IncomingMessage {+ const readable = Readable.from(Buffer.from(JSON.stringify(body))) as unknown as IncomingMessage;+ readable.method = 'POST';+ readable.url = '/api/agent-patch';+ readable.headers = {+ host: 'localhost',+ 'content-type': 'application/json',+ };+ return readable;+}++function makeRes(): { res: ServerResponse; captured: CapturedResponse } {+ const captured: CapturedResponse = { status: 0, body: '' };+ const res = {+ writeHead(status: number) {+ captured.status = status;+ },+ end(body?: string) {+ captured.body = body ?? '';+ },+ } as unknown as ServerResponse;+ return { res, captured };+}++async function callAgentPatch(+ hocuspocus: Hocuspocus,+ sessionManager: AgentSessionManager,+ contentDir: string,+ body: unknown,+): Promise<CapturedResponse> {+ const ext = createApiExtension({+ hocuspocus,+ sessionManager,+ contentDir,+ serverInstanceId: 'prd-6654-e2e-test-instance',+ getFileIndex: () => new Map(),+ });+ const req = makeJsonPostReq(body);+ const { res, captured } = makeRes();+ await (+ ext as {+ onRequest: (ctx: { request: IncomingMessage; response: ServerResponse }) => Promise<void>;+ }+ ).onRequest({ request: req, response: res });+ return captured;+}++function setup() {+ const projectDir = mkdtempSync(join(tmpdir(), 'ok-prd-6654-e2e-'));+ const contentDir = join(projectDir, 'content');+ mkdirSync(contentDir, { recursive: true });+ const hocuspocus = new Hocuspocus({+ quiet: true,+ extensions: [createServerObserverExtension({ mdManager, schema })],+ });+ const sessionManager = new AgentSessionManager(hocuspocus);+ return {+ projectDir,+ contentDir,+ hocuspocus,+ sessionManager,+ cleanup: async () => {+ await sessionManager.closeAll();+ rmSync(projectDir, { recursive: true, force: true });+ },+ };+}++const USER_TYPING_ORIGIN = {+ source: 'local' as const,+ context: { origin: 'user-typing' },+};++const SEED_ORIGIN = {+ source: 'local' as const,+ skipStoreHooks: false,+ context: { origin: 'test-seed', paired: true as const },+};++describe('PRD-6654 — full end-to-end gate (handleAgentPatch + WYSIWYG touch)', () => {+ test('row-without-trailing-pipe shape: the second find succeeds (200, no "Text not found")', async () => {+ const env = setup();+ try {+ const session = await env.sessionManager.getSession('test-doc');+ const ytext = session.dc.document.getText('source');+ const xmlFragment = session.dc.document.getXmlFragment('default');++ session.dc.document.transact(() => {+ ytext.delete(0, ytext.length);+ ytext.insert(0, '# Notes\n\nplaceholder\n');+ }, SEED_ORIGIN);++ const r0 = await callAgentPatch(env.hocuspocus, env.sessionManager, env.contentDir, {+ docName: 'test-doc',+ find: 'placeholder',+ replace: '| a | b |\n| - | - |\n| 1 | 2',+ });+ expect(r0.status).toBe(200);+ expect(ytext.toString().includes('| 1 | 2\n')).toBe(true);++ session.dc.document.transact(() => {+ const para = new Y.XmlElement('paragraph');+ para.insert(0, [new Y.XmlText('hi')]);+ xmlFragment.insert(xmlFragment.length, [para]);+ }, USER_TYPING_ORIGIN);++ const r2 = await callAgentPatch(env.hocuspocus, env.sessionManager, env.contentDir, {+ docName: 'test-doc',+ find: '| 1 | 2\n',+ replace: '| 1 | 2\n| 3 | 4\n',+ });+ expect(r2.status).toBe(200);+ expect(ytext.toString().includes('| 3 | 4')).toBe(true);+ } finally {+ await env.cleanup();+ }+ });++ test('leading-blank-line shape: the second find succeeds (200)', async () => {+ const env = setup();+ try {+ const session = await env.sessionManager.getSession('test-doc');+ const ytext = session.dc.document.getText('source');+ const xmlFragment = session.dc.document.getXmlFragment('default');++ session.dc.document.transact(() => {+ composeAndWriteRawBody(session.dc.document, '\n\nhello\n', 'agent');+ }, AGENT_WRITE_ORIGIN);++ session.dc.document.transact(() => {+ const para = new Y.XmlElement('paragraph');+ para.insert(0, [new Y.XmlText('z')]);+ xmlFragment.insert(xmlFragment.length, [para]);+ }, USER_TYPING_ORIGIN);++ const r2 = await callAgentPatch(env.hocuspocus, env.sessionManager, env.contentDir, {+ docName: 'test-doc',+ find: '\n\nhello',+ replace: '\n\nGOODBYE',+ });+ expect(r2.status).toBe(200);+ expect(ytext.toString().includes('GOODBYE')).toBe(true);+ } finally {+ await env.cleanup();+ }+ });++ test('reporter scenario: 12 rapid sequential edits with interleaved WYSIWYG noise all succeed', async () => {+ const env = setup();+ try {+ const session = await env.sessionManager.getSession('test-doc');+ const ytext = session.dc.document.getText('source');+ const xmlFragment = session.dc.document.getXmlFragment('default');++ const seed = '# Trace\n\n| step | note |\n| ---- | ---- |\n| 0 | start |\n';+ session.dc.document.transact(() => {+ ytext.delete(0, ytext.length);+ ytext.insert(0, seed);+ }, SEED_ORIGIN);++ let prevRow = '| 0 | start |';+ let successes = 0;+ const failures: string[] = [];+ for (let i = 1; i <= 12; i++) {+ const newRow = `| ${i} | iter${i} |`;+ const r = await callAgentPatch(env.hocuspocus, env.sessionManager, env.contentDir, {+ docName: 'test-doc',+ find: prevRow,+ replace: `${prevRow}\n${newRow}`,+ });+ if (r.status === 200) {+ successes++;+ prevRow = newRow;+ } else {+ failures.push(`iter ${i}: status=${r.status} body=${r.body.slice(0, 120)}`);+ }+ if (i % 2 === 0) {+ session.dc.document.transact(() => {+ const para = new Y.XmlElement('paragraph');+ para.insert(0, [new Y.XmlText(`n${i}`)]);+ xmlFragment.insert(xmlFragment.length, [para]);+ }, USER_TYPING_ORIGIN);+ }+ }++ expect(failures).toEqual([]);+ expect(successes).toBe(12);+ } finally {+ await env.cleanup();+ }+ });+});diff --git a/packages/server/src/prd-6654-multi-client-repro.test.ts b/packages/server/src/prd-6654-multi-client-repro.test.tsnew file mode 100644index 000000000..844b379f9--- /dev/null+++ b/packages/server/src/prd-6654-multi-client-repro.test.ts@@ -0,0 +1,220 @@+import { describe, expect, test } from 'bun:test';+import { MarkdownManager, sharedExtensions } from '@inkeep/open-knowledge-core';+import { getSchema } from '@tiptap/core';+import { updateYFragment } from '@tiptap/y-tiptap';+import * as Y from 'yjs';+import { AGENT_WRITE_ORIGIN } from './agent-sessions.ts';+import { composeAndWriteRawBody } from './bridge-intake.ts';+import { setupServerObservers } from './server-observers.ts';++const mdManager = new MarkdownManager({ extensions: sharedExtensions });+const schema = getSchema(sharedExtensions);++const USER_TYPING_ORIGIN = {+ source: 'connection' as const,+ context: { origin: 'user-typing' },+};++interface AgentThenWysiwygResult {+ readonly afterAgent: string;+ readonly afterWysiwyg: string;+}++function agentWriteThenWysiwygTouch(raw: string): AgentThenWysiwygResult {+ const doc = new Y.Doc();+ const xmlFragment = doc.getXmlFragment('default');+ const ytext = doc.getText('source');+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });++ doc.transact(() => {+ composeAndWriteRawBody(doc, raw, 'agent');+ }, AGENT_WRITE_ORIGIN);+ const afterAgent = ytext.toString();++ doc.transact(() => {+ const para = new Y.XmlElement('paragraph');+ para.insert(0, [new Y.XmlText('x')]);+ xmlFragment.insert(xmlFragment.length, [para]);+ }, USER_TYPING_ORIGIN);+ const afterWysiwyg = ytext.toString();++ cleanup();+ return { afterAgent, afterWysiwyg };+}++describe('PRD-6654 — agent bytes survive a concurrent non-paired WYSIWYG edit', () => {+ describe('canonical-form controls (serializer retention; pass on either Path-A variant)', () => {+ test('under-padded table row in a padded table round-trips byte-exact through serialize(parse)', () => {+ const rawTable = [+ '# Trace',+ '',+ '| Step | Time | Note | Status |',+ '| ---- | ---- | ----------------------------- | --------- |',+ '| 5 | T+5s | Your browser edits survived! | CONFIRMED |',+ '',+ ].join('\n');+ expect(mdManager.serialize(mdManager.parse(rawTable))).toBe(rawTable);+ });++ test('narrow row appended to a wide-padded table round-trips byte-exact through serialize(parse)', () => {+ const composed =+ '| step | description | status |\n' ++ '| ---- | ------------------------------------------ | ------ |\n' ++ '| 1 | This is a really really long row of stuff | DONE |\n' ++ '| 3 | short row | OK |\n';+ expect(mdManager.serialize(mdManager.parse(composed))).toBe(composed);+ });++ test('agent table row with under-padded columns stays verbatim after a WYSIWYG touch', () => {+ const agentRow = '| 5 | T+5s | Your browser edits survived! | CONFIRMED |';+ const rawTable = [+ '# Trace',+ '',+ '| Step | Time | Note | Status |',+ '| ---- | ---- | ----------------------------- | --------- |',+ agentRow,+ '',+ ].join('\n');++ const { afterAgent, afterWysiwyg } = agentWriteThenWysiwygTouch(rawTable);+ expect(afterAgent).toBe(rawTable);+ expect(afterWysiwyg.includes(agentRow)).toBe(true);+ });++ test('end-to-end find flow: indexOf of the agent-composed row succeeds after a WYSIWYG touch', () => {+ const findTarget = '| 5 | T+5s | Your browser edits survived! | CONFIRMED |';+ const initial = [+ '# Trace',+ '',+ '| Step | Time | Note | Status |',+ '| ---- | ---- | ----------------------------- | --------- |',+ findTarget,+ '',+ ].join('\n');++ const { afterWysiwyg } = agentWriteThenWysiwygTouch(initial);+ expect(afterWysiwyg.indexOf(findTarget)).not.toBe(-1);+ });+ });++ describe('splice discriminators (require the map-driven Path A; fail under whole-body line diff)', () => {+ test('table row WITHOUT trailing pipe stays verbatim when the WYSIWYG edit is in another block', () => {+ const raw = '# Notes\n\n| a | b |\n| - | - |\n| 1 | 2\n';+ const { afterAgent, afterWysiwyg } = agentWriteThenWysiwygTouch(raw);++ expect(afterAgent).toBe(raw);+ expect(mdManager.serialize(mdManager.parse(raw))).not.toBe(raw);+ expect(afterWysiwyg.indexOf('| 1 | 2\n')).not.toBe(-1);+ });++ test('multi-blank-line block separator stays verbatim when the WYSIWYG edit is in another block', () => {+ const raw = '# H\n\na\n\n\n\nb\n';+ const { afterAgent, afterWysiwyg } = agentWriteThenWysiwygTouch(raw);++ expect(afterAgent).toBe(raw);+ expect(afterWysiwyg.indexOf('a\n\n\n\nb')).not.toBe(-1);+ });++ test('leading blank lines stay verbatim when the WYSIWYG edit is in another block', () => {+ const raw = '\n\n## Section\n\nLorem ipsum.\n';+ const { afterAgent, afterWysiwyg } = agentWriteThenWysiwygTouch(raw);++ expect(afterAgent).toBe(raw);+ expect(afterWysiwyg.startsWith('\n\n## Section')).toBe(true);+ });+ });++ describe('row-no-trailing-pipe tolerance (Observer A already-in-sync gate + watchdog)', () => {+ const UNIFORM = '# Notes\n\n| a | b\n| - | -\n| 1 | 2\n';++ function stripTableCaptureAttrs(json: unknown): void {+ if (!json || typeof json !== 'object') return;+ const node = json as {+ type?: string;+ attrs?: Record<string, unknown>;+ content?: unknown[];+ };+ if (node.type === 'table' && node.attrs) {+ node.attrs.sourceOuterPipes = null;+ }+ for (const child of node.content ?? []) stripTableCaptureAttrs(child);+ }++ function rebuildFragmentFrom(+ doc: Y.Doc,+ xmlFragment: Y.XmlFragment,+ json: Record<string, unknown>,+ ): void {+ const pmNode = schema.nodeFromJSON(json);+ doc.transact(() => {+ const meta = { mapping: new Map(), isOMark: new Map() };+ updateYFragment(doc, xmlFragment, pmNode, meta);+ }, USER_TYPING_ORIGIN);+ }++ test('attr-only fragment churn (capture attrs lost, zero content change) leaves Y.Text byte-identical', () => {+ const doc = new Y.Doc();+ const xmlFragment = doc.getXmlFragment('default');+ const ytext = doc.getText('source');+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });++ doc.transact(() => {+ composeAndWriteRawBody(doc, UNIFORM, 'agent');+ }, AGENT_WRITE_ORIGIN);+ expect(ytext.toString()).toBe(UNIFORM);++ const stripped = mdManager.parse(UNIFORM) as unknown as Record<string, unknown>;+ stripTableCaptureAttrs(stripped);+ rebuildFragmentFrom(doc, xmlFragment, stripped);++ expect(ytext.toString()).toBe(UNIFORM);+ expect(ytext.toString().indexOf('| 1 | 2\n')).not.toBe(-1);++ cleanup();+ });++ test('source-mode typed no-trailing-pipe table settles without a bridge-invariant violation', () => {+ const doc = new Y.Doc();+ const xmlFragment = doc.getXmlFragment('default');+ const ytext = doc.getText('source');+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });++ const raw = '# Notes\n\n| a | b |\n| - | - |\n| 1 | 2\n';+ doc.transact(+ () => {+ ytext.insert(0, raw);+ },+ {+ source: 'connection' as const,+ context: { origin: 'source-typing' },+ },+ );++ expect(ytext.toString()).toBe(raw);++ cleanup();+ });++ test('genuine cell edit through the same attr-churn path still syncs (not absorbed by the tolerance)', () => {+ const doc = new Y.Doc();+ const xmlFragment = doc.getXmlFragment('default');+ const ytext = doc.getText('source');+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });++ doc.transact(() => {+ composeAndWriteRawBody(doc, UNIFORM, 'agent');+ }, AGENT_WRITE_ORIGIN);++ const edited = mdManager.parse(UNIFORM.replace('| 1 | 2', '| 1 | 99')) as unknown as Record<+ string,+ unknown+ >;+ stripTableCaptureAttrs(edited);+ rebuildFragmentFrom(doc, xmlFragment, edited);++ expect(ytext.toString()).toContain('99');++ cleanup();+ });+ });+});diff --git a/packages/server/src/server-observers.test.ts b/packages/server/src/server-observers.test.tsindex f206b1447..8dabe5377 100644--- a/packages/server/src/server-observers.test.ts+++ b/packages/server/src/server-observers.test.ts@@ -890,7 +890,7 @@ describe('Server Observer B — error recovery paths', () => {resetMetrics();const ngRaw =- '---\ntitle: NG recovery fixture\n---\n\n# Hello\n\nInline math: $a + b$ stays.\n\nBody text stays.\n';+ '---\ntitle: NG recovery fixture\n---\n\n# Hello\n\n> Lazy quote\nstays lazy.\n\nBody text stays.\n';const doc = new Y.Doc();const xmlFragment = doc.getXmlFragment('default');const ytext = doc.getText('source');@@ -1039,7 +1039,7 @@ describe('Server Observer B — Y.Text-is-truth contract (FR-31)', () => {});describe('Observer A routing — Path B fires iff Y.Text holds unabsorbed changes (FR-3)', () => {- const RESIDUAL_RAW = '---\ntitle: Routing fixture\n---\n\n# Hello\n\nBody text stays.\n';+ const RESIDUAL_RAW = '---\ntitle: Routing fixture\n---\n\n# Hello \n\nBody text stays.\n';function canonicalOf(raw: string): string {const { frontmatter, body } = stripFrontmatter(raw);@@ -1283,7 +1283,7 @@ describe('Observer A routing — Path B fires iff Y.Text holds unabsorbed change});const NG_RAW =- '---\ntitle: NG routing fixture\n---\n\n# Hello\n\nInline math: $a + b$ stays.\n\nBody text stays.\n';+ '---\ntitle: NG routing fixture\n---\n\n# Hello\n\n> Lazy quote\nstays lazy.\n\nBody text stays.\n';test('in-sync doc with beyond-tolerance residual: fragment change preserves NG bytes without a Path B fire', () => {__resetBridgeWatchdogForTests();@@ -1310,8 +1310,8 @@ describe('Observer A routing — Path B fires iff Y.Text holds unabsorbed changeconst finalText = ytext.toString();expect(finalText).toContain('User WYSIWYG edit.');- expect(finalText).toContain('Inline math: $a + b$ stays.');- expect(finalText).not.toContain('$$a + b$$');+ expect(finalText).toContain('> Lazy quote\nstays lazy.');+ expect(finalText).not.toContain('> stays lazy.');expect(finalText).toContain('Body text stays.');cleanup();@@ -1364,8 +1364,8 @@ describe('Observer A routing — Path B fires iff Y.Text holds unabsorbed change});expect(events1).toHaveLength(0);expect(getMetrics().observerAResidualMergeRuns).toBe(1);- expect(ytext.toString()).toContain('Inline math: $a + b$ stays.');- expect(ytext.toString()).not.toContain('$$a + b$$');+ expect(ytext.toString()).toContain('> Lazy quote\nstays lazy.');+ expect(ytext.toString()).not.toContain('> stays lazy.');const events2 = capturePathBEvents(() => {populateFragment(doc, xmlFragment, `${serializeFragmentBody(xmlFragment)}\nSecond edit.\n`);@@ -1377,8 +1377,8 @@ describe('Observer A routing — Path B fires iff Y.Text holds unabsorbed changeconst finalText = ytext.toString();expect(finalText).toContain('First edit.');expect(finalText).toContain('Second edit.');- expect(finalText).toContain('Inline math: $a + b$ stays.');- expect(finalText).not.toContain('$$a + b$$');+ expect(finalText).toContain('> Lazy quote\nstays lazy.');+ expect(finalText).not.toContain('> stays lazy.');cleanup();});@@ -1394,7 +1394,7 @@ describe('Observer A routing — Path B fires iff Y.Text holds unabsorbed changeexpect(ytext.toString()).toBe(NG_RAW);const pairedRaw =- '---\ntitle: NG routing fixture\n---\n\n# Hello\n\nInline math: $a + b$ stays.\n\nPaired body.\n';+ '---\ntitle: NG routing fixture\n---\n\n# Hello\n\n> Lazy quote\nstays lazy.\n\nPaired body.\n';expect(normalizeBridge(canonicalOf(pairedRaw))).not.toBe(normalizeBridge(pairedRaw));const pairedJson = mdManager.parse(stripFrontmatter(pairedRaw).body);const pairedNode = schema.nodeFromJSON(pairedJson);diff --git a/packages/server/src/server-observers.ts b/packages/server/src/server-observers.tsindex 623981441..043076f7a 100644--- a/packages/server/src/server-observers.ts+++ b/packages/server/src/server-observers.ts@@ -20,11 +20,15 @@ import {emitBridgeSplitBrainRederive,emitObserverAPathBFired,} from './bridge-watchdog.ts';+import { isConfigDoc, isSystemDoc } from './cc1-broadcast.ts';import { recordFrontmatterEditSurface } from './frontmatter-telemetry.ts';+import { computeMapDrivenBodySplice } from './map-driven-splice.ts';import {incrementBridgeMergeCheckpointCreated,incrementBridgeMergeContentLoss,incrementBridgeSplitBrainRederives,+ incrementMapDrivenSpliceApplied,+ incrementMapDrivenSpliceFallback,incrementObserverAPathBFires,incrementObserverAResidualMergeRuns,incrementServerObserverError,@@ -56,6 +60,56 @@ export function shouldRethrowBridgeMergeLoss(env: NodeJS.ProcessEnv = process.enreturn env.NODE_ENV === 'test' || env.OK_RETHROW_BRIDGE_LOSS === '1';}+interface YTextMapDrivenSplice {+ readonly spliceStart: number;+ readonly spliceEnd: number;+ readonly newSlice: string;+}++interface TryComputeMapDrivenSpliceArgs {+ readonly currentText: string;+ readonly lastSyncedXmlMd: string;+ readonly json: unknown;+ readonly mdManager: MarkdownManager;+ readonly docName: string | undefined;+}++function tryComputeMapDrivenSplice(+ args: TryComputeMapDrivenSpliceArgs,+): YTextMapDrivenSplice | null {+ const { currentText, lastSyncedXmlMd, json, mdManager, docName } = args;+ if (currentText !== lastSyncedXmlMd) {+ incrementMapDrivenSpliceFallback('text-mismatch');+ return null;+ }+ if (docName !== undefined && (isSystemDoc(docName) || isConfigDoc(docName))) {+ incrementMapDrivenSpliceFallback('synthetic-doc');+ return null;+ }++ const { body: oldBody } = stripFrontmatter(currentText);+ const bodyOffset = currentText.length - oldBody.length;+ const splice = computeMapDrivenBodySplice(+ oldBody,+ json as Parameters<typeof computeMapDrivenBodySplice>[1],+ mdManager,+ incrementMapDrivenSpliceFallback,+ );+ if (!splice) return null;++ return {+ spliceStart: bodyOffset + splice.spliceStart,+ spliceEnd: bodyOffset + splice.spliceEnd,+ newSlice: splice.newSlice,+ };+}++function applyMapDrivenSplice(ytext: Y.Text, splice: YTextMapDrivenSplice): void {+ const deleteLength = splice.spliceEnd - splice.spliceStart;+ if (deleteLength > 0) ytext.delete(splice.spliceStart, deleteLength);+ if (splice.newSlice.length > 0) ytext.insert(splice.spliceStart, splice.newSlice);+}+type ShadowAccessor = () => ShadowHandle | undefined;type BranchAccessor = () => string;@@ -206,7 +260,8 @@ export function setupServerObservers(opts: SetupServerObserversOpts): () => voidconst ytextCanonicalBody = mdManager.serialize(mdManager.parseWithFallback(stripFrontmatter(ytext.toString()).body, initParseOpts),);- if (ytextCanonicalBody === initialBody) {+ const stripDocBoundary = (b: string): string => b.replace(/^\n+/, '').replace(/\n+$/, '');+ if (stripDocBoundary(ytextCanonicalBody) === stripDocBoundary(initialBody)) {recordSettledBaselines(canonicalInit);} else {recordDivergedAttachBaselines(canonicalInit);@@ -261,8 +316,25 @@ export function setupServerObservers(opts: SetupServerObserversOpts): () => void: 'path-b',});const pathBState: { mergedText: string | null } = { mergedText: null };++ const mapDrivenSplice =+ ytextInSync && residualMergeEligible+ ? null+ : tryComputeMapDrivenSplice({+ currentText,+ lastSyncedXmlMd: lastSyncedYTextBytes,+ json,+ mdManager,+ docName: opts.docName,+ });+ if (mapDrivenSplice) {+ setActiveSpanAttributes({ 'observer.a.path': 'map-driven-splice' });+ }+doc.transact(() => {- if (ytextInSync && !residualMergeEligible) {+ if (mapDrivenSplice) {+ applyMapDrivenSplice(ytext, mapDrivenSplice);+ } else if (ytextInSync && !residualMergeEligible) {applyIncrementalDiff(ytext, currentText, md);} else {const mergeBase = ytextInSync ? lastSyncedCanonicalMd : preMergeBaseline;@@ -280,6 +352,8 @@ export function setupServerObservers(opts: SetupServerObserversOpts): () => void}}, OBSERVER_SYNC_ORIGIN);+ if (mapDrivenSplice) incrementMapDrivenSpliceApplied();+if (pathBState.mergedText !== null && !ytextInSync) {if (emitObserverAPathBFired(opts.docName)) {incrementObserverAPathBFires();diff --git a/packages/server/src/tolerance-telemetry-writer.test.ts b/packages/server/src/tolerance-telemetry-writer.test.tsnew file mode 100644index 000000000..0c878ffa1--- /dev/null+++ b/packages/server/src/tolerance-telemetry-writer.test.ts@@ -0,0 +1,129 @@+import { afterEach, beforeEach, describe, expect, test } from 'bun:test';+import { randomUUID } from 'node:crypto';+import { existsSync, mkdirSync, readFileSync, rmSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { resolve } from 'node:path';++import { emitToleranceFire } from '@inkeep/open-knowledge-core';++import {+ initToleranceTelemetryWriter,+ isToleranceTelemetryEnabled,+ teardownToleranceTelemetryWriter,+} from './tolerance-telemetry-writer.ts';++let tmpProjectDir: string;++beforeEach(() => {+ tmpProjectDir = resolve(tmpdir(), `ok-telemetry-test-${randomUUID()}`);+ mkdirSync(tmpProjectDir, { recursive: true });+ delete process.env.OK_BRIDGE_TOLERANCE_TELEMETRY;+});++afterEach(() => {+ teardownToleranceTelemetryWriter();+ try {+ rmSync(tmpProjectDir, { recursive: true, force: true });+ } catch {}+});++describe('isToleranceTelemetryEnabled', () => {+ test('returns false by default', () => {+ expect(isToleranceTelemetryEnabled({})).toBe(false);+ });++ test('returns true when OK_BRIDGE_TOLERANCE_TELEMETRY=1', () => {+ expect(isToleranceTelemetryEnabled({ OK_BRIDGE_TOLERANCE_TELEMETRY: '1' })).toBe(true);+ });++ test('returns false for other values', () => {+ expect(isToleranceTelemetryEnabled({ OK_BRIDGE_TOLERANCE_TELEMETRY: '0' })).toBe(false);+ expect(isToleranceTelemetryEnabled({ OK_BRIDGE_TOLERANCE_TELEMETRY: 'true' })).toBe(false);+ });+});++describe('initToleranceTelemetryWriter', () => {+ test('does nothing when flag is OFF', () => {+ process.env.OK_BRIDGE_TOLERANCE_TELEMETRY = '0';+ initToleranceTelemetryWriter(tmpProjectDir);++ emitToleranceFire(['crlf'], 'a\r\n', 'a\n', 'test-doc');++ const logPath = resolve(tmpProjectDir, '.ok', 'local', 'tolerance-telemetry.jsonl');+ expect(existsSync(logPath)).toBe(false);+ });++ test('writes JSONL when flag is ON', () => {+ process.env.OK_BRIDGE_TOLERANCE_TELEMETRY = '1';+ initToleranceTelemetryWriter(tmpProjectDir);++ emitToleranceFire(['crlf'], 'a\r\n', 'a\n', 'notes/meeting');++ const logPath = resolve(tmpProjectDir, '.ok', 'local', 'tolerance-telemetry.jsonl');+ expect(existsSync(logPath)).toBe(true);++ const content = readFileSync(logPath, 'utf-8');+ const lines = content.trim().split('\n');+ expect(lines).toHaveLength(1);++ const record = JSON.parse(lines[0] ?? '');+ expect(record.event).toBe('bridge-tolerance-fire');+ expect(record.class).toBe('crlf');+ expect(record.document).toBe('notes/meeting');+ expect(record.codeUnitPosition).toBe(1);+ expect(record.severity).toBe('parser-caused');+ expect(record.timestamp).toMatch(/^\d{4}-\d{2}-\d{2}T/);+ });++ test('appends multiple fires to same file', () => {+ process.env.OK_BRIDGE_TOLERANCE_TELEMETRY = '1';+ initToleranceTelemetryWriter(tmpProjectDir);++ emitToleranceFire(['crlf'], 'a\r\n', 'a\n', 'doc-1');+ emitToleranceFire(['bom'], 'hello', 'hello', 'doc-2');++ const logPath = resolve(tmpProjectDir, '.ok', 'local', 'tolerance-telemetry.jsonl');+ const lines = readFileSync(logPath, 'utf-8').trim().split('\n');+ expect(lines).toHaveLength(2);+ expect(JSON.parse(lines[0] ?? '').class).toBe('crlf');+ expect(JSON.parse(lines[1] ?? '').class).toBe('bom');+ });++ test('multi-class fire produces one line per class', () => {+ process.env.OK_BRIDGE_TOLERANCE_TELEMETRY = '1';+ initToleranceTelemetryWriter(tmpProjectDir);++ emitToleranceFire(['bom', 'crlf', 'trailing-whitespace'], 'a \r\n', 'a\n', 'doc');++ const logPath = resolve(tmpProjectDir, '.ok', 'local', 'tolerance-telemetry.jsonl');+ const lines = readFileSync(logPath, 'utf-8').trim().split('\n');+ expect(lines).toHaveLength(3);++ const classes = lines.map((l) => JSON.parse(l).class);+ expect(classes).toEqual(['bom', 'crlf', 'trailing-whitespace']);+ });++ test('null document name serializes as null', () => {+ process.env.OK_BRIDGE_TOLERANCE_TELEMETRY = '1';+ initToleranceTelemetryWriter(tmpProjectDir);++ emitToleranceFire(['trailing-newline'], 'hello\n', 'hello', undefined);++ const logPath = resolve(tmpProjectDir, '.ok', 'local', 'tolerance-telemetry.jsonl');+ const record = JSON.parse(readFileSync(logPath, 'utf-8').trim());+ expect(record.document).toBeNull();+ });++ test('teardown stops further writes', () => {+ process.env.OK_BRIDGE_TOLERANCE_TELEMETRY = '1';+ initToleranceTelemetryWriter(tmpProjectDir);++ emitToleranceFire(['crlf'], 'a\r', 'a', 'doc');+ teardownToleranceTelemetryWriter();+ emitToleranceFire(['crlf'], 'b\r', 'b', 'doc');++ const logPath = resolve(tmpProjectDir, '.ok', 'local', 'tolerance-telemetry.jsonl');+ const lines = readFileSync(logPath, 'utf-8').trim().split('\n');+ expect(lines).toHaveLength(1);+ });+});diff --git a/packages/server/src/tolerance-telemetry-writer.ts b/packages/server/src/tolerance-telemetry-writer.tsnew file mode 100644index 000000000..69f137759--- /dev/null+++ b/packages/server/src/tolerance-telemetry-writer.ts@@ -0,0 +1,40 @@+import { appendFileSync, mkdirSync } from 'node:fs';+import { resolve } from 'node:path';++import { setToleranceTelemetryHook, type ToleranceFireRecord } from '@inkeep/open-knowledge-core';++import { getLocalDir } from './config/paths.ts';++const TELEMETRY_FILENAME = 'tolerance-telemetry.jsonl';++export function isToleranceTelemetryEnabled(+ env: Record<string, string | undefined> = process.env,+): boolean {+ return env.OK_BRIDGE_TOLERANCE_TELEMETRY === '1';+}++export function initToleranceTelemetryWriter(projectDir: string): void {+ if (!isToleranceTelemetryEnabled()) return;++ const localDir = getLocalDir(projectDir);+ mkdirSync(localDir, { recursive: true });+ const logPath = resolve(localDir, TELEMETRY_FILENAME);++ setToleranceTelemetryHook((record: ToleranceFireRecord) => {+ const line = JSON.stringify({+ event: 'bridge-tolerance-fire',+ timestamp: record.timestamp,+ class: record.className,+ document: record.documentName ?? null,+ codeUnitPosition: record.codeUnitPosition,+ severity: record.severity,+ });+ try {+ appendFileSync(logPath, `${line}\n`);+ } catch {}+ });+}++export function teardownToleranceTelemetryWriter(): void {+ setToleranceTelemetryHook(null);+}