Skip to content

Beautiful, AI-native markdown IDE and LLM wiki

feat(open-knowledge): markdown byte-fidelity engine + conformance suite (#1775)

Nick Gomez5b5eda4893b1
diff --git a/.changeset/markdown-fidelity-e2e.md b/.changeset/markdown-fidelity-e2e.md
new file mode 100644
index 000000000..43f1a8473
--- /dev/null
+++ b/.changeset/markdown-fidelity-e2e.md
@@ -0,0 +1,14 @@
+---
+"@inkeep/open-knowledge-core": minor
+"@inkeep/open-knowledge-server": minor
+---
+
+feat(markdown): end-to-end byte fidelity — markdown round-trips byte-exact, agents' untouched bytes survive concurrent edits
+
+The markdown engine no longer canonicalizes source formatting on the WYSIWYG/cross-surface edit path. Every producer-fixable byte-residual axis now round-trips byte-exact through `serialize(parse(x))`: table outer pipes, delimiter-row dashes and alignment padding, ordered-list numbering (`1. 1. 1.` stays), list marker spacing and continuation indents, fence length and info-string padding, strikethrough delimiter (`~` vs `~~`), task-checkbox case, wiki-link inner padding, opaque-extension embeds (`![[notes.foo]]` no longer downcasts to a plain link), inline math (`$x$` stays `$x$` instead of rewriting to `$$x$$`, with currency safety preserved), doc-start `---` thematic breaks (no longer rewritten to `***`, and a doc-start `---` no longer corrupts adjacent blocks), BOM, leading/trailing blank lines, inter-block blank-line counts, and tab-indented code. One visible consequence: a literal frontmatter-shaped block (`---`/`key: val`/`---`) ingested directly into the parse engine now renders as visible markdown (thematic break + setext heading) instead of being silently dropped.
+
+Silent data-loss fixes: boundary-whitespace emphasis (`<em>foo </em>`) now serializes to a re-parseable form instead of degrading to plain text; entity references in a fenced-code info string survive instead of being stripped; `[link](</my uri>)` parses as a link instead of literal text; inline code containing `|` inside a table cell keeps its escape instead of corrupting the table. The serializer also stops escaping bare `*`/`_`/`~` and line-start `=` where CommonMark does not require it.
+
+Agent find-fidelity (PRD-6654): a concurrent human WYSIWYG edit no longer rewrites an agent's untouched blocks in `Y.Text` — the map-driven Observer A splice is now the default path and a parse-equivalent `row-no-trailing-pipe` divergence no longer forces a canonicalizing write-back — so exact-match find-and-replace on previously written content keeps succeeding.
+
+Clipboard/export rendering: `img`/`video`/`audio` compat components render as real HTML elements instead of placeholders, and entity references and `<>` destinations in link/image URLs and titles decode correctly (with URL sanitization preserved).
diff --git a/biome-plugins/README.md b/biome-plugins/README.md
index e4068bf77..7406ea599 100644
--- a/biome-plugins/README.md
+++ b/biome-plugins/README.md
@@ -99,6 +99,54 @@ The rule does NOT catch:
Plugin: [`biome-plugins/playwright-topass-budget.grit`](playwright-topass-budget.grit). Fixture: [`biome-plugins/__fixtures__/playwright-topass-budget.fixture.tsx`](__fixtures__/playwright-topass-budget.fixture.tsx). Test: [`packages/desktop/tests/integration/playwright-topass-budget.test.ts`](../packages/desktop/tests/integration/playwright-topass-budget.test.ts). See [PRECEDENTS.md #42](../PRECEDENTS.md#custom-lint-enforcement-precedent-42) for the GritQL-plugin convention.
+### `path-conditional-map-driven-origin.grit`
+
+Observer A origin discipline. Inside `packages/server/src/server-observers.ts`, every `Y.Doc.transact()` call MUST pass the sanctioned origin `OBSERVER_SYNC_ORIGIN` as its second argument (`doc.transact(fn, OBSERVER_SYNC_ORIGIN)`). Bare `doc.transact(fn)` - or a wrong origin - routes the write to `openknowledge-service` and breaks per-session UndoManager attribution (the `trackedOrigins` Set-identity match skips the transaction).
+
+**Scoped via `overrides[].plugins`** to `packages/server/src/server-observers.ts`. Other server files routing through `session.dc.document.transact(fn, session.origin)` are out of scope - that contract is enforced by `paired-write-enforcement.test.ts`.
+
+The rule checks the **argument position** (the second argument node), not whether the call subtree contains the identifier somewhere. It matches every `transact(...)` call, then excludes the two sanctioned shapes (`transact($_, OBSERVER_SYNC_ORIGIN)` and the 3-arg `transact($_, OBSERVER_SYNC_ORIGIN, $...)`). A callback body that mentions `OBSERVER_SYNC_ORIGIN` for an unrelated reason therefore neither clears a bare/wrong call nor trips a correct one - the prior `contains`-based shape was falsely cleared by such a mention.
+
+The rule does NOT catch:
+- Transact calls outside `server-observers.ts` (scoped to the one observer-dispatch file)
+- An origin passed in a position other than the second argument (the contract is second-argument placement; no real call site does otherwise)
+
+Plugin: [`biome-plugins/path-conditional-map-driven-origin.grit`](path-conditional-map-driven-origin.grit). Fixture: [`biome-plugins/__fixtures__/path-conditional-map-driven-origin.fixture.tsx`](__fixtures__/path-conditional-map-driven-origin.fixture.tsx). Test: [`packages/server/src/path-conditional-map-driven-origin.test.ts`](../packages/server/src/path-conditional-map-driven-origin.test.ts).
+
+### `cst-pm-handler-todo-stub.grit`
+
+Codemod handler TODO-stub grep. Flags handler files under `packages/md-conformance/src/substrates/*/handlers/` that still contain the codemod's stub body (`throw new Error("TODO: implement <substrate>:<dir>/<key>")`), so a codemod stub that survives to lint time is caught.
+
+This is a TODO-stub grep, NOT an exhaustiveness check. Whether a substrate covers every PM node type (the node-set traversal over `packages/core/schema-snapshot.json`) is a separate concern owned by a suite-self-consistency gate. The compile-time backstop for missing methods is the TypeScript handlers-table type check against `ICstEngine`.
+
+**Scoped via `overrides[].plugins`** to `packages/md-conformance/src/substrates/*/handlers/**/*.ts`. The codemod implementation, harness tests, and oracles are out of scope (they legitimately produce strings containing "TODO: implement" in their own contexts).
+
+The rule does NOT catch:
+- Missing handler FILES (file-presence is out of GritQL's scope; TypeScript + codemod coverage handle this)
+- Whether the handler set is complete per the PM schema (a separate node-set traversal, deferred to the suite-self-consistency gate)
+- Error subclasses (only matches `new Error(...)`)
+- Non-anchored "TODO" mentions (regex requires the message to start with `TODO: implement`)
+
+Plugin: [`biome-plugins/cst-pm-handler-todo-stub.grit`](cst-pm-handler-todo-stub.grit). Fixture: [`biome-plugins/__fixtures__/cst-pm-handler-todo-stub.fixture.tsx`](__fixtures__/cst-pm-handler-todo-stub.fixture.tsx). Test: [`packages/md-conformance/src/lint-plugins/cst-pm-handler-todo-stub.test.ts`](../packages/md-conformance/src/lint-plugins/cst-pm-handler-todo-stub.test.ts).
+
+### `class-proof-registration-discipline.grit`
+
+Class-proof DSL contract enforcement. Two patterns (GritQL `or` — short-circuit, Pattern A wins when both apply):
+
+- **Pattern A (missing args):** flags `defineClassProof(name, opts)` when `opts` doesn't contain both `predicate:` and `proof:` property assignments.
+- **Pattern B (outside canonical dir):** flags any `defineClassProof(...)` call. The override scope excludes `packages/md-conformance/src/class-proofs/proofs/**`, so this fires only on registrations outside the sanctioned location.
+
+Inside the canonical proofs/ dir, neither pattern fires — TypeScript's `ClassProofOptions<M>` signature backstops the missing-args check, and any `defineClassProof` call there is in the sanctioned location.
+
+**Scoped via `overrides[].plugins`** to all `.ts`/`.tsx` files EXCEPT `packages/md-conformance/src/class-proofs/proofs/**` and `*.test.ts`/`*.test.tsx` (test files that exercise the DSL are exempt — `dsl.test.ts` legitimately calls `defineClassProof` outside the canonical dir).
+
+The rule does NOT catch:
+- Missing-args violations inside the canonical proofs/ dir (TypeScript catches those)
+- Functions with similar names (`myDefineClassProofVariant`) — pattern matches the exact function name
+- Type references to `defineClassProof` — pattern scopes to call expressions
+
+Plugin: [`biome-plugins/class-proof-registration-discipline.grit`](class-proof-registration-discipline.grit). Fixture: [`biome-plugins/__fixtures__/class-proof-registration-discipline.fixture.tsx`](__fixtures__/class-proof-registration-discipline.fixture.tsx). Test: [`packages/md-conformance/src/lint-plugins/class-proof-registration-discipline.test.ts`](../packages/md-conformance/src/lint-plugins/class-proof-registration-discipline.test.ts).
+
### `playwright-prefer-to-have-count.grit`
Flags `expect(await locator.count())` — the one-shot count snapshot that never retries. Under CI CPU contention the DOM settles a few frames after the read, so the assertion flakes while the auto-retrying web-first form `await expect(locator).toHaveCount(n)` passes deterministically (the no-retry read was one of the hidden-flake shapes in the 2026-06 e2e CI audit). The pattern matches the probe sub-expression regardless of the matcher that follows (`.toBe`, `.toEqual`, `.toBeGreaterThanOrEqual`, ...). Upstream precedent: eslint-plugin-playwright `prefer-to-have-count`.
diff --git a/biome-plugins/__fixtures__/class-proof-registration-discipline.fixture.tsx b/biome-plugins/__fixtures__/class-proof-registration-discipline.fixture.tsx
new file mode 100644
index 000000000..54246c886
--- /dev/null
+++ b/biome-plugins/__fixtures__/class-proof-registration-discipline.fixture.tsx
@@ -0,0 +1,73 @@
+// FIXTURE — drives `class-proof-registration-discipline.test.ts` via
+// shell-out to `bunx biome check`. Not part of the main lint (override
+// scope keeps the rule off this fixture's path under normal `bun run lint`;
+// the test explicitly invokes biome on this file path).
+//
+// GritQL's `or { ... }` is short-circuit: when a call matches the first
+// branch (Pattern A: missing predicate/proof), the second branch (Pattern
+// B: any defineClassProof) is NOT also considered. Missing-args calls
+// therefore get the more-specific Pattern A diagnostic; well-formed calls
+// outside the canonical dir get Pattern B's diagnostic.
+//
+// 3 expected diagnostic fires (one per violating call):
+// - Positive A1: missing predicate (1 fire — Pattern A diagnostic)
+// - Positive A2: missing proof (1 fire — Pattern A diagnostic)
+// - Positive B1: well-formed but outside canonical dir
+// (1 fire — Pattern B diagnostic)
+// - Negative cases: 0 fires (must NOT fire)
+//
+// Exact-equality (`toBe(3)`) catches both false-negative regressions (drop
+// below 3) and false-positive widenings (above 3).
+
+// biome-ignore lint/correctness/noExplicitAny: fixture-only — production types unimportant here
+declare const defineClassProof: any;
+// biome-ignore lint/correctness/noExplicitAny: fixture-only — production types unimportant here
+declare const enumerateThings: any;
+// biome-ignore lint/correctness/noExplicitAny: fixture-only — production types unimportant here
+declare const someOtherFunction: any;
+
+// === Positive cases — must fire ===
+
+// (A1) Missing `predicate` field — Pattern A's missing-args diagnostic
+// wins over Pattern B's outside-canonical diagnostic via `or`'s
+// short-circuit. Total: 1 diagnostic.
+export const badProofMissingPredicate = defineClassProof('cp-bad-1', {
+ enumerate: enumerateThings,
+ proof: () => ({ pass: true }),
+});
+
+// (A2) Missing `proof` field — same as A1: Pattern A fires. 1 diagnostic.
+export const badProofMissingProof = defineClassProof('cp-bad-2', {
+ enumerate: enumerateThings,
+ predicate: () => true,
+});
+
+// (B1) All three fields present, but the file is outside the canonical
+// dir — Pattern A doesn't match (args complete), so Pattern B fires.
+// Total: 1 diagnostic.
+export const wellFormedButWrongLocation = defineClassProof('cp-bad-3', {
+ enumerate: enumerateThings,
+ predicate: () => true,
+ proof: () => ({ pass: true }),
+});
+
+// === Negative cases — must NOT fire ===
+
+// (1) An unrelated function call with the same arg shape — must NOT fire.
+// The rule's pattern matches the literal function name `defineClassProof`.
+export const unrelated = someOtherFunction('cp-1', {
+ enumerate: enumerateThings,
+ predicate: () => true,
+});
+
+// (2) An identifier whose name contains `defineClassProof` substring but is
+// not exactly that — must NOT fire. The rule matches the bare
+// identifier, not arbitrary affixes.
+export const customDef = (() => {
+ const myDefineClassProofVariant = (_n: string, _o: unknown) => ({ ok: true });
+ return myDefineClassProofVariant('cp-2', { enumerate: enumerateThings });
+})();
+
+// (3) A type reference that mentions `defineClassProof` (e.g. for an alias)
+// — must NOT fire. The rule scopes to call expressions.
+export type DefineClassProofAlias = typeof defineClassProof;
diff --git a/biome-plugins/__fixtures__/cst-pm-handler-todo-stub.fixture.tsx b/biome-plugins/__fixtures__/cst-pm-handler-todo-stub.fixture.tsx
new file mode 100644
index 000000000..1c261d335
--- /dev/null
+++ b/biome-plugins/__fixtures__/cst-pm-handler-todo-stub.fixture.tsx
@@ -0,0 +1,71 @@
+// FIXTURE - drives `cst-pm-handler-todo-stub.test.ts` via shell-out
+// to `bunx biome check`. Not part of the main lint (override scope keeps
+// the rule off this fixture's path under normal `bun run lint`; the test
+// explicitly invokes biome on this file path).
+//
+// 4 positive cases (deliberate codemod-stub bodies — plugin must fire) +
+// 4 negative cases (filled-in bodies + non-handler error throws that must
+// NOT fire). Exact-equality (`toBe(4)`) in the test catches both
+// false-negative regressions (drop below 4) and false-positive widenings
+// (above 4).
+//
+// Parameters are prefixed with `_` per the TS / biome convention for
+// intentionally unused — the rule scopes to the throw-statement shape,
+// not parameter usage. The codemod's emitted stubs name params `node`
+// without the underscore; in real substrate adapters those names will
+// be referenced once the body is filled in.
+
+// === Positive cases — codemod-emitted stubs that must fire ===
+
+// (1) Single-quoted message - the codemod's `JSON.stringify(...)` output
+// renders as a double-quoted literal in TS source. Single-quoted here
+// covers the prettier-rewritten / hand-edited variant.
+export function handleParagraph(_node: unknown): unknown {
+ throw new Error('TODO: implement micromark-event-fold:mdast-to-pm/paragraph');
+}
+
+// (2) Double-quoted message — the codemod's emitted shape verbatim.
+// biome-ignore lint/style/useSingleQuote: fixture covers both quote styles
+export function handleHeading(_node: unknown): unknown {
+ throw new Error("TODO: implement micromark-event-fold:mdast-to-pm/heading");
+}
+
+// (3) Template-literal message — the third quote style; still a TODO stub.
+export function handleStrong(_node: unknown): unknown {
+ throw new Error(`TODO: implement micromark-event-fold:pm-to-mdast-mark/strong`);
+}
+
+// (4) Stub for a different substrate id — pattern is substrate-agnostic.
+export function handleEmphasis(_node: unknown): unknown {
+ throw new Error('TODO: implement lezer-markdown-derived:pm-to-mdast-mark/emphasis');
+}
+
+// === Negative cases — implemented handlers + unrelated throws ===
+
+// (1) A filled-in handler that does real work — must NOT fire.
+export function handleBlockquote(node: unknown): unknown {
+ return { type: 'blockquote', content: [node] };
+}
+
+// (2) An error thrown for a legitimate runtime contract violation — must
+// NOT fire. Message doesn't start with the codemod's `TODO: implement`
+// marker.
+export function handleCodeBlock(node: unknown): unknown {
+ if (typeof node !== 'object' || node === null) {
+ throw new Error('handleCodeBlock: expected mdast code node, got non-object');
+ }
+ return { type: 'codeBlock', content: [] };
+}
+
+// (3) An error thrown via a different constructor — must NOT fire.
+// The rule only matches `Error` (the JS built-in) by name.
+class HandlerSubstrateError extends Error {}
+export function handleListItem(_node: unknown): unknown {
+ throw new HandlerSubstrateError('TODO: implement list item handler');
+}
+
+// (4) An error whose message MENTIONS "TODO" inline but doesn't start with
+// `TODO: implement` — the rule's regex is anchored. Must NOT fire.
+export function handleTable(_node: unknown): unknown {
+ throw new Error('Failed to handle table (TODO: revisit after column-align support lands)');
+}
diff --git a/biome-plugins/__fixtures__/path-conditional-map-driven-origin.fixture.tsx b/biome-plugins/__fixtures__/path-conditional-map-driven-origin.fixture.tsx
new file mode 100644
index 000000000..ee862aa96
--- /dev/null
+++ b/biome-plugins/__fixtures__/path-conditional-map-driven-origin.fixture.tsx
@@ -0,0 +1,97 @@
+// FIXTURE — drives `path-conditional-map-driven-origin.test.ts` via shell-out
+// to `biome check`. Not part of the main lint (override scope keeps the rule
+// off this fixture's path under normal `bun run lint`; the test explicitly
+// invokes biome on this file path).
+//
+// 7 positive cases (deliberate violations — plugin must fire) + 3 negative
+// cases (sanctioned shapes that must NOT fire). Exact-equality (`toBe(7)`) in
+// the test catches both false-negative regressions (drop below 7) and
+// false-positive widenings (above 7).
+//
+// The rule checks the ORIGIN ARGUMENT POSITION (the second argument), not
+// "the call subtree contains the identifier anywhere". Positive7 + Negative3
+// are the load-bearing pair: a callback body that mentions OBSERVER_SYNC_ORIGIN
+// must NOT clear a call whose origin argument is missing/wrong (Positive7), and
+// must NOT trip a call whose origin argument is correct (Negative3).
+
+// biome-ignore lint/correctness/noExplicitAny: fixture-only — production types unimportant here
+declare const doc: any;
+// biome-ignore lint/correctness/noExplicitAny: fixture-only — production types unimportant here
+declare const session: any;
+// biome-ignore lint/correctness/noExplicitAny: fixture-only — production types unimportant here
+declare const OBSERVER_SYNC_ORIGIN: any;
+// biome-ignore lint/correctness/noExplicitAny: fixture-only — production types unimportant here
+declare const SOME_OTHER_ORIGIN: any;
+// biome-ignore lint/correctness/noExplicitAny: fixture-only — production types unimportant here
+declare const current: any;
+
+const noop = (): void => undefined;
+
+// === Positive cases — must fire ===
+
+// (1) Bare `doc.transact(fn)` — no origin at all.
+export function Positive1(): void {
+ doc.transact(noop);
+}
+
+// (2) Bare `session.doc.transact(fn)` via a different receiver — still bare.
+export function Positive2(): void {
+ session.doc.transact(noop);
+}
+
+// (3) 2-arg with a non-sanctioned origin identifier.
+export function Positive3(): void {
+ doc.transact(noop, SOME_OTHER_ORIGIN);
+}
+
+// (4) 2-arg with a string-literal origin (a common shortcut that bypasses the
+// typed allowlist).
+export function Positive4(): void {
+ doc.transact(noop, 'made-up-origin');
+}
+
+// (5) 2-arg using `session.origin` — the per-session origin used by non-observer
+// paths. Inside observer dispatch only `OBSERVER_SYNC_ORIGIN` is sanctioned,
+// so `session.origin` is a violation in observer scope.
+export function Positive5(): void {
+ doc.transact(noop, session.origin);
+}
+
+// (6) 3-arg form `(fn, origin, local)` with a wrong origin in the middle slot.
+export function Positive6(): void {
+ doc.transact(noop, SOME_OTHER_ORIGIN, true);
+}
+
+// (7) Nested-identifier: the callback body mentions OBSERVER_SYNC_ORIGIN but the
+// call itself is bare — no origin argument. The old `contains`-based rule
+// was falsely cleared by the nested mention; the argument-position rule
+// fires because the second argument is absent.
+export function Positive7(): void {
+ doc.transact(() => {
+ if (current === OBSERVER_SYNC_ORIGIN) return;
+ noop();
+ });
+}
+
+// === Negative cases — must NOT fire ===
+
+// (1) Sanctioned origin in the second position.
+export function Negative1(): void {
+ doc.transact(noop, OBSERVER_SYNC_ORIGIN);
+}
+
+// (2) 3-arg form with the sanctioned origin in the second position.
+export function Negative2(): void {
+ doc.transact(noop, OBSERVER_SYNC_ORIGIN, true);
+}
+
+// (3) Nested-identifier WITH the correct origin — mirrors the real
+// `server-observers.ts` drain shape: the callback references
+// OBSERVER_SYNC_ORIGIN and the second argument is OBSERVER_SYNC_ORIGIN.
+// Must NOT fire.
+export function Negative3(): void {
+ doc.transact(() => {
+ if (current === OBSERVER_SYNC_ORIGIN) return;
+ noop();
+ }, OBSERVER_SYNC_ORIGIN);
+}
diff --git a/biome-plugins/class-proof-registration-discipline.grit b/biome-plugins/class-proof-registration-discipline.grit
new file mode 100644
index 000000000..de27b7735
--- /dev/null
+++ b/biome-plugins/class-proof-registration-discipline.grit
@@ -0,0 +1,53 @@
+// class-proof-registration-discipline — class-proof DSL contract enforcement.
+//
+// `defineClassProof()` (from `packages/md-conformance/src/class-proofs/dsl.ts`)
+// registers a class predicate + per-class proof obligation against ts-morph
+// reflective enumeration. Two discipline rules:
+//
+// (a) Registrations live ONLY in
+// `packages/md-conformance/src/class-proofs/proofs/<name>.ts`. Authoring
+// proofs elsewhere splinters the run-all manifest and breaks
+// `runAllClassProofs(engine)`'s single-import contract.
+// (b) Every call must pass both `predicate` and `proof` options. TypeScript
+// enforces this at compile time via `ClassProofOptions<M>`; this rule
+// is the lint-time backstop for the (rare) case where a registration
+// is written outside the canonical dir AND escapes type-checking (e.g.
+// a `.test.ts` file demoed before the test runs).
+//
+// Scoped via `overrides[].plugins` to ALL .ts/.tsx files EXCEPT the canonical
+// `packages/md-conformance/src/class-proofs/proofs/` directory. Inside the
+// canonical dir, neither pattern fires — TypeScript's
+// `ClassProofOptions<M>` signature backstops the missing-args check. Outside
+// the canonical dir, ANY `defineClassProof` call fires (= "wrong location"),
+// and a missing-args call additionally fires a second diagnostic with a
+// dedicated message so the author sees both reasons.
+//
+// Suppress legitimate uses (e.g. a test file that constructs a class-proof
+// inline as part of a DSL-machinery test) with:
+// // biome-ignore lint/plugin/class-proof-registration-discipline: <reason>
+
+language js
+
+or {
+ // Pattern A: missing predicate or proof option. `$opts` is the options
+ // bag; `contains` traverses into the object literal to find each key.
+ `defineClassProof($_, $opts)` as $call where {
+ or {
+ not $opts <: contains `predicate: $_`,
+ not $opts <: contains `proof: $_`
+ },
+ register_diagnostic(
+ span = $call,
+ message = "Class-proof registration missing `predicate` or `proof` option — both are required by `ClassProofOptions<M>`. Provide all three of `{ enumerate, predicate, proof }` to `defineClassProof()`. See https://github.com/inkeep/open-knowledge-legacy/blob/main/biome-plugins/README.md#class-proof-registration-disciplinegrit"
+ )
+ },
+ // Pattern B: any `defineClassProof` call at all. The override scope
+ // excludes the canonical proofs/ dir, so this fires only on registrations
+ // outside the sanctioned location.
+ `defineClassProof($_, $_)` as $call where {
+ register_diagnostic(
+ span = $call,
+ message = "Class-proof registered outside the canonical dir. Move the `defineClassProof(...)` call to `packages/md-conformance/src/class-proofs/proofs/<name>.ts` and export it from `run-all.ts` so it participates in the orchestrator manifest. See https://github.com/inkeep/open-knowledge-legacy/blob/main/biome-plugins/README.md#class-proof-registration-disciplinegrit"
+ )
+ }
+}
diff --git a/biome-plugins/cst-pm-handler-todo-stub.grit b/biome-plugins/cst-pm-handler-todo-stub.grit
new file mode 100644
index 000000000..17ba7184b
--- /dev/null
+++ b/biome-plugins/cst-pm-handler-todo-stub.grit
@@ -0,0 +1,34 @@
+// cst-pm-handler-todo-stub - codemod handler TODO-stub grep.
+//
+// The substrate-adapter codemod seeds a per-substrate handler scaffold by
+// emitting one stub file per PM node type, each whose body is
+// `throw new Error("TODO: implement <substrate>:<dir>/<key>")`. This rule is a
+// TODO-stub grep: it flags any `throw new Error` whose message starts with
+// `TODO: implement ` inside a handler-directory file, so a codemod stub that
+// survives to lint time is caught.
+//
+// It is NOT an exhaustiveness check. Whether a substrate covers every PM node
+// type (the node-set traversal over `packages/core/schema-snapshot.json`) is a
+// separate concern: the TypeScript handlers-table type check against
+// `ICstEngine` is the compile-time backstop for missing methods, and the
+// per-substrate node-set traversal belongs to a suite-self-consistency gate.
+//
+// Scoped via `overrides[].plugins` to
+// `packages/md-conformance/src/substrates/*/handlers/**/*.ts` (and the fixture).
+// Out of scope: the codemod implementation itself, harness tests, oracles - all
+// of which legitimately throw error tags that include the literal string
+// `TODO: implement` (e.g. an oracle producing a synthetic failure reason).
+//
+// Suppress legitimate uses (a handler that legitimately throws "TODO" pending
+// upstream work) with:
+// // biome-ignore lint/plugin/cst-pm-handler-todo-stub: <reason>
+
+language js
+
+`throw new Error($msg)` as $stmt where {
+ $msg <: r"^['\"`]TODO: implement.*",
+ register_diagnostic(
+ span = $stmt,
+ message = "Codemod-emitted handler stub still throws TODO marker - fill in the substrate-specific body so the handler is implemented per the ICstEngine contract. See https://github.com/inkeep/open-knowledge-legacy/blob/main/biome-plugins/README.md#cst-pm-handler-todo-stubgrit"
+ )
+}
diff --git a/biome-plugins/path-conditional-map-driven-origin.grit b/biome-plugins/path-conditional-map-driven-origin.grit
new file mode 100644
index 000000000..054f7ea5e
--- /dev/null
+++ b/biome-plugins/path-conditional-map-driven-origin.grit
@@ -0,0 +1,37 @@
+// path-conditional-map-driven-origin — Observer A origin discipline.
+//
+// Inside the server-side observer module, every `Y.Doc.transact()` call MUST
+// pass the sanctioned origin `OBSERVER_SYNC_ORIGIN` as its SECOND argument:
+//
+// doc.transact(() => { ... }, OBSERVER_SYNC_ORIGIN);
+//
+// Bare `doc.transact(fn)` — no origin at all — silently classifies the write as
+// `openknowledge-service` (precedent #25) and breaks per-session UndoManager
+// attribution (the `trackedOrigins` Set-identity match skips the txn). A wrong
+// origin in the second position is the same failure with a different cause.
+//
+// The rule checks the ARGUMENT POSITION (the second argument), not whether the
+// call subtree contains the identifier somewhere. A callback body that mentions
+// OBSERVER_SYNC_ORIGIN for an unrelated reason therefore neither clears a
+// bare/wrong call nor is required to — only the second-argument node decides.
+//
+// Scoped via `overrides[].plugins` to `packages/server/src/server-observers.ts`
+// (and the plugin fixture). The file's transact-call surface is the entire
+// observer dispatch spine — every call site there is observer-cross-CRDT and
+// must honor the discipline. Other server files routing through
+// `session.dc.document.transact(fn, session.origin)` are out of scope (that
+// contract is enforced by `paired-write-enforcement.test.ts`).
+//
+// Suppress legitimate exceptions (none expected today) with:
+// // biome-ignore lint/plugin/path-conditional-map-driven-origin: <reason>
+
+language js
+
+`$_.transact($...)` as $call where {
+ not $call <: `$_.transact($_, OBSERVER_SYNC_ORIGIN)`,
+ not $call <: `$_.transact($_, OBSERVER_SYNC_ORIGIN, $...)`,
+ register_diagnostic(
+ span = $call,
+ message = "Observer-side transact call missing sanctioned origin. Pass `OBSERVER_SYNC_ORIGIN` as the second argument: `doc.transact(fn, OBSERVER_SYNC_ORIGIN)`. Bare `doc.transact(fn)` (or a wrong origin) routes the write to `openknowledge-service` and breaks per-session UndoManager attribution. See https://github.com/inkeep/open-knowledge-legacy/blob/main/biome-plugins/README.md#path-conditional-map-driven-origingrit"
+ )
+}
diff --git a/biome.jsonc b/biome.jsonc
index 09ff7b6ca..04258ddf4 100644
--- a/biome.jsonc
+++ b/biome.jsonc
@@ -54,6 +54,63 @@
}
},
"overrides": [
+ {
+ // Observer A origin discipline. Scoped to the one file that owns the
+ // observer-cross-CRDT transact spine. Inside this file every
+ // `doc.transact(fn, origin)` must pass `OBSERVER_SYNC_ORIGIN` as the
+ // second argument; bare `doc.transact(fn)` (or a wrong origin) routes the
+ // write to `openknowledge-service` and breaks per-session UndoManager
+ // attribution. Other server files routing through
+ // `session.dc.document.transact(fn, session.origin)` are out of scope -
+ // that contract is enforced by `paired-write-enforcement.test.ts`.
+ "includes": [
+ "packages/server/src/server-observers.ts",
+ "biome-plugins/__fixtures__/path-conditional-map-driven-origin.fixture.tsx"
+ ],
+ "plugins": ["./biome-plugins/path-conditional-map-driven-origin.grit"]
+ },
+ {
+ // Codemod handler TODO-stub grep. The substrate-adapter codemod seeds
+ // per-substrate handler scaffolds under
+ // packages/md-conformance/src/substrates/<id>/handlers/{mdast-to-pm,
+ // pm-to-mdast-node,pm-to-mdast-mark}/<key>.ts, each containing
+ // `throw new Error("TODO: implement <substrate>:<dir>/<key>")`. This rule
+ // greps for stubs that survive to lint time so an unfinished handler
+ // can't pass review. It is NOT an exhaustiveness check: the per-substrate
+ // node-set traversal belongs to a suite-self-consistency gate, and the
+ // handlers-table type check against ICstEngine is the compile-time
+ // backstop for missing methods. Scoped narrowly to the handlers/ subtree
+ // under any substrate dir, so the codemod implementation + harness tests +
+ // oracles (which legitimately produce strings containing "TODO: implement"
+ // in their own contexts) are out of scope.
+ "includes": [
+ "packages/md-conformance/src/substrates/*/handlers/**/*.ts",
+ "biome-plugins/__fixtures__/cst-pm-handler-todo-stub.fixture.tsx"
+ ],
+ "plugins": ["./biome-plugins/cst-pm-handler-todo-stub.grit"]
+ },
+ {
+ // Class-proof DSL registration discipline. Per the dsl.ts contract
+ // (`defineClassProof` from packages/md-conformance/src/class-proofs/dsl.ts),
+ // class-proof registrations MUST live in the canonical proofs/ dir so
+ // run-all.ts's manifest stays the single source of truth, and EVERY
+ // registration MUST pass `predicate` + `proof` options. The plugin's
+ // override scope EXCLUDES the canonical proofs/ dir — inside the dir,
+ // TypeScript's `ClassProofOptions<M>` signature is the missing-args
+ // backstop. Outside the dir, any `defineClassProof(...)` call fires
+ // (= wrong location), and missing-args fires an additional diagnostic.
+ "includes": [
+ "**/*.ts",
+ "**/*.tsx",
+ "!**/node_modules/**",
+ "!**/dist/**",
+ "!**/*.test.ts",
+ "!**/*.test.tsx",
+ "!packages/md-conformance/src/class-proofs/proofs/**",
+ "biome-plugins/__fixtures__/class-proof-registration-discipline.fixture.tsx"
+ ],
+ "plugins": ["./biome-plugins/class-proof-registration-discipline.grit"]
+ },
{
// Apple-Event / IPC roundtrip toPass budget enforcement is scoped to
// the desktop-smoke files (and the plugin's own fixture for testing).
@@ -191,7 +248,26 @@
"!**/tests/perf/results",
"!**/tests/perf/playwright-results",
"!packages/desktop/out",
- "!packages/desktop/dist-desktop"
+ "!packages/desktop/dist-desktop",
+ // The md-audit framework was folded under packages/md-conformance/md-audit/
+ // but is a self-contained unit with its own tsconfig + conventions; it was
+ // never OK-biome-formatted (it lived outside the lint scan targets before
+ // the fold) and ships a generated docs/byte-preservation-audit.json. Its
+ // own typecheck + tests run via `bun --cwd packages/md-conformance/md-audit`
+ // and the md-conformance test task; keep it out of OK's biome format scan.
+ "!packages/md-conformance/md-audit",
+ // Mechanically generated by scripts/enumerate-conformance.ts (JSON.stringify);
+ // its byte format is owned by the generator, not biome (biome would collapse
+ // the short per-entry testTier arrays the generator emits expanded). Content
+ // correctness is enforced by check-conformance-fresh.ts (sha256 pin), so the
+ // format scan adds nothing. Path is copybara-excluded, like the md-audit line.
+ "!packages/md-conformance/conformance-catalog.json",
+ // Mechanically extracted by src/corpora/authoring/extract-gfm-extensions.ts
+ // (JSON.stringify); its byte format is owned by the extractor, not biome
+ // (biome would collapse the single-element `extensions` arrays the extractor
+ // emits expanded). Content correctness is enforced by the re-extraction
+ // freshness gate (gfm-extensions-freshness.test.ts). Path is copybara-excluded.
+ "!packages/md-conformance/corpora/gfm-extensions.json"
]
}
}
diff --git a/bun.lock b/bun.lock
index ad102179e..8ca39f0e0 100644
--- a/bun.lock
+++ b/bun.lock
@@ -249,15 +249,20 @@
"@tiptap/extension-image": "^3.22.3",
"@tiptap/extension-italic": "^3.22.3",
"@tiptap/extension-link": "^3.22.3",
+ "@tiptap/extension-strike": "^3.22.3",
"@tiptap/extension-table": "^3.22.3",
"@tiptap/pm": "^3.22.3",
"@tiptap/starter-kit": "^3.22.3",
+ "decode-named-character-reference": "^1.3.0",
"diff-match-patch": "^1.0.5",
"mdast-util-from-markdown": "^2.0.3",
"mdast-util-math": "^3.0.0",
"mdast-util-mdx": "3.0.0",
"mdast-util-to-markdown": "^2.1.2",
"micromark-extension-mdx": "2.1.0",
+ "micromark-util-classify-character": "^2.0.1",
+ "micromark-util-decode-numeric-character-reference": "^2.0.2",
+ "micromark-util-sanitize-uri": "^2.0.1",
"micromark-util-types": "^2.0.2",
"node-diff3": "^3.2.0",
"rehype-parse": "^9.0.1",
@@ -295,6 +300,8 @@
"culori": "^4.0.2",
"fast-check": "^4.6.0",
"mdast-util-to-hast": "^13.2.1",
+ "micromark": "^4.0.2",
+ "micromark-util-symbol": "^2.0.1",
"tsdown": "^0.21.7",
"typescript": "^5.9.3",
},
@@ -373,6 +380,7 @@
"zod": "^4.3.6",
},
"devDependencies": {
+ "@types/mdast": "^4.0.4",
"@types/shell-quote": "^1.7.5",
"@types/ws": "^8.18.1",
"@types/yazl": "^2.4.6",
@@ -1734,7 +1742,7 @@
"@types/ms": ["@types/[email protected]", "", {}, "sha512-GsCCIZDE/p3i96vtEqx+7dBUGXrc7zeSK3wwPHIaRThS+9OhWIXRqzs4d6k1SVU8g91DrNRWxWUGhp5KXQb2VA=="],
- "@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],
+ "@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],
"@types/node-fetch": ["@types/[email protected]", "", { "dependencies": { "@types/node": "*", "form-data": "^4.0.4" } }, "sha512-QGpRVpzSaUs30JBSGPjOg4Uveu384erbHBoT1zeONvyCfwQxIkUshLAOqN/k9EjGviPRmWTTe6aH2qySWKTVSw=="],
@@ -3954,7 +3962,7 @@
"undici": ["[email protected]", "", {}, "sha512-xXnp4kTyor2Zq+J1FfPI6Eq3ew5h6Vl0F/8d9XU5zZQf1tX9s2Su1/3PiMmUANFULpmksxkClamIZcaUqryHsQ=="],
- "undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],
+ "undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],
"unicorn-magic": ["[email protected]", "", {}, "sha512-+QBBXBCvifc56fsbuxZQ6Sic3wqqc3WWaqxs58gvJrcOuN83HGTCwz3oS5phzU9LthRNE9VrJCFCLUgHeeFnfA=="],
@@ -4204,7 +4212,11 @@
"@inkeep/cxkit-styled/tailwind-merge": ["[email protected]", "", {}, "sha512-P+Vu1qXfzediirmHOC3xKGAYeZtPcV9g76X+xg2FD4tYgR71ewMA35Y3sCz3zhiN/dwefRpJX0yBcgwi1fXNQA=="],
- "@inkeep/open-knowledge-core/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],
+ "@inkeep/open-knowledge/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],
+
+ "@inkeep/open-knowledge-desktop/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],
+
+ "@inkeep/open-knowledge-docs/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],
"@inkeep/open-knowledge-docs/lucide-react": ["[email protected]", "", { "peerDependencies": { "react": "^16.5.1 || ^17.0.0 || ^18.0.0 || ^19.0.0" } }, "sha512-HGGkdlPWQ0vTF8jJ5TdIqhQXZi6uh3LnNgfZ8MHiuxFfX3RZeA79r2MW2tHAZKlAVfoNE8esm3p+O6VkIvpj6w=="],
@@ -4230,8 +4242,6 @@
"@isaacs/cliui/wrap-ansi": ["[email protected]", "", { "dependencies": { "ansi-styles": "^6.1.0", "string-width": "^5.0.1", "strip-ansi": "^7.0.1" } }, "sha512-si7QWI6zUMq56bESFvagtmzMdGOtoxfR+Sez11Mobfc7tm+VkUckk9bW2UeffTGVUbOksxmSw0AA2gs8g71NCQ=="],
- "@jest/types/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],
-
"@lingui/cli/jiti": ["[email protected]", "", { "bin": { "jiti": "lib/jiti-cli.mjs" } }, "sha512-ekilCSN1jwRvIbgeg/57YFh8qQDNbwDb9xT/qu2DAHbFFZUicIl4ygVaAvzveMhMVr3LnpSKTNnwt8PoOfmKhQ=="],
"@lingui/conf/jiti": ["[email protected]", "", { "bin": { "jiti": "lib/jiti-cli.mjs" } }, "sha512-ekilCSN1jwRvIbgeg/57YFh8qQDNbwDb9xT/qu2DAHbFFZUicIl4ygVaAvzveMhMVr3LnpSKTNnwt8PoOfmKhQ=="],
@@ -4318,13 +4328,23 @@
"@tiptap/starter-kit/@tiptap/pm": ["@tiptap/[email protected]", "", { "dependencies": { "prosemirror-changeset": "^2.3.0", "prosemirror-collab": "^1.3.1", "prosemirror-commands": "^1.6.2", "prosemirror-dropcursor": "^1.8.1", "prosemirror-gapcursor": "^1.3.2", "prosemirror-history": "^1.4.1", "prosemirror-inputrules": "^1.4.0", "prosemirror-keymap": "^1.2.2", "prosemirror-markdown": "^1.13.1", "prosemirror-menu": "^1.2.4", "prosemirror-model": "^1.24.1", "prosemirror-schema-basic": "^1.2.3", "prosemirror-schema-list": "^1.5.0", "prosemirror-state": "^1.4.3", "prosemirror-tables": "^1.6.4", "prosemirror-trailing-node": "^3.0.0", "prosemirror-transform": "^1.10.2", "prosemirror-view": "^1.38.1" } }, "sha512-NjfWjZuvrqmpICT+GZWNIjtOdhPyqFKDMtQy7tsQ5rErM9L2ZQdy/+T/BKSO1JdTeBhdg9OP+0yfsqoYp2aT6A=="],
- "@types/jsdom/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],
+ "@types/busboy/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],
+
+ "@types/cacheable-request/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],
+
+ "@types/fs-extra/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],
"@types/jsdom/undici-types": ["[email protected]", "", {}, "sha512-AXNgS1Byr27fTI+2bsPEkV9CxkT8H6xNyRI68b3TatlZo3RkzlqQBLL+w7SmGPVpokjHbcuNVQUWE7FRTg+LRA=="],
- "@types/node-fetch/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],
+ "@types/keyv/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],
- "@types/yazl/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],
+ "@types/plist/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],
+
+ "@types/responselike/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],
+
+ "@types/ws/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],
+
+ "@types/yauzl/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],
"ajv-keywords/ajv": ["[email protected]", "", { "dependencies": { "fast-deep-equal": "^3.1.1", "fast-json-stable-stringify": "^2.0.0", "json-schema-traverse": "^0.4.1", "uri-js": "^4.2.2" } }, "sha512-IWrosm/yrn43eiKqkfkHis7QioDleaXQHdDVPKg0FSwwd/DuvyX79TZnFOnYpB7dcsFAMmtFztZuXPDvSePkFw=="],
@@ -4348,8 +4368,6 @@
"builder-util/fs-extra": ["[email protected]", "", { "dependencies": { "graceful-fs": "^4.2.0", "jsonfile": "^6.0.1", "universalify": "^2.0.0" } }, "sha512-oRXApq54ETRj4eMiFzGnHWGy+zo5raudjuxN0b8H7s/RU2oW0Wvsx9O0ACRN/kRq9E8Vu/ReskGB5o3ji+FzHQ=="],
- "bun-types/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],
-
"cacache/glob": ["[email protected]", "", { "dependencies": { "foreground-child": "^3.1.0", "jackspeak": "^3.1.2", "minimatch": "^9.0.4", "minipass": "^7.1.2", "package-json-from-dist": "^1.0.0", "path-scurry": "^1.11.1" }, "bin": { "glob": "dist/esm/bin.mjs" } }, "sha512-DfXN8DfhJ7NH3Oe7cFmu3NCu1wKbkReJ8TorzSAFbSKrlNaQSKfIzqYqVY8zlbs2NLBbWpRiU52GX2PbaBVNkg=="],
"cacache/lru-cache": ["[email protected]", "", {}, "sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ=="],
@@ -4388,6 +4406,8 @@
"dotenv-expand/dotenv": ["[email protected]", "", {}, "sha512-uBq4egWHTcTt33a72vpSG0z3HnPuIl6NqYcTrKEg2azoEyl2hpW0zqlxysq2pK9HlDIHyHyakeYaYnSAwd8bow=="],
+ "electron/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g=="],
+
"electron-builder/fs-extra": ["[email protected]", "", { "dependencies": { "graceful-fs": "^4.2.0", "jsonfile": "^6.0.1", "universalify": "^2.0.0" } }, "sha512-oRXApq54ETRj4eMiFzGnHWGy+zo5raudjuxN0b8H7s/RU2oW0Wvsx9O0ACRN/kRq9E8Vu/ReskGB5o3ji+FzHQ=="],
"electron-publish/fs-extra": ["[email protected]", "", { "dependencies": { "graceful-fs": "^4.2.0", "jsonfile": "^6.0.1", "universalify": "^2.0.0" } }, "sha512-oRXApq54ETRj4eMiFzGnHWGy+zo5raudjuxN0b8H7s/RU2oW0Wvsx9O0ACRN/kRq9E8Vu/ReskGB5o3ji+FzHQ=="],
@@ -4512,8 +4532,6 @@
"prosemirror-trailing-node/escape-string-regexp": ["[email protected]", "", {}, "sha512-TtpcNJ3XAzx3Gq8sWRzJaVajRs0uVxA2YAkdb1jm2YkPz4G6egUFAyA3n5vtEIZefPk5Wa4UXbKuS5fKkJWdgA=="],
- "protobufjs/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],
-
"proxy-addr/ipaddr.js": ["[email protected]", "", {}, "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g=="],
"proxy-agent/lru-cache": ["[email protected]", "", {}, "sha512-jumlc0BIUrS3qJGgIkWZsyfAM7NCWiBcCDhnd+3NNM5KbBmLTgHVfWBcg6W+rLUsIpzpERPsvwUP7CckAQSOoA=="],
@@ -4728,9 +4746,11 @@
"@inkeep/cxkit-primitives/@radix-ui/react-tooltip/@radix-ui/react-visually-hidden": ["@radix-ui/[email protected]", "", { "dependencies": { "@radix-ui/react-primitive": "2.0.1" }, "peerDependencies": { "@types/react": "*", "@types/react-dom": "*", "react": "^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc", "react-dom": "^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc" }, "optionalPeers": ["@types/react", "@types/react-dom"] }, "sha512-vVfA2IZ9q/J+gEamvj761Oq1FpWgCDaNOOIfbPVp2MVPLEomUr5+Vf7kJGwQ24YxZSlQVar7Bes8kyTo5Dshpg=="],
- "@inkeep/open-knowledge-core/@types/node/undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],
+ "@inkeep/open-knowledge-desktop/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],
- "@inkeep/open-knowledge-server/@types/yazl/@types/node": ["@types/[email protected]", "", { "dependencies": { "undici-types": "~7.19.0" } }, "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ=="],
+ "@inkeep/open-knowledge-docs/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],
+
+ "@inkeep/open-knowledge/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],
"@inquirer/confirm/@inquirer/core/@inquirer/ansi": ["@inquirer/[email protected]", "", {}, "sha512-S8qNSZiYzFd0wAcyG5AXCvUHC5Sr7xpZ9wZ2py9XR88jUz8wooStVx5M6dRzczbBWjic9NP7+rY0Xi7qqK/aMQ=="],
@@ -4748,8 +4768,6 @@
"@isaacs/cliui/wrap-ansi/ansi-styles": ["[email protected]", "", {}, "sha512-4Dj6M28JB+oAH8kFkTLUo+a2jwOFkuqb3yucU0CANcRRUbxS0cP0nZYCGjcc3BNXwRIsUVmDGgzawme7zvJHvg=="],
- "@jest/types/@types/node/undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],
-
"@malept/flatpak-bundler/fs-extra/jsonfile": ["[email protected]", "", { "dependencies": { "universalify": "^2.0.0" }, "optionalDependencies": { "graceful-fs": "^4.1.6" } }, "sha512-FGuPw30AdOIUTRMC2OMRtQV+jkVj2cfPqSeWXv1NEAJ1qZ5zb1X6z1mFhbfOB/iy3ssJCD+3KuZ8r8C3uVFlAg=="],
"@malept/flatpak-bundler/fs-extra/universalify": ["[email protected]", "", {}, "sha512-gptHNQghINnc/vTGIk0SOFGFNXw7JVrlRUtConJRlvaw6DuX0wO5Jeko9sWrMBhh+PsYAZ7oXAiOnf/UKogyiw=="],
@@ -4760,11 +4778,21 @@
"@opentelemetry/instrumentation-fetch/@opentelemetry/sdk-trace-web/@opentelemetry/sdk-trace-base": ["@opentelemetry/[email protected]", "", { "dependencies": { "@opentelemetry/core": "2.1.0", "@opentelemetry/resources": "2.1.0", "@opentelemetry/semantic-conventions": "^1.29.0" }, "peerDependencies": { "@opentelemetry/api": ">=1.3.0 <1.10.0" } }, "sha512-uTX9FBlVQm4S2gVQO1sb5qyBLq/FPjbp+tmGoxu4tIgtYGmBYB44+KX/725RFDe30yBSaA9Ml9fqphe1hbUyLQ=="],
- "@types/jsdom/@types/node/undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],
+ "@types/busboy/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],
+
+ "@types/cacheable-request/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],
+
+ "@types/fs-extra/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],
+
+ "@types/keyv/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],
+
+ "@types/plist/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],
- "@types/node-fetch/@types/node/undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],
+ "@types/responselike/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],
- "@types/yazl/@types/node/undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],
+ "@types/ws/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],
+
+ "@types/yauzl/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],
"ajv-keywords/ajv/json-schema-traverse": ["[email protected]", "", {}, "sha512-xbbCH5dCYU5T8LcEhhuh7HJ88HXuW3qsI3Y0zOZFKfZEHcpWiHU/Jxzk629Brsab/mMiHQti9wMP+845RPe3Vg=="],
@@ -4788,8 +4816,6 @@
"builder-util/fs-extra/universalify": ["[email protected]", "", {}, "sha512-gptHNQghINnc/vTGIk0SOFGFNXw7JVrlRUtConJRlvaw6DuX0wO5Jeko9sWrMBhh+PsYAZ7oXAiOnf/UKogyiw=="],
- "bun-types/@types/node/undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],
-
"cacache/glob/minimatch": ["[email protected]", "", { "dependencies": { "brace-expansion": "^2.0.2" } }, "sha512-OBwBN9AL4dqmETlpS2zasx+vTeWclWzkblfZk7KTA5j3jeOONz/tRCnZomUyvNg83wL5Zv9Ss6HMJXAgL8R2Yg=="],
"cli-truncate/string-width/strip-ansi": ["[email protected]", "", { "dependencies": { "ansi-regex": "^6.2.2" } }, "sha512-yDPMNjp4WyfYBkHnjIRLfca1i6KMyGCtsVgoKe/z1+6vukgaENdgGBZt+ZmKPc4gavvEZ5OgHfHdrazhgNyG7w=="],
@@ -4818,6 +4844,8 @@
"electron-updater/fs-extra/universalify": ["[email protected]", "", {}, "sha512-gptHNQghINnc/vTGIk0SOFGFNXw7JVrlRUtConJRlvaw6DuX0wO5Jeko9sWrMBhh+PsYAZ7oXAiOnf/UKogyiw=="],
+ "electron/@types/node/undici-types": ["[email protected]", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],
+
"filelist/minimatch/brace-expansion": ["[email protected]", "", { "dependencies": { "balanced-match": "^1.0.0" } }, "sha512-TN1kCZAgdgweJhWWpgKYrQaMNHcDULHkWwQIspdtjV4Y5aurRdZpjAqn6yX3FPqTA9ngHCc4hJxMAMgGfve85w=="],
"form-data/mime-types/mime-db": ["[email protected]", "", {}, "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg=="],
@@ -4914,8 +4942,6 @@
"prebuild-install/tar-fs/tar-stream": ["[email protected]", "", { "dependencies": { "bl": "^4.0.3", "end-of-stream": "^1.4.1", "fs-constants": "^1.0.0", "inherits": "^2.0.3", "readable-stream": "^3.1.1" } }, "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ=="],
- "protobufjs/@types/node/undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],
-
"puppeteer-core/chromium-bidi/zod": ["[email protected]", "", {}, "sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ=="],
"puppeteer/puppeteer-core/webdriver-bidi-protocol": ["[email protected]", "", {}, "sha512-ARrjNjtWRRs2w4Tk7nqrf2gBI0QXWuOmMCx2hU+1jUt6d00MjMxURrhxhGbrsoiZKJrhTSTzbIrc554iKI10qw=="],
@@ -5104,8 +5130,6 @@
"@inkeep/cxkit-primitives/@radix-ui/react-tooltip/@radix-ui/react-use-controllable-state/@radix-ui/react-use-callback-ref": ["@radix-ui/[email protected]", "", { "peerDependencies": { "@types/react": "*", "react": "^16.8 || ^17.0 || ^18.0 || ^19.0 || ^19.0.0-rc" }, "optionalPeers": ["@types/react"] }, "sha512-CasTfvsy+frcFkbXtSJ2Zu9JHpN8TYKxkgJGWbjiZhFivxaeW7rMeZt7QELGVLaYVfFMsKHjb7Ak0nMEe+2Vfw=="],
- "@inkeep/open-knowledge-server/@types/yazl/@types/node/undici-types": ["[email protected]", "", {}, "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg=="],
-
"@opentelemetry/instrumentation-fetch/@opentelemetry/sdk-trace-web/@opentelemetry/sdk-trace-base/@opentelemetry/resources": ["@opentelemetry/[email protected]", "", { "dependencies": { "@opentelemetry/core": "2.1.0", "@opentelemetry/semantic-conventions": "^1.29.0" }, "peerDependencies": { "@opentelemetry/api": ">=1.3.0 <1.10.0" } }, "sha512-1CJjf3LCvoefUOgegxi8h6r4B/wLSzInyhGP2UmIBYNlo4Qk5CZ73e1eEyWmfXvFtm1ybkmfb2DqWvspsYLrWw=="],
"app-builder-lib/@electron/fuses/fs-extra/jsonfile": ["[email protected]", "", { "dependencies": { "universalify": "^2.0.0" }, "optionalDependencies": { "graceful-fs": "^4.1.6" } }, "sha512-FGuPw30AdOIUTRMC2OMRtQV+jkVj2cfPqSeWXv1NEAJ1qZ5zb1X6z1mFhbfOB/iy3ssJCD+3KuZ8r8C3uVFlAg=="],
diff --git a/knip.config.ts b/knip.config.ts
index 676e4b77a..58556b66b 100644
--- a/knip.config.ts
+++ b/knip.config.ts
@@ -7,6 +7,7 @@ export default {
'husky',
'@lingui/babel-plugin-lingui-macro',
'@lingui/format-po',
+ 'micromark',
],
ignoreBinaries: ['printf'],
ignoreIssues: {
@@ -15,7 +16,7 @@ export default {
'docs/source.config.ts': ['exports'],
'packages/app/src/editor/extensions/internal-link.ts': ['exports', 'types'],
'packages/app/src/editor/clipboard/serialize.ts': ['types'],
- '{tech-probes,reports,specs}/**': ['files'],
+ '{tech-probes,reports,specs}/**': ['files', 'exports', 'types'],
'tests/integration/**': ['files'],
'packages/core/src/desktop-bridge.ts': ['files', 'types'],
'packages/desktop/src/shared/ipc-events.ts': ['files'],
diff --git a/package.json b/package.json
index e6b288412..c459fbace 100644
--- a/package.json
+++ b/package.json
@@ -16,7 +16,6 @@
"knip": "turbo run build --filter=@inkeep/open-knowledge-server --filter=@inkeep/open-knowledge-core --output-logs=errors-only && knip",
"notices": "bun scripts/generate-third-party-notices.mjs",
"schema:dump": "bun packages/core/scripts/dump-schema.ts",
- "md-audit:measure-comment-discipline": "bun run --cwd md-audit measure-comment-discipline",
"build": "turbo run build",
"build:desktop": "turbo run build:desktop",
"build:desktop:dir": "turbo run build:desktop:dir",
diff --git a/packages/app/tests/fidelity/doc-boundary-fragment-drop.test.ts b/packages/app/tests/fidelity/doc-boundary-fragment-drop.test.ts
new file mode 100644
index 000000000..db5fe1228
--- /dev/null
+++ b/packages/app/tests/fidelity/doc-boundary-fragment-drop.test.ts
@@ -0,0 +1,25 @@
+import { describe, expect, test } from 'bun:test';
+import { MarkdownManager, sharedExtensions } from '@inkeep/open-knowledge-core';
+import { getSchema } from '@tiptap/core';
+import { updateYFragment, yXmlFragmentToProseMirrorRootNode } from '@tiptap/y-tiptap';
+import * as Y from 'yjs';
+
+const mdManager = new MarkdownManager({ extensions: sharedExtensions });
+const schema = getSchema(sharedExtensions);
+
+describe('sourceDocBoundary never survives the Y.XmlFragment path', () => {
+ test('a captured boundary snapshot resets to the schema default through the fragment round-trip', () => {
+ const json = mdManager.parse('\u{FEFF}\n\nHello\n\n\nWorld\n');
+ expect(json.attrs?.sourceDocBoundary ?? null).not.toBeNull();
+
+ const doc = new Y.Doc();
+ const fragment = doc.getXmlFragment('default');
+ const meta = { mapping: new Map(), isOMark: new Map() };
+ updateYFragment(doc, fragment, schema.nodeFromJSON(json), meta);
+
+ const back = yXmlFragmentToProseMirrorRootNode(fragment, schema).toJSON();
+ expect(back.attrs?.sourceDocBoundary ?? null).toBeNull();
+ expect(mdManager.serialize(back)).toBe('Hello\n\nWorld\n');
+ doc.destroy();
+ });
+});
diff --git a/packages/app/tests/fidelity/invariant-i5.test.ts b/packages/app/tests/fidelity/invariant-i5.test.ts
index 62d212c29..4f6f391a0 100644
--- a/packages/app/tests/fidelity/invariant-i5.test.ts
+++ b/packages/app/tests/fidelity/invariant-i5.test.ts
@@ -1,5 +1,6 @@
import { describe, expect, test } from 'bun:test';
import { MarkdownManager, sharedExtensions } from '@inkeep/open-knowledge-core';
+import type { JSONContent } from '@tiptap/core';
import { getSchema } from '@tiptap/core';
import { updateYFragment, yXmlFragmentToProseMirrorRootNode } from '@tiptap/y-tiptap';
import * as fc from 'fast-check';
@@ -10,14 +11,21 @@ import { NUM_RUNS, normalize } from './helpers';
const mdManager = new MarkdownManager({ extensions: sharedExtensions });
const schema = getSchema(sharedExtensions);
+/** Drop the doc-boundary snapshot so both layers consume the identical
+ * fragment-representable doc (the Y path cannot carry doc attrs). */
+function stripDocBoundary(json: JSONContent): JSONContent {
+ const { sourceDocBoundary: _dropped, ...attrs } = (json.attrs ?? {}) as Record<string, unknown>;
+ return { ...json, attrs };
+}
+
function layerA(md: string): string {
- return mdManager.serialize(mdManager.parse(md));
+ return mdManager.serialize(stripDocBoundary(mdManager.parse(md)));
}
function layerB(md: string): string {
const doc = new Y.Doc();
const fragment = doc.getXmlFragment('default');
- const json = mdManager.parse(md);
+ const json = stripDocBoundary(mdManager.parse(md));
const pmNode = schema.nodeFromJSON(json);
const meta = { mapping: new Map(), isOMark: new Map() };
updateYFragment(doc, fragment, pmNode, meta);
diff --git a/packages/app/tests/fidelity/invariant-i8.test.ts b/packages/app/tests/fidelity/invariant-i8.test.ts
deleted file mode 100644
index d2e6233a5..000000000
--- a/packages/app/tests/fidelity/invariant-i8.test.ts
+++ /dev/null
@@ -1,134 +0,0 @@
-import { describe, test } from 'bun:test';
-import { VFileMessage } from '@inkeep/open-knowledge-core';
-import * as fc from 'fast-check';
-import { block } from './arbitraries';
-import { mdManager, NUM_RUNS, PBT_TIMEOUT_MS } from './helpers';
-
-function isExpectedParseError(err: unknown): boolean {
- if (err instanceof SyntaxError) return true;
- if (err instanceof VFileMessage) return true;
- if (err instanceof RangeError && err.message.includes('Invalid content for node')) return true;
- return false;
-}
-
-function assertNoCrash(input: string): void {
- try {
- mdManager.parse(input);
- } catch (err) {
- if (!isExpectedParseError(err)) {
- throw err; // Unexpected error type — test fails
- }
- }
-}
-
-const dangerousText = fc
- .array(
- fc.oneof(
- fc.constantFrom(...'abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.,;?!\n'),
- fc.constantFrom(...'<>&[]{}:@`#\\*_$~!/()'),
- ),
- { maxLength: 150 },
- )
- .map((chars) => chars.join(''));
-
-describe('I8 — crash resistance: parse() never throws unexpected errors', () => {
- test(
- 'arbitrary prose with dangerous characters',
- () => {
- fc.assert(
- fc.property(dangerousText, (s) => {
- assertNoCrash(s);
- }),
- { numRuns: NUM_RUNS, seed: 42 },
- );
- },
- PBT_TIMEOUT_MS,
- );
-
- test(
- 'multi-block documents with dangerous chars adjacent to valid blocks',
- () => {
- fc.assert(
- fc.property(
- fc
- .tuple(dangerousText, block, dangerousText)
- .map(([pre, b, post]) => `${pre}\n\n${b}\n\n${post}`),
- (md) => {
- assertNoCrash(md);
- },
- ),
- { numRuns: NUM_RUNS, seed: 42 },
- );
- },
- PBT_TIMEOUT_MS,
- );
-
- test('hardcoded regression cases — known-crashworthy inputs', () => {
- const mustNotCrash = [
- 'text <foo bar',
- '<Component',
- '<',
- 'a<b',
- 'a<B>c<D',
- '<$special>',
- '<_private>',
- '<$',
- '<_',
- '</',
- '</foo',
- '</Callout',
- '<<<merge>>>',
- '<<<<<<< HEAD',
- '>>>>>>> main',
- '<script>alert(1)</script>',
- ':::note\nopen directive',
- '::leafDirective',
- '{',
- '{ ',
- 'text {',
- '{ unclosed',
- 'a{b',
- '{a',
- '{{',
- '{{{',
- '{a{b',
- '{a:b}',
- '{a b}',
- '{a;b}',
- '{if(x)y}',
- '{a {b}}',
- '{<>}',
- '{&}',
- '{*}',
- '{#}',
- '{expression}',
- '{/* comment */}',
- '{}',
- '~:a~',
- '*:a*',
- '---\n\n---',
- '---\n---',
- '< < < <',
- 'a<b<c<d',
- 'end of line <',
- '< start of line',
- 'mid < dle',
- '<div',
- '<div class="x"',
- '<span style=',
- '**bold** and <foo unclosed',
- 'if (x < y) { z > w }',
- 'a < b && c > d',
- '<foo and {bar',
- '<Callout>{content}</Callout>',
- '',
- ' ',
- '\n',
- '\n\n\n',
- ];
-
- for (const input of mustNotCrash) {
- assertNoCrash(input);
- }
- });
-});
diff --git a/packages/app/tests/fidelity/invariant-wiki-link-embed-conversion.test.ts b/packages/app/tests/fidelity/invariant-wiki-link-embed-conversion.test.ts
index bd2ce37cb..aae22ae26 100644
--- a/packages/app/tests/fidelity/invariant-wiki-link-embed-conversion.test.ts
+++ b/packages/app/tests/fidelity/invariant-wiki-link-embed-conversion.test.ts
@@ -10,7 +10,6 @@ const nonImageExt = fc.constantFrom('mp4', 'webm', 'mov', 'mp3', 'wav', 'ogg', '
const fileAttachmentExt = fc.constantFrom('pdf', 'zip', 'docx', 'csv');
const opaqueExt = fc.constantFrom('xyz', 'qux', 'wat');
-const renderableExt = fc.oneof(imageExt, nonImageExt, fileAttachmentExt);
const allExts = fc.oneof(imageExt, nonImageExt, fileAttachmentExt, opaqueExt);
const anchor = fc.option(fc.stringMatching(/^[a-zA-Z0-9_=-]{1,12}$/), { nil: null });
@@ -23,19 +22,16 @@ function buildEmbed(s: string, e: string, a: string | null, l: string | null): s
return `${out}]]`;
}
-const renderableEmbedMd = fc
- .tuple(stem, renderableExt, anchor, alias)
- .map(([s, e, a, l]) => buildEmbed(s, e, a, l));
const anyEmbedMd = fc
.tuple(stem, allExts, anchor, alias)
.map(([s, e, a, l]) => buildEmbed(s, e, a, l));
describe('wiki-embed conversion invariants — mdManager path (US-010)', () => {
test(
- 'I1 — parse → serialize is byte-identical for renderable extensions (image + non-image wikiembed)',
+ 'I1 — parse → serialize is byte-identical for every extension class (renderable + opaque)',
() => {
assertAcrossSeeds(
- fc.property(renderableEmbedMd, (md) => {
+ fc.property(anyEmbedMd, (md) => {
const out = normalize(mdRoundTrip(md));
expect(out).toBe(md);
}),
@@ -58,9 +54,9 @@ describe('wiki-embed conversion invariants — mdManager path (US-010)', () => {
PBT_TIMEOUT_MS,
);
- test('truly-opaque wikiembed round-trip normalizes to plain markdown link', () => {
- expect(normalize(mdRoundTrip('![[archive.xyz]]'))).toBe('[archive.xyz](archive.xyz)');
- expect(normalize(mdRoundTrip('![[archive.xyz|Download]]'))).toBe('[Download](archive.xyz)');
+ test('truly-opaque wikiembed round-trips byte-identical (no downcast to plain link)', () => {
+ expect(normalize(mdRoundTrip('![[archive.xyz]]'))).toBe('![[archive.xyz]]');
+ expect(normalize(mdRoundTrip('![[archive.xyz|Download]]'))).toBe('![[archive.xyz|Download]]');
});
test('block-context image-extension embed dispatches to jsxComponent(WikiEmbedImage)', () => {
@@ -86,14 +82,15 @@ describe('wiki-embed conversion invariants — mdManager path (US-010)', () => {
expect(props?.alias).toBe('Draft');
});
- test('US-013 — truly-opaque extensions dispatch to plain link (no sourceForm)', () => {
+ test('truly-opaque extensions dispatch to the wikiembed-tagged link chip, never a PM wikiLinkEmbed node', () => {
const json = mdManager.parse('![[archive.xyz]]');
const para = json.content?.[0];
const text = para?.content?.[0];
expect(text?.type).toBe('text');
const linkMark = text?.marks?.find((mk) => mk.type === 'link');
expect(linkMark).toBeDefined();
- expect(linkMark?.attrs?.sourceForm).toBeNull();
+ expect(linkMark?.attrs?.sourceForm).toBe('wikiembed');
+ expect(linkMark?.attrs?.target).toBe('archive.xyz');
expect(linkMark?.attrs?.href).toBe('archive.xyz');
});
diff --git a/packages/app/tests/fidelity/mark-rename-verification.test.ts b/packages/app/tests/fidelity/mark-rename-verification.test.ts
index 238874169..ea0e239bd 100644
--- a/packages/app/tests/fidelity/mark-rename-verification.test.ts
+++ b/packages/app/tests/fidelity/mark-rename-verification.test.ts
@@ -72,8 +72,8 @@ describe('renamed marks round-trip markdown correctly', () => {
expect(normalize(mdRoundTrip('*em*\n'))).toBe(normalize('*em*\n'));
});
- test('doc-start --- normalizes to *** (NG10)', () => {
- expect(normalize(mdRoundTrip('---\n'))).toBe(normalize('***\n'));
+ test('doc-start --- round-trips byte-identical (NG10 retired)', () => {
+ expect(normalize(mdRoundTrip('---\n'))).toBe(normalize('---\n'));
});
test('non-doc-start --- preserves authoring form', () => {
diff --git a/packages/app/tests/fidelity/ng-pinned.test.ts b/packages/app/tests/fidelity/ng-pinned.test.ts
index 7e34b44dd..a427c62bc 100644
--- a/packages/app/tests/fidelity/ng-pinned.test.ts
+++ b/packages/app/tests/fidelity/ng-pinned.test.ts
@@ -4,18 +4,17 @@ import { loadNgPinnedCases } from '../../../core/src/markdown/fixtures/index.ts'
const mdManager = new MarkdownManager({ extensions: sharedExtensions });
-describe('NG1 — blank-line count normalization (byte-identity)', () => {
- test('four newlines between blocks collapse to one blank line', () => {
+describe('NG1 — blank-line count between blocks (byte-identity)', () => {
+ test('four newlines between blocks round-trip byte-identical', () => {
const input = '# H\n\n\n\nP\n';
const output = mdManager.serialize(mdManager.parse(input));
- expect(output).toBe('# H\n\nP\n');
+ expect(output).toBe('# H\n\n\n\nP\n');
});
});
-describe('NG11 — ignore-typed-only doc triggers ensureNonEmptyDoc synthesis (byte-identity)', () => {
- test('yaml frontmatter alone → PM doc has one empty paragraph; serializes to empty string', () => {
- const input = '---\ntitle: X\n---\n';
- const pm = mdManager.parse(input);
+describe('NG11 — empty doc triggers ensureNonEmptyDoc synthesis (byte-identity)', () => {
+ test('empty input → PM doc has one empty paragraph; serializes to empty string', () => {
+ const pm = mdManager.parse('');
expect(pm).toMatchObject({
type: 'doc',
@@ -28,6 +27,12 @@ describe('NG11 — ignore-typed-only doc triggers ensureNonEmptyDoc synthesis (b
const output = mdManager.serialize(pm);
expect(output).toBe('');
});
+
+ test('frontmatter-shaped input no longer empties the doc (content stays visible)', () => {
+ const pm = mdManager.parse('---\ntitle: X\n---\n');
+ const output = mdManager.serialize(pm);
+ expect(output).toBe('---\n\ntitle: X\n---\n');
+ });
});
describe('NG12 — edited-node quoting normalization (idempotence probe)', () => {
diff --git a/packages/app/tests/integration/disk-authority-reconcile.test.ts b/packages/app/tests/integration/disk-authority-reconcile.test.ts
index 1aa90256d..fbbf276af 100644
--- a/packages/app/tests/integration/disk-authority-reconcile.test.ts
+++ b/packages/app/tests/integration/disk-authority-reconcile.test.ts
@@ -5,9 +5,11 @@ import { join } from 'node:path';
import {
agentPatch,
agentWriteMd,
+ createTestClients,
createTestServer,
pollUntil,
readTestDoc,
+ type TestClient,
type TestServer,
} from './test-harness.ts';
@@ -104,6 +106,51 @@ describe('PRD-6832 β L1: agent write reconciles a newer out-of-band disk edit',
expect(after).toContain('native-body-line'); // out-of-band body edit preserved
});
+ test('concurrent un-flushed CRDT edit: the L1 wholesale ingest drops it (current behavior, recorded honestly)', async () => {
+ server = await createTestServer({ debounce: 300_000, maxDebounce: 600_000 });
+ const { port, contentDir } = server;
+ const docName = `reconcile-concurrent-${randomUUID()}`;
+ const filePath = join(contentDir, `${docName}.md`);
+
+ writeFileSync(filePath, '# Doc\n\nseed-body\n', 'utf-8');
+
+ let clients: TestClient[] = [];
+ try {
+ clients = await createTestClients(port, { count: 1, docName });
+ const client = clients[0];
+ if (!client) throw new Error('client setup failed');
+ await pollUntil(() => client.ytext.toString().includes('seed-body'));
+
+ client.doc.transact(() => {
+ client.ytext.insert(client.ytext.length, '\ncrdt-unflushed-line\n');
+ });
+ const serverYtext = () =>
+ server?.instance.hocuspocus.documents.get(docName)?.getText('source').toString() ?? '';
+ await pollUntil(() => serverYtext().includes('crdt-unflushed-line'));
+ expect(readTestDoc(contentDir, docName)).not.toContain('crdt-unflushed-line');
+
+ writeFileSync(filePath, '# Doc\n\nseed-body\n\ndisk-oob-line\n', 'utf-8');
+
+ const res = await fetch(`http://localhost:${port}/api/agent-write-md`, {
+ method: 'POST',
+ headers: { 'Content-Type': 'application/json' },
+ body: JSON.stringify({ docName, markdown: 'agent-line\n', position: 'append' }),
+ });
+ expect(res.status).toBe(200);
+ const body = (await res.json()) as { warning?: { kind?: string } };
+ expect(body.warning?.kind).toBe('disk-edit-reconciled');
+
+ await pollUntil(() => serverYtext().includes('agent-line'));
+ expect(serverYtext()).toContain('disk-oob-line');
+
+ expect(serverYtext()).not.toContain('crdt-unflushed-line');
+ } finally {
+ for (const c of clients) {
+ await c.cleanup();
+ }
+ }
+ });
+
test('rename: the renamed doc carries the newer out-of-band disk content', async () => {
server = await createTestServer({ debounce: 50, maxDebounce: 200 });
const { port, contentDir } = server;
diff --git a/packages/app/tests/integration/init-load-byte-stable-corpus-coverage.test.ts b/packages/app/tests/integration/init-load-byte-stable-corpus-coverage.test.ts
deleted file mode 100644
index 4f38b15f1..000000000
--- a/packages/app/tests/integration/init-load-byte-stable-corpus-coverage.test.ts
+++ /dev/null
@@ -1,126 +0,0 @@
-/**
- * Mega-combo corpus coverage drift check.
- *
- * Asserts that the mega-combo entry's `ng[]` field, combined with the
- * explicit MEGA_COMBO_EXCLUSIONS set defined here, exhaustively covers
- * the canonical `NG_TAXONOMY` from `md-audit/src/data/ng-taxonomy.ts`.
- *
- * Failure modes this catches:
- * - A new NG slug is added to the canonical taxonomy without being
- * addressed in this corpus → drift assertion fails.
- * - A typo in mega-combo's `ng[]` field references a slug that doesn't
- * exist in the canonical taxonomy → second assertion fails.
- * - A stale entry remains in MEGA_COMBO_EXCLUSIONS after the canonical
- * taxonomy retired the corresponding slug → third assertion fails.
- *
- * If this test fails because a new canonical NG class was added, the fix is
- * one of:
- * (a) extend the mega-combo body (in
- * `_fixtures/init-load-byte-stable-corpus.ts`) to include a snippet
- * exercising the new class, AND add the new slug to mega-combo's
- * `ng[]` field;
- * (b) add the new slug to MEGA_COMBO_EXCLUSIONS below WITH a one-line
- * inline rationale explaining why it cannot compose into a single .md
- * (e.g., requires .mdx, structurally no-body, etc.);
- * (c) decide the new class isn't relevant to load-without-mutate at all
- * (rare — most NG classes are byte-shift behaviors that should be
- * absorbed by the markdownSemanticallyUnchanged short-circuit).
- *
- * NOT a behavioral test of byte-stability — that's
- * `init-load-byte-stable.test.ts`. This file is a metadata-consistency
- * check between two artifacts.
- */
-
-import { describe, expect, test } from 'bun:test';
-import { NG_TAXONOMY, type NgFloorId } from '../../../../md-audit/src/data/ng-taxonomy.ts';
-import { CORPUS, type CorpusEntry } from './_fixtures/init-load-byte-stable-corpus.ts';
-
-const MEGA_COMBO_EXCLUSIONS = new Set<NgFloorId>([
- 'storage-no-html-sanitize', // Storage-policy structural property (HTML pass-through, no sanitization). Not a content shape exercisable via byte-stability — the storage layer accepts whatever bytes the user content emits, no normalization gate. mega-combo's HTML entity content covers entity-ref-preservation, not storage-no-html-sanitize.
- 'mdx-jsx-thematic-break', // Requires .mdx — JSX-context `---` parsing is MDX-grammar-only. Covered at integration tier as standalone .mdx entry (ng5-mdx-yaml-in-jsx.mdx).
- 'mdx-inline-gfm-block-flatten', // Requires .mdx — block-GFM-inside-inline-JSX flattening is MDX-grammar-only. Covered at integration tier as standalone .mdx entry (ng6-block-inside-jsx.mdx).
- 'empty-doc-paragraph-synthesis', // Structurally "no body" — the synthesized-empty-paragraph behavior only fires when the parsed mdast body is empty. Adding body content to mega-combo destroys the property. Covered at integration tier as standalone entry (ng8-frontmatter-only.md).
- 'code-fence-quoting-canonicalize', // Edited-node quoting normalization. Not yet reviewed for inclusion in this corpus.
- 'jsx-component-single-anchor', // Requires .mdx — jsxComponent is the PM-schema anchor for user JSX components (Note/Card/Callout/...). MDX-grammar-only.
- 'jsx-inline-atom', // Requires .mdx — jsxInline is the PM atom for inline-position MDX content (`<Icon />`, `<Mention />`). MDX-grammar-only.
- 'setext-heading-contiguous-paragraph', // Taxonomy slot retained for cell-resolver matcher reference. Produces byte-identical output; no load-without-mutate coverage gap to fill.
- 'backslash-byte-identity', // Taxonomy slot retained for cell-resolver matcher continuity (position-slice extends CommonMark §2.4 ESCAPABLE_CHARS coverage). Produces byte-identical output; no load-without-mutate coverage gap to fill.
-]);
-
-function findMegaCombo(): CorpusEntry {
- const entry = CORPUS.find((c) => c.filename === 'mega-combo-8ng.md');
- if (!entry) {
- throw new Error('CORPUS invariant violated: mega-combo-8ng.md not found');
- }
- return entry;
-}
-
-describe('mega-combo corpus coverage drift check', () => {
- test('every slug in the canonical taxonomy is either covered by mega-combo or explicitly excluded', () => {
- const megaCombo = findMegaCombo();
- const covered = new Set<string>(megaCombo.ng);
- const taxonomyIds = NG_TAXONOMY.map((entry) => entry.id);
-
- const unaddressed = taxonomyIds.filter(
- (id) => !covered.has(id) && !(MEGA_COMBO_EXCLUSIONS as Set<string>).has(id),
- );
-
- if (unaddressed.length > 0) {
- throw new Error(
- `Canonical NG_TAXONOMY contains entries not addressed in mega-combo or MEGA_COMBO_EXCLUSIONS: ${unaddressed.join(', ')}. ` +
- `Either extend the mega-combo body to exercise the new class(es) and add the slugs to mega-combo.ng[], ` +
- `OR add the slugs to MEGA_COMBO_EXCLUSIONS in init-load-byte-stable-corpus-coverage.test.ts with an inline rationale.`,
- );
- }
-
- expect(unaddressed).toEqual([]);
- });
-
- test('every slug in mega-combo.ng[] exists in the canonical taxonomy (no typos)', () => {
- const megaCombo = findMegaCombo();
- const taxonomyIds = new Set<string>(NG_TAXONOMY.map((entry) => entry.id));
-
- const unknown = megaCombo.ng.filter((id) => !taxonomyIds.has(id));
-
- if (unknown.length > 0) {
- throw new Error(
- `mega-combo.ng[] references slugs that do not exist in the canonical NG_TAXONOMY: ${unknown.join(', ')}. ` +
- `Either fix the typo in the corpus fixture, or add the missing entries to md-audit/src/data/ng-taxonomy.ts.`,
- );
- }
-
- expect(unknown).toEqual([]);
- });
-
- test('every slug in MEGA_COMBO_EXCLUSIONS exists in the canonical taxonomy (no stale exclusions)', () => {
- const taxonomyIds = new Set<string>(NG_TAXONOMY.map((entry) => entry.id));
-
- const stale = [...MEGA_COMBO_EXCLUSIONS].filter((id) => !taxonomyIds.has(id));
-
- if (stale.length > 0) {
- throw new Error(
- `MEGA_COMBO_EXCLUSIONS references slugs that do not exist in the canonical NG_TAXONOMY: ${stale.join(', ')}. ` +
- `Either remove the stale exclusion (the canonical taxonomy retired the entry), ` +
- `or fix the typo in the exclusion slug.`,
- );
- }
-
- expect(stale).toEqual([]);
- });
-
- test('mega-combo.ng[] and MEGA_COMBO_EXCLUSIONS are disjoint (no contradictions)', () => {
- const megaCombo = findMegaCombo();
- const covered = new Set<string>(megaCombo.ng);
-
- const overlap = [...MEGA_COMBO_EXCLUSIONS].filter((id) => covered.has(id));
-
- if (overlap.length > 0) {
- throw new Error(
- `Contradiction: slug(s) appear in BOTH mega-combo.ng[] AND MEGA_COMBO_EXCLUSIONS: ${overlap.join(', ')}. ` +
- `An NG class is either covered by mega-combo or explicitly excluded from it; it cannot be both.`,
- );
- }
-
- expect(overlap).toEqual([]);
- });
-});
diff --git a/packages/app/tests/integration/map-driven-observer-a-crdt.test.ts b/packages/app/tests/integration/map-driven-observer-a-crdt.test.ts
new file mode 100644
index 000000000..a83f8ad53
--- /dev/null
+++ b/packages/app/tests/integration/map-driven-observer-a-crdt.test.ts
@@ -0,0 +1,98 @@
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import { setTimeout as wait } from 'node:timers/promises';
+import * as Y from 'yjs';
+import {
+ assertBridgeInvariant,
+ createTestClients,
+ createTestServer,
+ pollUntil,
+ type TestClient,
+ type TestServer,
+} from './test-harness';
+
+let server: TestServer;
+
+beforeAll(async () => {
+ server = await createTestServer();
+});
+
+afterAll(async () => {
+ await server.cleanup();
+});
+
+function appendParagraph(client: TestClient, text: string): void {
+ const paragraph = new Y.XmlElement('paragraph');
+ const ytext = new Y.XmlText();
+ ytext.applyDelta([{ insert: text }]);
+ paragraph.insert(0, [ytext]);
+ client.fragment.push([paragraph]);
+}
+
+async function awaitConvergence(clients: TestClient[], markers: string[]): Promise<void> {
+ for (const marker of markers) {
+ for (const client of clients) {
+ await pollUntil(() => client.ytext.toString().includes(marker), 5000);
+ }
+ }
+ await wait(500);
+}
+
+describe('map-driven Observer A — cross-CRDT integration', () => {
+ test('(a) two-client concurrent edits converge to byte-identical Y.Text', async () => {
+ const clients = await createTestClients(server.port, {
+ count: 2,
+ perClientOptions: { skipInvariantWatcher: true },
+ });
+ try {
+ appendParagraph(clients[0], 'MAP-DRIVEN-A-WYSIWYG');
+ appendParagraph(clients[1], 'MAP-DRIVEN-B-WYSIWYG');
+
+ await awaitConvergence(clients, ['MAP-DRIVEN-A-WYSIWYG', 'MAP-DRIVEN-B-WYSIWYG']);
+
+ const ytexts = clients.map((c) => c.ytext.toString());
+ expect(ytexts[1]).toBe(ytexts[0]);
+
+ for (const client of clients) {
+ assertBridgeInvariant(client.ytext, client.fragment);
+ }
+ } finally {
+ for (const c of clients) await c.cleanup();
+ }
+ });
+
+ test('(b) per-replica AC1 ≡ post-convergence AC1 — non-overlapping splices preserve each others bytes', async () => {
+ const clients = await createTestClients(server.port, {
+ count: 2,
+ perClientOptions: { skipInvariantWatcher: true },
+ });
+ try {
+ appendParagraph(clients[0], 'AC1-NONOVERLAP-CLIENT-A');
+ await awaitConvergence(clients, ['AC1-NONOVERLAP-CLIENT-A']);
+
+ appendParagraph(clients[0], 'AC1-NONOVERLAP-EDIT-A');
+ appendParagraph(clients[1], 'AC1-NONOVERLAP-EDIT-B');
+
+ await awaitConvergence(clients, [
+ 'AC1-NONOVERLAP-CLIENT-A',
+ 'AC1-NONOVERLAP-EDIT-A',
+ 'AC1-NONOVERLAP-EDIT-B',
+ ]);
+
+ for (const client of clients) {
+ const text = client.ytext.toString();
+ expect(text).toContain('AC1-NONOVERLAP-CLIENT-A');
+ expect(text).toContain('AC1-NONOVERLAP-EDIT-A');
+ expect(text).toContain('AC1-NONOVERLAP-EDIT-B');
+ }
+
+ const ytexts = clients.map((c) => c.ytext.toString());
+ expect(ytexts[1]).toBe(ytexts[0]);
+
+ for (const client of clients) {
+ assertBridgeInvariant(client.ytext, client.fragment);
+ }
+ } finally {
+ for (const c of clients) await c.cleanup();
+ }
+ });
+});
diff --git a/packages/app/tests/integration/single-file-mode.test.ts b/packages/app/tests/integration/single-file-mode.test.ts
index 918df632c..1d74930d1 100644
--- a/packages/app/tests/integration/single-file-mode.test.ts
+++ b/packages/app/tests/integration/single-file-mode.test.ts
@@ -10,6 +10,7 @@ import {
} from 'node:fs';
import { tmpdir } from 'node:os';
import { dirname, join } from 'node:path';
+import { MarkdownManager, sharedExtensions } from '@inkeep/open-knowledge-core';
import * as Y from 'yjs';
import { createTestClient, createTestServer, pollUntil, wait } from './test-harness';
@@ -154,8 +155,20 @@ describe('single-file mode — write-back (FR3)', () => {
});
describe('single-file mode — no rewrite on open (FR4 / G8)', () => {
- const RAW = '> q\nq2\n';
- const CANONICAL = '> q\n> q2\n';
+ const RAW = '\tfoo\n';
+ const CANONICAL = ' foo\n';
+
+ /** Parse-side tab dropper: reproduces the pre-capture engine's leading-tab
+ * expansion (a parse-time loss) while serialize stays the real engine. */
+ class TabDroppingManager extends MarkdownManager {
+ override parseWithFallback(
+ markdown: string,
+ opts?: Parameters<MarkdownManager['parseWithFallback']>[1],
+ ): ReturnType<MarkdownManager['parseWithFallback']> {
+ return super.parseWithFallback(markdown.replace(/^\t/gm, ' '), opts);
+ }
+ }
+ const tabDroppingManager = new TabDroppingManager({ extensions: sharedExtensions });
test('a reconciliation to the file’s own canonical form is suppressed; a genuine edit persists', async () => {
const contentDir = ephemeralContentDir({ 'notes.md': RAW });
@@ -165,6 +178,7 @@ describe('single-file mode — no rewrite on open (FR4 / G8)', () => {
keepContentDir: true,
singleDocRelPath: 'notes.md',
debounce: 100,
+ mdManager: tabDroppingManager,
});
const client = await createTestClient(server.port, 'notes');
try {
@@ -186,7 +200,12 @@ describe('single-file mode — no rewrite on open (FR4 / G8)', () => {
test('CONTRAST: a regular (non-ephemeral) project persists the same canonicalization on open', async () => {
const contentDir = ephemeralContentDir({ 'notes.md': RAW });
- const server = await createTestServer({ contentDir, keepContentDir: true, debounce: 100 });
+ const server = await createTestServer({
+ contentDir,
+ keepContentDir: true,
+ debounce: 100,
+ mdManager: tabDroppingManager,
+ });
const client = await createTestClient(server.port, 'notes');
try {
setSource(client.ytext, CANONICAL);
diff --git a/packages/app/tests/integration/test-harness.ts b/packages/app/tests/integration/test-harness.ts
index 5a2bc074c..31dee4441 100644
--- a/packages/app/tests/integration/test-harness.ts
+++ b/packages/app/tests/integration/test-harness.ts
@@ -88,6 +88,7 @@ export interface CreateTestServerOptions {
ephemeral?: boolean;
projectDir?: string;
singleDocRelPath?: string;
+ mdManager?: MarkdownManager;
}
export async function createTestServer(options: CreateTestServerOptions = {}): Promise<TestServer> {
@@ -128,6 +129,7 @@ export async function createTestServer(options: CreateTestServerOptions = {}): P
contentRoot: options.gitEnabled === true ? '.' : undefined,
enableTestRoutes: true,
localOpCliArgs: options.localOpCliArgs,
+ mdManager: options.mdManager,
...(ephemeral ? { ephemeral: true, singleDocRelPath: options.singleDocRelPath } : {}),
skipStateManifestCheck: true,
});
diff --git a/packages/app/tests/stress/bridge-convergence.fuzz.test.ts b/packages/app/tests/stress/bridge-convergence.fuzz.test.ts
index feead0e7a..30090ce1c 100644
--- a/packages/app/tests/stress/bridge-convergence.fuzz.test.ts
+++ b/packages/app/tests/stress/bridge-convergence.fuzz.test.ts
@@ -246,6 +246,7 @@ async function applyOp(
}
case 'external-change': {
try {
+ writeFileSync(join(server.contentDir, `${docName}.md`), op.newContent, 'utf-8');
applyExternalChange(server.instance.hocuspocus, docName, op.newContent);
} catch {}
break;
@@ -649,18 +650,27 @@ describe('bridge-convergence fuzzer (FR-17)', () => {
acceptableForPrefix.set(prefix, accepts);
}
+ const chunkGlueRe = /^M\d+-chunked-/;
+ const hasChunkedPaste = ops.some((o) => o.kind === 'chunked-source-paste');
+
const missingContent: Array<{ clientIdx: number; prefix: string }> = [];
for (let ci = 0; ci < clients.length; ci++) {
const client = clients[ci];
if (!client) continue;
- const gotLines = new Set(
- client.ytext
- .toString()
- .split('\n')
- .map((l) => l.trimEnd()),
- );
+ const gotLineList = client.ytext
+ .toString()
+ .split('\n')
+ .map((l) => l.trimEnd());
+ const gotLines = new Set(gotLineList);
for (const [prefix, accepts] of acceptableForPrefix) {
- const matched = [...accepts].some((l) => gotLines.has(l));
+ const matched = [...accepts].some(
+ (l) =>
+ gotLines.has(l) ||
+ (hasChunkedPaste &&
+ gotLineList.some(
+ (line) => line.startsWith(l) && chunkGlueRe.test(line.slice(l.length)),
+ )),
+ );
if (!matched) {
missingContent.push({ clientIdx: ci, prefix });
}
diff --git a/packages/cli/scripts/aggregate-tolerance-class-fires.ts b/packages/cli/scripts/aggregate-tolerance-class-fires.ts
new file mode 100644
index 000000000..6acabdbd5
--- /dev/null
+++ b/packages/cli/scripts/aggregate-tolerance-class-fires.ts
@@ -0,0 +1,104 @@
+#!/usr/bin/env bun
+
+import { readFileSync } from 'node:fs';
+
+interface FireRecord {
+ event: string;
+ timestamp: string;
+ class: string;
+ document: string | null;
+ codeUnitPosition: number;
+ severity: string;
+}
+
+interface ClassStats {
+ totalFires: number;
+ uniqueDocuments: number;
+ documents: Record<string, number>;
+ severity: string;
+ codeUnitPositions: { min: number; max: number; median: number };
+}
+
+function main(): void {
+ const logPath = process.argv[2];
+ if (!logPath) {
+ console.error(
+ 'Usage: bun run aggregate-tolerance-class-fires.ts <path-to-tolerance-telemetry.jsonl>',
+ );
+ process.exit(1);
+ }
+
+ const raw = readFileSync(logPath, 'utf-8');
+ const lines = raw.split('\n').filter((l) => l.trim().length > 0);
+ const records: FireRecord[] = [];
+
+ for (const line of lines) {
+ try {
+ const parsed = JSON.parse(line) as FireRecord;
+ if (parsed.event === 'bridge-tolerance-fire') {
+ records.push(parsed);
+ }
+ } catch {}
+ }
+
+ if (records.length === 0) {
+ console.log(JSON.stringify({ totalRecords: 0, classes: {}, bySeverity: {} }, null, 2));
+ return;
+ }
+
+ const byClass = new Map<string, FireRecord[]>();
+ const bySeverity = new Map<string, number>();
+
+ for (const r of records) {
+ const arr = byClass.get(r.class) ?? [];
+ arr.push(r);
+ byClass.set(r.class, arr);
+ bySeverity.set(r.severity, (bySeverity.get(r.severity) ?? 0) + 1);
+ }
+
+ const classStats: Record<string, ClassStats> = {};
+ for (const [cls, fires] of byClass) {
+ const docCounts = new Map<string, number>();
+ const positions: number[] = [];
+
+ for (const f of fires) {
+ const doc = f.document ?? '__unknown__';
+ docCounts.set(doc, (docCounts.get(doc) ?? 0) + 1);
+ if (f.codeUnitPosition >= 0) positions.push(f.codeUnitPosition);
+ }
+
+ positions.sort((a, b) => a - b);
+ const median = positions.length > 0 ? (positions[Math.floor(positions.length / 2)] ?? -1) : -1;
+
+ classStats[cls] = {
+ totalFires: fires.length,
+ uniqueDocuments: docCounts.size,
+ documents: Object.fromEntries(docCounts),
+ severity: fires[0]?.severity,
+ codeUnitPositions: {
+ min: positions.length > 0 ? (positions[0] ?? -1) : -1,
+ max: positions.length > 0 ? (positions[positions.length - 1] ?? -1) : -1,
+ median,
+ },
+ };
+ }
+
+ const timeRange =
+ records.length > 0
+ ? {
+ earliest: records.reduce((a, b) => (a.timestamp < b.timestamp ? a : b)).timestamp,
+ latest: records.reduce((a, b) => (a.timestamp > b.timestamp ? a : b)).timestamp,
+ }
+ : null;
+
+ const report = {
+ totalRecords: records.length,
+ timeRange,
+ bySeverity: Object.fromEntries(bySeverity),
+ classes: classStats,
+ };
+
+ console.log(JSON.stringify(report, null, 2));
+}
+
+main();
diff --git a/packages/core/package.json b/packages/core/package.json
index d508e8aaf..445d06abd 100644
--- a/packages/core/package.json
+++ b/packages/core/package.json
@@ -51,15 +51,20 @@
"@tiptap/extension-image": "^3.22.3",
"@tiptap/extension-italic": "^3.22.3",
"@tiptap/extension-link": "^3.22.3",
+ "@tiptap/extension-strike": "^3.22.3",
"@tiptap/extension-table": "^3.22.3",
"@tiptap/pm": "^3.22.3",
"@tiptap/starter-kit": "^3.22.3",
+ "decode-named-character-reference": "^1.3.0",
"diff-match-patch": "^1.0.5",
"mdast-util-from-markdown": "^2.0.3",
"mdast-util-math": "^3.0.0",
"mdast-util-mdx": "3.0.0",
"mdast-util-to-markdown": "^2.1.2",
"micromark-extension-mdx": "2.1.0",
+ "micromark-util-classify-character": "^2.0.1",
+ "micromark-util-decode-numeric-character-reference": "^2.0.2",
+ "micromark-util-sanitize-uri": "^2.0.1",
"micromark-util-types": "^2.0.2",
"node-diff3": "^3.2.0",
"rehype-parse": "^9.0.1",
@@ -91,7 +96,8 @@
"test:perf:regression": "bun run tests/perf/run-regression-gate.ts",
"test:health:unit": "bun test tests/health/parse-health-gate.test.ts",
"test:health": "bun run tests/health/parse-health-gate.ts tests/health/baseline.json",
- "regen-schema-snapshot": "bun run scripts/regen-schema-snapshot.ts"
+ "regen-schema-snapshot": "bun run scripts/regen-schema-snapshot.ts",
+ "gen:guard-flanking-matrix": "OK_UPDATE_GUARD_FLANKING_MATRIX=1 bun test src/markdown/guard-flanking-matrix.test.ts && biome check --write src/markdown/guard-flanking-matrix.json"
},
"devDependencies": {
"@standard-schema/spec": "^1.1.0",
@@ -110,6 +116,8 @@
"culori": "^4.0.2",
"fast-check": "^4.6.0",
"mdast-util-to-hast": "^13.2.1",
+ "micromark": "^4.0.2",
+ "micromark-util-symbol": "^2.0.1",
"tsdown": "^0.21.7",
"typescript": "^5.9.3"
}
diff --git a/packages/core/schema-snapshot.json b/packages/core/schema-snapshot.json
index cfa4a0a80..ff2962da2 100644
--- a/packages/core/schema-snapshot.json
+++ b/packages/core/schema-snapshot.json
@@ -35,6 +35,22 @@
"hasDefault": true,
"default": null
},
+ "sourceClosingFenceLength": {
+ "hasDefault": true,
+ "default": null
+ },
+ "sourceFenceIndent": {
+ "hasDefault": true,
+ "default": null
+ },
+ "sourceIndents": {
+ "hasDefault": true,
+ "default": null
+ },
+ "sourceInfoPadding": {
+ "hasDefault": true,
+ "default": null
+ },
"sourceStyle": {
"hasDefault": true,
"default": "fenced"
@@ -68,7 +84,12 @@
"isolating": false
},
"doc": {
- "attrs": {},
+ "attrs": {
+ "sourceDocBoundary": {
+ "hasDefault": true,
+ "default": null
+ }
+ },
"content": "block+",
"marks": "",
"group": "",
@@ -144,6 +165,14 @@
"hasDefault": true,
"default": false
},
+ "sourceInteriorSpacing": {
+ "hasDefault": true,
+ "default": null
+ },
+ "sourceLeadingIndent": {
+ "hasDefault": true,
+ "default": null
+ },
"sourceTrailingHashes": {
"hasDefault": true,
"default": null
@@ -351,6 +380,22 @@
"hasDefault": true,
"default": null
},
+ "sourceCheckboxChar": {
+ "hasDefault": true,
+ "default": null
+ },
+ "sourceContinuationIndent": {
+ "hasDefault": true,
+ "default": null
+ },
+ "sourceMarkerSpacing": {
+ "hasDefault": true,
+ "default": null
+ },
+ "sourceOrdinal": {
+ "hasDefault": true,
+ "default": null
+ },
"spread": {
"hasDefault": true,
"default": false
@@ -377,6 +422,10 @@
"language": {
"hasDefault": true,
"default": "latex"
+ },
+ "sourceDelimiter": {
+ "hasDefault": true,
+ "default": null
}
},
"content": "",
@@ -421,9 +470,17 @@
},
"table": {
"attrs": {
+ "sourceAlignmentPadding": {
+ "hasDefault": true,
+ "default": null
+ },
"sourceDashCounts": {
"hasDefault": true,
"default": null
+ },
+ "sourceOuterPipes": {
+ "hasDefault": true,
+ "default": null
}
},
"content": "tableRow+",
@@ -560,6 +617,18 @@
"hasDefault": true,
"default": false
},
+ "sourceAlias": {
+ "hasDefault": true,
+ "default": null
+ },
+ "sourceAnchor": {
+ "hasDefault": true,
+ "default": null
+ },
+ "sourceTarget": {
+ "hasDefault": true,
+ "default": null
+ },
"target": {
"hasDefault": true,
"default": ""
@@ -611,6 +680,10 @@
"sourceFenceLength": {
"hasDefault": true,
"default": 1
+ },
+ "sourcePadded": {
+ "hasDefault": true,
+ "default": false
}
},
"excludes": "",
@@ -726,7 +799,12 @@
"spanning": true
},
"strike": {
- "attrs": {},
+ "attrs": {
+ "sourceDelimiter": {
+ "hasDefault": true,
+ "default": "~~"
+ }
+ },
"excludes": "strike",
"group": "",
"inclusive": true,
@@ -758,11 +836,6 @@
"version": "11.0.0",
"hasOptions": false
},
- {
- "name": "remark-frontmatter",
- "version": "5.0.0",
- "hasOptions": true
- },
{
"name": "remark-mdx-agnostic",
"version": null,
diff --git a/packages/core/src/bridge/index.ts b/packages/core/src/bridge/index.ts
index 2315275e3..d96ecc3aa 100644
--- a/packages/core/src/bridge/index.ts
+++ b/packages/core/src/bridge/index.ts
@@ -54,3 +54,12 @@ export {
normalizeBridge,
} from './normalize.ts';
export { defaultScheduler, type Scheduler } from './scheduler.ts';
+export {
+ classifySeverity,
+ emitToleranceFire,
+ findFirstDivergenceIndex,
+ setToleranceTelemetryHook,
+ type ToleranceClassSeverity,
+ type ToleranceFireRecord,
+ type ToleranceTelemetryHook,
+} from './tolerance-telemetry.ts';
diff --git a/packages/core/src/bridge/normalize.test.ts b/packages/core/src/bridge/normalize.test.ts
index 80587e2f3..22970543d 100644
--- a/packages/core/src/bridge/normalize.test.ts
+++ b/packages/core/src/bridge/normalize.test.ts
@@ -1,4 +1,6 @@
import { describe, expect, test } from 'bun:test';
+import { sharedExtensions } from '../extensions/shared.ts';
+import { MarkdownManager } from '../markdown/index.ts';
import {
BRIDGE_TOLERANCE_CLASSES,
detectAppliedToleranceClasses,
@@ -23,7 +25,7 @@ describe('per-line trailing whitespace strip', () => {
});
});
-describe('3+ newline collapse to 2 (NG1 floor)', () => {
+describe('3+ newline collapse to 2 (NG1 comparison-only tolerance)', () => {
test('three newlines collapse to two', () => {
expect(normalizeBridge('P1\n\n\nP2')).toBe(normalizeBridge('P1\n\nP2'));
});
@@ -63,7 +65,7 @@ describe('trailing newline policy', () => {
});
});
-describe('leading newline policy (NG1 architectural floor at doc-start)', () => {
+describe('leading newline policy (NG1 comparison-only tolerance at doc-start)', () => {
test('no leading newline', () => {
expect(normalizeBridge('P')).toBe(normalizeBridge('P'));
});
@@ -507,6 +509,7 @@ describe('detectAppliedToleranceClasses (FR-41)', () => {
'doc-start-thematic',
'block-separator-collapse',
'table-align-row-spacing',
+ 'row-no-trailing-pipe',
'list-indent-canonical',
'ordered-list-marker-number',
'trailing-whitespace',
@@ -679,3 +682,163 @@ describe('detectAppliedToleranceClasses (FR-41)', () => {
}
});
});
+
+describe('fence-opener tracking (CommonMark 4.5: closer must match opener char)', () => {
+ const FENCED_A = '```ts\n~~~\n| a | b |\n| - | - |\n| 1 | 2\n```\n';
+ const FENCED_B = '```ts\n~~~\n| a | b |\n| - | - |\n| 1 | 2 |\n```\n';
+
+ test('does not absorb a trailing-pipe divergence inside an interleaved fence', () => {
+ expect(normalizeBridge(FENCED_A)).not.toBe(normalizeBridge(FENCED_B));
+ });
+
+ test('still tolerates the divergence outside any fence (control)', () => {
+ const a = '| a | b |\n| - | - |\n| 1 | 2\n';
+ const b = '| a | b |\n| - | - |\n| 1 | 2 |\n';
+ expect(normalizeBridge(a)).toBe(normalizeBridge(b));
+ });
+
+ test('matching-char closers still close the fence (control)', () => {
+ const a = '```ts\ncode\n```\n\n| a | b |\n| - | - |\n| 1 | 2\n';
+ const b = '```ts\ncode\n```\n\n| a | b |\n| - | - |\n| 1 | 2 |\n';
+ expect(normalizeBridge(a)).toBe(normalizeBridge(b));
+ });
+});
+
+describe('table-row trailing-pipe tolerance (row-no-trailing-pipe)', () => {
+ const mm = new MarkdownManager({ extensions: sharedExtensions });
+
+ const NON_UNIFORM = '# Notes\n\n| a | b |\n| - | - |\n| 1 | 2\n';
+ const UNIFORM = '# Notes\n\n| a | b\n| - | -\n| 1 | 2\n';
+
+ function semanticTree(md: string): unknown {
+ const strip = (node: unknown): unknown => {
+ if (Array.isArray(node)) return node.map(strip);
+ if (!node || typeof node !== 'object') return node;
+ const out: Record<string, unknown> = {};
+ for (const [k, v] of Object.entries(node as Record<string, unknown>)) {
+ if (k === 'position' || k === 'data') continue;
+ out[k] = strip(v);
+ }
+ return out;
+ };
+ return strip(mm.parseToMdast(md));
+ }
+
+ function stripTableCaptureAttrs(json: unknown): void {
+ if (!json || typeof json !== 'object') return;
+ const node = json as { type?: string; attrs?: Record<string, unknown>; content?: unknown[] };
+ if (node.type === 'table' && node.attrs) {
+ node.attrs.sourceOuterPipes = null;
+ }
+ for (const child of node.content ?? []) stripTableCaptureAttrs(child);
+ }
+
+ test('non-uniform witness ≡ its real serialize(parse) twin (β-proof: bytes differ, comparator equal, parses structurally equal)', () => {
+ const canon = mm.serialize(mm.parse(NON_UNIFORM));
+ expect(canon).not.toBe(NON_UNIFORM);
+ expect(normalizeBridge(NON_UNIFORM)).toBe(normalizeBridge(canon));
+ expect(semanticTree(NON_UNIFORM)).toEqual(semanticTree(canon));
+ });
+
+ test('uniform witness ≡ the capture-attr-loss canonical twin (every row gains a bare trailing pipe)', () => {
+ const stripped = mm.parse(UNIFORM) as unknown as Record<string, unknown>;
+ stripTableCaptureAttrs(stripped);
+ const canon = mm.serialize(stripped);
+ expect(canon).toBe('# Notes\n\n| a | b|\n| - | -|\n| 1 | 2|\n');
+ expect(normalizeBridge(UNIFORM)).toBe(normalizeBridge(canon));
+ expect(semanticTree(UNIFORM)).toEqual(semanticTree(canon));
+ });
+
+ test('padded trailing pipe on a data row tolerated in table context', () => {
+ expect(normalizeBridge('| a | b |\n| - | - |\n| 1 | 2 |')).toBe(
+ normalizeBridge('| a | b |\n| - | - |\n| 1 | 2'),
+ );
+ });
+
+ test('delimiter-row trailing pipe tolerated in table context', () => {
+ expect(normalizeBridge('| a | b\n| - | -|\n| 1 | 2')).toBe(
+ normalizeBridge('| a | b\n| - | -\n| 1 | 2'),
+ );
+ });
+
+ test('fully piped table ≡ uniformly unpiped-trailing table (per-row class application)', () => {
+ expect(normalizeBridge('| a | b |\n| - | - |\n| 1 | 2 |\n| 3 | 4 |')).toBe(
+ normalizeBridge('| a | b\n| - | -\n| 1 | 2\n| 3 | 4'),
+ );
+ });
+
+ test('normalizeBridge stays idempotent with the class active', () => {
+ for (const input of [
+ NON_UNIFORM,
+ UNIFORM,
+ mm.serialize(mm.parse(NON_UNIFORM)),
+ '| a | b |\n| - | - |\n| 1 | 2 |',
+ ]) {
+ const once = normalizeBridge(input);
+ expect(normalizeBridge(once)).toBe(once);
+ }
+ });
+
+ test('detectAppliedToleranceClasses reports the class on the witness pair', () => {
+ const canon = mm.serialize(mm.parse(NON_UNIFORM));
+ expect(detectAppliedToleranceClasses(NON_UNIFORM, canon)).toContain('row-no-trailing-pipe');
+ });
+
+ test('detectAppliedToleranceClasses does not report the class on a tableless pair', () => {
+ expect(detectAppliedToleranceClasses('plain paragraph', 'plain paragraph\n')).not.toContain(
+ 'row-no-trailing-pipe',
+ );
+ });
+
+ test('BRIDGE_TOLERANCE_CLASSES carries the class label', () => {
+ expect(BRIDGE_TOLERANCE_CLASSES).toContain('row-no-trailing-pipe');
+ });
+
+ describe('context gating — pipe-leading lines outside a table are untouched', () => {
+ test('paragraph with vs without trailing pipe is a genuine divergence', () => {
+ expect(normalizeBridge('| foo')).not.toBe(normalizeBridge('| foo |'));
+ });
+
+ test('pipe-leading paragraph elsewhere in a table-bearing doc keeps its trailing pipe', () => {
+ const withPipe = '| p | q |\n\n| a | b |\n| - | - |\n| 1 | 2 |';
+ const withoutPipe = '| p | q\n\n| a | b |\n| - | - |\n| 1 | 2 |';
+ expect(normalizeBridge(withPipe)).not.toBe(normalizeBridge(withoutPipe));
+ });
+
+ test('table-shaped lines inside a fenced code block are untouched', () => {
+ const piped = '```\n| a | b |\n| - | - |\n| 1 | 2 |\n```';
+ const unpiped = '```\n| a | b |\n| - | - |\n| 1 | 2\n```';
+ expect(normalizeBridge(piped)).not.toBe(normalizeBridge(unpiped));
+ });
+ });
+
+ describe('guards — genuine non-β-safe divergence is NOT absorbed by the tolerance set', () => {
+ test('embed downcast is a genuine divergence', () => {
+ expect(normalizeBridge('![[note]]')).not.toBe(normalizeBridge('[note](note)'));
+ });
+
+ test('touched-cell edit is a genuine divergence', () => {
+ expect(normalizeBridge('| a | b |\n| - | - |\n| 1 | 2 |')).not.toBe(
+ normalizeBridge('| a | b |\n| - | - |\n| 1 | 99 |'),
+ );
+ });
+
+ test('trailing EMPTY cell is a genuine divergence (single strip, not fixpoint)', () => {
+ expect(normalizeBridge('| a | b |\n| - | - |\n| 1 | 2 | |')).not.toBe(
+ normalizeBridge('| a | b |\n| - | - |\n| 1 | 2 |'),
+ );
+ });
+
+ test('unescaped content pipe at cell end is a genuine divergence', () => {
+ expect(normalizeBridge('| a | b |\n| - | - |\n| 1 | 2| |')).not.toBe(
+ normalizeBridge('| a | b |\n| - | - |\n| 1 | 2 |'),
+ );
+ });
+
+ test('missing LEADING pipe is out of class (genuine divergence)', () => {
+ expect(normalizeBridge('| a | b |\n| - | - |\n1 | 2')).not.toBe(
+ normalizeBridge('| a | b |\n| - | - |\n| 1 | 2 |'),
+ );
+ });
+ });
+});
diff --git a/packages/core/src/bridge/normalize.ts b/packages/core/src/bridge/normalize.ts
index 5dea59752..fe758e0d1 100644
--- a/packages/core/src/bridge/normalize.ts
+++ b/packages/core/src/bridge/normalize.ts
@@ -13,8 +13,45 @@ const ORDERED_LIST_MARKER_CANONICAL = '1';
const EMPHASIS_AROUND_CODE_RE = /\*\*\s*(`[^`]+`)\s*\*\*/g;
+const FENCE_LINE_RE = /^(`{3,}|~{3,})/;
+
+function findTableRowLines(lines: readonly string[]): Set<number> {
+ const rows = new Set<number>();
+ let fenceOpener: '`' | '~' | null = null;
+ for (let i = 0; i < lines.length; i++) {
+ const trimmed = lines[i].trimEnd();
+ const fence = FENCE_LINE_RE.exec(trimmed);
+ if (fence) {
+ const char = fence[1][0] as '`' | '~';
+ if (fenceOpener === null) {
+ fenceOpener = char;
+ continue;
+ }
+ if (fenceOpener === char) {
+ fenceOpener = null;
+ continue;
+ }
+ }
+ if (fenceOpener !== null || i === 0) continue;
+ if (!TABLE_ALIGN_ROW_RE.test(trimmed)) continue;
+ const prev = lines[i - 1].trimEnd();
+ if (!prev.startsWith('|') || TABLE_ALIGN_ROW_RE.test(prev)) continue;
+ rows.add(i - 1);
+ rows.add(i);
+ for (let j = i + 1; j < lines.length; j++) {
+ if (!lines[j].trimEnd().startsWith('|')) break;
+ rows.add(j);
+ }
+ }
+ return rows;
+}
+
+function stripTrailingPipe(line: string): string {
+ return line.slice(0, -1).trimEnd();
+}
+
export function normalizeBridge(s: string): string {
- return s
+ const lines = s
.replace(/^/, '')
.replace(/\r/g, '')
.replace(COMMONMARK_ESCAPE_RE, '$1')
@@ -23,16 +60,25 @@ export function normalizeBridge(s: string): string {
.replace(/^[*-]{3,}(?=\n|$)/, '---')
.replace(/(\n)([#>+-]|\d+[.)]|`{3,}|~{3,})/g, '\n\n$2')
.replace(/^([#>+-].*|\d+[.)].*|`{3,}.*|~{3,}.*)\n([^\n])/gm, '$1\n\n$2')
- .split('\n')
- .map((l) => {
+ .split('\n');
+ const tableRowLines = findTableRowLines(lines);
+ return lines
+ .map((l, i) => {
const trimmed = l.trimEnd();
if (TABLE_ALIGN_ROW_RE.test(trimmed)) {
- return trimmed.replace(/\s+/g, '');
+ const collapsed = trimmed.replace(/\s+/g, '');
+ return tableRowLines.has(i) &&
+ collapsed.length > 1 &&
+ collapsed.startsWith('|') &&
+ collapsed.endsWith('|')
+ ? stripTrailingPipe(collapsed)
+ : collapsed;
}
- const deindented = LIST_ITEM_INDENT_RE.test(trimmed)
- ? trimmed.replace(/^[ \t]+/, '')
- : trimmed;
- return deindented.replace(ORDERED_LIST_MARKER_RE, `${ORDERED_LIST_MARKER_CANONICAL}$1`);
+ let line = LIST_ITEM_INDENT_RE.test(trimmed) ? trimmed.replace(/^[ \t]+/, '') : trimmed;
+ if (tableRowLines.has(i) && line.length > 1 && line.startsWith('|') && line.endsWith('|')) {
+ line = stripTrailingPipe(line);
+ }
+ return line.replace(ORDERED_LIST_MARKER_RE, `${ORDERED_LIST_MARKER_CANONICAL}$1`);
})
.join('\n')
.replace(/\n{3,}/g, '\n\n')
@@ -48,6 +94,7 @@ export const BRIDGE_TOLERANCE_CLASSES = [
'doc-start-thematic',
'block-separator-collapse',
'table-align-row-spacing',
+ 'row-no-trailing-pipe',
'list-indent-canonical',
'ordered-list-marker-number',
'trailing-whitespace',
@@ -112,6 +159,12 @@ export function detectAppliedToleranceClasses(left: string, right: string): Brid
classes.push('table-align-row-spacing');
}
+ const hasUnterminatedPipeRow = (s: string): boolean =>
+ tableAlignRowMultiline.test(s) && /^\|[^\n]*[^|\s][ \t]*$/m.test(s);
+ if (hasUnterminatedPipeRow(leftLf) !== hasUnterminatedPipeRow(rightLf)) {
+ classes.push('row-no-trailing-pipe');
+ }
+
const listIndentMultiline = /^[ \t]+([-+*]|\d+[.)]|[a-zA-Z][.)])\s/m;
if (listIndentMultiline.test(leftLf) || listIndentMultiline.test(rightLf)) {
classes.push('list-indent-canonical');
diff --git a/packages/core/src/bridge/tolerance-telemetry.test.ts b/packages/core/src/bridge/tolerance-telemetry.test.ts
new file mode 100644
index 000000000..99f61bef2
--- /dev/null
+++ b/packages/core/src/bridge/tolerance-telemetry.test.ts
@@ -0,0 +1,149 @@
+import { afterEach, describe, expect, test } from 'bun:test';
+
+import { BRIDGE_TOLERANCE_CLASSES, type BridgeToleranceClass } from './normalize.ts';
+import {
+ classifySeverity,
+ emitToleranceFire,
+ findFirstDivergenceIndex,
+ setToleranceTelemetryHook,
+ type ToleranceFireRecord,
+} from './tolerance-telemetry.ts';
+
+afterEach(() => {
+ setToleranceTelemetryHook(null);
+});
+
+describe('classifySeverity', () => {
+ const pmModelCaused: BridgeToleranceClass[] = ['emphasis-around-code'];
+ const parserCaused: BridgeToleranceClass[] = [
+ 'bom',
+ 'crlf',
+ 'commonmark-escape',
+ 'leading-newline',
+ 'doc-start-thematic',
+ ];
+ const serializerCaused: BridgeToleranceClass[] = [
+ 'block-separator-collapse',
+ 'table-align-row-spacing',
+ 'row-no-trailing-pipe',
+ 'ordered-list-marker-number',
+ 'list-indent-canonical',
+ 'trailing-whitespace',
+ 'blank-line-collapse',
+ 'trailing-newline',
+ ];
+
+ test('pm-model-caused classes', () => {
+ for (const cls of pmModelCaused) {
+ expect(classifySeverity(cls)).toBe('pm-model-caused');
+ }
+ });
+
+ test('parser-caused classes', () => {
+ for (const cls of parserCaused) {
+ expect(classifySeverity(cls)).toBe('parser-caused');
+ }
+ });
+
+ test('serializer-caused classes', () => {
+ for (const cls of serializerCaused) {
+ expect(classifySeverity(cls)).toBe('serializer-caused');
+ }
+ });
+
+ test('the severity buckets partition every bridge tolerance class', () => {
+ const bucketed = [...pmModelCaused, ...parserCaused, ...serializerCaused];
+ expect(new Set(bucketed).size).toBe(bucketed.length);
+ expect([...bucketed].sort()).toEqual([...BRIDGE_TOLERANCE_CLASSES].sort());
+ });
+});
+
+describe('findFirstDivergenceIndex', () => {
+ test('identical strings return -1', () => {
+ expect(findFirstDivergenceIndex('hello', 'hello')).toBe(-1);
+ });
+
+ test('first char differs returns 0', () => {
+ expect(findFirstDivergenceIndex('abc', 'xyz')).toBe(0);
+ });
+
+ test('middle char differs', () => {
+ expect(findFirstDivergenceIndex('abcdef', 'abcXef')).toBe(3);
+ });
+
+ test('length difference at end', () => {
+ expect(findFirstDivergenceIndex('abc', 'abcdef')).toBe(3);
+ });
+
+ test('empty vs non-empty', () => {
+ expect(findFirstDivergenceIndex('', 'a')).toBe(0);
+ });
+
+ test('both empty returns -1', () => {
+ expect(findFirstDivergenceIndex('', '')).toBe(-1);
+ });
+});
+
+describe('emitToleranceFire', () => {
+ test('does nothing when no hook is set', () => {
+ emitToleranceFire(['crlf'], 'a\r\n', 'a\n', 'test-doc');
+ });
+
+ test('fires hook once per class', () => {
+ const records: ToleranceFireRecord[] = [];
+ setToleranceTelemetryHook((r) => records.push(r));
+
+ emitToleranceFire(['bom', 'crlf'], 'a\r\n', 'a\n', 'my-doc');
+
+ expect(records).toHaveLength(2);
+ expect(records[0]?.className).toBe('bom');
+ expect(records[1]?.className).toBe('crlf');
+ });
+
+ test('record has all required fields', () => {
+ const records: ToleranceFireRecord[] = [];
+ setToleranceTelemetryHook((r) => records.push(r));
+
+ emitToleranceFire(['trailing-newline'], 'hello\n', 'hello', 'notes/foo');
+
+ expect(records).toHaveLength(1);
+ const r = records[0];
+ expect(r).toBeDefined();
+ expect(r?.timestamp).toMatch(/^\d{4}-\d{2}-\d{2}T/);
+ expect(r?.className).toBe('trailing-newline');
+ expect(r?.documentName).toBe('notes/foo');
+ expect(r?.codeUnitPosition).toBe(5);
+ expect(r?.severity).toBe('serializer-caused');
+ });
+
+ test('empty classes array does not fire hook', () => {
+ let called = false;
+ setToleranceTelemetryHook(() => {
+ called = true;
+ });
+
+ emitToleranceFire([], 'same', 'same', 'doc');
+
+ expect(called).toBe(false);
+ });
+
+ test('undefined documentName passes through', () => {
+ const records: ToleranceFireRecord[] = [];
+ setToleranceTelemetryHook((r) => records.push(r));
+
+ emitToleranceFire(['crlf'], 'a\r', 'a', undefined);
+
+ expect(records[0]?.documentName).toBeUndefined();
+ });
+
+ test('clearing hook stops emissions', () => {
+ const records: ToleranceFireRecord[] = [];
+ setToleranceTelemetryHook((r) => records.push(r));
+ emitToleranceFire(['crlf'], 'a\r', 'a', 'doc');
+ expect(records).toHaveLength(1);
+
+ setToleranceTelemetryHook(null);
+ emitToleranceFire(['crlf'], 'a\r', 'a', 'doc');
+ expect(records).toHaveLength(1);
+ });
+});
diff --git a/packages/core/src/bridge/tolerance-telemetry.ts b/packages/core/src/bridge/tolerance-telemetry.ts
new file mode 100644
index 000000000..3fbde92ee
--- /dev/null
+++ b/packages/core/src/bridge/tolerance-telemetry.ts
@@ -0,0 +1,69 @@
+import type { BridgeToleranceClass } from './normalize.ts';
+
+export type ToleranceClassSeverity = 'pm-model-caused' | 'parser-caused' | 'serializer-caused';
+
+export interface ToleranceFireRecord {
+ timestamp: string;
+ className: BridgeToleranceClass;
+ documentName: string | undefined;
+ codeUnitPosition: number;
+ severity: ToleranceClassSeverity;
+}
+
+export type ToleranceTelemetryHook = (record: ToleranceFireRecord) => void;
+
+const SEVERITY_BY_CLASS = {
+ 'emphasis-around-code': 'pm-model-caused',
+ bom: 'parser-caused',
+ crlf: 'parser-caused',
+ 'commonmark-escape': 'parser-caused',
+ 'leading-newline': 'parser-caused',
+ 'doc-start-thematic': 'parser-caused',
+ 'block-separator-collapse': 'serializer-caused',
+ 'table-align-row-spacing': 'serializer-caused',
+ 'row-no-trailing-pipe': 'serializer-caused',
+ 'ordered-list-marker-number': 'serializer-caused',
+ 'list-indent-canonical': 'serializer-caused',
+ 'trailing-whitespace': 'serializer-caused',
+ 'blank-line-collapse': 'serializer-caused',
+ 'trailing-newline': 'serializer-caused',
+} as const satisfies Record<BridgeToleranceClass, ToleranceClassSeverity>;
+
+export function classifySeverity(cls: BridgeToleranceClass): ToleranceClassSeverity {
+ return SEVERITY_BY_CLASS[cls];
+}
+
+export function findFirstDivergenceIndex(left: string, right: string): number {
+ const len = Math.min(left.length, right.length);
+ for (let i = 0; i < len; i++) {
+ if (left.charCodeAt(i) !== right.charCodeAt(i)) return i;
+ }
+ if (left.length !== right.length) return len;
+ return -1;
+}
+
+let hook: ToleranceTelemetryHook | null = null;
+
+export function setToleranceTelemetryHook(h: ToleranceTelemetryHook | null): void {
+ hook = h;
+}
+
+export function emitToleranceFire(
+ classes: readonly BridgeToleranceClass[],
+ left: string,
+ right: string,
+ documentName: string | undefined,
+): void {
+ if (!hook || classes.length === 0) return;
+ const codeUnitPosition = findFirstDivergenceIndex(left, right);
+ const timestamp = new Date().toISOString();
+ for (const cls of classes) {
+ hook({
+ timestamp,
+ className: cls,
+ documentName,
+ codeUnitPosition,
+ severity: classifySeverity(cls),
+ });
+ }
+}
diff --git a/packages/core/src/extensions/code-block-fidelity.ts b/packages/core/src/extensions/code-block-fidelity.ts
index 1442f0563..021509e54 100644
--- a/packages/core/src/extensions/code-block-fidelity.ts
+++ b/packages/core/src/extensions/code-block-fidelity.ts
@@ -10,6 +10,10 @@ export const CodeBlockFidelity = CodeBlock.extend({
fenceLength: { default: 3 },
meta: { default: null },
sourceStyle: { default: 'fenced' },
+ sourceClosingFenceLength: { default: null, rendered: false },
+ sourceFenceIndent: { default: null, rendered: false },
+ sourceInfoPadding: { default: null, rendered: false },
+ sourceIndents: { default: null, rendered: false },
};
},
});
diff --git a/packages/core/src/extensions/code-mark-fidelity.ts b/packages/core/src/extensions/code-mark-fidelity.ts
index 158debba0..21f82352b 100644
--- a/packages/core/src/extensions/code-mark-fidelity.ts
+++ b/packages/core/src/extensions/code-mark-fidelity.ts
@@ -8,6 +8,7 @@ export const CodeMarkFidelity = Code.extend({
...this.parent?.(),
sourceFenceChar: { default: '`' },
sourceFenceLength: { default: 1 },
+ sourcePadded: { default: false, rendered: false },
};
},
});
diff --git a/packages/core/src/extensions/doc-fidelity.ts b/packages/core/src/extensions/doc-fidelity.ts
new file mode 100644
index 000000000..9281ad11a
--- /dev/null
+++ b/packages/core/src/extensions/doc-fidelity.ts
@@ -0,0 +1,16 @@
+import { Extension } from '@tiptap/core';
+
+export const DocFidelity = Extension.create({
+ name: 'docFidelity',
+
+ addGlobalAttributes() {
+ return [
+ {
+ types: ['doc'],
+ attributes: {
+ sourceDocBoundary: { default: null, rendered: false },
+ },
+ },
+ ];
+ },
+});
diff --git a/packages/core/src/extensions/heading-fidelity.ts b/packages/core/src/extensions/heading-fidelity.ts
index 6f58479eb..d7efe9a8e 100644
--- a/packages/core/src/extensions/heading-fidelity.ts
+++ b/packages/core/src/extensions/heading-fidelity.ts
@@ -10,6 +10,8 @@ export const HeadingFidelity = Heading.extend({
sourceTrailingHashes: { default: null },
sourceUnderlineLength: { default: null },
sourceContiguousNext: { default: false },
+ sourceLeadingIndent: { default: null, rendered: false },
+ sourceInteriorSpacing: { default: null, rendered: false },
};
},
});
diff --git a/packages/core/src/extensions/list.ts b/packages/core/src/extensions/list.ts
index c892ef982..dca5a9bcc 100644
--- a/packages/core/src/extensions/list.ts
+++ b/packages/core/src/extensions/list.ts
@@ -263,6 +263,10 @@ export const ListItemNode = Node.create({
return {
checked: { default: null },
spread: { default: false },
+ sourceMarkerSpacing: { default: null, rendered: false },
+ sourceOrdinal: { default: null, rendered: false },
+ sourceCheckboxChar: { default: null, rendered: false },
+ sourceContinuationIndent: { default: null, rendered: false },
};
},
diff --git a/packages/core/src/extensions/math-inline.ts b/packages/core/src/extensions/math-inline.ts
index d41afc663..66c5772c2 100644
--- a/packages/core/src/extensions/math-inline.ts
+++ b/packages/core/src/extensions/math-inline.ts
@@ -21,6 +21,7 @@ export const MathInline = Node.create({
formula: { default: '' },
id: { default: null },
language: { default: 'latex' },
+ sourceDelimiter: { default: null, rendered: false },
};
},
diff --git a/packages/core/src/extensions/shared.ts b/packages/core/src/extensions/shared.ts
index da806a319..0fcce76dd 100644
--- a/packages/core/src/extensions/shared.ts
+++ b/packages/core/src/extensions/shared.ts
@@ -6,6 +6,7 @@ import { CodeBlockFidelity } from './code-block-fidelity.ts';
import { CodeMarkFidelity } from './code-mark-fidelity.ts';
import { CommentBlock } from './comment-block.ts';
import { CommentMark } from './comment-mark.ts';
+import { DocFidelity } from './doc-fidelity.ts';
import { EmphasisFidelity, StrongFidelity } from './emphasis-fidelity.ts';
import { EscapeMark } from './escape-mark.ts';
import { FootnoteDefinition } from './footnote-definition.ts';
@@ -23,6 +24,7 @@ import { List, ListItem } from './list.ts';
import { MathInline } from './math-inline.ts';
import { RawMdxFallback } from './raw-mdx-fallback.ts';
import { SourceLiteralMark } from './source-literal-mark.ts';
+import { StrikeFidelity } from './strike-fidelity.ts';
import { TableCellFidelity, TableFidelity, TableHeaderFidelity } from './table-fidelity.ts';
import { Tag } from './tag.ts';
import { ThematicBreakFidelity } from './thematic-break-fidelity.ts';
@@ -41,6 +43,7 @@ export const sharedExtensions = [
ListItem,
EmphasisFidelity,
StrongFidelity,
+ StrikeFidelity,
CodeMarkFidelity,
CodeBlockFidelity,
BlockquoteFidelity,
@@ -51,6 +54,7 @@ export const sharedExtensions = [
LinkRefDefFidelity,
HardBreakFidelity,
EscapeMark,
+ DocFidelity,
SourceLiteralMark,
StarterKit.configure({
undoRedo: false,
@@ -59,6 +63,7 @@ export const sharedExtensions = [
listItem: false,
italic: false,
bold: false,
+ strike: false,
code: false,
codeBlock: false,
heading: false,
diff --git a/packages/core/src/extensions/strike-fidelity.ts b/packages/core/src/extensions/strike-fidelity.ts
new file mode 100644
index 000000000..296177bd6
--- /dev/null
+++ b/packages/core/src/extensions/strike-fidelity.ts
@@ -0,0 +1,12 @@
+import Strike from '@tiptap/extension-strike';
+
+export const StrikeFidelity = Strike.extend({
+ priority: 60,
+
+ addAttributes() {
+ return {
+ ...this.parent?.(),
+ sourceDelimiter: { default: '~~', rendered: false },
+ };
+ },
+});
diff --git a/packages/core/src/extensions/table-fidelity.ts b/packages/core/src/extensions/table-fidelity.ts
index 259c11444..f85c22edc 100644
--- a/packages/core/src/extensions/table-fidelity.ts
+++ b/packages/core/src/extensions/table-fidelity.ts
@@ -7,6 +7,8 @@ export const TableFidelity = Table.extend({
return {
...this.parent?.(),
sourceDashCounts: { default: null },
+ sourceOuterPipes: { default: null, rendered: false },
+ sourceAlignmentPadding: { default: null, rendered: false },
};
},
});
diff --git a/packages/core/src/extensions/wiki-link.ts b/packages/core/src/extensions/wiki-link.ts
index 45b8c84a1..ea96e40cd 100644
--- a/packages/core/src/extensions/wiki-link.ts
+++ b/packages/core/src/extensions/wiki-link.ts
@@ -77,6 +77,18 @@ export const WikiLink = Node.create({
resolved: {
default: false,
},
+ sourceTarget: {
+ default: null,
+ rendered: false,
+ },
+ sourceAnchor: {
+ default: null,
+ rendered: false,
+ },
+ sourceAlias: {
+ default: null,
+ rendered: false,
+ },
};
},
diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts
index 826fdc5d5..5d97a741b 100644
--- a/packages/core/src/index.ts
+++ b/packages/core/src/index.ts
@@ -18,11 +18,13 @@ export {
type BridgeMergeContentLossWhich,
type BridgeToleranceClass,
bindFrontmatterDoc,
+ classifySeverity,
type DiffChange,
defaultScheduler,
detectAppliedToleranceClasses,
detectFmRegion,
diffLinesFast,
+ emitToleranceFire,
type FmEditError,
type FmEditResult,
FORM_WRITE_ORIGIN,
@@ -36,6 +38,7 @@ export {
type FrontmatterBindingUnsubscribe,
type FrontmatterDocProvider,
type FrontmatterSnapshot,
+ findFirstDivergenceIndex,
fnv1aDigest,
type InvariantViolation,
MAX_FM_REGION_BYTES,
@@ -48,6 +51,10 @@ export {
readFmMap,
readFmRegionWithError,
type Scheduler,
+ setToleranceTelemetryHook,
+ type ToleranceClassSeverity,
+ type ToleranceFireRecord,
+ type ToleranceTelemetryHook,
toBridgeInvariantLog,
} from './bridge/index.ts';
export {
diff --git a/packages/core/src/markdown/autolink-promotion.ts b/packages/core/src/markdown/autolink-promotion.ts
index a2a9067db..7745bb11c 100644
--- a/packages/core/src/markdown/autolink-promotion.ts
+++ b/packages/core/src/markdown/autolink-promotion.ts
@@ -1,8 +1,9 @@
import type { Link, Parent, Text } from 'mdast';
+import { deriveFragmentPosition } from './promoter-position.ts';
const AUTOLINK_IN_TEXT_RE = /<([a-zA-Z][a-zA-Z0-9+.-]*:[^\s<>]+)>/g;
-export function promoteInParent(parent: Parent): void {
+export function promoteInParent(parent: Parent, source: string = ''): void {
const newChildren: Parent['children'] = [];
let changed = false;
@@ -26,16 +27,34 @@ export function promoteInParent(parent: Parent): void {
const matchStart = match.index;
if (matchStart > lastIndex) {
- segments.push({ type: 'text', value: text.slice(lastIndex, matchStart) } as Text);
+ const lead: Text = { type: 'text', value: text.slice(lastIndex, matchStart) };
+ const pos = deriveFragmentPosition(source, child as Text, lastIndex, matchStart);
+ if (pos) lead.position = pos;
+ segments.push(lead);
}
+ const innerText: Text = { type: 'text', value: uri };
+ const innerPos = deriveFragmentPosition(
+ source,
+ child as Text,
+ matchStart + 1,
+ matchStart + 1 + uri.length,
+ );
+ if (innerPos) innerText.position = innerPos;
const linkNode: Link & { data: { sourceStyle: string } } = {
type: 'link',
url: uri,
title: null,
- children: [{ type: 'text', value: uri } as Text],
+ children: [innerText],
data: { sourceStyle: 'autolink' },
};
+ const linkPos = deriveFragmentPosition(
+ source,
+ child as Text,
+ matchStart,
+ matchStart + fullMatch.length,
+ );
+ if (linkPos) linkNode.position = linkPos;
segments.push(linkNode);
lastIndex = matchStart + fullMatch.length;
@@ -46,7 +65,10 @@ export function promoteInParent(parent: Parent): void {
newChildren.push(child);
} else {
if (lastIndex < text.length) {
- segments.push({ type: 'text', value: text.slice(lastIndex) } as Text);
+ const tail: Text = { type: 'text', value: text.slice(lastIndex) };
+ const pos = deriveFragmentPosition(source, child as Text, lastIndex, text.length);
+ if (pos) tail.position = pos;
+ segments.push(tail);
}
newChildren.push(...segments);
}
diff --git a/packages/core/src/markdown/autolink-void-html-guard.test.ts b/packages/core/src/markdown/autolink-void-html-guard.test.ts
index 08848b87e..66eed34af 100644
--- a/packages/core/src/markdown/autolink-void-html-guard.test.ts
+++ b/packages/core/src/markdown/autolink-void-html-guard.test.ts
@@ -1,7 +1,9 @@
import { describe, expect, test } from 'bun:test';
import type { JSONContent } from '@tiptap/core';
import { sharedExtensions } from '../extensions/shared.ts';
+import { protectFromMdx } from './autolink-void-html-guard.ts';
import { MarkdownManager } from './index.ts';
+import { mdastToHtml } from './mdast-to-html.ts';
const mdManager = new MarkdownManager({ extensions: sharedExtensions });
@@ -274,3 +276,115 @@ describe('R6a: safeText idempotency for §2.4 ambiguous chars', () => {
expect(() => mdManager.parse('\\\\{a}\n')).not.toThrow();
});
});
+
+describe('R23 guard: angle-bracket link destinations', () => {
+ function findNodes(node: unknown, type: string): Array<Record<string, unknown>> {
+ const out: Array<Record<string, unknown>> = [];
+ const rec = node as Record<string, unknown>;
+ if (rec && typeof rec === 'object') {
+ if (rec.type === type) out.push(rec);
+ const children = rec.children;
+ if (Array.isArray(children)) {
+ for (const child of children) out.push(...findNodes(child, type));
+ }
+ }
+ return out;
+ }
+
+ function expectByteStable(md: string): void {
+ const r1 = roundTrip(md);
+ expect(r1).toBe(md);
+ expect(roundTrip(r1)).toBe(r1);
+ }
+
+ test('[link](</my uri>) parses to a link node with the space-bearing url', () => {
+ const tree = mdManager.parseToMdast('[link](</my uri>)\n');
+ const links = findNodes(tree, 'link');
+ expect(links.length).toBe(1);
+ expect(links[0].url).toBe('/my uri');
+ });
+
+ test('the destination < is not PUA-protected when the resource forms', () => {
+ const src = '[link](</my uri>)';
+ expect(protectFromMdx(src)).toBe(src);
+ });
+
+ test('angle destination link renders href /my%20uri', () => {
+ const html = mdastToHtml(mdManager.parseToMdast('[link](</my uri>)\n'));
+ expect(html).toContain('href="/my%20uri"');
+ expect(html).toContain('>link</a>');
+ });
+
+ test('angle destination link round-trips byte-identically and idempotently', () => {
+ expectByteStable('[link](</my uri>)\n');
+ });
+
+ test('image form ![img](</a b.png>) parses to an image and round-trips', () => {
+ const tree = mdManager.parseToMdast('![img](</a b.png>)\n');
+ const images = [...findNodes(tree, 'image'), ...findNodes(tree, 'mdxJsxFlowElement')];
+ expect(images.length).toBeGreaterThanOrEqual(1);
+ expectByteStable('![img](</a b.png>)\n');
+ });
+
+ test('uppercase-name space-bearing destination forms a link', () => {
+ const tree = mdManager.parseToMdast('[x](<My uri>)\n');
+ const links = findNodes(tree, 'link');
+ expect(links.length).toBe(1);
+ expect(links[0].url).toBe('My uri');
+ expectByteStable('[x](<My uri>)\n');
+ });
+
+ const mustStayGuarded: Array<[string, string]> = [
+ ['[link](</my uri', 'mid-typing: unclosed destination'],
+ ['[link](</my uri>', 'mid-typing: destination closed, resource open'],
+ ['[link](</my uri> junk)', 'resource fails after destination'],
+ ['x](</a b>)', 'no label-start before ]('],
+ ['link](</a b>)', 'label-start deleted mid-edit'],
+ ['a](<', 'bare ](< at EOF'],
+ ['a](</', 'bare ](</ at EOF'],
+ ['[x] y](</a b>)', 'label already closed before ]('],
+ ['[[page]](</a b>)', 'wikilink-consumed brackets before ]('],
+ ['[a\\](</b c>)', 'escaped ] cannot end a label'],
+ ['`a[b` ](</x y>)', 'code-span-consumed [ before ]('],
+ ['[^fn](</a b>)', 'footnote-shaped label'],
+ ['[a][b](</c d>)', 'reference-link-shaped label'],
+ ['[a](</x \\> y>)', 'escaped > inside destination'],
+ ['[a](</x <i:j> k>)', 'autolink inside destination'],
+ ['[a](<x\ny>)', 'destination spans a line ending'],
+ ['[link](</my uri> "t")', 'title after destination'],
+ ['[link]( </my uri>)', 'whitespace between ( and <'],
+ ['[a](</b c> )', 'whitespace between > and )'],
+ [`[a](<${' '.repeat(2000)}x y>)`, 'destination exceeding the scan cap'],
+ ];
+
+ for (const [input, label] of mustStayGuarded) {
+ test(`does not throw: ${label}`, () => {
+ expect(() => mdManager.parse(`${input}\n`)).not.toThrow();
+ });
+ }
+
+ test('conservative rejections form no link and stay byte-stable', () => {
+ for (const src of [
+ '[link](</my uri> "t")\n',
+ '[link]( </my uri>)\n',
+ '[a](</b c> )\n',
+ 'x](</a b>)\n',
+ 'link](</a b>)\n',
+ ]) {
+ const links = findNodes(mdManager.parseToMdast(src), 'link');
+ expect(links.length).toBe(0);
+ expectByteStable(src);
+ }
+ });
+
+ test('rejected destination < stays PUA-protected', () => {
+ expect(protectFromMdx('x](</a b>)')).toContain('\uE000');
+ expect(protectFromMdx('[link](</my uri')).toContain('\uE000');
+ });
+
+ test('no-space angle destinations keep their current byte-stable behavior', () => {
+ expectByteStable('[t](<u>)\n');
+ expectByteStable('[link](<foo%20bar>)\n');
+ expectByteStable('[link](</no-space>)\n');
+ });
+});
diff --git a/packages/core/src/markdown/autolink-void-html-guard.ts b/packages/core/src/markdown/autolink-void-html-guard.ts
index 4f863ab10..101069790 100644
--- a/packages/core/src/markdown/autolink-void-html-guard.ts
+++ b/packages/core/src/markdown/autolink-void-html-guard.ts
@@ -7,6 +7,27 @@ const GUARD_COLON = '\uE002';
const GUARD_AT = '\uE003';
const GUARD_OPEN_BRACE = '\uE004';
+const LITERAL_SENTINEL_ESCAPES: ReadonlyArray<readonly [string, string]> = [
+ [GUARD_OPEN, '\uE005'],
+ [GUARD_CLOSE, '\uE006'],
+ [GUARD_COLON, '\uE007'],
+ [GUARD_AT, '\uE008'],
+ [GUARD_OPEN_BRACE, '\uE009'],
+];
+const HAS_LITERAL_SENTINEL_RE = /[\uE000-\uE004]/;
+const HAS_ESCAPED_LITERAL_SENTINEL_RE = /[\uE005-\uE009]/;
+
+export const R23_GUARD_SUBSTITUTIONS: ReadonlyArray<{ from: string; to: string }> = [
+ { from: '<', to: GUARD_OPEN },
+ { from: '>', to: GUARD_CLOSE },
+ { from: ':', to: GUARD_COLON },
+ { from: '@', to: GUARD_AT },
+ { from: '{', to: GUARD_OPEN_BRACE },
+];
+
+export const R23_SENTINEL_ESCAPE_SUBSTITUTIONS: ReadonlyArray<{ from: string; to: string }> =
+ LITERAL_SENTINEL_ESCAPES.map(([from, to]) => ({ from, to }));
+
const AUTOLINK_RE = /<([a-zA-Z][a-zA-Z0-9+.-]*:[^\s<>]+)>/g;
const HTML_COMMENT_RE = /<!--[\s\S]*?-->/g;
@@ -76,6 +97,57 @@ function indexGreaterThan(source: string): number[] {
return positions;
}
+const ANGLE_DEST_SCAN_CAP = 1024;
+
+function isAngleBracketDestinationOpen(offset: number, result: string): boolean {
+ if (result[offset - 1] !== '(' || result[offset - 2] !== ']') return false;
+ const labelEnd = offset - 2;
+ let bs = 0;
+ for (let j = labelEnd - 1; j >= 0 && result[j] === '\\'; j--) bs++;
+ if (bs % 2 === 1) return false;
+
+ let foundLabelStart = false;
+ const scanFloor = Math.max(0, labelEnd - ANGLE_DEST_SCAN_CAP);
+ for (let j = labelEnd - 1; j >= scanFloor; j--) {
+ const ch = result[j];
+ if (ch === '\n' || ch === '\r' || ch === '`') return false;
+ if (ch !== '[' && ch !== ']') continue;
+ let k = 0;
+ for (let m = j - 1; m >= 0 && result[m] === '\\'; m--) k++;
+ if (k % 2 === 1) continue;
+ if (ch === ']') return false;
+ if (result[j + 1] === '^') return false;
+ if (result[j - 1] === ']') return false;
+ foundLabelStart = true;
+ break;
+ }
+ if (!foundLabelStart) return false;
+
+ let sawWhitespace = false;
+ let destClose = -1;
+ const scanCeil = Math.min(result.length, offset + 1 + ANGLE_DEST_SCAN_CAP);
+ for (let j = offset + 1; j < scanCeil; j++) {
+ const ch = result[j];
+ if (ch === '>') {
+ destClose = j;
+ break;
+ }
+ if (ch === '<' || ch === '\\' || ch === '\n' || ch === '\r') return false;
+ if (
+ ch === GUARD_OPEN ||
+ ch === GUARD_CLOSE ||
+ ch === GUARD_COLON ||
+ ch === GUARD_AT ||
+ ch === GUARD_OPEN_BRACE
+ ) {
+ return false;
+ }
+ if (ch === ' ' || ch === '\t') sawWhitespace = true;
+ }
+ if (destClose === -1 || !sawWhitespace) return false;
+ return result[destClose + 1] === ')';
+}
+
function isSelfClosingTagAt(
offset: number,
result: string,
@@ -94,6 +166,12 @@ function isSelfClosingTagAt(
export function protectFromMdx(source: string): string {
let result = source;
+ if (HAS_LITERAL_SENTINEL_RE.test(result)) {
+ for (const [sentinel, escapeChar] of LITERAL_SENTINEL_ESCAPES) {
+ result = result.replaceAll(sentinel, escapeChar);
+ }
+ }
+
result = result.replace(HTML_COMMENT_RE, (match) => {
return match.replace(/</g, GUARD_OPEN).replace(/>/g, GUARD_CLOSE);
});
@@ -129,6 +207,8 @@ export function protectFromMdx(source: string): string {
const greaterThanOffsets = indexGreaterThan(result);
result = result.replace(/</g, (match, offset) => {
+ if (isAngleBracketDestinationOpen(offset, result)) return match;
+
const lookahead = result.slice(offset, offset + 256);
if (lookahead[1] === '/') {
@@ -229,13 +309,7 @@ export function protectFromMdx(source: string): string {
}
function hasSentinels(s: string): boolean {
- return (
- s.includes('\uE000') ||
- s.includes('\uE001') ||
- s.includes('\uE002') ||
- s.includes('\uE003') ||
- s.includes('\uE004')
- );
+ return HAS_LITERAL_SENTINEL_RE.test(s) || HAS_ESCAPED_LITERAL_SENTINEL_RE.test(s);
}
export function restoreFromMdx() {
@@ -254,15 +328,27 @@ export function restoreFromMdx() {
if (typeof rec.alt === 'string' && hasSentinels(rec.alt)) {
rec.alt = restoreString(rec.alt);
}
+ if (typeof rec.lang === 'string' && hasSentinels(rec.lang)) {
+ rec.lang = restoreString(rec.lang);
+ }
+ if (typeof rec.meta === 'string' && hasSentinels(rec.meta)) {
+ rec.meta = restoreString(rec.meta);
+ }
});
};
}
function restoreString(s: string): string {
- return s
+ let out = s
.replaceAll(GUARD_OPEN, '<')
.replaceAll(GUARD_CLOSE, '>')
.replaceAll(GUARD_COLON, ':')
.replaceAll(GUARD_AT, '@')
.replaceAll(GUARD_OPEN_BRACE, '{');
+ if (HAS_ESCAPED_LITERAL_SENTINEL_RE.test(out)) {
+ for (const [sentinel, escapeChar] of LITERAL_SENTINEL_ESCAPES) {
+ out = out.replaceAll(escapeChar, sentinel);
+ }
+ }
+ return out;
}
diff --git a/packages/core/src/markdown/backslash-escape-guard.ts b/packages/core/src/markdown/backslash-escape-guard.ts
index b7f3b7faa..69f415f4b 100644
--- a/packages/core/src/markdown/backslash-escape-guard.ts
+++ b/packages/core/src/markdown/backslash-escape-guard.ts
@@ -5,6 +5,10 @@ import { protectPattern } from './entity-ref-guard.ts';
const BACKSLASH_ESCAPE_PUA_MARK = '';
+export const BACKSLASH_GUARD_SUBSTITUTIONS: ReadonlyArray<{ from: string; to: string }> = [
+ { from: '\\', to: BACKSLASH_ESCAPE_PUA_MARK },
+];
+
const BACKSLASH_ESCAPE_RE = /\\</g;
export function encodeBackslashEscapes(source: string): string {
diff --git a/packages/core/src/markdown/comment-promoter.ts b/packages/core/src/markdown/comment-promoter.ts
index 11e9217d5..193e286b0 100644
--- a/packages/core/src/markdown/comment-promoter.ts
+++ b/packages/core/src/markdown/comment-promoter.ts
@@ -1,13 +1,16 @@
import type { Nodes, Paragraph, PhrasingContent, Root, RootContent, Text } from 'mdast';
import { SKIP, visit } from 'unist-util-visit';
+import type { VFile } from 'vfile';
import type { CommentBlockMdast, CommentMdast } from './mdast-augmentation.ts';
+import { deriveFragmentPosition } from './promoter-position.ts';
const PERCENT_COMMENT_RE = /(?<!%)%%([^\n]*?[^\n%])%%(?!%)/g;
const HTML_COMMENT_INLINE_RE = /<!--([\s\S]*?)-->/g;
export function commentPromoterPlugin() {
- return (tree: Root) => {
+ return (tree: Root, file: VFile) => {
+ const source = typeof file.value === 'string' ? file.value : '';
handleBlockCommentsAtRoot(tree);
visit(tree, 'text', (node: Text, index, parent) => {
@@ -23,18 +26,27 @@ export function commentPromoterPlugin() {
let cursor = 0;
for (const match of matches) {
if (match.start > cursor) {
- replacements.push({ type: 'text', value: value.slice(cursor, match.start) });
+ const lead: Text = { type: 'text', value: value.slice(cursor, match.start) };
+ const pos = deriveFragmentPosition(source, node, cursor, match.start);
+ if (pos) lead.position = pos;
+ replacements.push(lead);
}
+ const innerText: Text = { type: 'text', value: match.body };
const commentNode: CommentMdast = {
type: 'comment',
- children: [{ type: 'text', value: match.body }],
+ children: [innerText],
data: { sourceForm: match.sourceForm },
};
+ const commentPos = deriveFragmentPosition(source, node, match.start, match.end);
+ if (commentPos) commentNode.position = commentPos;
replacements.push(commentNode as unknown as PhrasingContent);
cursor = match.end;
}
if (cursor < value.length) {
- replacements.push({ type: 'text', value: value.slice(cursor) });
+ const tail: Text = { type: 'text', value: value.slice(cursor) };
+ const pos = deriveFragmentPosition(source, node, cursor, value.length);
+ if (pos) tail.position = pos;
+ replacements.push(tail);
}
const arr = (parent as { children: PhrasingContent[] }).children;
diff --git a/packages/core/src/markdown/doc-start-thematic-fix.test.ts b/packages/core/src/markdown/doc-start-thematic-fix.test.ts
deleted file mode 100644
index 3f5f6f9a3..000000000
--- a/packages/core/src/markdown/doc-start-thematic-fix.test.ts
+++ /dev/null
@@ -1,63 +0,0 @@
-import { describe, expect, test } from 'bun:test';
-import { MarkdownManager, sharedExtensions } from '@inkeep/open-knowledge-core';
-import type { JSONContent } from '@tiptap/core';
-
-const mdManager = new MarkdownManager({ extensions: sharedExtensions });
-
-function roundTrip(md: string): string {
- return mdManager.serialize(mdManager.parse(md));
-}
-
-function findNodeType(json: JSONContent, type: string): boolean {
- if (json.type === type) return true;
- for (const child of json.content ?? []) {
- if (findNodeType(child, type)) return true;
- }
- return false;
-}
-
-describe('doc-start-thematic-fix: empty yaml → thematicBreak', () => {
- test('---\\n\\n--- (empty yaml) produces thematicBreak PM nodes', () => {
- const json = mdManager.parse('---\n\n---');
- expect(findNodeType(json, 'thematicBreak')).toBe(true);
- });
-
- test('---\\n--- (empty yaml, no blank) produces thematicBreak PM nodes', () => {
- const json = mdManager.parse('---\n---');
- expect(findNodeType(json, 'thematicBreak')).toBe(true);
- });
-
- test('empty yaml round-trip is idempotent', () => {
- const r1 = roundTrip('---\n\n---');
- const r2 = roundTrip(r1);
- expect(r2).toBe(r1);
- });
-});
-
-describe('doc-start-thematic-fix: real frontmatter is untouched', () => {
- test('yaml with content stays as yaml (ignored by PM → empty doc with paragraph)', () => {
- const json = mdManager.parse('---\ntitle: x\n---\n\nbody\n');
- const hasBody = JSON.stringify(json).includes('body');
- expect(hasBody).toBe(true);
- });
-
- test('real frontmatter + body round-trips without frontmatter (stripped by PM layer)', () => {
- const r = roundTrip('---\ntitle: x\n---\n\nbody\n');
- expect(r.trim()).toBe('body');
- });
-});
-
-describe('doc-start-thematic-fix: unclosed --- fallback', () => {
- test('single --- (unclosed) is already thematicBreak by remark-frontmatter', () => {
- const json = mdManager.parse('---\n');
- expect(findNodeType(json, 'thematicBreak')).toBe(true);
- });
-
- test('--- followed by content (no closing ---) round-trips with NG10 normalization', () => {
- const r1 = roundTrip('---\n\ntext\n');
- expect(r1).toContain('***');
- expect(r1).toContain('text');
- const r2 = roundTrip(r1);
- expect(r2).toBe(r1);
- });
-});
diff --git a/packages/core/src/markdown/doc-start-thematic-fix.ts b/packages/core/src/markdown/doc-start-thematic-fix.ts
deleted file mode 100644
index a6c8dc0c3..000000000
--- a/packages/core/src/markdown/doc-start-thematic-fix.ts
+++ /dev/null
@@ -1,44 +0,0 @@
-import type { Root, ThematicBreak } from 'mdast';
-import type { VFile } from 'vfile';
-
-export function applyDocStartThematicFix(tree: Root, file: VFile): void {
- if (tree.children.length === 0) return;
-
- const first = tree.children[0];
- if (first.type !== 'yaml') return;
-
- const value = (first as { value?: string }).value ?? '';
- if (value.trim() !== '') return;
-
- const startOff = first.position?.start?.offset;
- if (startOff !== 0) return;
-
- const endOff = first.position?.end?.offset;
- if (typeof endOff !== 'number') return;
-
- const source = typeof file.value === 'string' ? file.value : '';
- if (!source) return;
-
- const slice = source.slice(startOff, endOff);
- const lines = slice.split('\n');
-
- const hrPattern = /^-{3,}\s*$/;
- const hrLines = lines.filter((l) => hrPattern.test(l));
-
- if (hrLines.length === 0) return;
-
- const replacements: ThematicBreak[] = hrLines.map((raw) => ({
- type: 'thematicBreak' as const,
- data: { sourceRaw: raw.replace(/\s+$/, '') },
- }));
-
- if (replacements.length > 0) {
- const firstLineLen = hrLines[0].length;
- replacements[0].position = {
- start: { line: 1, column: 1, offset: 0 },
- end: { line: 1, column: firstLineLen + 1, offset: firstLineLen },
- };
- }
-
- tree.children.splice(0, 1, ...replacements);
-}
diff --git a/packages/core/src/markdown/entity-ref-guard.test.ts b/packages/core/src/markdown/entity-ref-guard.test.ts
index e505d04a0..732320661 100644
--- a/packages/core/src/markdown/entity-ref-guard.test.ts
+++ b/packages/core/src/markdown/entity-ref-guard.test.ts
@@ -253,6 +253,32 @@ describe('restoreEntityRefsPlugin — mdast walk', () => {
expect(image.alt).toBe('Tom &amp; Jerry');
});
+ test('restores encoded entity in code-node `lang` field (fenced-code info string)', () => {
+ const encoded = encodeEntityRefs('f&ouml;&ouml;');
+ const tree: Root = {
+ type: 'root',
+ children: [{ type: 'code', lang: encoded, meta: null, value: 'code' }],
+ };
+ runPlugin(tree);
+ const code = tree.children[0];
+ if (code?.type !== 'code') throw new Error('expected code');
+ expect(code.lang).toBe('f&ouml;&ouml;');
+ expect((code as { data?: unknown }).data).toBeUndefined();
+ });
+
+ test('restores encoded entity in code-node `meta` field (fenced-code info string)', () => {
+ const encoded = encodeEntityRefs('title=&quot;x&quot;');
+ const tree: Root = {
+ type: 'root',
+ children: [{ type: 'code', lang: 'js', meta: encoded, value: 'code' }],
+ };
+ runPlugin(tree);
+ const code = tree.children[0];
+ if (code?.type !== 'code') throw new Error('expected code');
+ expect(code.meta).toBe('title=&quot;x&quot;');
+ expect(code.lang).toBe('js');
+ });
+
test('does NOT tag entityRefSpans on non-text nodes (e.g. inlineCode value)', () => {
const encoded = encodeEntityRefs('foo &amp; bar');
const tree: Root = {
@@ -291,6 +317,47 @@ describe('restoreEntityRefsPlugin — mdast walk', () => {
});
});
+describe('fenced-code info-string entity refs — real-engine round-trip', () => {
+ const mgr = new MarkdownManager({ extensions: sharedExtensions });
+ const roundTrip = (src: string) => mgr.serialize(mgr.parse(src));
+
+ test('named entity in lang round-trips byte-exact', () => {
+ const src = '```f&ouml;&ouml;\ncode\n```\n';
+ expect(roundTrip(src)).toBe(src);
+ });
+
+ test('round-trip is idempotent (second pass emits identical bytes)', () => {
+ const src = '```f&ouml;&ouml;\ncode\n```\n';
+ const once = roundTrip(src);
+ expect(roundTrip(once)).toBe(once);
+ });
+
+ test('named entity in meta round-trips byte-exact', () => {
+ const src = '```js meta&amp;stuff\ncode\n```\n';
+ expect(roundTrip(src)).toBe(src);
+ });
+
+ test('entities in both lang and meta round-trip byte-exact', () => {
+ const src = '```f&ouml;&ouml; title=&quot;x&quot;\ncode\n```\n';
+ expect(roundTrip(src)).toBe(src);
+ });
+
+ test('numeric hex entity in lang round-trips byte-exact', () => {
+ const src = '```f&#xF6;&#xF6;\ncode\n```\n';
+ expect(roundTrip(src)).toBe(src);
+ });
+
+ test('tilde-fence info-string entity round-trips byte-exact', () => {
+ const src = '~~~f&ouml;&ouml;\ncode\n~~~\n';
+ expect(roundTrip(src)).toBe(src);
+ });
+
+ test('plain info strings stay byte-identical (no encode/restore detour)', () => {
+ const src = '```typescript title="x"\ncode\n```\n';
+ expect(roundTrip(src)).toBe(src);
+ });
+});
+
describe('escape + entity-ref marker interleaving on the same text node', () => {
test('text node with BOTH escapedChars AND entityRefSpans round-trips through parse → serialize byte-equal', () => {
const mgr = new MarkdownManager({ extensions: sharedExtensions });
diff --git a/packages/core/src/markdown/entity-ref-guard.ts b/packages/core/src/markdown/entity-ref-guard.ts
index 3991b0e81..5a3996fbf 100644
--- a/packages/core/src/markdown/entity-ref-guard.ts
+++ b/packages/core/src/markdown/entity-ref-guard.ts
@@ -1,10 +1,17 @@
+import { decodeNamedCharacterReference } from 'decode-named-character-reference';
import type { Nodes, Root } from 'mdast';
+import { decodeNumericCharacterReference } from 'micromark-util-decode-numeric-character-reference';
import type { Plugin } from 'unified';
import { visit } from 'unist-util-visit';
const ENTITY_REF_PUA_OPEN = '';
const ENTITY_REF_PUA_CLOSE = '';
+export const ENTITY_REF_GUARD_SUBSTITUTIONS: ReadonlyArray<{ from: string; to: string }> = [
+ { from: '&', to: ENTITY_REF_PUA_OPEN },
+ { from: ';', to: ENTITY_REF_PUA_CLOSE },
+];
+
const ENTITY_REF_RE = /&(?:[A-Za-z][A-Za-z0-9]*|#[0-9]+|#[xX][0-9A-Fa-f]+);/g;
function countPrecedingBackslashes(source: string, before: number): number {
@@ -26,6 +33,19 @@ export function protectPattern(
});
}
+export function decodeEntityRefs(value: string): string {
+ ENTITY_REF_RE.lastIndex = 0;
+ return value.replace(ENTITY_REF_RE, (match) => {
+ const body = match.slice(1, -1);
+ if (body.startsWith('#')) {
+ const hex = body[1] === 'x' || body[1] === 'X';
+ return decodeNumericCharacterReference(body.slice(hex ? 2 : 1), hex ? 16 : 10);
+ }
+ const decoded = decodeNamedCharacterReference(body);
+ return decoded === false ? match : decoded;
+ });
+}
+
export function encodeEntityRefs(source: string): string {
return protectPattern(source, ENTITY_REF_RE, (match) => {
const body = match.slice(1, -1);
@@ -84,7 +104,7 @@ export function restoreEntityRefsPlugin(): ReturnType<Plugin<[], Root>> {
textNode.data.entityRefSpans = spans;
}
}
- for (const key of ['url', 'title', 'alt'] as const) {
+ for (const key of ['url', 'title', 'alt', 'lang', 'meta'] as const) {
const v = rec[key];
if (typeof v === 'string' && v.includes(ENTITY_REF_PUA_OPEN)) {
rec[key] = restoreEntityRefsInString(v).value;
diff --git a/packages/core/src/markdown/guard-flanking-matrix.json b/packages/core/src/markdown/guard-flanking-matrix.json
new file mode 100644
index 000000000..c6f92c8c0
--- /dev/null
+++ b/packages/core/src/markdown/guard-flanking-matrix.json
@@ -0,0 +1,471 @@
+{
+ "delimiters": ["*", "_", "**", "~~", "=="],
+ "substitutions": [
+ {
+ "alphabet": "entity-ref",
+ "from": "&",
+ "toCodePoint": "U+E100",
+ "fromClass": "punctuation",
+ "toClass": "other",
+ "classChanged": true
+ },
+ {
+ "alphabet": "entity-ref",
+ "from": ";",
+ "toCodePoint": "U+E101",
+ "fromClass": "punctuation",
+ "toClass": "other",
+ "classChanged": true
+ },
+ {
+ "alphabet": "r23",
+ "from": "<",
+ "toCodePoint": "U+E000",
+ "fromClass": "punctuation",
+ "toClass": "other",
+ "classChanged": true
+ },
+ {
+ "alphabet": "r23",
+ "from": ">",
+ "toCodePoint": "U+E001",
+ "fromClass": "punctuation",
+ "toClass": "other",
+ "classChanged": true
+ },
+ {
+ "alphabet": "r23",
+ "from": ":",
+ "toCodePoint": "U+E002",
+ "fromClass": "punctuation",
+ "toClass": "other",
+ "classChanged": true
+ },
+ {
+ "alphabet": "r23",
+ "from": "@",
+ "toCodePoint": "U+E003",
+ "fromClass": "punctuation",
+ "toClass": "other",
+ "classChanged": true
+ },
+ {
+ "alphabet": "r23",
+ "from": "{",
+ "toCodePoint": "U+E004",
+ "fromClass": "punctuation",
+ "toClass": "other",
+ "classChanged": true
+ },
+ {
+ "alphabet": "r23-sentinel-escape",
+ "from": "U+E000",
+ "toCodePoint": "U+E005",
+ "fromClass": "other",
+ "toClass": "other",
+ "classChanged": false
+ },
+ {
+ "alphabet": "r23-sentinel-escape",
+ "from": "U+E001",
+ "toCodePoint": "U+E006",
+ "fromClass": "other",
+ "toClass": "other",
+ "classChanged": false
+ },
+ {
+ "alphabet": "r23-sentinel-escape",
+ "from": "U+E002",
+ "toCodePoint": "U+E007",
+ "fromClass": "other",
+ "toClass": "other",
+ "classChanged": false
+ },
+ {
+ "alphabet": "r23-sentinel-escape",
+ "from": "U+E003",
+ "toCodePoint": "U+E008",
+ "fromClass": "other",
+ "toClass": "other",
+ "classChanged": false
+ },
+ {
+ "alphabet": "r23-sentinel-escape",
+ "from": "U+E004",
+ "toCodePoint": "U+E009",
+ "fromClass": "other",
+ "toClass": "other",
+ "classChanged": false
+ },
+ {
+ "alphabet": "backslash-escape",
+ "from": "\\",
+ "toCodePoint": "U+E102",
+ "fromClass": "punctuation",
+ "toClass": "other",
+ "classChanged": true
+ }
+ ],
+ "cells": [
+ {
+ "alphabet": "entity-ref",
+ "from": "&",
+ "toCodePoint": "U+E100",
+ "delimiter": "*",
+ "witness": "*.y.*&#x41;b",
+ "side": "after-close",
+ "roundTrip": "\\*.y.\\*&#x41;b\n"
+ },
+ {
+ "alphabet": "entity-ref",
+ "from": "&",
+ "toCodePoint": "U+E100",
+ "delimiter": "_",
+ "witness": "_.y._&#x41;b",
+ "side": "after-close",
+ "roundTrip": "\\_.y.\\_&#x41;b\n"
+ },
+ {
+ "alphabet": "entity-ref",
+ "from": "&",
+ "toCodePoint": "U+E100",
+ "delimiter": "**",
+ "witness": "**.y.**&#x41;b",
+ "side": "after-close",
+ "roundTrip": "\\*\\*.y.\\*\\*&#x41;b\n"
+ },
+ {
+ "alphabet": "entity-ref",
+ "from": "&",
+ "toCodePoint": "U+E100",
+ "delimiter": "~~",
+ "witness": "~~.y.~~&#x41;b",
+ "side": "after-close",
+ "roundTrip": "\\~\\~.y.\\~\\~&#x41;b\n"
+ },
+ {
+ "alphabet": "entity-ref",
+ "from": "&",
+ "toCodePoint": "U+E100",
+ "delimiter": "==",
+ "witness": "==.y.==&#x41;b",
+ "side": "after-close",
+ "roundTrip": "==.y.==\\&#x41;b\n"
+ },
+ {
+ "alphabet": "entity-ref",
+ "from": ";",
+ "toCodePoint": "U+E101",
+ "delimiter": "*",
+ "witness": "a&#x41;*.y.*",
+ "side": "before-open",
+ "roundTrip": "a&#x41;\\*.y.\\*\n"
+ },
+ {
+ "alphabet": "entity-ref",
+ "from": ";",
+ "toCodePoint": "U+E101",
+ "delimiter": "_",
+ "witness": "a&#x41;_.y._",
+ "side": "before-open",
+ "roundTrip": "a&#x41;\\_.y.\\_\n"
+ },
+ {
+ "alphabet": "entity-ref",
+ "from": ";",
+ "toCodePoint": "U+E101",
+ "delimiter": "**",
+ "witness": "a&#x41;**.y.**",
+ "side": "before-open",
+ "roundTrip": "a&#x41;\\*\\*.y.\\*\\*\n"
+ },
+ {
+ "alphabet": "entity-ref",
+ "from": ";",
+ "toCodePoint": "U+E101",
+ "delimiter": "~~",
+ "witness": "a&#x41;~~.y.~~",
+ "side": "before-open",
+ "roundTrip": "a&#x41;\\~\\~.y.\\~\\~\n"
+ },
+ {
+ "alphabet": "entity-ref",
+ "from": ";",
+ "toCodePoint": "U+E101",
+ "delimiter": "==",
+ "witness": "a&#x41;==.y.==",
+ "side": "before-open",
+ "roundTrip": "a\\&#x41;==.y.==\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": "<",
+ "toCodePoint": "U+E000",
+ "delimiter": "*",
+ "witness": "*.y.*<br>b",
+ "side": "after-close",
+ "roundTrip": "\\*.y.\\*<br>b\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": "<",
+ "toCodePoint": "U+E000",
+ "delimiter": "_",
+ "witness": "_.y._<br>b",
+ "side": "after-close",
+ "roundTrip": "\\_.y.\\_<br>b\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": "<",
+ "toCodePoint": "U+E000",
+ "delimiter": "**",
+ "witness": "**.y.**<br>b",
+ "side": "after-close",
+ "roundTrip": "\\*\\*.y.\\*\\*<br>b\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": "<",
+ "toCodePoint": "U+E000",
+ "delimiter": "~~",
+ "witness": "~~.y.~~<br>b",
+ "side": "after-close",
+ "roundTrip": "\\~\\~.y.\\~\\~<br>b\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": "<",
+ "toCodePoint": "U+E000",
+ "delimiter": "==",
+ "witness": "==.y.==<br>b",
+ "side": "after-close",
+ "roundTrip": "==.y.==<br>b\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": ">",
+ "toCodePoint": "U+E001",
+ "delimiter": "*",
+ "witness": "a<br>*.y.*",
+ "side": "before-open",
+ "roundTrip": "a<br>\\*.y.\\*\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": ">",
+ "toCodePoint": "U+E001",
+ "delimiter": "_",
+ "witness": "a<br>_.y._",
+ "side": "before-open",
+ "roundTrip": "a<br>\\_.y.\\_\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": ">",
+ "toCodePoint": "U+E001",
+ "delimiter": "**",
+ "witness": "a<br>**.y.**",
+ "side": "before-open",
+ "roundTrip": "a<br>\\*\\*.y.\\*\\*\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": ">",
+ "toCodePoint": "U+E001",
+ "delimiter": "~~",
+ "witness": "a<br>~~.y.~~",
+ "side": "before-open",
+ "roundTrip": "a<br>\\~\\~.y.\\~\\~\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": ">",
+ "toCodePoint": "U+E001",
+ "delimiter": "==",
+ "witness": "a<br>==.y.==",
+ "side": "before-open",
+ "roundTrip": "a<br>==.y.==\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": ":",
+ "toCodePoint": "U+E002",
+ "delimiter": "*",
+ "witness": "<a:*.y.*>",
+ "side": "before-open",
+ "roundTrip": "<a:*.y.*>\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": ":",
+ "toCodePoint": "U+E002",
+ "delimiter": "_",
+ "witness": "<a:_.y._>",
+ "side": "before-open",
+ "roundTrip": "<a:_.y._>\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": ":",
+ "toCodePoint": "U+E002",
+ "delimiter": "**",
+ "witness": "<a:**.y.**>",
+ "side": "before-open",
+ "roundTrip": "<a:**.y.**>\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": ":",
+ "toCodePoint": "U+E002",
+ "delimiter": "~~",
+ "witness": "<a:~~.y.~~>",
+ "side": "before-open",
+ "roundTrip": "<a:~~.y.~~>\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": ":",
+ "toCodePoint": "U+E002",
+ "delimiter": "==",
+ "witness": "<a:==.y.==>",
+ "side": "before-open",
+ "roundTrip": "<a:==.y.==>\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": "@",
+ "toCodePoint": "U+E003",
+ "delimiter": "*",
+ "witness": "<mailto:a@*.y.*>",
+ "side": "before-open",
+ "roundTrip": "<mailto:a@*.y.*>\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": "@",
+ "toCodePoint": "U+E003",
+ "delimiter": "_",
+ "witness": "<mailto:a@_.y._>",
+ "side": "before-open",
+ "roundTrip": "<mailto:a@_.y._>\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": "@",
+ "toCodePoint": "U+E003",
+ "delimiter": "**",
+ "witness": "<mailto:a@**.y.**>",
+ "side": "before-open",
+ "roundTrip": "<mailto:a@**.y.**>\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": "@",
+ "toCodePoint": "U+E003",
+ "delimiter": "~~",
+ "witness": "<mailto:a@~~.y.~~>",
+ "side": "before-open",
+ "roundTrip": "<mailto:a@~~.y.~~>\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": "@",
+ "toCodePoint": "U+E003",
+ "delimiter": "==",
+ "witness": "<mailto:a@==.y.==>",
+ "side": "before-open",
+ "roundTrip": "<mailto:a@==.y.==>\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": "{",
+ "toCodePoint": "U+E004",
+ "delimiter": "*",
+ "witness": "*.y.*{b",
+ "side": "after-close",
+ "roundTrip": "\\*.y.\\*\\{b\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": "{",
+ "toCodePoint": "U+E004",
+ "delimiter": "_",
+ "witness": "_.y._{b",
+ "side": "after-close",
+ "roundTrip": "\\_.y.\\_\\{b\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": "{",
+ "toCodePoint": "U+E004",
+ "delimiter": "**",
+ "witness": "**.y.**{b",
+ "side": "after-close",
+ "roundTrip": "\\*\\*.y.\\*\\*\\{b\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": "{",
+ "toCodePoint": "U+E004",
+ "delimiter": "~~",
+ "witness": "~~.y.~~{b",
+ "side": "after-close",
+ "roundTrip": "\\~\\~.y.\\~\\~\\{b\n"
+ },
+ {
+ "alphabet": "r23",
+ "from": "{",
+ "toCodePoint": "U+E004",
+ "delimiter": "==",
+ "witness": "==.y.=={b",
+ "side": "after-close",
+ "roundTrip": "==.y.==\\{b\n"
+ },
+ {
+ "alphabet": "backslash-escape",
+ "from": "\\",
+ "toCodePoint": "U+E102",
+ "delimiter": "*",
+ "witness": "*.y.*\\<b",
+ "side": "after-close",
+ "roundTrip": "\\*.y.\\*\\<b\n"
+ },
+ {
+ "alphabet": "backslash-escape",
+ "from": "\\",
+ "toCodePoint": "U+E102",
+ "delimiter": "_",
+ "witness": "_.y._\\<b",
+ "side": "after-close",
+ "roundTrip": "\\_.y.\\_\\<b\n"
+ },
+ {
+ "alphabet": "backslash-escape",
+ "from": "\\",
+ "toCodePoint": "U+E102",
+ "delimiter": "**",
+ "witness": "**.y.**\\<b",
+ "side": "after-close",
+ "roundTrip": "\\*\\*.y.\\*\\*\\<b\n"
+ },
+ {
+ "alphabet": "backslash-escape",
+ "from": "\\",
+ "toCodePoint": "U+E102",
+ "delimiter": "~~",
+ "witness": "~~.y.~~\\<b",
+ "side": "after-close",
+ "roundTrip": "\\~\\~.y.\\~\\~\\<b\n"
+ },
+ {
+ "alphabet": "backslash-escape",
+ "from": "\\",
+ "toCodePoint": "U+E102",
+ "delimiter": "==",
+ "witness": "==.y.==\\<b",
+ "side": "after-close",
+ "roundTrip": "==.y.==\\<b\n"
+ }
+ ]
+}
diff --git a/packages/core/src/markdown/guard-flanking-matrix.test.ts b/packages/core/src/markdown/guard-flanking-matrix.test.ts
new file mode 100644
index 000000000..c7a185819
--- /dev/null
+++ b/packages/core/src/markdown/guard-flanking-matrix.test.ts
@@ -0,0 +1,131 @@
+import { describe, expect, test } from 'bun:test';
+import { readFileSync, writeFileSync } from 'node:fs';
+import path from 'node:path';
+import { sharedExtensions } from '../extensions/shared.ts';
+import {
+ ATTENTION_DELIMITERS,
+ buildGuardFlankingMatrix,
+ computeGuardFlankingCells,
+ computeGuardSubstitutionRows,
+ diffGuardFlankingMatrix,
+ type FlankClass,
+ type GuardFlankingMatrix,
+} from './guard-flanking-matrix.ts';
+import { MarkdownManager } from './index.ts';
+
+const md = new MarkdownManager({ extensions: sharedExtensions });
+const roundTrip = (source: string) => md.serialize(md.parse(source));
+
+function textContent(source: string): string {
+ let out = '';
+ const walk = (node: unknown): void => {
+ if (node === null || typeof node !== 'object') return;
+ const rec = node as { value?: unknown; children?: unknown[] };
+ if (typeof rec.value === 'string') out += rec.value;
+ if (Array.isArray(rec.children)) rec.children.forEach(walk);
+ };
+ walk(md.parseToMdast(source));
+ return out;
+}
+
+const MATRIX_PATH = path.join(import.meta.dir, 'guard-flanking-matrix.json');
+
+function readCommitted(): GuardFlankingMatrix {
+ return JSON.parse(readFileSync(MATRIX_PATH, 'utf8')) as GuardFlankingMatrix;
+}
+
+if (process.env.OK_UPDATE_GUARD_FLANKING_MATRIX === '1') {
+ writeFileSync(MATRIX_PATH, `${JSON.stringify(buildGuardFlankingMatrix(roundTrip), null, 2)}\n`);
+}
+
+describe('guard-flanking matrix - computation', () => {
+ test('every guard substitution is classified; the punctuation->PUA rows are class-changing', () => {
+ const rows = computeGuardSubstitutionRows();
+ const changed = rows.filter((r) => r.classChanged);
+ const preserved = rows.filter((r) => !r.classChanged);
+ expect(changed.map((r) => `${r.alphabet}:${r.from}`).sort()).toEqual([
+ 'backslash-escape:\\',
+ 'entity-ref:&',
+ 'entity-ref:;',
+ 'r23::',
+ 'r23:<',
+ 'r23:>',
+ 'r23:@',
+ 'r23:{',
+ ]);
+ const expectedDelta: { from: FlankClass; to: FlankClass } = {
+ from: 'punctuation',
+ to: 'other',
+ };
+ for (const row of changed) {
+ expect(row.fromClass).toBe(expectedDelta.from);
+ expect(row.toClass).toBe(expectedDelta.to);
+ }
+ for (const row of preserved) {
+ expect(row.alphabet).toBe('r23-sentinel-escape');
+ expect(row.fromClass).toBe('other');
+ expect(row.toClass).toBe('other');
+ }
+ });
+
+ test('every class-changing substitution x delimiter carries a cell with protect-layer-verified adjacency', () => {
+ const cells = computeGuardFlankingCells();
+ expect(cells.length).toBe(8 * ATTENTION_DELIMITERS.length);
+ const keys = new Set(cells.map((c) => `${c.alphabet}:${c.from} x ${c.delimiter}`));
+ for (const row of computeGuardSubstitutionRows().filter((r) => r.classChanged)) {
+ for (const delimiter of ATTENTION_DELIMITERS) {
+ expect(keys.has(`${row.alphabet}:${row.from} x ${delimiter}`)).toBe(true);
+ }
+ }
+ });
+});
+
+describe('guard-flanking matrix - freshness gate', () => {
+ test('committed snapshot matches a live recompute (regen: bun run gen:guard-flanking-matrix)', () => {
+ const live = buildGuardFlankingMatrix(roundTrip);
+ const mismatches = diffGuardFlankingMatrix(live, readCommitted());
+ expect(mismatches).toEqual([]);
+ for (const [index, cell] of live.cells.entries()) {
+ expect(cell.roundTrip).toBe(readCommitted().cells[index]?.roundTrip ?? '');
+ }
+ });
+
+ test('every committed cell is lossless, idempotent, and preserves the guarded bytes verbatim', () => {
+ for (const cell of readCommitted().cells) {
+ const rt1 = roundTrip(cell.witness);
+ expect(rt1).toBe(cell.roundTrip);
+ expect(roundTrip(rt1)).toBe(rt1);
+ expect(textContent(rt1)).toBe(textContent(cell.witness));
+ if (cell.alphabet === 'entity-ref') {
+ expect(rt1).toContain('&#x41;');
+ }
+ }
+ });
+});
+
+describe('guard-flanking matrix - non-vacuity (tamper)', () => {
+ test('a dropped committed cell fails the freshness diff', () => {
+ const live = buildGuardFlankingMatrix(roundTrip);
+ const tampered: GuardFlankingMatrix = { ...live, cells: live.cells.slice(1) };
+ const mismatches = diffGuardFlankingMatrix(live, tampered);
+ expect(mismatches.some((m) => m.includes('not committed'))).toBe(true);
+ });
+
+ test('a drifted round-trip pin fails the per-cell byte compare', () => {
+ const committed = readCommitted();
+ const first = committed.cells[0];
+ if (!first) throw new Error('empty committed matrix');
+ expect(roundTrip(first.witness)).not.toBe(`${first.roundTrip}tampered`);
+ });
+
+ test('a substitution removed from a guard alphabet fails the freshness diff', () => {
+ const live = buildGuardFlankingMatrix(roundTrip);
+ const tampered: GuardFlankingMatrix = {
+ ...live,
+ substitutions: live.substitutions.filter((r) => r.from !== '&'),
+ cells: live.cells.filter((c) => c.from !== '&'),
+ };
+ const mismatches = diffGuardFlankingMatrix(live, tampered);
+ expect(mismatches.some((m) => m.includes('entity-ref:&'))).toBe(true);
+ });
+});
diff --git a/packages/core/src/markdown/guard-flanking-matrix.ts b/packages/core/src/markdown/guard-flanking-matrix.ts
new file mode 100644
index 000000000..420dc0fe5
--- /dev/null
+++ b/packages/core/src/markdown/guard-flanking-matrix.ts
@@ -0,0 +1,184 @@
+import { classifyCharacter } from 'micromark-util-classify-character';
+import {
+ protectFromMdx,
+ R23_GUARD_SUBSTITUTIONS,
+ R23_SENTINEL_ESCAPE_SUBSTITUTIONS,
+} from './autolink-void-html-guard.ts';
+import { BACKSLASH_GUARD_SUBSTITUTIONS, encodeBackslashEscapes } from './backslash-escape-guard.ts';
+import { ENTITY_REF_GUARD_SUBSTITUTIONS, encodeEntityRefs } from './entity-ref-guard.ts';
+
+export const ATTENTION_DELIMITERS = ['*', '_', '**', '~~', '=='] as const;
+
+export type FlankClass = 'whitespace' | 'punctuation' | 'other';
+
+export interface GuardSubstitutionRow {
+ alphabet: string;
+ from: string;
+ toCodePoint: string;
+ fromClass: FlankClass;
+ toClass: FlankClass;
+ classChanged: boolean;
+}
+
+export interface GuardFlankingCell {
+ alphabet: string;
+ from: string;
+ toCodePoint: string;
+ delimiter: string;
+ witness: string;
+ side: 'before-open' | 'after-close';
+}
+
+export interface GuardFlankingMatrix {
+ delimiters: readonly string[];
+ substitutions: GuardSubstitutionRow[];
+ cells: Array<GuardFlankingCell & { roundTrip: string }>;
+}
+
+const ALPHABETS: ReadonlyArray<{
+ alphabet: string;
+ substitutions: ReadonlyArray<{ from: string; to: string }>;
+}> = [
+ { alphabet: 'entity-ref', substitutions: ENTITY_REF_GUARD_SUBSTITUTIONS },
+ { alphabet: 'r23', substitutions: R23_GUARD_SUBSTITUTIONS },
+ { alphabet: 'r23-sentinel-escape', substitutions: R23_SENTINEL_ESCAPE_SUBSTITUTIONS },
+ { alphabet: 'backslash-escape', substitutions: BACKSLASH_GUARD_SUBSTITUTIONS },
+];
+
+const WITNESS_TEMPLATES: Record<
+ string,
+ { side: GuardFlankingCell['side']; make: (delim: string) => string }
+> = {
+ 'entity-ref:&': { side: 'after-close', make: (d) => `${d}.y.${d}&#x41;b` },
+ 'entity-ref:;': { side: 'before-open', make: (d) => `a&#x41;${d}.y.${d}` },
+ 'r23:<': { side: 'after-close', make: (d) => `${d}.y.${d}<br>b` },
+ 'r23:>': { side: 'before-open', make: (d) => `a<br>${d}.y.${d}` },
+ 'r23::': { side: 'before-open', make: (d) => `<a:${d}.y.${d}>` },
+ 'r23:@': { side: 'before-open', make: (d) => `<mailto:a@${d}.y.${d}>` },
+ 'r23:{': { side: 'after-close', make: (d) => `${d}.y.${d}{b` },
+ 'backslash-escape:\\': { side: 'after-close', make: (d) => `${d}.y.${d}\\<b` },
+};
+
+function classify(char: string): FlankClass {
+ const code = char.codePointAt(0);
+ if (code === undefined) throw new Error('empty substitution char');
+ const group = classifyCharacter(code);
+ if (group === 1) return 'whitespace';
+ if (group === 2) return 'punctuation';
+ return 'other';
+}
+
+function formatCodePoint(char: string): string {
+ const code = char.codePointAt(0);
+ if (code === undefined) throw new Error('empty substitution char');
+ return `U+${code.toString(16).toUpperCase().padStart(4, '0')}`;
+}
+
+function displayChar(char: string): string {
+ const code = char.codePointAt(0);
+ if (code === undefined) throw new Error('empty substitution char');
+ return code >= 0x20 && code <= 0x7e ? char : formatCodePoint(char);
+}
+
+function runProtectChain(source: string): string {
+ return encodeEntityRefs(protectFromMdx(encodeBackslashEscapes(source)));
+}
+
+export function computeGuardSubstitutionRows(): GuardSubstitutionRow[] {
+ const rows: GuardSubstitutionRow[] = [];
+ for (const { alphabet, substitutions } of ALPHABETS) {
+ for (const { from, to } of substitutions) {
+ const fromClass = classify(from);
+ const toClass = classify(to);
+ rows.push({
+ alphabet,
+ from: displayChar(from),
+ toCodePoint: formatCodePoint(to),
+ fromClass,
+ toClass,
+ classChanged: fromClass !== toClass,
+ });
+ }
+ }
+ return rows;
+}
+
+export function computeGuardFlankingCells(): GuardFlankingCell[] {
+ const cells: GuardFlankingCell[] = [];
+ for (const { alphabet, substitutions } of ALPHABETS) {
+ for (const { from, to } of substitutions) {
+ if (classify(from) === classify(to)) continue;
+ const template = WITNESS_TEMPLATES[`${alphabet}:${from}`];
+ if (!template) {
+ throw new Error(`no witness template for class-changing substitution ${alphabet}:${from}`);
+ }
+ for (const delimiter of ATTENTION_DELIMITERS) {
+ const witness = template.make(delimiter);
+ const protectedSource = runProtectChain(witness);
+ const adjacencyProbe =
+ template.side === 'before-open' ? `${to}${delimiter}` : `${delimiter}${to}`;
+ if (!protectedSource.includes(adjacencyProbe)) {
+ throw new Error(
+ `witness for ${alphabet}:${from} x ${delimiter} lost sentinel-delimiter adjacency at the protect layer`,
+ );
+ }
+ cells.push({
+ alphabet,
+ from,
+ toCodePoint: formatCodePoint(to),
+ delimiter,
+ witness,
+ side: template.side,
+ });
+ }
+ }
+ }
+ return cells;
+}
+
+export function buildGuardFlankingMatrix(roundTrip: (md: string) => string): GuardFlankingMatrix {
+ return {
+ delimiters: [...ATTENTION_DELIMITERS],
+ substitutions: computeGuardSubstitutionRows(),
+ cells: computeGuardFlankingCells().map((cell) => ({
+ ...cell,
+ roundTrip: roundTrip(cell.witness),
+ })),
+ };
+}
+
+export function diffGuardFlankingMatrix(
+ live: GuardFlankingMatrix,
+ committed: GuardFlankingMatrix,
+): string[] {
+ const mismatches: string[] = [];
+ if (JSON.stringify(live.delimiters) !== JSON.stringify(committed.delimiters)) {
+ mismatches.push('delimiter set drifted');
+ }
+ const rowKey = (r: GuardSubstitutionRow) => `${r.alphabet}:${r.from}`;
+ const liveRows = new Map(live.substitutions.map((r) => [rowKey(r), r]));
+ const committedRows = new Map(committed.substitutions.map((r) => [rowKey(r), r]));
+ for (const [key, row] of liveRows) {
+ const prior = committedRows.get(key);
+ if (!prior) mismatches.push(`substitution ${key} is live but not committed`);
+ else if (JSON.stringify(row) !== JSON.stringify(prior)) {
+ mismatches.push(`substitution ${key} drifted`);
+ }
+ }
+ for (const key of committedRows.keys()) {
+ if (!liveRows.has(key)) mismatches.push(`substitution ${key} is committed but no longer live`);
+ }
+ const cellKey = (c: GuardFlankingCell) =>
+ `${c.alphabet}:${c.from} x ${JSON.stringify(c.delimiter)}`;
+ const liveCells = new Map(live.cells.map((c) => [cellKey(c), c]));
+ const committedCells = new Map(committed.cells.map((c) => [cellKey(c), c]));
+ for (const [key, cell] of liveCells) {
+ const prior = committedCells.get(key);
+ if (!prior) mismatches.push(`cell ${key} is live but not committed`);
+ else if (JSON.stringify(cell) !== JSON.stringify(prior)) mismatches.push(`cell ${key} drifted`);
+ }
+ for (const key of committedCells.keys()) {
+ if (!liveCells.has(key)) mismatches.push(`cell ${key} is committed but no longer live`);
+ }
+ return mismatches;
+}
diff --git a/packages/core/src/markdown/handler-shadow-audit.test.ts b/packages/core/src/markdown/handler-shadow-audit.test.ts
new file mode 100644
index 000000000..b717ef4d0
--- /dev/null
+++ b/packages/core/src/markdown/handler-shadow-audit.test.ts
@@ -0,0 +1,99 @@
+import { describe, expect, test } from 'bun:test';
+import { sharedExtensions } from '../extensions/shared.ts';
+import {
+ collectExtensionHandlerKeys,
+ computeShadowedHandlerKeys,
+ HANDLER_SHADOW_ADJUDICATIONS,
+ type HandlerShadowWitness,
+} from './handler-shadow-audit.ts';
+import { MarkdownManager } from './index.ts';
+import { createSerializeProcessor } from './pipeline.ts';
+import { toMarkdownHandlers } from './to-markdown-handlers.ts';
+
+const md = new MarkdownManager({ extensions: sharedExtensions });
+const roundTrip = (source: string) => md.serialize(md.parse(source));
+
+function computeLiveShadowedKeys(): string[] {
+ const processor = createSerializeProcessor({});
+ const extensions = processor.data('toMarkdownExtensions');
+ if (!Array.isArray(extensions) || extensions.length === 0) {
+ throw new Error('serialize processor registered no toMarkdown extensions');
+ }
+ return computeShadowedHandlerKeys(
+ collectExtensionHandlerKeys(extensions),
+ Object.keys(toMarkdownHandlers),
+ );
+}
+
+describe('handler-shadow audit - closure', () => {
+ test('every shadowed handler key carries an adjudication and none is stale', () => {
+ const shadowed = computeLiveShadowedKeys();
+ expect(shadowed.length).toBeGreaterThan(0);
+ const adjudicated = Object.keys(HANDLER_SHADOW_ADJUDICATIONS).sort();
+ expect(shadowed).toEqual(adjudicated);
+ });
+
+ test('the known shadow surface is exactly the six adjudicated keys', () => {
+ expect(computeLiveShadowedKeys()).toEqual([
+ 'delete',
+ 'inlineCode',
+ 'inlineMath',
+ 'mdxJsxFlowElement',
+ 'mdxJsxTextElement',
+ 'table',
+ ]);
+ });
+});
+
+describe('handler-shadow audit - live witnesses', () => {
+ test('every adjudication witness round-trips byte-exactly through the real engine', () => {
+ const runWitness = (key: string, witness: HandlerShadowWitness): void => {
+ const out = roundTrip(witness.input);
+ expect(
+ out === witness.input || out === `${witness.input}\n`,
+ `${key} witness drifted: ${JSON.stringify(witness.input)} -> ${JSON.stringify(out)}`,
+ ).toBe(true);
+ if (witness.expect === 'byte-and-reparse-type') {
+ const reparsed = md.parseToMdast(out) as { children?: Array<{ type: string }> };
+ expect(reparsed.children?.[0]?.type).toBe(witness.reparseType ?? '');
+ }
+ };
+ for (const [key, adjudication] of Object.entries(HANDLER_SHADOW_ADJUDICATIONS)) {
+ expect(adjudication.witnesses.length).toBeGreaterThan(0);
+ for (const witness of adjudication.witnesses) runWitness(key, witness);
+ }
+ });
+});
+
+describe('handler-shadow audit - non-vacuity (tamper)', () => {
+ test('removing the inlineCode (table-cell pipe re-escape) adjudication fails the closure', () => {
+ const shadowed = computeLiveShadowedKeys();
+ const { inlineCode: _omitted, ...rest } = HANDLER_SHADOW_ADJUDICATIONS;
+ expect(shadowed).not.toEqual(Object.keys(rest).sort());
+ const missing = shadowed.filter((key) => !(key in rest));
+ expect(missing).toEqual(['inlineCode']);
+ });
+
+ test('a planted adjudication for an unshadowed key is detected as stale', () => {
+ const shadowed = new Set(computeLiveShadowedKeys());
+ const planted = {
+ ...HANDLER_SHADOW_ADJUDICATIONS,
+ paragraph: HANDLER_SHADOW_ADJUDICATIONS.delete,
+ };
+ const stale = Object.keys(planted).filter((key) => !shadowed.has(key));
+ expect(stale).toEqual(['paragraph']);
+ });
+
+ test('the key collector sees through nested extension arrays', () => {
+ const keys = collectExtensionHandlerKeys([
+ {
+ extensions: [
+ { handlers: { alpha: () => '' } },
+ { extensions: [{ handlers: { beta: () => '' } }] },
+ ],
+ },
+ { handlers: { gamma: () => '' } },
+ ]);
+ expect(keys).toEqual(['alpha', 'beta', 'gamma']);
+ });
+});
diff --git a/packages/core/src/markdown/handler-shadow-audit.ts b/packages/core/src/markdown/handler-shadow-audit.ts
new file mode 100644
index 000000000..2653473da
--- /dev/null
+++ b/packages/core/src/markdown/handler-shadow-audit.ts
@@ -0,0 +1,102 @@
+export interface HandlerShadowWitness {
+ input: string;
+ expect: 'byte' | 'byte-and-reparse-type';
+ reparseType?: string;
+}
+
+export interface HandlerShadowAdjudication {
+ shadows: string;
+ obligation: string;
+ witnesses: ReadonlyArray<HandlerShadowWitness>;
+}
+
+export const HANDLER_SHADOW_ADJUDICATIONS: Record<string, HandlerShadowAdjudication> = {
+ delete: {
+ shadows: 'mdast-util-gfm-strikethrough handlers.delete (via remark-gfm)',
+ obligation:
+ 'emit tilde-delimited strikethrough; OK additionally re-emits the captured source ' +
+ 'delimiter so single-tilde forms survive',
+ witnesses: [
+ { input: 'a ~~b~~ c', expect: 'byte' },
+ { input: 'a ~b~ c', expect: 'byte' },
+ ],
+ },
+ inlineCode: {
+ shadows: 'mdast-util-gfm-table inlineCodeWithTable (via remark-gfm)',
+ obligation:
+ 'inside a tableCell, re-escape | in the emitted code span (GFM: an unescaped pipe splits ' +
+ 'the cell, so dropping the escape re-parses the table as a paragraph); outside tables, ' +
+ 'preserve fence char/length and padding',
+ witnesses: [
+ {
+ input: '| `a\\|b` |\n| - |',
+ expect: 'byte-and-reparse-type',
+ reparseType: 'table',
+ },
+ { input: 'x `a|b` y', expect: 'byte' },
+ ],
+ },
+ inlineMath: {
+ shadows: 'mdast-util-math handlers.inlineMath (via remark-math)',
+ obligation:
+ 'emit dollar-delimited inline math with upstream sizing/padding; OK additionally re-emits ' +
+ 'the captured single-dollar source form when the currency-safe promoter re-accepts it',
+ witnesses: [
+ { input: '$$x$$', expect: 'byte' },
+ { input: '$x$', expect: 'byte' },
+ ],
+ },
+ mdxJsxFlowElement: {
+ shadows: 'mdast-util-mdx-jsx handlers.mdxJsxFlowElement (via the mdx utilities)',
+ obligation:
+ 'serialize JSX flow elements (name, attributes, children) as parseable MDX; OK re-emits ' +
+ 'captured source for byte fidelity',
+ witnesses: [{ input: '<Callout type="info">\nbody\n</Callout>', expect: 'byte' }],
+ },
+ mdxJsxTextElement: {
+ shadows: 'mdast-util-mdx-jsx handlers.mdxJsxTextElement (via the mdx utilities)',
+ obligation:
+ 'serialize inline JSX elements as parseable MDX; OK re-emits captured source for byte fidelity',
+ witnesses: [{ input: 'a <File name="x" /> b', expect: 'byte' }],
+ },
+ table: {
+ shadows: 'mdast-util-gfm-table handlers.table (via remark-gfm)',
+ obligation:
+ 'emit the GFM table grid (header, delimiter row, body; cell |-escaping; alignment ' +
+ 'markers); OK additionally threads captured padding/dash/outer-pipe source forms',
+ witnesses: [
+ {
+ input: '| a | b |\n| --- | --- |\n| 1 | 2 |',
+ expect: 'byte-and-reparse-type',
+ reparseType: 'table',
+ },
+ {
+ input: '| l | r |\n| :-- | --: |\n| 1 | 2 |',
+ expect: 'byte-and-reparse-type',
+ reparseType: 'table',
+ },
+ ],
+ },
+};
+
+export function collectExtensionHandlerKeys(extensions: readonly unknown[]): string[] {
+ const keys = new Set<string>();
+ const visit = (ext: unknown): void => {
+ if (ext === null || typeof ext !== 'object') return;
+ const rec = ext as { handlers?: unknown; extensions?: unknown };
+ if (rec.handlers !== null && typeof rec.handlers === 'object') {
+ for (const key of Object.keys(rec.handlers as Record<string, unknown>)) keys.add(key);
+ }
+ if (Array.isArray(rec.extensions)) rec.extensions.forEach(visit);
+ };
+ extensions.forEach(visit);
+ return [...keys].sort();
+}
+
+export function computeShadowedHandlerKeys(
+ extensionHandlerKeys: readonly string[],
+ overrideKeys: readonly string[],
+): string[] {
+ const overrides = new Set(overrideKeys);
+ return extensionHandlerKeys.filter((key) => overrides.has(key)).sort();
+}
diff --git a/packages/core/src/markdown/highlight-promoter.ts b/packages/core/src/markdown/highlight-promoter.ts
index 16e39cb1f..f758cf737 100644
--- a/packages/core/src/markdown/highlight-promoter.ts
+++ b/packages/core/src/markdown/highlight-promoter.ts
@@ -1,7 +1,9 @@
import type { Nodes, Parent, PhrasingContent, Root, Text } from 'mdast';
import type { MdxJsxTextElement } from 'mdast-util-mdx';
import { SKIP, visit } from 'unist-util-visit';
+import type { VFile } from 'vfile';
import type { MarkMdast } from './mdast-augmentation.ts';
+import { deriveFragmentPosition } from './promoter-position.ts';
const HIGHLIGHT_RE = /(?<!=)==(?=\S)([^\n]*?[^\s=])==(?!=)/g;
@@ -178,7 +180,8 @@ function walkPromote(node: Nodes): void {
}
export function highlightPromoterPlugin() {
- return (tree: Root) => {
+ return (tree: Root, file: VFile) => {
+ const source = typeof file.value === 'string' ? file.value : '';
visit(tree, 'text', (node: Text, index, parent) => {
if (parent === undefined || index === undefined || index === null) return;
@@ -201,17 +204,26 @@ export function highlightPromoterPlugin() {
const end = start + match[0].length;
if (start > cursor) {
const lead: Text = { type: 'text', value: value.slice(cursor, start) };
+ const pos = deriveFragmentPosition(source, node, cursor, start);
+ if (pos) lead.position = pos;
replacements.push(lead);
}
+ const innerText: Text = { type: 'text', value: match[1] };
+ const innerPos = deriveFragmentPosition(source, node, start + 2, end - 2);
+ if (innerPos) innerText.position = innerPos;
const markNode: MarkMdast = {
type: 'mark',
- children: [{ type: 'text', value: match[1] }],
+ children: [innerText],
};
+ const markPos = deriveFragmentPosition(source, node, start, end);
+ if (markPos) markNode.position = markPos;
replacements.push(markNode as unknown as PhrasingContent);
cursor = end;
}
if (cursor < value.length) {
const tail: Text = { type: 'text', value: value.slice(cursor) };
+ const pos = deriveFragmentPosition(source, node, cursor, value.length);
+ if (pos) tail.position = pos;
replacements.push(tail);
}
diff --git a/packages/core/src/markdown/index.ts b/packages/core/src/markdown/index.ts
index 248a9e766..05203d59b 100644
--- a/packages/core/src/markdown/index.ts
+++ b/packages/core/src/markdown/index.ts
@@ -15,6 +15,7 @@ import type {
Break,
Code,
Definition,
+ Delete,
Emphasis,
FootnoteDefinition,
FootnoteReference,
@@ -109,13 +110,7 @@ export class MarkdownManager {
pmNodeHandlers: this.pmNodeHandlers,
pmMarkHandlers: this.pmMarkHandlers,
});
- this.serializeProcessor = createSerializeProcessor({
- schema: this.schema,
- handlers: this.handlers,
- pmNodeHandlers: this.pmNodeHandlers,
- pmMarkHandlers: this.pmMarkHandlers,
- toMarkdownHandlers,
- });
+ this.serializeProcessor = createSerializeProcessor({ toMarkdownHandlers });
}
parse(markdown: string, opts?: ParseContext): JSONContent {
@@ -240,6 +235,7 @@ function isEmptyMdastParagraph(node: MdastNodes): boolean {
interface InlineCodeFidelityData {
sourceFenceChar?: string;
sourceFenceLength?: number;
+ sourcePadded?: boolean;
}
function withInlineCodeData(
@@ -252,6 +248,9 @@ function withInlineCodeData(
if (typeof data.sourceFenceLength === 'number') {
dataEntry.sourceFenceLength = data.sourceFenceLength;
}
+ if (data.sourcePadded === true) {
+ dataEntry.sourcePadded = true;
+ }
if (Object.keys(dataEntry).length === 0) return node as unknown as MdastNodes;
return { ...node, data: dataEntry } as unknown as MdastNodes;
}
@@ -299,7 +298,6 @@ import {
FILE_ATTACHMENT_EXTENSIONS,
IMAGE_EXTENSIONS,
VIDEO_EXTENSIONS,
- WIKI_EMBED_EXTENSIONS,
} from '../constants/upload.ts';
const WIKI_EMBED_IMAGE_EXTS = IMAGE_EXTENSIONS;
@@ -386,7 +384,12 @@ function buildMdastToPmHandlers(
if (rowNode) pmRows.push(rowNode);
}
const sourceDashCounts = node.data?.sourceDashCounts ?? null;
- return n.table.createAndFill({ sourceDashCounts }, pmRows.length > 0 ? pmRows : null);
+ const sourceOuterPipes = node.data?.sourceOuterPipes ?? null;
+ const sourceAlignmentPadding = node.data?.sourceAlignmentPadding ?? null;
+ return n.table.createAndFill(
+ { sourceDashCounts, sourceOuterPipes, sourceAlignmentPadding },
+ pmRows.length > 0 ? pmRows : null,
+ );
};
}
@@ -489,12 +492,17 @@ function buildMdastToPmHandlers(
sourceFenceChar: node.data?.sourceFenceChar ?? '`',
sourceFenceLength:
typeof node.data?.sourceFenceLength === 'number' ? node.data.sourceFenceLength : 1,
+ sourcePadded: node.data?.sourcePadded === true,
}),
]);
}
const strikeMark = m.strike ?? m.delete;
- if (strikeMark) handlers.delete = toPmMark(strikeMark);
+ if (strikeMark) {
+ handlers.delete = toPmMark(strikeMark, (node: Delete) => ({
+ sourceDelimiter: node.data?.sourceDelimiter === '~' ? '~' : '~~',
+ }));
+ }
if (m.highlight) handlers.mark = toPmMark(m.highlight);
@@ -530,6 +538,8 @@ function buildMdastToPmHandlers(
sourceTrailingHashes: node.data?.sourceTrailingHashes ?? null,
sourceUnderlineLength: node.data?.sourceUnderlineLength ?? null,
sourceContiguousNext: node.data?.sourceContiguousNext ?? false,
+ sourceLeadingIndent: node.data?.sourceLeadingIndent ?? null,
+ sourceInteriorSpacing: node.data?.sourceInteriorSpacing ?? null,
}));
}
@@ -543,6 +553,10 @@ function buildMdastToPmHandlers(
fenceDelimiter: node.data?.sourceFenceChar ?? '`',
fenceLength: node.data?.sourceFenceLength ?? 3,
sourceStyle: node.data?.sourceStyle ?? 'fenced',
+ sourceClosingFenceLength: node.data?.sourceClosingFenceLength ?? null,
+ sourceFenceIndent: node.data?.sourceFenceIndent ?? null,
+ sourceInfoPadding: node.data?.sourceInfoPadding ?? null,
+ sourceIndents: node.data?.sourceIndents ?? null,
},
textContent,
);
@@ -576,6 +590,10 @@ function buildMdastToPmHandlers(
handlers.listItem = toPmNode(n.listItem, (node: ListItem) => ({
checked: node.checked ?? null,
spread: !!node.spread,
+ sourceMarkerSpacing: node.data?.sourceMarkerSpacing ?? null,
+ sourceOrdinal: node.data?.sourceOrdinal ?? null,
+ sourceCheckboxChar: node.data?.sourceCheckboxChar ?? null,
+ sourceContinuationIndent: node.data?.sourceContinuationIndent ?? null,
}));
}
@@ -742,6 +760,9 @@ function buildMdastToPmHandlers(
target: node.data?.target ?? '',
alias: node.data?.alias ?? null,
anchor: node.data?.anchor ?? null,
+ sourceTarget: node.data?.sourceTarget ?? null,
+ sourceAnchor: node.data?.sourceAnchor ?? null,
+ sourceAlias: node.data?.sourceAlias ?? null,
});
}
@@ -833,7 +854,7 @@ function buildMdastToPmHandlers(
}
const label = alias || (anchor ? `${target}#${anchor}` : target);
- if (WIKI_EMBED_EXTENSIONS.has(ext) && m.link) {
+ if (m.link) {
const linkMark = m.link.create({
href: srcOrTarget,
title: null,
@@ -847,16 +868,6 @@ function buildMdastToPmHandlers(
return schema.text(label, [linkMark]);
}
- if (m.link) {
- const linkMark = m.link.create({
- href: target,
- title: null,
- linkStyle: 'inline',
- refLabel: null,
- });
- return schema.text(label, [linkMark]);
- }
-
throw new Error(
'[wikiLinkEmbed handler] schema lacks `link` mark — cannot dispatch ' +
'without violating the STOP rule against emitting PM wikiLinkEmbed server-side',
@@ -898,8 +909,15 @@ function buildMdastToPmHandlers(
};
if (n.mathInline) {
- handlers.inlineMath = (node: { type: 'inlineMath'; value?: string }) =>
- n.mathInline.create({ formula: node.value ?? '' });
+ handlers.inlineMath = (node: {
+ type: 'inlineMath';
+ value?: string;
+ data?: { sourceDelimiter?: string };
+ }) =>
+ n.mathInline.create({
+ formula: node.value ?? '',
+ sourceDelimiter: node.data?.sourceDelimiter ?? null,
+ });
}
if (!handlers.math) handlers.math = blockUnknownHandler;
@@ -995,7 +1013,7 @@ function buildPmToMdastHandlers(schema: Schema): {
if (n.blockquote) {
nodeHandlers.blockquote = fromPmNode('blockquote', (pmNode: PmNode) => {
const spacings = pmNode.attrs.sourceMarkerSpacings;
- const data: { sourceMarkerSpacings?: Array<'single' | 'none'> } = {};
+ const data: { sourceMarkerSpacings?: Array<number | 'single' | 'none'> } = {};
if (Array.isArray(spacings) && spacings.length > 0) {
data.sourceMarkerSpacings = spacings;
}
@@ -1011,6 +1029,8 @@ function buildPmToMdastHandlers(schema: Schema): {
sourceTrailingHashes: pmNode.attrs.sourceTrailingHashes ?? undefined,
sourceUnderlineLength: pmNode.attrs.sourceUnderlineLength ?? undefined,
sourceContiguousNext: pmNode.attrs.sourceContiguousNext === true ? true : undefined,
+ sourceLeadingIndent: pmNode.attrs.sourceLeadingIndent ?? undefined,
+ sourceInteriorSpacing: pmNode.attrs.sourceInteriorSpacing ?? undefined,
},
}));
}
@@ -1030,6 +1050,10 @@ function buildPmToMdastHandlers(schema: Schema): {
sourceFenceChar: pmNode.attrs.fenceDelimiter,
sourceFenceLength: pmNode.attrs.fenceLength,
sourceStyle,
+ sourceClosingFenceLength: pmNode.attrs.sourceClosingFenceLength ?? undefined,
+ sourceFenceIndent: pmNode.attrs.sourceFenceIndent ?? undefined,
+ sourceInfoPadding: pmNode.attrs.sourceInfoPadding ?? undefined,
+ sourceIndents: pmNode.attrs.sourceIndents ?? undefined,
},
};
};
@@ -1066,10 +1090,24 @@ function buildPmToMdastHandlers(schema: Schema): {
const children = state.all(pmNode);
const stripped =
children.length > 1 && isEmptyMdastParagraph(children[0]) ? children.slice(1) : children;
+ const itemData: NonNullable<ListItem['data']> = {};
+ if (typeof pmNode.attrs.sourceMarkerSpacing === 'number') {
+ itemData.sourceMarkerSpacing = pmNode.attrs.sourceMarkerSpacing;
+ }
+ if (typeof pmNode.attrs.sourceOrdinal === 'number') {
+ itemData.sourceOrdinal = pmNode.attrs.sourceOrdinal;
+ }
+ if (pmNode.attrs.sourceCheckboxChar === 'X') {
+ itemData.sourceCheckboxChar = 'X';
+ }
+ if (typeof pmNode.attrs.sourceContinuationIndent === 'number') {
+ itemData.sourceContinuationIndent = pmNode.attrs.sourceContinuationIndent;
+ }
return {
type: 'listItem' as const,
checked: pmNode.attrs.checked ?? null,
spread: pmNode.attrs.spread ?? false,
+ ...(Object.keys(itemData).length > 0 ? { data: itemData } : {}),
children: stripped,
} as ListItem;
};
@@ -1090,10 +1128,25 @@ function buildPmToMdastHandlers(schema: Schema): {
(c): c is TableRow => (c as MdastNodes).type === 'tableRow',
);
const sourceDashCounts = pmNode.attrs.sourceDashCounts;
+ const tableData: NonNullable<Table['data']> = {};
+ if (Array.isArray(sourceDashCounts) && sourceDashCounts.length > 0) {
+ tableData.sourceDashCounts = sourceDashCounts as number[];
+ }
+ const outerPipes = pmNode.attrs.sourceOuterPipes;
+ if (
+ outerPipes !== null &&
+ typeof outerPipes === 'object' &&
+ typeof (outerPipes as { leading?: unknown }).leading === 'boolean' &&
+ typeof (outerPipes as { trailing?: unknown }).trailing === 'boolean'
+ ) {
+ tableData.sourceOuterPipes = outerPipes as { leading: boolean; trailing: boolean };
+ }
+ const alignPadding = pmNode.attrs.sourceAlignmentPadding;
+ if (Array.isArray(alignPadding) && alignPadding.length > 0) {
+ tableData.sourceAlignmentPadding = alignPadding as Array<{ left: number; right: number }>;
+ }
const data: Table['data'] | undefined =
- Array.isArray(sourceDashCounts) && sourceDashCounts.length > 0
- ? { sourceDashCounts: sourceDashCounts as number[] }
- : undefined;
+ Object.keys(tableData).length > 0 ? tableData : undefined;
const result: Table = {
type: 'table' as const,
align,
@@ -1262,9 +1315,13 @@ function buildPmToMdastHandlers(schema: Schema): {
children: [],
} as unknown as MdastNodes;
}
+ const sourceDelimiter = pmNode.attrs.sourceDelimiter;
return {
type: 'inlineMath' as const,
value: formula,
+ ...(typeof sourceDelimiter === 'string' && sourceDelimiter.length > 0
+ ? { data: { sourceDelimiter } }
+ : {}),
} as unknown as MdastNodes;
};
}
@@ -1275,10 +1332,16 @@ function buildPmToMdastHandlers(schema: Schema): {
const anchor: string | null = pmNode.attrs.anchor ?? null;
const alias: string | null = pmNode.attrs.alias ?? null;
const label = alias ? alias : anchor ? `${target}#${anchor}` : target;
+ const sourceTarget: string | null =
+ typeof pmNode.attrs.sourceTarget === 'string' ? pmNode.attrs.sourceTarget : null;
+ const sourceAnchor: string | null =
+ typeof pmNode.attrs.sourceAnchor === 'string' ? pmNode.attrs.sourceAnchor : null;
+ const sourceAlias: string | null =
+ typeof pmNode.attrs.sourceAlias === 'string' ? pmNode.attrs.sourceAlias : null;
return {
type: 'wikiLink' as const,
value: label,
- data: { target, anchor, alias },
+ data: { target, anchor, alias, sourceTarget, sourceAnchor, sourceAlias },
children: [{ type: 'text' as const, value: label }],
} as unknown as MdastNodes;
};
@@ -1323,14 +1386,17 @@ function buildPmToMdastHandlers(schema: Schema): {
markHandlers.code = (mark: PmMark, _parent: PmNode, children: MdastNodes[]) => {
const sourceFenceChar = mark.attrs.sourceFenceChar as string | undefined;
const sourceFenceLength = mark.attrs.sourceFenceLength as number | undefined;
- return wrapAsInlineCode(children, { sourceFenceChar, sourceFenceLength });
+ const sourcePadded = mark.attrs.sourcePadded === true ? true : undefined;
+ return wrapAsInlineCode(children, { sourceFenceChar, sourceFenceLength, sourcePadded });
};
}
const strikeMark = m.strike ?? m.delete;
if (strikeMark) {
const name = m.strike ? 'strike' : 'delete';
- markHandlers[name] = fromPmMark('delete');
+ markHandlers[name] = fromPmMark('delete', (mark: PmMark) => ({
+ data: { sourceDelimiter: mark.attrs.sourceDelimiter === '~' ? '~' : '~~' },
+ }));
}
if (m.highlight) {
@@ -1380,11 +1446,18 @@ function buildPmToMdastHandlers(schema: Schema): {
typeof mark.attrs.anchor === 'string' && mark.attrs.anchor.length > 0
? mark.attrs.anchor
: null;
- const alias: string | null =
+ let alias: string | null =
typeof mark.attrs.alias === 'string' && mark.attrs.alias.length > 0
? mark.attrs.alias
: null;
- const label = alias ? alias : anchor ? `${target}#${anchor}` : target;
+ let label = alias ? alias : anchor ? `${target}#${anchor}` : target;
+ const visibleText = children
+ .map((child) => ('value' in child && typeof child.value === 'string' ? child.value : ''))
+ .join('');
+ if (visibleText !== label && visibleText.trim().length > 0 && !visibleText.includes(']]')) {
+ alias = visibleText;
+ label = visibleText;
+ }
return {
type: 'wikiLinkEmbed' as const,
value: label,
diff --git a/packages/core/src/markdown/inline-promoter-position-preservation.test.ts b/packages/core/src/markdown/inline-promoter-position-preservation.test.ts
new file mode 100644
index 000000000..e99ac4d24
--- /dev/null
+++ b/packages/core/src/markdown/inline-promoter-position-preservation.test.ts
@@ -0,0 +1,219 @@
+import { describe, expect, test } from 'bun:test';
+import type { Root as MdastRoot, Nodes } from 'mdast';
+import { visit } from 'unist-util-visit';
+
+import { sharedExtensions } from '../extensions/shared.ts';
+import { createParseProcessor, parseMdToMdast } from './pipeline.ts';
+
+const proc = createParseProcessor({ extensions: sharedExtensions });
+
+function parse(source: string): MdastRoot {
+ return parseMdToMdast(source, proc);
+}
+
+function findNodesByType(tree: MdastRoot, type: string): Nodes[] {
+ const found: Nodes[] = [];
+ visit(tree, type, (node) => {
+ found.push(node as Nodes);
+ });
+ return found;
+}
+
+function assertHasPosition(node: Nodes) {
+ expect(node.position).toBeDefined();
+ expect(typeof node.position?.start.offset).toBe('number');
+ expect(typeof node.position?.end?.offset).toBe('number');
+}
+
+function getStartOffset(node: Nodes): number {
+ return node.position?.start?.offset ?? -1;
+}
+
+function getEndOffset(node: Nodes): number {
+ return node.position?.end?.offset ?? -1;
+}
+
+function slicePosition(source: string, node: Nodes): string {
+ const s = getStartOffset(node);
+ const e = getEndOffset(node);
+ if (s < 0 || e < 0) return '';
+ return source.slice(s, e);
+}
+
+describe('inline-promoter position preservation', () => {
+ describe('single-dollar-math ($x$)', () => {
+ test('inlineMath node has position pointing to $', () => {
+ const source = 'Hello $x$ world';
+ const tree = parse(source);
+ const nodes = findNodesByType(tree, 'inlineMath');
+ expect(nodes.length).toBe(1);
+ assertHasPosition(nodes[0]);
+ expect(source[getStartOffset(nodes[0])]).toBe('$');
+ });
+
+ test('text fragments around $a$ have position', () => {
+ const source = 'Hello $a$ world';
+ const tree = parse(source);
+ const paragraph = tree.children[0] as Nodes & { children: Nodes[] };
+ for (const child of paragraph.children) {
+ assertHasPosition(child);
+ }
+ });
+
+ test('multiple math nodes preserve ordered positions', () => {
+ const source = 'See $a$ and $b$ here';
+ const tree = parse(source);
+ const nodes = findNodesByType(tree, 'inlineMath');
+ expect(nodes.length).toBe(2);
+ for (const n of nodes) assertHasPosition(n);
+ expect(getStartOffset(nodes[0])).toBeLessThan(getStartOffset(nodes[1]));
+ });
+ });
+
+ describe('highlight (==text==)', () => {
+ test('mark node has position starting at ==', () => {
+ const source = 'Hello ==word== world';
+ const tree = parse(source);
+ const nodes = findNodesByType(tree, 'mark');
+ expect(nodes.length).toBe(1);
+ assertHasPosition(nodes[0]);
+ expect(source.slice(getStartOffset(nodes[0]), getStartOffset(nodes[0]) + 2)).toBe('==');
+ });
+
+ test('text fragments around ==a== have position', () => {
+ const source = 'Hello ==a== world';
+ const tree = parse(source);
+ const paragraph = tree.children[0] as Nodes & { children: Nodes[] };
+ for (const child of paragraph.children) {
+ assertHasPosition(child);
+ }
+ });
+ });
+
+ describe('comment (%%text%%)', () => {
+ test('%%note%% comment has position starting at %%', () => {
+ const source = 'Hello %%note%% world';
+ const tree = parse(source);
+ const nodes = findNodesByType(tree, 'comment');
+ expect(nodes.length).toBe(1);
+ assertHasPosition(nodes[0]);
+ expect(source.slice(getStartOffset(nodes[0]), getStartOffset(nodes[0]) + 2)).toBe('%%');
+ });
+
+ test('<!-- html --> comment has position', () => {
+ const source = 'Hello <!-- note --> world';
+ const tree = parse(source);
+ const nodes = findNodesByType(tree, 'comment');
+ expect(nodes.length).toBe(1);
+ assertHasPosition(nodes[0]);
+ });
+ });
+
+ describe('tag (#tagname)', () => {
+ test('#hello tag has position pointing to #', () => {
+ const source = 'See #hello there';
+ const tree = parse(source);
+ const nodes = findNodesByType(tree, 'tag');
+ expect(nodes.length).toBe(1);
+ assertHasPosition(nodes[0]);
+ expect(source[getStartOffset(nodes[0])]).toBe('#');
+ });
+
+ test('multiple tags preserve position', () => {
+ const source = '#a and #b';
+ const tree = parse(source);
+ const nodes = findNodesByType(tree, 'tag');
+ expect(nodes.length).toBe(2);
+ for (const n of nodes) assertHasPosition(n);
+ });
+ });
+
+ describe('autolink (<scheme:uri>)', () => {
+ test('autolink link has position', () => {
+ const source = 'Visit <https://x.com> now';
+ const tree = parse(source);
+ const links: Nodes[] = [];
+ visit(tree, 'link', (node) => {
+ const data = (node as Nodes & { data?: Record<string, unknown> }).data;
+ if (data?.sourceStyle === 'autolink') links.push(node as Nodes);
+ });
+ expect(links.length).toBe(1);
+ assertHasPosition(links[0]);
+ });
+ });
+
+ describe('chain: mixed promoters in one paragraph', () => {
+ test('$x$ + ==highlight== both have position', () => {
+ const source = 'Math $x$ and ==hi== text';
+ const tree = parse(source);
+ const math = findNodesByType(tree, 'inlineMath');
+ const marks = findNodesByType(tree, 'mark');
+ expect(math.length).toBe(1);
+ expect(marks.length).toBe(1);
+ assertHasPosition(math[0]);
+ assertHasPosition(marks[0]);
+ });
+
+ test('#tag + %%comment%% both have position', () => {
+ const source = 'See #tag %%note%% end';
+ const tree = parse(source);
+ const tags = findNodesByType(tree, 'tag');
+ const comments = findNodesByType(tree, 'comment');
+ expect(tags.length).toBe(1);
+ expect(comments.length).toBe(1);
+ assertHasPosition(tags[0]);
+ assertHasPosition(comments[0]);
+ });
+ });
+
+ describe('escape-aware offset accuracy', () => {
+ test('\\* before $x$ — math offset points to $', () => {
+ const source = '\\*text\\* and $x$ here';
+ const tree = parse(source);
+ const nodes = findNodesByType(tree, 'inlineMath');
+ expect(nodes.length).toBeGreaterThanOrEqual(1);
+ expect(source[getStartOffset(nodes[0])]).toBe('$');
+ });
+
+ test('\\# is not promoted to tag', () => {
+ const source = 'See \\#notag here';
+ const tree = parse(source);
+ const tags = findNodesByType(tree, 'tag');
+ expect(tags.length).toBe(0);
+ });
+ });
+
+ describe('position byte accuracy', () => {
+ test('mark position spans ==world==', () => {
+ const source = 'Hello ==world== there';
+ const tree = parse(source);
+ const marks = findNodesByType(tree, 'mark');
+ expect(marks.length).toBe(1);
+ expect(slicePosition(source, marks[0])).toBe('==world==');
+ });
+
+ test('inlineMath position spans $abc$', () => {
+ const source = 'Hello $abc$ there';
+ const tree = parse(source);
+ const nodes = findNodesByType(tree, 'inlineMath');
+ expect(nodes.length).toBe(1);
+ expect(slicePosition(source, nodes[0])).toBe('$abc$');
+ });
+
+ test('tag position spans #hello', () => {
+ const source = 'See #hello there';
+ const tree = parse(source);
+ const tags = findNodesByType(tree, 'tag');
+ expect(tags.length).toBe(1);
+ expect(slicePosition(source, tags[0])).toBe('#hello');
+ });
+
+ test('comment position spans %%note%%', () => {
+ const source = 'See %%note%% there';
+ const tree = parse(source);
+ const comments = findNodesByType(tree, 'comment');
+ expect(comments.length).toBe(1);
+ expect(slicePosition(source, comments[0])).toBe('%%note%%');
+ });
+ });
+});
diff --git a/packages/core/src/markdown/math-block-guard.precision.test.ts b/packages/core/src/markdown/math-block-guard.precision.test.ts
index 9a16dbdb7..af928eac7 100644
--- a/packages/core/src/markdown/math-block-guard.precision.test.ts
+++ b/packages/core/src/markdown/math-block-guard.precision.test.ts
@@ -77,11 +77,11 @@ describe('inline math (single-line `$$x$$` only) → mathInline atom', () => {
expect(out).toBe(source);
});
- test('single-dollar `$x^2$` normalizes to `$$x^2$$` on serialize (D-M5b normalization)', () => {
+ test('single-dollar `$x^2$` round-trips byte-identical on serialize', () => {
const source = 'Result: $x^2$.\n';
const json = mdManager.parse(source);
const out = mdManager.serialize(json);
- expect(out).toBe('Result: $$x^2$$.\n');
+ expect(out).toBe('Result: $x^2$.\n');
});
test('`<InlineMath formula="x" id="eq-1" />` round-trips byte-identical (id-bearing JSX form)', () => {
diff --git a/packages/core/src/markdown/mdast-augmentation.ts b/packages/core/src/markdown/mdast-augmentation.ts
index 04b84e4d6..757424675 100644
--- a/packages/core/src/markdown/mdast-augmentation.ts
+++ b/packages/core/src/markdown/mdast-augmentation.ts
@@ -23,6 +23,9 @@ export interface WikiLinkMdast {
target: string;
alias: string | null;
anchor: string | null;
+ sourceTarget?: string | null;
+ sourceAnchor?: string | null;
+ sourceAlias?: string | null;
[key: string]: unknown;
};
children: Array<{ type: 'text'; value: string }>;
@@ -36,6 +39,12 @@ export interface WikiLinkEmbedMdast {
target: string;
alias: string | null;
anchor: string | null;
+ /** Untrimmed source segments — see WikiLinkMdast. Captured by the
+ * shared micromark exits; the embed serialization path currently
+ * ignores them (embed fidelity is tracked separately). */
+ sourceTarget?: string | null;
+ sourceAnchor?: string | null;
+ sourceAlias?: string | null;
[key: string]: unknown;
};
children: Array<{ type: 'text'; value: string }>;
@@ -91,7 +100,20 @@ export interface TagMdast {
position?: Position;
}
+export interface SourceDocBoundary {
+ bom?: true;
+ leading?: string;
+ trailing?: string;
+ gapBlankLines?: ReadonlyArray<number | null>;
+}
+
declare module 'mdast' {
+ interface Data {
+ sourcePrecedingBlankLines?: number;
+ }
+ interface RootData {
+ sourceDocBoundary?: SourceDocBoundary;
+ }
interface TextData {
escapedChars?: Array<{ offset: number; char: string }>;
sourceRaw?: string;
@@ -121,27 +143,45 @@ declare module 'mdast' {
interface HeadingData {
sourceStyle?: string;
sourceTrailingHashes?: number;
+ sourceLeadingIndent?: number;
+ sourceInteriorSpacing?: number;
sourceUnderlineLength?: number;
sourceContiguousNext?: boolean;
}
interface CodeData {
sourceFenceChar?: string;
sourceFenceLength?: number;
+ sourceClosingFenceLength?: number;
+ sourceFenceIndent?: number;
+ sourceInfoPadding?: number;
sourceStyle?: 'indented' | 'fenced';
+ sourceIndents?: string[];
}
interface InlineCodeData {
sourceFenceChar?: string;
sourceFenceLength?: number;
+ sourcePadded?: boolean;
}
interface ListData {
bulletMarker?: string;
listMarkerDelimiter?: string;
}
+ interface ListItemData {
+ sourceMarkerSpacing?: number;
+ sourceOrdinal?: number;
+ sourceCheckboxChar?: 'X';
+ sourceContinuationIndent?: number;
+ }
+ interface DeleteData {
+ sourceDelimiter?: '~' | '~~';
+ }
interface BlockquoteData {
- sourceMarkerSpacings?: Array<'single' | 'none'>;
+ sourceMarkerSpacings?: Array<number | 'single' | 'none'>;
}
interface TableData {
sourceDashCounts?: number[];
+ sourceOuterPipes?: { leading: boolean; trailing: boolean };
+ sourceAlignmentPadding?: Array<{ left: number; right: number }>;
}
interface TableCellData {
sourcePadding?: { left: number; right: number };
@@ -179,3 +219,9 @@ declare module 'mdast-util-mdx-expression' {
sourceRaw?: string;
}
}
+
+declare module 'mdast-util-math' {
+ interface InlineMathData {
+ sourceDelimiter?: string;
+ }
+}
diff --git a/packages/core/src/markdown/mdast-to-hast-handlers.ts b/packages/core/src/markdown/mdast-to-hast-handlers.ts
index 97f9c528c..679c72ef7 100644
--- a/packages/core/src/markdown/mdast-to-hast-handlers.ts
+++ b/packages/core/src/markdown/mdast-to-hast-handlers.ts
@@ -2,6 +2,8 @@ import type { Comment, Element, ElementContent, Properties } from 'hast';
import type { FootnoteDefinition, FootnoteReference } from 'mdast';
import type { MdxJsxFlowElement, MdxJsxTextElement } from 'mdast-util-mdx';
import type { Handler, Handlers } from 'mdast-util-to-hast';
+import { builtInComponents } from '../registry/built-ins.ts';
+import type { CompatMeta } from '../registry/types.ts';
import { wikiLinkHref } from '../utils/slug.ts';
import type {
MarkMdast,
@@ -14,25 +16,58 @@ import type {
const HTML_PRIMITIVE_TAGS = new Set(['img', 'video', 'audio', 'mark']);
-function tryNativeHtmlPrimitive(node: MdxJsxFlowElement | MdxJsxTextElement): Element | null {
- const name = node.name;
- if (!name || !HTML_PRIMITIVE_TAGS.has(name)) return null;
- const properties: Properties = {};
+const compatPrimitiveByName: ReadonlyMap<string, CompatMeta> = new Map(
+ builtInComponents.flatMap((meta) =>
+ meta.surface === 'compat' && HTML_PRIMITIVE_TAGS.has(meta.rendersAs)
+ ? [[meta.name, meta] as const]
+ : [],
+ ),
+);
+
+function collectStaticJsxAttributes(
+ node: MdxJsxFlowElement | MdxJsxTextElement,
+): Record<string, string | true> | null {
+ const bag: Record<string, string | true> = {};
for (const attr of node.attributes) {
if (attr.type !== 'mdxJsxAttribute') return null;
const lowerName = attr.name.toLowerCase();
if (lowerName.length >= 3 && lowerName.startsWith('on')) continue;
if (attr.value === null) {
- properties[attr.name] = true;
+ bag[attr.name] = true;
} else if (typeof attr.value === 'string') {
- properties[attr.name] = attr.value;
+ bag[attr.name] = attr.value;
} else {
return null;
}
}
+ return bag;
+}
+
+function tryNativeHtmlPrimitive(node: MdxJsxFlowElement | MdxJsxTextElement): Element | null {
+ const name = node.name;
+ if (!name || !HTML_PRIMITIVE_TAGS.has(name)) return null;
+ const properties = collectStaticJsxAttributes(node);
+ if (properties === null) return null;
return { type: 'element', tagName: name, properties, children: [] };
}
+function tryCompatPrimitive(node: MdxJsxFlowElement | MdxJsxTextElement): Element | null {
+ const name = node.name;
+ if (!name) return null;
+ const meta = compatPrimitiveByName.get(name);
+ if (!meta) return null;
+ const bag = collectStaticJsxAttributes(node);
+ if (bag === null) return null;
+ const translated = meta.translateProps(bag);
+ const properties: Properties = {};
+ for (const [key, value] of Object.entries(translated)) {
+ if (typeof value === 'string' || typeof value === 'boolean') {
+ properties[key] = value;
+ }
+ }
+ return { type: 'element', tagName: meta.rendersAs, properties, children: [] };
+}
+
const wikiLinkHandler: Handler = (state, node) => {
const wiki = node as WikiLinkMdast;
const { target, anchor, alias } = wiki.data;
@@ -81,6 +116,15 @@ const mdxJsxFlowHandler: Handler = (state, node) => {
state.patch(node, native);
return state.applyData(node, native);
}
+ const compat = tryCompatPrimitive(jsx);
+ if (compat) {
+ state.patch(node, compat);
+ const resolved = state.applyData(node, compat) as Element;
+ if (resolved.tagName === 'img') {
+ return { type: 'element', tagName: 'p', properties: {}, children: [resolved] };
+ }
+ return resolved;
+ }
const raw = typeof jsx.data?.sourceRaw === 'string' ? jsx.data.sourceRaw : '';
const code: Element = {
type: 'element',
@@ -108,6 +152,11 @@ const mdxJsxTextHandler: Handler = (state, node) => {
state.patch(node, native);
return state.applyData(node, native);
}
+ const compat = tryCompatPrimitive(jsx);
+ if (compat) {
+ state.patch(node, compat);
+ return state.applyData(node, compat);
+ }
const raw = typeof jsx.data?.sourceRaw === 'string' ? jsx.data.sourceRaw : '';
const span: Element = {
type: 'element',
diff --git a/packages/core/src/markdown/mdast-to-html.test.ts b/packages/core/src/markdown/mdast-to-html.test.ts
index 90c34609a..732140210 100644
--- a/packages/core/src/markdown/mdast-to-html.test.ts
+++ b/packages/core/src/markdown/mdast-to-html.test.ts
@@ -1,6 +1,14 @@
import { describe, expect, test } from 'bun:test';
+import { sharedExtensions } from '../extensions/shared.ts';
+import { MarkdownManager } from './index.ts';
import { markdownToHtml, mdastToHtml } from './mdast-to-html.ts';
+const mdManager = new MarkdownManager({ extensions: sharedExtensions });
+
+function renderFaithful(md: string): string {
+ return mdastToHtml(mdManager.parseToMdast(md));
+}
+
describe('markdownToHtml — markdown string → HTML', () => {
test('paragraph', () => {
expect(markdownToHtml('hello world')).toBe('<p>hello world</p>');
@@ -258,3 +266,96 @@ describe('URL scheme filter — outbound clipboard HTML sanitization', () => {
expect(html2).not.toMatch(/data:/i);
});
});
+
+describe('faithful render path — compat media descriptors resolve to native elements', () => {
+ test('CommonMarkImage emits <p><img></p>, not the mdx-component placeholder', () => {
+ expect(renderFaithful('![foo](/url "title")')).toBe(
+ '<p><img src="/url" alt="foo" title="title"></p>',
+ );
+ });
+
+ test('empty alt is preserved as alt="" (decorative opt-in), not dropped', () => {
+ expect(renderFaithful('![](/url)')).toBe('<p><img src="/url" alt=""></p>');
+ });
+
+ test('nested image description flattens to plain-text alt', () => {
+ expect(renderFaithful('![foo ![bar](/url)](/url2)')).toBe(
+ '<p><img src="/url2" alt="foo bar"></p>',
+ );
+ });
+
+ test('inline image inside prose stays a native inline <img>', () => {
+ expect(renderFaithful('before ![pic](/i.png) after')).toBe(
+ '<p>before <img src="/i.png" alt="pic"> after</p>',
+ );
+ });
+
+ test('lowercase MDX <img> keeps its bare flow shape (no paragraph wrap)', () => {
+ expect(renderFaithful('<img src="/x.png" alt="y" />')).toBe('<img src="/x.png" alt="y">');
+ });
+
+ test('Callout component descriptor keeps the mdx-component placeholder', () => {
+ const html = renderFaithful('> [!NOTE]\n> body');
+ expect(html).toContain('mdx-component');
+ expect(html).not.toContain('<img');
+ });
+
+ test('event-handler attributes are skipped on the compat path', () => {
+ const html = renderFaithful('<CommonMarkImage src="/x.png" alt="y" onerror="alert(1)" />');
+ expect(html).toContain('<img');
+ expect(html).not.toContain('onerror');
+ });
+});
+
+describe('faithful render path — source-preserved url/title attrs decode at render', () => {
+ test('empty angle-bracket destination renders href=""', () => {
+ expect(renderFaithful('[link](<>)')).toBe('<p><a href="">link</a></p>');
+ });
+
+ test('empty angle-bracket destination via reference definition renders href=""', () => {
+ expect(renderFaithful('[foo]: <>\n\n[foo]')).toBe('<p><a href="">foo</a></p>');
+ });
+
+ test('entity refs in a link title decode to their characters', () => {
+ expect(renderFaithful('[link](/url "title \\"&quot;")')).toBe(
+ '<p><a href="/url" title="title &#x22;&#x22;">link</a></p>',
+ );
+ });
+
+ test('entity refs in a destination decode then percent-encode like the spec', () => {
+ expect(renderFaithful('[foo](/f&ouml;&ouml; "f&ouml;&ouml;")')).toBe(
+ '<p><a href="/f%C3%B6%C3%B6" title="föö">foo</a></p>',
+ );
+ });
+
+ test('entity-SHAPED-but-invalid references stay literal', () => {
+ const html = renderFaithful('[x](/a "t&hl;t")');
+ expect(html).toContain('t&#x26;hl;t');
+ });
+
+ test('entity refs in a fenced-code info string decode in the language- class', () => {
+ expect(renderFaithful('```f&ouml;&ouml;\ncode\n```')).toBe(
+ '<pre><code class="language-föö">code\n</code></pre>',
+ );
+ });
+
+ test('entity-SHAPED-but-invalid info-string reference stays literal in the class', () => {
+ const html = renderFaithful('```f&hl;\ncode\n```');
+ expect(html).toContain('language-f&#x26;hl;');
+ });
+
+ test('body-text entity refs are NOT decoded (storage-fidelity dialect untouched)', () => {
+ const html = renderFaithful('&amp; in prose');
+ expect(html).toContain('&#x26;amp; in prose');
+ });
+
+ test('entity-obfuscated javascript: scheme is still stripped', () => {
+ const html = renderFaithful('[x](java&#x73;cript:alert(1))');
+ expect(html).not.toMatch(/javascript:/i);
+ });
+
+ test('entity-obfuscated scheme on an image src is still stripped', () => {
+ const html = renderFaithful('![x](java&#x73;cript:alert(1))');
+ expect(html).not.toMatch(/javascript:/i);
+ });
+});
diff --git a/packages/core/src/markdown/mdast-to-html.ts b/packages/core/src/markdown/mdast-to-html.ts
index be6c79033..79072375c 100644
--- a/packages/core/src/markdown/mdast-to-html.ts
+++ b/packages/core/src/markdown/mdast-to-html.ts
@@ -1,5 +1,6 @@
import type { Element, Root as HastRoot } from 'hast';
import type { Root as MdastRoot } from 'mdast';
+import { normalizeUri } from 'micromark-util-sanitize-uri';
import rehypeStringify from 'rehype-stringify';
import remarkFrontmatter from 'remark-frontmatter';
import remarkGfm from 'remark-gfm';
@@ -7,6 +8,7 @@ import remarkParse from 'remark-parse';
import remarkRehype from 'remark-rehype';
import { type Plugin, unified } from 'unified';
import { visit } from 'unist-util-visit';
+import { decodeEntityRefs } from './entity-ref-guard.ts';
import { customNodeHandlers } from './mdast-to-hast-handlers.ts';
import { remarkMdxAgnostic } from './remark-mdx-agnostic.ts';
import { isSafeUrl } from './safe-url.ts';
@@ -33,6 +35,53 @@ function isSrcsetValueSafe(srcset: string): boolean {
return true;
}
+const DECODE_ATTRS_BY_TAG: Record<string, readonly string[]> = {
+ a: ['href', 'title'],
+ img: ['src', 'alt', 'title'],
+ video: ['src', 'title'],
+ audio: ['src', 'title'],
+ code: ['className'],
+};
+
+const URL_ATTRS = new Set(['href', 'src']);
+
+const NORMALIZED_EMPTY_ANGLE_DEST = normalizeUri('<>');
+
+const rehypeDecodeSourcePreservedAttrs: Plugin<[], HastRoot> = () => {
+ return (tree) => {
+ visit(tree, 'element', (node: Element) => {
+ const attrs = DECODE_ATTRS_BY_TAG[node.tagName.toLowerCase()];
+ const props = node.properties;
+ if (!attrs || !props) return;
+ for (const key of attrs) {
+ const value = props[key];
+ if (Array.isArray(value)) {
+ const decodedItems = value.map((item) =>
+ typeof item === 'string' ? decodeEntityRefs(item) : item,
+ );
+ if (decodedItems.some((item, i) => item !== value[i])) {
+ props[key] = decodedItems;
+ }
+ continue;
+ }
+ if (typeof value !== 'string') continue;
+ let decoded: string;
+ if (URL_ATTRS.has(key)) {
+ if (value === '<>' || value === NORMALIZED_EMPTY_ANGLE_DEST) {
+ decoded = '';
+ } else {
+ const chars = decodeEntityRefs(value);
+ decoded = chars === value ? value : normalizeUri(chars);
+ }
+ } else {
+ decoded = decodeEntityRefs(value);
+ }
+ if (decoded !== value) props[key] = decoded;
+ }
+ });
+ };
+};
+
const rehypeSanitizeUrls: Plugin<[], HastRoot> = () => {
return (tree) => {
visit(tree, 'element', (node: Element) => {
@@ -93,6 +142,7 @@ const rehypeSanitizeUrls: Plugin<[], HastRoot> = () => {
export function mdastToHtml(tree: MdastRoot): string {
const processor = unified()
.use(remarkRehype, { handlers: customNodeHandlers })
+ .use(rehypeDecodeSourcePreservedAttrs)
.use(rehypeSanitizeUrls)
.use(rehypeStringify);
const hast = processor.runSync(tree) as unknown as HastRoot;
@@ -107,6 +157,7 @@ export function markdownToHtml(md: string): string {
.use(remarkGfm)
.use(remarkWikiLink)
.use(remarkRehype, { handlers: customNodeHandlers })
+ .use(rehypeDecodeSourcePreservedAttrs)
.use(rehypeSanitizeUrls)
.use(rehypeStringify);
return String(processor.processSync(md));
diff --git a/packages/core/src/markdown/merged-walker.test.ts b/packages/core/src/markdown/merged-walker.test.ts
index 7055a192b..a2611acd3 100644
--- a/packages/core/src/markdown/merged-walker.test.ts
+++ b/packages/core/src/markdown/merged-walker.test.ts
@@ -34,7 +34,7 @@ describe('mergedPostParseWalkerPlugin — Phase B dispatch', () => {
expect(para.children[0].data?.sourceStyle).toBe('autolink');
});
- test('pass 3: doc-start empty yaml → thematicBreak(s) with data.sourceRaw', () => {
+ test('pass 3 retired: a root-position empty yaml node is left untouched', () => {
const src = '---\n\n---\n';
const tree: MdastRoot = {
type: 'root',
@@ -45,11 +45,8 @@ describe('mergedPostParseWalkerPlugin — Phase B dispatch', () => {
};
mergedPostParseWalkerPlugin()(tree, new VFile(src));
- expect(tree.children.length).toBe(2);
- expect(tree.children[0].type).toBe('thematicBreak');
- expect(tree.children[1].type).toBe('thematicBreak');
- expect(tree.children[0].data?.sourceRaw).toBe('---');
- expect(tree.children[1].data?.sourceRaw).toBe('---');
+ expect(tree.children.length).toBe(1);
+ expect(tree.children[0].type).toBe('yaml');
});
test('pass 4: attaches sourceDelimiter to emphasis node based on source', () => {
@@ -183,7 +180,7 @@ describe('mergedPostParseWalkerPlugin — Phase B dispatch', () => {
expect(fallback.originalType).toBe('brandNew');
});
- test('non-root-first yaml is left untouched (pass 3 is root[0]-specific)', () => {
+ test('non-root-first yaml is left untouched', () => {
const tree = {
type: 'root',
children: [
diff --git a/packages/core/src/markdown/merged-walker.ts b/packages/core/src/markdown/merged-walker.ts
index 873e3f2aa..35d74b85e 100644
--- a/packages/core/src/markdown/merged-walker.ts
+++ b/packages/core/src/markdown/merged-walker.ts
@@ -2,7 +2,6 @@ import type { Nodes, Parent, Root } from 'mdast';
import { SKIP, visit } from 'unist-util-visit';
import type { VFile } from 'vfile';
import { promoteInParent } from './autolink-promotion.ts';
-import { applyDocStartThematicFix } from './doc-start-thematic-fix.ts';
import { applyPositionSliceToNode } from './position-slice.ts';
import { promoteTagsInParent } from './tag-promotion.ts';
import { KNOWN_MDAST_TYPES, toRawMdxFallbackMdast } from './unknown-mdast-guard.ts';
@@ -11,8 +10,6 @@ export function mergedPostParseWalkerPlugin() {
return (tree: Root, file: VFile) => {
const source = typeof file.value === 'string' ? file.value : '';
- applyDocStartThematicFix(tree, file);
-
const debug = typeof process !== 'undefined' && process.env?.OK_DEBUG_POSITION_SLICE === '1';
visit(tree, (node, index, parent) => {
@@ -30,12 +27,17 @@ export function mergedPostParseWalkerPlugin() {
if ('children' in node && Array.isArray((node as Parent).children)) {
const parentLike = node as Parent;
if (parentLike.children.some((c) => c.type === 'text')) {
- promoteInParent(parentLike);
+ promoteInParent(parentLike, source);
promoteTagsInParent(parentLike, source);
}
}
- applyPositionSliceToNode(node as Nodes, source, debug);
+ applyPositionSliceToNode(
+ node as Nodes,
+ source,
+ debug,
+ parent as { type?: string } | undefined,
+ );
if (
node.type === 'heading' &&
diff --git a/packages/core/src/markdown/parser-drop-closure.ts b/packages/core/src/markdown/parser-drop-closure.ts
new file mode 100644
index 000000000..e6ab4fd57
--- /dev/null
+++ b/packages/core/src/markdown/parser-drop-closure.ts
@@ -0,0 +1,411 @@
+export type DroppedTokenAdjudication =
+ | {
+ kind: 'format-dof-axis';
+ axisIds: readonly string[];
+ rationale: string;
+ }
+ | {
+ kind: 'structural-only';
+ rationale: string;
+ }
+ | {
+ kind: 'retained-by-capture';
+ witness: string;
+ rationale: string;
+ }
+ | {
+ kind: 'documented-residual';
+ witnesses: ReadonlyArray<{ input: string; roundTrip: string }>;
+ rationale: string;
+ };
+
+export const DROPPED_TOKEN_ADJUDICATIONS: Record<string, DroppedTokenAdjudication> = {
+ atxHeadingText: {
+ kind: 'structural-only',
+ rationale: 'heading-text container; bytes ride inside its phrasing children',
+ },
+ autolinkMarker: {
+ kind: 'format-dof-axis',
+ axisIds: ['autolink:angle'],
+ rationale: 'the < > wrapper of an autolink; the angle form is the witnessed axis',
+ },
+ blockQuoteMarker: {
+ kind: 'format-dof-axis',
+ axisIds: ['blockquote:marker-spacing'],
+ rationale: 'the > marker byte; its spacing variation is the witnessed axis',
+ },
+ blockQuotePrefix: {
+ kind: 'format-dof-axis',
+ axisIds: ['blockquote:marker-spacing'],
+ rationale: 'marker + following whitespace; covered by the marker-spacing axis',
+ },
+ blockQuotePrefixWhitespace: {
+ kind: 'format-dof-axis',
+ axisIds: ['blockquote:marker-spacing'],
+ rationale:
+ 'the whitespace after the > marker; the marker-spacing axis varies exactly these bytes',
+ },
+ characterReferenceMarker: {
+ kind: 'format-dof-axis',
+ axisIds: ['lexical:char-reference'],
+ rationale:
+ 'the & and ; of an entity reference; preserved by the entity-ref guard, witnessed by the lexical axis',
+ },
+ chunkContent: {
+ kind: 'structural-only',
+ rationale: 'tokenizer re-chunking artifact; spans only bytes owned by nested tokens',
+ },
+ chunkDocument: {
+ kind: 'structural-only',
+ rationale: 'tokenizer re-chunking artifact; spans only bytes owned by nested tokens',
+ },
+ chunkFlow: {
+ kind: 'structural-only',
+ rationale: 'tokenizer re-chunking artifact; spans only bytes owned by nested tokens',
+ },
+ chunkString: {
+ kind: 'structural-only',
+ rationale: 'tokenizer re-chunking artifact; spans only bytes owned by nested tokens',
+ },
+ chunkText: {
+ kind: 'structural-only',
+ rationale: 'tokenizer re-chunking artifact; spans only bytes owned by nested tokens',
+ },
+ codeFencedFenceSequence: {
+ kind: 'format-dof-axis',
+ axisIds: ['code-fence:char', 'code-fence:symmetric-length', 'code-fence:closing-length'],
+ rationale:
+ 'the fence run itself; char, length, and closing-length variation are the witnessed axes',
+ },
+ codeTextPadding: {
+ kind: 'retained-by-capture',
+ witness: '` x `',
+ rationale: 'inline-code pad space; retained via the inlineCode sourcePadded capture',
+ },
+ codeTextSequence: {
+ kind: 'retained-by-capture',
+ witness: '``x``',
+ rationale: 'inline-code fence run; retained via the inlineCode fence-length capture',
+ },
+ content: {
+ kind: 'structural-only',
+ rationale: 'flow-content container; spans only bytes owned by nested tokens',
+ },
+ definitionDestination: {
+ kind: 'structural-only',
+ rationale:
+ 'destination container; bytes ride in its literal/raw child tokens into mdast definition.url',
+ },
+ definitionDestinationLiteral: {
+ kind: 'retained-by-capture',
+ witness: '[a]: </u v>',
+ rationale: 'angle-bracketed definition destination; the angle form round-trips',
+ },
+ definitionDestinationLiteralMarker: {
+ kind: 'retained-by-capture',
+ witness: '[a]: </u v>',
+ rationale: 'the < > of an angle-bracketed definition destination; round-trips with the literal',
+ },
+ definitionDestinationRaw: {
+ kind: 'retained-by-capture',
+ witness: '[a]: /u',
+ rationale: 'raw definition destination; bytes land in mdast definition.url and re-emit',
+ },
+ definitionLabel: {
+ kind: 'retained-by-capture',
+ witness: '[A]: /u\n\n[a]',
+ rationale: 'definition label with authored casing; the label byte form re-emits',
+ },
+ definitionLabelMarker: {
+ kind: 'structural-only',
+ rationale: 'fixed [ ] bytes around the definition label; no source-form freedom',
+ },
+ definitionMarker: {
+ kind: 'structural-only',
+ rationale: 'the fixed : byte after the definition label; no source-form freedom',
+ },
+ definitionTitle: {
+ kind: 'retained-by-capture',
+ witness: "[a]: /u 'ti'",
+ rationale: 'definition title with authored quote style; the quote form re-emits',
+ },
+ definitionTitleMarker: {
+ kind: 'retained-by-capture',
+ witness: "[a]: /u 'ti'",
+ rationale: 'the quote bytes of a definition title; round-trip with the title',
+ },
+ emphasisSequence: {
+ kind: 'format-dof-axis',
+ axisIds: ['emphasis:delimiter'],
+ rationale: 'the * or _ run; delimiter choice is the witnessed axis',
+ },
+ emphasisText: {
+ kind: 'structural-only',
+ rationale: 'emphasis-content container; bytes ride inside its phrasing children',
+ },
+ escapeMarker: {
+ kind: 'format-dof-axis',
+ axisIds: ['lexical:backslash-escape'],
+ rationale: 'the backslash of an escape; preserved end-to-end, witnessed by the lexical axis',
+ },
+ labelEnd: {
+ kind: 'structural-only',
+ rationale: 'fixed ] byte; the reference-form freedom is retained via mdast referenceType',
+ },
+ labelImage: {
+ kind: 'structural-only',
+ rationale: 'image-label container; fixed ![ ] bytes plus child phrasing',
+ },
+ labelImageMarker: {
+ kind: 'structural-only',
+ rationale: 'the fixed ! byte of an image label; no source-form freedom',
+ },
+ labelLink: {
+ kind: 'structural-only',
+ rationale: 'link-label container; fixed [ ] bytes plus child phrasing',
+ },
+ labelMarker: {
+ kind: 'structural-only',
+ rationale: 'fixed [ or ] byte; no source-form freedom',
+ },
+ lineEndingBlank: {
+ kind: 'format-dof-axis',
+ axisIds: [
+ 'block-separator:blank-line-count',
+ 'doc-boundary:doc-leading-blank',
+ 'doc-boundary:doc-trailing-blank',
+ ],
+ rationale:
+ 'blank-line newlines; leading/trailing/inter-block blank runs are the witnessed axes',
+ },
+ linePrefix: {
+ kind: 'format-dof-axis',
+ axisIds: ['lexical:tab-expansion', 'atx:leading-indent', 'code-fence:leading-indent'],
+ rationale: 'leading line indent (tabs or spaces); the indent axes vary exactly these bytes',
+ },
+ lineSuffix: {
+ kind: 'documented-residual',
+ witnesses: [
+ { input: 'a \nb', roundTrip: 'a\nb\n' },
+ { input: 'a \nb', roundTrip: 'a \nb\n' },
+ ],
+ rationale:
+ 'trailing intra-line whitespace: a single trailing space is stripped (CommonMark soft line ' +
+ 'breaks treat it as presentational) and a hard-break space run beyond two normalizes to two. ' +
+ 'Not an enumerated format-DOF axis and not captured today; pinned here so any shift is loud.',
+ },
+ listItemIndent: {
+ kind: 'format-dof-axis',
+ axisIds: ['list:nested-indent-width'],
+ rationale: 'continuation indent of a list item; the nested-indent axis varies these bytes',
+ },
+ listItemMarker: {
+ kind: 'format-dof-axis',
+ axisIds: [
+ 'list:bullet-plus',
+ 'list:ordered-paren-delim',
+ 'list:ordered-start-number',
+ 'list:ordered-renumber',
+ ],
+ rationale: 'the bullet or ordinal marker; marker char and ordinal axes vary these bytes',
+ },
+ listItemPrefix: {
+ kind: 'format-dof-axis',
+ axisIds: ['list:item-marker-spacing'],
+ rationale: 'marker + following whitespace; the marker-spacing axis varies these bytes',
+ },
+ listItemPrefixWhitespace: {
+ kind: 'format-dof-axis',
+ axisIds: ['list:item-marker-spacing'],
+ rationale:
+ 'the whitespace after a list marker; the marker-spacing axis varies exactly these bytes',
+ },
+ referenceMarker: {
+ kind: 'structural-only',
+ rationale:
+ 'fixed [ ] bytes of a reference; the form freedom is retained via mdast referenceType',
+ },
+ resourceDestination: {
+ kind: 'structural-only',
+ rationale: 'destination container; bytes ride in its literal/raw child tokens into mdast url',
+ },
+ resourceDestinationLiteral: {
+ kind: 'format-dof-axis',
+ axisIds: ['link:angle-url'],
+ rationale: 'angle-bracketed resource destination; the angle form is the witnessed axis',
+ },
+ resourceDestinationLiteralMarker: {
+ kind: 'format-dof-axis',
+ axisIds: ['link:angle-url'],
+ rationale: 'the < > of an angle-bracketed destination; covered by the angle-url axis',
+ },
+ resourceDestinationRaw: {
+ kind: 'retained-by-capture',
+ witness: '[t](/u)',
+ rationale: 'raw resource destination; bytes land in mdast url and re-emit via the link handler',
+ },
+ resourceMarker: {
+ kind: 'structural-only',
+ rationale: 'fixed ( ) bytes of a resource; no source-form freedom',
+ },
+ resourceTitle: {
+ kind: 'structural-only',
+ rationale:
+ 'title container; the title value rides into mdast title, quote style via the title-quote axis',
+ },
+ resourceTitleMarker: {
+ kind: 'format-dof-axis',
+ axisIds: ['link:title-quote'],
+ rationale: 'the quote bytes of a resource title; quote style is the witnessed axis',
+ },
+ setextHeadingLine: {
+ kind: 'format-dof-axis',
+ axisIds: ['setext:underline'],
+ rationale: 'the setext underline run; the underline form is the witnessed axis',
+ },
+ strongSequence: {
+ kind: 'format-dof-axis',
+ axisIds: ['strong:delimiter'],
+ rationale: 'the ** or __ run; delimiter choice is the witnessed axis',
+ },
+ strongText: {
+ kind: 'structural-only',
+ rationale: 'strong-content container; bytes ride inside its phrasing children',
+ },
+ thematicBreakSequence: {
+ kind: 'format-dof-axis',
+ axisIds: ['thematic:marker-mid-doc', 'thematic:doc-start-forcing'],
+ rationale:
+ 'the thematic-break marker run; marker form and doc-start forcing are the witnessed axes',
+ },
+ whitespace: {
+ kind: 'documented-residual',
+ witnesses: [{ input: '[a]: /u', roundTrip: '[a]: /u\n' }],
+ rationale:
+ 'inter-part whitespace inside definition/resource constructs collapses to a single space on ' +
+ 'serialize. Not an enumerated format-DOF axis and not captured today; pinned here so any ' +
+ 'shift is loud.',
+ },
+};
+
+export type PreprocessTransformAdjudication =
+ | { kind: 'format-dof-axis'; axisIds: readonly string[]; rationale: string }
+ | { kind: 'structural-only'; rationale: string }
+ | {
+ kind: 'spec-mandated-replacement';
+ witness: { input: string; roundTrip: string };
+ rationale: string;
+ };
+
+export const PREPROCESS_TRANSFORM_ADJUDICATIONS: Record<string, PreprocessTransformAdjudication> = {
+ '\0': {
+ kind: 'spec-mandated-replacement',
+ witness: { input: 'a\0b', roundTrip: 'a�b\n' },
+ rationale:
+ 'CommonMark 2.3 (insecure characters): U+0000 must be replaced with U+FFFD. A divergence no ' +
+ 'fix may close; doubles as the instrument-alive control in the format-DOF gates.',
+ },
+ '\t': {
+ kind: 'format-dof-axis',
+ axisIds: ['lexical:tab-expansion'],
+ rationale:
+ 'tabs encode as virtual spaces in preprocess; the indent-tab byte form is the witnessed axis',
+ },
+ '\n': {
+ kind: 'structural-only',
+ rationale:
+ 'LF chunk encoding; LF is the canonical line ending and round-trips byte-identically',
+ },
+ '\r': {
+ kind: 'format-dof-axis',
+ axisIds: ['doc-boundary:line-ending-normalize'],
+ rationale:
+ 'CR / CRLF encoding; line-ending form is the witnessed axis (OK retains interior CRLF)',
+ },
+ 'bom-head-check': {
+ kind: 'format-dof-axis',
+ axisIds: ['doc-boundary:bom-strip'],
+ rationale:
+ 'preprocess strips a document-head U+FEFF before any event exists; OK captures it with its ' +
+ 'own pre-parse head check, witnessed by the bom-strip axis',
+ },
+};
+
+export const PINNED_PREPROCESS_SOURCE_SHA256 =
+ 'b582f16d9bc04d93d388004544be9c5293fda614eb86f8d02b97c3e57b9409e1';
+
+export function extractFromMarkdownHandlerKeys(libSource: string): string[] {
+ const keys = new Set<string>();
+ for (const label of ['enter', 'exit']) {
+ const start = libSource.indexOf(`${label}: {`);
+ if (start < 0) throw new Error(`from-markdown source lost its "${label}" handler block`);
+ let index = libSource.indexOf('{', start);
+ let depth = 0;
+ const lines: string[] = [];
+ let lineStart = index + 1;
+ for (; index < libSource.length; index++) {
+ const ch = libSource[index];
+ if (ch === '{') depth++;
+ else if (ch === '}') {
+ depth--;
+ if (depth === 0) break;
+ } else if (ch === '\n') {
+ if (depth === 1) lines.push(libSource.slice(lineStart, index));
+ lineStart = index + 1;
+ }
+ }
+ if (depth !== 0) throw new Error(`unbalanced "${label}" handler block`);
+ for (const line of lines) {
+ const match = /^\s*([a-zA-Z][\w]*):/.exec(line);
+ if (match?.[1]) keys.add(match[1]);
+ }
+ }
+ if (keys.size === 0) throw new Error('extracted zero from-markdown handler keys');
+ return [...keys].sort();
+}
+
+export function extractPreprocessTransforms(preprocessSource: string): {
+ transformChars: string[];
+ hasBomHeadCheck: boolean;
+} {
+ const match = /const search = \/\[(.+?)\]\/g/.exec(preprocessSource);
+ if (!match?.[1]) throw new Error('preprocess source lost its search regex');
+ const chars: string[] = [];
+ const body = match[1];
+ for (let i = 0; i < body.length; i++) {
+ const ch = body[i];
+ if (ch === '\\') {
+ const next = body[i + 1];
+ i++;
+ if (next === '0') chars.push('\0');
+ else if (next === 't') chars.push('\t');
+ else if (next === 'n') chars.push('\n');
+ else if (next === 'r') chars.push('\r');
+ else if (next !== undefined) chars.push(next);
+ } else if (ch !== undefined) {
+ chars.push(ch);
+ }
+ }
+ return {
+ transformChars: chars,
+ hasBomHeadCheck: /65279|byteOrderMarker/.test(preprocessSource),
+ };
+}
+
+export interface ClosureCheckResult {
+ unadjudicated: string[];
+ stale: string[];
+}
+
+export function checkClosure(
+ computed: readonly string[],
+ adjudicated: Readonly<Record<string, unknown>>,
+): ClosureCheckResult {
+ const computedSet = new Set(computed);
+ return {
+ unadjudicated: computed.filter((key) => !(key in adjudicated)).sort(),
+ stale: Object.keys(adjudicated)
+ .filter((key) => !computedSet.has(key))
+ .sort(),
+ };
+}
diff --git a/packages/core/src/markdown/pipeline.doc-boundary.test.ts b/packages/core/src/markdown/pipeline.doc-boundary.test.ts
new file mode 100644
index 000000000..59ce65552
--- /dev/null
+++ b/packages/core/src/markdown/pipeline.doc-boundary.test.ts
@@ -0,0 +1,139 @@
+import { describe, expect, test } from 'bun:test';
+import type { JSONContent } from '@tiptap/core';
+import { sharedExtensions } from '../extensions/shared.ts';
+import { MarkdownManager } from './index.ts';
+
+const md = new MarkdownManager({ extensions: sharedExtensions });
+
+function rt(source: string): string {
+ return md.serialize(md.parse(source));
+}
+
+function expectByteStable(out: string): void {
+ expect(rt(out)).toBe(out);
+}
+
+describe('indented-code leading-indent capture (tab-expansion)', () => {
+ test('a tab-indented code line round-trips byte-exact', () => {
+ expect(rt('\tfoo\tbar')).toBe('\tfoo\tbar\n');
+ expectByteStable('\tfoo\tbar\n');
+ });
+
+ test('multi-line tab indents round-trip per line', () => {
+ expect(rt('\tfoo\n\tbar')).toBe('\tfoo\n\tbar\n');
+ });
+
+ test('mixed tab/space indents round-trip per line', () => {
+ expect(rt('\tfoo\n bar')).toBe('\tfoo\n bar\n');
+ });
+
+ test('an interior blank line keeps its own (possibly partial) indent bytes', () => {
+ expect(rt('\tfoo\n\n\tbar')).toBe('\tfoo\n\n\tbar\n');
+ });
+
+ test('canonical 4-space indents are unchanged (no capture noise)', () => {
+ expect(rt(' foo')).toBe(' foo\n');
+ expect(rt(' foo')).toBe(' foo\n');
+ });
+
+ test('a WYSIWYG-shaped edit that changes the line count falls back to the canonical indent', () => {
+ const json = md.parse('\tfoo\n\tbar') as JSONContent;
+ const code = json.content?.[0];
+ expect(code?.type).toBe('codeBlock');
+ if (code) code.content = [{ type: 'text', text: 'foo\nbar\nbaz' }];
+ const out = md.serialize(json);
+ expect(out).toBe(' foo\n bar\n baz\n');
+ expectByteStable(out);
+ });
+});
+
+describe('document-head BOM capture', () => {
+ test('a BOM-led document round-trips the BOM', () => {
+ expect(rt('\uFEFFhello')).toBe('\uFEFFhello\n');
+ expectByteStable('\uFEFFhello\n');
+ });
+
+ test('the BOM composes with other captured boundary state', () => {
+ expect(rt('\uFEFF\n\nhello')).toBe('\uFEFF\n\nhello\n');
+ });
+});
+
+describe('BOM offset-rebase latent bug (first-node source-form mis-slice)', () => {
+ test('a BOM-led spaced thematic break keeps its exact glyphs (non-doc-start)', () => {
+ const out = rt('\uFEFFx\n\n- - -');
+ expect(out).toBe('\uFEFFx\n\n- - -\n');
+ expectByteStable(out);
+ });
+
+ test('a BOM-led ATX heading keeps its interior spacing capture', () => {
+ expect(rt('\uFEFF## x')).toBe('\uFEFF## x\n');
+ });
+
+ test('a BOM-led doc-start dash break behaves like its non-BOM twin (glyphs preserved)', () => {
+ expect(rt('\uFEFF- - -')).toBe('\uFEFF- - -\n');
+ expect(rt('- - -')).toBe('- - -\n');
+ expect(md.parseToMdast(rt('- - -')).children[0]?.type).toBe('thematicBreak');
+ });
+
+ test('non-BOM controls are byte-identical to their pre-capture behavior', () => {
+ expect(rt('x\n\n- - -')).toBe('x\n\n- - -\n');
+ expect(rt('## x')).toBe('## x\n');
+ });
+});
+
+describe('doc-leading blank-line capture', () => {
+ test('leading blank lines round-trip', () => {
+ expect(rt('\n\nhello')).toBe('\n\nhello\n');
+ expect(rt('\nhello')).toBe('\nhello\n');
+ expectByteStable('\n\nhello\n');
+ });
+});
+
+describe('doc-trailing blank-line capture', () => {
+ test('trailing blank lines round-trip byte-exact', () => {
+ expect(rt('hello\n\n')).toBe('hello\n\n');
+ expect(rt('hello\n\n\n')).toBe('hello\n\n\n');
+ });
+
+ test('the canonical single final newline is unchanged', () => {
+ expect(rt('hello')).toBe('hello\n');
+ expect(rt('hello\n')).toBe('hello\n');
+ });
+});
+
+describe('inter-block blank-line-count capture', () => {
+ test('extra blank lines between blocks round-trip', () => {
+ expect(rt('a\n\n\n\nb')).toBe('a\n\n\n\nb\n');
+ expect(rt('a\n\n\nb')).toBe('a\n\n\nb\n');
+ expectByteStable('a\n\n\n\nb\n');
+ });
+
+ test('per-boundary counts are independent across three blocks', () => {
+ expect(rt('a\n\n\n\nb\n\n\nc')).toBe('a\n\n\n\nb\n\n\nc\n');
+ });
+
+ test('the canonical single blank line is unchanged', () => {
+ expect(rt('a\n\nb')).toBe('a\n\nb\n');
+ });
+
+ test('a 0-blank adjacency still normalizes to the canonical blank line', () => {
+ expect(rt('# H\nP')).toBe('# H\n\nP\n');
+ });
+
+ test('a WYSIWYG-shaped block deletion drops the stale gap capture', () => {
+ const json = md.parse('a\n\n\n\nb\n\n\n\nc') as JSONContent;
+ expect(json.content?.length).toBe(3);
+ json.content = json.content?.slice(0, 2) ?? [];
+ const out = md.serialize(json);
+ expect(out).toBe('a\n\nb\n');
+ expectByteStable(out);
+ });
+});
+
+describe('composed doc-boundary state', () => {
+ test('BOM + leading + gaps + trailing all round-trip together', () => {
+ const doc = '\uFEFF\n\na\n\n\n\nb\n\n';
+ expect(rt(doc)).toBe(doc);
+ expectByteStable(doc);
+ });
+});
diff --git a/packages/core/src/markdown/pipeline.doc-start.test.ts b/packages/core/src/markdown/pipeline.doc-start.test.ts
new file mode 100644
index 000000000..9f4edc781
--- /dev/null
+++ b/packages/core/src/markdown/pipeline.doc-start.test.ts
@@ -0,0 +1,92 @@
+import { describe, expect, test } from 'bun:test';
+import { stripFrontmatter } from '../extensions/frontmatter.ts';
+import { sharedExtensions } from '../extensions/shared.ts';
+import { MarkdownManager } from './index.ts';
+
+const mdManager = new MarkdownManager({ extensions: sharedExtensions });
+
+function rt(source: string): string {
+ return mdManager.serialize(mdManager.parse(source));
+}
+
+function topLevelMdastTypes(source: string): string[] {
+ return mdManager.parseToMdast(source).children.map((child) => child.type);
+}
+
+describe('content after a doc-start thematic break parses per CommonMark', () => {
+ test('a list after a doc-start --- stays a list (no escaped-paragraph collapse)', () => {
+ expect(topLevelMdastTypes('---\n\n- a\n- b')).toEqual(['thematicBreak', 'list']);
+ expect(rt('---\n\n- a\n- b')).toBe('---\n\n- a\n- b\n');
+ });
+
+ test('the recovered-list output is byte-stable across a second round-trip', () => {
+ const r1 = rt('---\n\n- a\n- b');
+ expect(rt(r1)).toBe(r1);
+ });
+});
+
+describe('frontmatter-shaped paste is visible content, never silently dropped', () => {
+ const fmPaste = '---\nkey: val\n---\n\nbody';
+
+ test('every content byte of a pasted frontmatter block stays visible', () => {
+ const out = rt(fmPaste);
+ expect(out).toContain('key: val');
+ expect(out).toContain('body');
+ expect(out).toBe('---\n\nkey: val\n---\n\nbody\n');
+ });
+
+ test('the pasted block parses as thematic break + setext heading + paragraph', () => {
+ expect(topLevelMdastTypes(fmPaste)).toEqual(['thematicBreak', 'heading', 'paragraph']);
+ const heading = mdManager.parseToMdast(fmPaste).children[1];
+ expect(heading.type === 'heading' && heading.depth).toBe(2);
+ });
+
+ test('the visible-content output is byte-stable across a second round-trip', () => {
+ const r1 = rt(fmPaste);
+ expect(rt(r1)).toBe(r1);
+ });
+
+ test('a yaml node never enters the parse tree', () => {
+ for (const input of [fmPaste, '---\ntitle: x\n---\n', '---\n---', '---\n\n---']) {
+ expect(topLevelMdastTypes(input)).not.toContain('yaml');
+ }
+ });
+});
+
+describe('doc-start --- is preserved verbatim (the *** override is retired)', () => {
+ test('a doc-start --- round-trips as ---', () => {
+ expect(rt('---\n\nfoo')).toBe('---\n\nfoo\n');
+ expect(rt('---\n')).toBe('---\n');
+ });
+
+ test('empty-fence shapes keep both --- lines', () => {
+ expect(rt('---\n---')).toBe('---\n\n---\n');
+ expect(rt('---\n\n---')).toBe('---\n\n---\n');
+ });
+
+ test('*** authored as *** still round-trips as ***', () => {
+ expect(rt('***\n\nfoo')).toBe('***\n\nfoo\n');
+ });
+
+ test('mid-doc frontmatter-shaped content is untouched (change is doc-start-scoped)', () => {
+ expect(rt('x\n\n---\nkey: val\n---')).toBe('x\n\n---\n\nkey: val\n---\n');
+ });
+});
+
+describe('the pre-existing intake-strip edge (documented, not changed)', () => {
+ test('a doc-start --- HR paired with a later --- line is claimed as frontmatter', () => {
+ const doc = '---\n\nintro prose\n\n---\n\nbody';
+ expect(stripFrontmatter(doc)).toEqual({
+ frontmatter: '---\n\nintro prose\n\n---\n',
+ body: '\nbody',
+ });
+ });
+
+ test('a doc-start --- with no later --- line passes through to the parser intact', () => {
+ expect(stripFrontmatter('---\n\nbody with no later dashes')).toEqual({
+ frontmatter: '',
+ body: '---\n\nbody with no later dashes',
+ });
+ expect(stripFrontmatter('---\n')).toEqual({ frontmatter: '', body: '---\n' });
+ });
+});
diff --git a/packages/core/src/markdown/pipeline.test.ts b/packages/core/src/markdown/pipeline.test.ts
index 1516bd91c..a691bb6ea 100644
--- a/packages/core/src/markdown/pipeline.test.ts
+++ b/packages/core/src/markdown/pipeline.test.ts
@@ -139,21 +139,31 @@ describe('position-aware blank-line Join (FR-12)', () => {
});
});
- describe('PM round-trip path (positions absent → default 1 blank line)', () => {
+ describe('PM round-trip path (doc-boundary snapshot carries the counts)', () => {
test('1 blank line between paragraphs is the dominant case (byte-identical)', () => {
const input = 'P1\n\nP2\n';
const json = mdManager.parse(input);
expect(mdManager.serialize(json)).toBe(input);
});
- test('2+ blank lines collapse to 1 (NG1 floor on PM route)', () => {
+ test('2+ blank lines round-trip via the sourceDocBoundary attr', () => {
const json = mdManager.parse('P1\n\n\nP2\n');
- expect(mdManager.serialize(json)).toBe('P1\n\nP2\n');
+ expect(mdManager.serialize(json)).toBe('P1\n\n\nP2\n');
});
- test('4 blank lines collapse to 1 on PM route (NG1 floor)', () => {
+ test('4 blank lines round-trip via the sourceDocBoundary attr', () => {
const json = mdManager.parse('P1\n\n\n\n\nP2\n');
- expect(mdManager.serialize(json)).toBe('P1\n\nP2\n');
+ expect(mdManager.serialize(json)).toBe('P1\n\n\n\n\nP2\n');
+ });
+
+ test('a doc without the attr (the CRDT/fragment shape) collapses to the canonical 1 blank line', () => {
+ const json = mdManager.parse('P1\n\n\n\n\nP2\n');
+ const { sourceDocBoundary: _dropped, ...attrs } = (json.attrs ?? {}) as Record<
+ string,
+ unknown
+ >;
+ const stripped = { ...json, attrs };
+ expect(mdManager.serialize(stripped)).toBe('P1\n\nP2\n');
});
});
@@ -165,7 +175,7 @@ describe('position-aware blank-line Join (FR-12)', () => {
expect(round).toEqual(initial);
});
- test('PM identity holds even when source has 2+ blank lines (PM does not carry counts)', () => {
+ test('PM identity holds even when source has 2+ blank lines', () => {
const md = 'P1\n\n\n\nP2\n';
const initial = mdManager.parse(md);
const round = mdManager.parse(mdManager.serialize(initial));
diff --git a/packages/core/src/markdown/pipeline.ts b/packages/core/src/markdown/pipeline.ts
index d9333f6ec..674e08091 100644
--- a/packages/core/src/markdown/pipeline.ts
+++ b/packages/core/src/markdown/pipeline.ts
@@ -26,6 +26,7 @@ import { highlightPromoterPlugin } from './highlight-promoter.ts';
import { imagePromoterPlugin } from './image-promoter.ts';
import { indentedCodePromoterPlugin } from './indented-code-promoter.ts';
import { mathPromoterPlugin } from './math-promoter.ts';
+import type { SourceDocBoundary } from './mdast-augmentation.ts';
import { mergedPostParseWalkerPlugin } from './merged-walker.ts';
import { mermaidPromoterPlugin } from './mermaid-promoter.ts';
import { positionAwareBlankLineJoin } from './position-aware-join.ts';
@@ -65,7 +66,6 @@ function ensureNonEmptyDoc(tree: MdastRoot): MdastRoot {
export const ACTIVE_MDAST_PLUGINS = [
{ name: 'remark-parse', plugin: remarkParse },
- { name: 'remark-frontmatter', plugin: remarkFrontmatter, options: ['yaml'] },
{ name: 'remark-mdx-agnostic', plugin: remarkMdxAgnostic },
{ name: 'remark-gfm', plugin: remarkGfm },
{ name: 'remark-math', plugin: remarkMath, options: { singleDollarTextMath: false } },
@@ -115,7 +115,9 @@ export function createParseProcessor(opts: PipelineOptions): Processor {
return processor;
}
-export function createSerializeProcessor(opts: PipelineOptions): Processor {
+export function createSerializeProcessor(
+ opts: Pick<PipelineOptions, 'toMarkdownHandlers'>,
+): Processor {
const processor = unified()
.use(remarkFrontmatter, ['yaml'])
.use(remarkGfm, { tablePipeAlign: false })
@@ -134,7 +136,90 @@ export function createSerializeProcessor(opts: PipelineOptions): Processor {
return processor as unknown as Processor;
}
-export function parseMd(source: string, processor: Processor): PmNode {
+function splitDocumentHeadBom(source: string): { source: string; hadBom: boolean } {
+ return source.charCodeAt(0) === 0xfeff
+ ? { source: source.slice(1), hadBom: true }
+ : { source, hadBom: false };
+}
+
+function captureDocBoundary(
+ root: MdastRoot,
+ source: string,
+ hadBom: boolean,
+): SourceDocBoundary | undefined {
+ let leading: string | undefined;
+ let trailing: string | undefined;
+ let gapBlankLines: Array<number | null> | undefined;
+
+ const children = root.children;
+ if (children.length > 0) {
+ const firstStart = children[0]?.position?.start?.offset;
+ if (typeof firstStart === 'number' && firstStart > 0) {
+ const gap = source.slice(0, firstStart);
+ if (/^\n+$/.test(gap)) leading = gap;
+ }
+ const lastEnd = children[children.length - 1]?.position?.end?.offset;
+ if (typeof lastEnd === 'number' && lastEnd <= source.length) {
+ const gap = source.slice(lastEnd);
+ if (/^\n{2,}$/.test(gap)) trailing = gap;
+ }
+ const gaps: Array<number | null> = [];
+ let anyGap = false;
+ for (let i = 1; i < children.length; i++) {
+ const prevEnd = children[i - 1]?.position?.end?.offset;
+ const nextStart = children[i]?.position?.start?.offset;
+ let blanks: number | null = null;
+ if (typeof prevEnd === 'number' && typeof nextStart === 'number' && nextStart >= prevEnd) {
+ const gap = source.slice(prevEnd, nextStart);
+ if (/^\n{3,}$/.test(gap)) {
+ blanks = gap.length - 1;
+ anyGap = true;
+ }
+ }
+ gaps.push(blanks);
+ }
+ if (anyGap) gapBlankLines = gaps;
+ }
+
+ if (!hadBom && leading === undefined && trailing === undefined && !gapBlankLines) {
+ return undefined;
+ }
+ const boundary: SourceDocBoundary = {
+ ...(hadBom ? { bom: true as const } : {}),
+ ...(leading !== undefined ? { leading } : {}),
+ ...(trailing !== undefined ? { trailing } : {}),
+ ...(gapBlankLines ? { gapBlankLines } : {}),
+ };
+ root.data ??= {};
+ root.data.sourceDocBoundary = boundary;
+ return boundary;
+}
+
+function readDocBoundary(value: unknown): SourceDocBoundary | undefined {
+ if (typeof value !== 'object' || value === null) return undefined;
+ const v = value as Record<string, unknown>;
+ const out: {
+ bom?: true;
+ leading?: string;
+ trailing?: string;
+ gapBlankLines?: Array<number | null>;
+ } = {};
+ if (v.bom === true) out.bom = true;
+ if (typeof v.leading === 'string' && /^\n+$/.test(v.leading)) out.leading = v.leading;
+ if (typeof v.trailing === 'string' && /^\n{2,}$/.test(v.trailing)) out.trailing = v.trailing;
+ if (
+ Array.isArray(v.gapBlankLines) &&
+ v.gapBlankLines.every(
+ (g) => g === null || (typeof g === 'number' && Number.isInteger(g) && g >= 2),
+ )
+ ) {
+ out.gapBlankLines = v.gapBlankLines as Array<number | null>;
+ }
+ return out.bom || out.leading || out.trailing || out.gapBlankLines ? out : undefined;
+}
+
+export function parseMd(rawSource: string, processor: Processor): PmNode {
+ const { source, hadBom } = splitDocumentHeadBom(rawSource);
const protectedFr14 = encodeBackslashEscapes(source);
const protectedR23 = protectFromMdx(protectedFr14);
const protected_ = encodeEntityRefs(protectedR23);
@@ -142,16 +227,22 @@ export function parseMd(source: string, processor: Processor): PmNode {
const file = new VFile(protected_);
const tree = processor.parse(file);
file.value = source;
- const transformed = processor.runSync(tree, file);
- return (processor as unknown as { stringify(tree: unknown): PmNode }).stringify(transformed);
+ const transformed = processor.runSync(tree, file) as MdastRoot;
+ const boundary = captureDocBoundary(transformed, source, hadBom);
+ const doc = (processor as unknown as { stringify(tree: unknown): PmNode }).stringify(transformed);
+ if (!boundary) return doc;
+ return doc.type.create({ ...doc.attrs, sourceDocBoundary: boundary }, doc.content, doc.marks);
}
-export function parseMdToMdast(source: string, processor: Processor): MdastRoot {
+export function parseMdToMdast(rawSource: string, processor: Processor): MdastRoot {
+ const { source, hadBom } = splitDocumentHeadBom(rawSource);
const protected_ = encodeEntityRefs(protectFromMdx(encodeBackslashEscapes(source)));
const file = new VFile(protected_);
const tree = processor.parse(file);
file.value = source;
- return processor.runSync(tree, file) as MdastRoot;
+ const transformed = processor.runSync(tree, file) as MdastRoot;
+ captureDocBoundary(transformed, source, hadBom);
+ return transformed;
}
export function serializeMd(doc: PmNode, processor: Processor, opts: SerializeMdOptions): string {
@@ -161,5 +252,24 @@ export function serializeMd(doc: PmNode, processor: Processor, opts: SerializeMd
markHandlers: opts.pmMarkHandlers,
});
- return String(processor.stringify(mdast));
+ const boundary = readDocBoundary(doc.attrs?.sourceDocBoundary);
+ const gaps = boundary?.gapBlankLines;
+ if (gaps && gaps.length === mdast.children.length - 1) {
+ for (let i = 1; i < mdast.children.length; i++) {
+ const blanks = gaps[i - 1];
+ if (typeof blanks === 'number') {
+ const child = mdast.children[i];
+ child.data ??= {};
+ child.data.sourcePrecedingBlankLines = blanks;
+ }
+ }
+ }
+
+ let out = String(processor.stringify(mdast));
+ if (boundary?.leading) out = boundary.leading + out;
+ if (boundary?.trailing) {
+ out = out.replace(/\n+$/, '') + boundary.trailing;
+ }
+ if (boundary?.bom) out = `\uFEFF${out}`;
+ return out;
}
diff --git a/packages/core/src/markdown/position-aware-join.ts b/packages/core/src/markdown/position-aware-join.ts
index 524ac285e..248685cfe 100644
--- a/packages/core/src/markdown/position-aware-join.ts
+++ b/packages/core/src/markdown/position-aware-join.ts
@@ -2,7 +2,10 @@ import type { Join } from 'mdast-util-to-markdown';
import type { Position } from 'unist';
type MaybePositioned = { position?: Position };
-type MaybeTyped = { type?: string; data?: { sourceContiguousNext?: boolean } };
+type MaybeTyped = {
+ type?: string;
+ data?: { sourceContiguousNext?: boolean; sourcePrecedingBlankLines?: number };
+};
type FlowNode = Parameters<Join>[0];
function isContiguousSetextWithParagraph(left: FlowNode, right: FlowNode): boolean {
@@ -13,6 +16,8 @@ function isContiguousSetextWithParagraph(left: FlowNode, right: FlowNode): boole
export const positionAwareBlankLineJoin: Join = (left, right) => {
if (isContiguousSetextWithParagraph(left, right)) return 0;
+ const dataGap = (right as MaybeTyped).data?.sourcePrecedingBlankLines;
+ if (typeof dataGap === 'number' && dataGap >= 2) return dataGap;
const lp = (left as MaybePositioned).position;
const rp = (right as MaybePositioned).position;
if (!lp || !rp) return undefined;
diff --git a/packages/core/src/markdown/position-slice.test.ts b/packages/core/src/markdown/position-slice.test.ts
index f13c09d61..6a302a9b0 100644
--- a/packages/core/src/markdown/position-slice.test.ts
+++ b/packages/core/src/markdown/position-slice.test.ts
@@ -643,74 +643,74 @@ describe('position-slice: inline-link title marker recovery (FR-20)', () => {
});
describe('position-slice: blockquote marker spacing recovery (FR-23)', () => {
- test('single-space marker → sourceMarkerSpacings=["single"]', () => {
+ test('single-space marker → sourceMarkerSpacings=[1]', () => {
const tree = parseMdast('> foo\n');
const bq = findNode(tree, 'blockquote');
- expect(bq?.data?.sourceMarkerSpacings).toEqual(['single']);
+ expect(bq?.data?.sourceMarkerSpacings).toEqual([1]);
});
- test('no-space marker → sourceMarkerSpacings=["none"]', () => {
+ test('no-space marker → sourceMarkerSpacings=[0]', () => {
const tree = parseMdast('>foo\n');
const bq = findNode(tree, 'blockquote');
- expect(bq?.data?.sourceMarkerSpacings).toEqual(['none']);
+ expect(bq?.data?.sourceMarkerSpacings).toEqual([0]);
});
test('multi-line single-space → array of length 2', () => {
const tree = parseMdast('> line 1\n> line 2\n');
const bq = findNode(tree, 'blockquote');
- expect(bq?.data?.sourceMarkerSpacings).toEqual(['single', 'single']);
+ expect(bq?.data?.sourceMarkerSpacings).toEqual([1, 1]);
});
test('multi-line no-space → array of length 2', () => {
const tree = parseMdast('>line 1\n>line 2\n');
const bq = findNode(tree, 'blockquote');
- expect(bq?.data?.sourceMarkerSpacings).toEqual(['none', 'none']);
+ expect(bq?.data?.sourceMarkerSpacings).toEqual([0, 0]);
});
test('mixed single + none → per-line array', () => {
const tree = parseMdast('> line 1\n>line 2\n');
const bq = findNode(tree, 'blockquote');
- expect(bq?.data?.sourceMarkerSpacings).toEqual(['single', 'none']);
+ expect(bq?.data?.sourceMarkerSpacings).toEqual([1, 0]);
});
test('mixed none + single → per-line array', () => {
const tree = parseMdast('>line 1\n> line 2\n');
const bq = findNode(tree, 'blockquote');
- expect(bq?.data?.sourceMarkerSpacings).toEqual(['none', 'single']);
+ expect(bq?.data?.sourceMarkerSpacings).toEqual([0, 1]);
});
test('blank-line `>` continuation EXCLUDED from capture', () => {
const tree = parseMdast('> p1\n>\n> p2\n');
const bq = findNode(tree, 'blockquote');
- expect(bq?.data?.sourceMarkerSpacings).toEqual(['single', 'single']);
+ expect(bq?.data?.sourceMarkerSpacings).toEqual([1, 1]);
});
test('nested blockquote captures outer marker spacing on outer node', () => {
const tree = parseMdast('> > nested\n');
const allBqs = findNodes(tree, 'blockquote');
expect(allBqs.length).toBe(2);
- expect(allBqs[0]?.data?.sourceMarkerSpacings).toEqual(['single']);
- expect(allBqs[1]?.data?.sourceMarkerSpacings).toEqual(['single']);
+ expect(allBqs[0]?.data?.sourceMarkerSpacings).toEqual([1]);
+ expect(allBqs[1]?.data?.sourceMarkerSpacings).toEqual([1]);
});
test('nested blockquote `>>` → outer none, inner single', () => {
const tree = parseMdast('>> nested\n');
const allBqs = findNodes(tree, 'blockquote');
expect(allBqs.length).toBe(2);
- expect(allBqs[0]?.data?.sourceMarkerSpacings).toEqual(['none']);
- expect(allBqs[1]?.data?.sourceMarkerSpacings).toEqual(['single']);
+ expect(allBqs[0]?.data?.sourceMarkerSpacings).toEqual([0]);
+ expect(allBqs[1]?.data?.sourceMarkerSpacings).toEqual([1]);
});
test('tab-after-marker treated as single-spacing', () => {
const tree = parseMdast('>\tfoo\n');
const bq = findNode(tree, 'blockquote');
- expect(bq?.data?.sourceMarkerSpacings).toEqual(['single']);
+ expect(bq?.data?.sourceMarkerSpacings).toEqual([1]);
});
test('1-3 leading spaces of indent before `>` tolerated (CommonMark §5.1)', () => {
const tree = parseMdast(' > foo\n');
const bq = findNode(tree, 'blockquote');
- expect(bq?.data?.sourceMarkerSpacings).toEqual(['single']);
+ expect(bq?.data?.sourceMarkerSpacings).toEqual([1]);
});
});
diff --git a/packages/core/src/markdown/position-slice.ts b/packages/core/src/markdown/position-slice.ts
index 7a49c4aa8..e6f7a26e2 100644
--- a/packages/core/src/markdown/position-slice.ts
+++ b/packages/core/src/markdown/position-slice.ts
@@ -34,6 +34,7 @@ export function applyPositionSliceToNode(
node: Nodes,
source: string,
debug: boolean = false,
+ parent?: { type?: string },
): void {
if (!source) return;
const pos = node.position;
@@ -116,6 +117,18 @@ export function applyPositionSliceToNode(
const segment = source.slice(startOff, endOff);
if (prefix === '#') {
node.data.sourceStyle = 'atx';
+ if (parent?.type === 'root') {
+ let lineStart = startOff;
+ while (lineStart > 0 && source[lineStart - 1] !== '\n') lineStart--;
+ const indentSlice = source.slice(lineStart, startOff);
+ if (indentSlice.length >= 1 && indentSlice.length <= 3 && /^ +$/.test(indentSlice)) {
+ node.data.sourceLeadingIndent = indentSlice.length;
+ }
+ }
+ const interior = /^#+( {2,})[^ ]/.exec(segment);
+ if (interior) {
+ node.data.sourceInteriorSpacing = interior[1].length;
+ }
const trailing = /^.*?[ \t](#+)[ \t]*$/.exec(segment);
if (trailing) {
node.data.sourceTrailingHashes = trailing[1].length;
@@ -147,7 +160,7 @@ export function applyPositionSliceToNode(
const slice = source.slice(startOff, endOff);
const lines = slice.split('\n');
- const markerSpacings: Array<'single' | 'none'> = [];
+ const markerSpacings: number[] = [];
for (let lineIdx = 0; lineIdx < lines.length; lineIdx++) {
const line = lines[lineIdx];
if (line.length === 0) continue;
@@ -156,11 +169,12 @@ export function applyPositionSliceToNode(
if (stripped[0] !== '>') continue;
const restAfterMarker = stripped.slice(1).trimEnd();
if (restAfterMarker === '') continue;
- const afterMarker = stripped[1];
- if (afterMarker === ' ' || afterMarker === '\t') {
- markerSpacings.push('single');
+ if (stripped[1] === '\t') {
+ markerSpacings.push(1);
} else {
- markerSpacings.push('none');
+ let spaceRun = 0;
+ while (stripped[1 + spaceRun] === ' ') spaceRun++;
+ markerSpacings.push(spaceRun);
}
}
if (markerSpacings.length > 0) {
@@ -187,6 +201,58 @@ export function applyPositionSliceToNode(
break;
}
+ case 'listItem': {
+ const slice = source.slice(startOff, endOff);
+ const m = /^(?:([-+*])|(\d{1,9})[.)])( *)/.exec(slice);
+ if (!m) break;
+ const markerLen = m[1] ? 1 : (m[2]?.length ?? 0) + 1;
+ const spacing = m[3].length;
+ if (spacing >= 2 && spacing <= 4) {
+ node.data.sourceMarkerSpacing = spacing;
+ }
+ if (m[2]) {
+ node.data.sourceOrdinal = Number.parseInt(m[2], 10);
+ }
+ if (node.checked != null) {
+ const checkbox = /^\[([xX])\] /.exec(slice.slice(markerLen + Math.max(spacing, 1)));
+ if (checkbox?.[1] === 'X') {
+ node.data.sourceCheckboxChar = 'X';
+ }
+ }
+ const effectiveSpacing = spacing >= 2 && spacing <= 4 ? spacing : 1;
+ const contentCol = markerLen + effectiveSpacing;
+ let itemLineStart = startOff;
+ while (itemLineStart > 0 && source[itemLineStart - 1] !== '\n') itemLineStart--;
+ const itemCol = pos.start.column;
+ for (const child of node.children ?? []) {
+ if (child.type !== 'list') continue;
+ const childStart = child.position?.start?.offset;
+ const childCol = child.position?.start?.column;
+ if (typeof childStart !== 'number' || typeof childCol !== 'number') continue;
+ let childLineStart = childStart;
+ while (childLineStart > 0 && source[childLineStart - 1] !== '\n') childLineStart--;
+ if (childLineStart === itemLineStart) continue; // same-line list — no pad applies
+ const indent = childCol - itemCol;
+ if (
+ indent !== contentCol &&
+ indent >= contentCol &&
+ indent <= contentCol + 3 &&
+ /^ +$/.test(source.slice(childLineStart, childStart))
+ ) {
+ node.data.sourceContinuationIndent = indent;
+ }
+ break;
+ }
+ break;
+ }
+
+ case 'delete': {
+ if (source[startOff] === '~') {
+ node.data.sourceDelimiter = source[startOff + 1] === '~' ? '~~' : '~';
+ }
+ break;
+ }
+
case 'code': {
const ch = source[startOff];
if (ch === '`' || ch === '~') {
@@ -199,8 +265,58 @@ export function applyPositionSliceToNode(
if (count >= 3) {
node.data.sourceFenceLength = count;
}
+
+ if (parent?.type === 'root') {
+ let fenceLineStart = startOff;
+ while (fenceLineStart > 0 && source[fenceLineStart - 1] !== '\n') fenceLineStart--;
+ const fenceIndent = source.slice(fenceLineStart, startOff);
+ if (fenceIndent.length >= 1 && fenceIndent.length <= 3 && /^ +$/.test(fenceIndent)) {
+ node.data.sourceFenceIndent = fenceIndent.length;
+ }
+ }
+
+ if (node.lang) {
+ let gap = 0;
+ while (source[startOff + count + gap] === ' ') gap++;
+ if (gap >= 1) {
+ node.data.sourceInfoPadding = gap;
+ }
+ }
+
+ const fenceSlice = source.slice(startOff, endOff);
+ const closing = new RegExp(`\\n[ \\t]*(\\${ch}+)[ \\t]*$`).exec(fenceSlice);
+ if (closing && closing[1].length > count) {
+ node.data.sourceClosingFenceLength = closing[1].length;
+ }
} else {
node.data.sourceStyle = 'indented';
+ if (parent?.type === 'root') {
+ const sliceLines = source.slice(startOff, endOff).split('\n');
+ const valueLines = (node.value ?? '').split('\n');
+ if (sliceLines.length === valueLines.length) {
+ const indents: string[] = [];
+ let nonCanonical = false;
+ let valid = true;
+ for (let i = 0; i < sliceLines.length; i++) {
+ const sliceLine = sliceLines[i];
+ const valueLine = valueLines[i];
+ if (!sliceLine.endsWith(valueLine)) {
+ valid = false;
+ break;
+ }
+ const indent = sliceLine.slice(0, sliceLine.length - valueLine.length);
+ if (!/^[ \t]*$/.test(indent)) {
+ valid = false;
+ break;
+ }
+ indents.push(indent);
+ if (indent !== (valueLine.length > 0 ? ' ' : '')) nonCanonical = true;
+ }
+ if (valid && nonCanonical) {
+ node.data.sourceIndents = indents;
+ }
+ }
+ }
}
break;
}
@@ -216,6 +332,22 @@ export function applyPositionSliceToNode(
if (count >= 1) {
node.data.sourceFenceLength = count;
}
+ const inner = source.slice(startOff + count, endOff - count);
+ const value: string = node.value ?? '';
+ if (inner !== value && inner === ` ${value} `) {
+ node.data.sourcePadded = true;
+ }
+ }
+ break;
+ }
+
+ case 'inlineMath': {
+ if (source[startOff] === '$') {
+ let count = 0;
+ while (startOff + count < source.length && source[startOff + count] === '$') {
+ count++;
+ }
+ node.data.sourceDelimiter = '$'.repeat(count);
}
break;
}
@@ -245,6 +377,50 @@ export function applyPositionSliceToNode(
node.data.sourceDashCounts = dashCounts;
}
+ {
+ const trimmedAlign = alignmentLine.trim();
+ const segments = splitGfmCellSegments(trimmedAlign);
+ if (segments.length > 0 && segments[0] === '' && trimmedAlign.startsWith('|')) {
+ segments.shift();
+ }
+ if (
+ segments.length > 0 &&
+ segments[segments.length - 1] === '' &&
+ trimmedAlign.endsWith('|')
+ ) {
+ segments.pop();
+ }
+ const allDelimiterCells = segments.every((seg) => /^[-: ]+$/.test(seg));
+ const alignPadding: Array<{ left: number; right: number }> = [];
+ for (const seg of segments) {
+ let left = 0;
+ while (left < seg.length && seg[left] === ' ') left++;
+ let right = 0;
+ while (right < seg.length - left && seg[seg.length - 1 - right] === ' ') right++;
+ alignPadding.push({ left, right });
+ }
+ if (allDelimiterCells && alignPadding.length > 0) {
+ node.data.sourceAlignmentPadding = alignPadding;
+ }
+ }
+
+ {
+ const tableLines = tableSrc.split('\n').filter((l) => l.trim().length > 0);
+ if (tableLines.length >= 2) {
+ const leadingStates = tableLines.map((l) => l.trimStart().startsWith('|'));
+ const trailingStates = tableLines.map((l) => /(^|[^\\])\|$/.test(l.trimEnd()));
+ const uniformLeading = leadingStates.every((s) => s === leadingStates[0]);
+ const uniformTrailing = trailingStates.every((s) => s === trailingStates[0]);
+ if (uniformLeading && uniformTrailing) {
+ const leading = leadingStates[0];
+ const trailing = trailingStates[0];
+ if (!leading || !trailing) {
+ node.data.sourceOuterPipes = { leading, trailing };
+ }
+ }
+ }
+ }
+
for (const row of node.children ?? []) {
if (row.type !== 'tableRow' || !row.position) continue;
const rowStart = row.position.start.offset;
@@ -370,8 +546,8 @@ export function positionSlicePlugin() {
const debug = typeof process !== 'undefined' && process.env?.OK_DEBUG_POSITION_SLICE === '1';
- visit(tree, (node: Nodes) => {
- applyPositionSliceToNode(node, source, debug);
+ visit(tree, (node: Nodes, _index, parent) => {
+ applyPositionSliceToNode(node, source, debug, parent ?? undefined);
});
};
}
diff --git a/packages/core/src/markdown/promoter-position.ts b/packages/core/src/markdown/promoter-position.ts
new file mode 100644
index 000000000..ea7949ee1
--- /dev/null
+++ b/packages/core/src/markdown/promoter-position.ts
@@ -0,0 +1,62 @@
+import type { Text } from 'mdast';
+import type { Point, Position } from 'unist';
+
+function valueOffsetToSourceOffset(
+ source: string,
+ parentSourceStart: number,
+ valueText: string,
+ targetValueOffset: number,
+): number {
+ let srcCursor = parentSourceStart;
+ let valCursor = 0;
+ while (valCursor < targetValueOffset && srcCursor < source.length) {
+ if (
+ source[srcCursor] === '\\' &&
+ srcCursor + 1 < source.length &&
+ valueText[valCursor] === source[srcCursor + 1]
+ ) {
+ srcCursor += 2;
+ valCursor += 1;
+ } else {
+ srcCursor += 1;
+ valCursor += 1;
+ }
+ }
+ return srcCursor;
+}
+
+function offsetToPoint(source: string, offset: number): Point {
+ let line = 1;
+ let column = 1;
+ for (let i = 0; i < offset && i < source.length; i++) {
+ if (source[i] === '\n') {
+ line += 1;
+ column = 1;
+ } else {
+ column += 1;
+ }
+ }
+ return { line, column, offset };
+}
+
+function makePosition(source: string, startOffset: number, endOffset: number): Position {
+ return {
+ start: offsetToPoint(source, startOffset),
+ end: offsetToPoint(source, endOffset),
+ };
+}
+
+export function deriveFragmentPosition(
+ source: string,
+ parentNode: Text,
+ valueStart: number,
+ valueEnd: number,
+): Position | undefined {
+ if (!source || !parentNode.position || typeof parentNode.position.start?.offset !== 'number') {
+ return undefined;
+ }
+ const parentOff = parentNode.position.start.offset;
+ const srcStart = valueOffsetToSourceOffset(source, parentOff, parentNode.value, valueStart);
+ const srcEnd = valueOffsetToSourceOffset(source, parentOff, parentNode.value, valueEnd);
+ return makePosition(source, srcStart, srcEnd);
+}
diff --git a/packages/core/src/markdown/single-dollar-math-promoter.ts b/packages/core/src/markdown/single-dollar-math-promoter.ts
index a610c4663..be97d4d62 100644
--- a/packages/core/src/markdown/single-dollar-math-promoter.ts
+++ b/packages/core/src/markdown/single-dollar-math-promoter.ts
@@ -1,11 +1,14 @@
import type { PhrasingContent, Root, Text } from 'mdast';
import type { InlineMath } from 'mdast-util-math';
import { SKIP, visit } from 'unist-util-visit';
+import type { VFile } from 'vfile';
+import { deriveFragmentPosition } from './promoter-position.ts';
const SINGLE_DOLLAR_MATH_RE = /(?<!\\)\$(?=\S)([^$\n]*?[^\s$])\$(?!\d)/g;
export function singleDollarMathPromoterPlugin() {
- return (tree: Root) => {
+ return (tree: Root, file: VFile) => {
+ const source = typeof file.value === 'string' ? file.value : '';
visit(tree, 'text', (node: Text, index, parent) => {
if (parent === undefined || index === undefined || index === null) return;
@@ -28,14 +31,20 @@ export function singleDollarMathPromoterPlugin() {
const end = start + match[0].length;
if (start > cursor) {
const lead: Text = { type: 'text', value: value.slice(cursor, start) };
+ const pos = deriveFragmentPosition(source, node, cursor, start);
+ if (pos) lead.position = pos;
replacements.push(lead);
}
const mathNode: InlineMath = { type: 'inlineMath', value: match[1] };
+ const fullPos = deriveFragmentPosition(source, node, start, end);
+ if (fullPos) mathNode.position = fullPos;
replacements.push(mathNode as unknown as PhrasingContent);
cursor = end;
}
if (cursor < value.length) {
const tail: Text = { type: 'text', value: value.slice(cursor) };
+ const pos = deriveFragmentPosition(source, node, cursor, value.length);
+ if (pos) tail.position = pos;
replacements.push(tail);
}
diff --git a/packages/core/src/markdown/tag-promotion.ts b/packages/core/src/markdown/tag-promotion.ts
index aa0bb9efb..6d71623fa 100644
--- a/packages/core/src/markdown/tag-promotion.ts
+++ b/packages/core/src/markdown/tag-promotion.ts
@@ -1,5 +1,6 @@
import type { Parent, Text } from 'mdast';
import type { TagMdast } from './mdast-augmentation.ts';
+import { deriveFragmentPosition } from './promoter-position.ts';
export const TAG_IN_TEXT_PATTERN_SOURCE = '(^|\\s)#([a-zA-Z][\\w/-]*)';
export function createTagInTextRegex(): RegExp {
@@ -71,10 +72,20 @@ export function promoteTagsInParent(parent: Parent, source: string = ''): void {
}
if (tagStart > lastIndex) {
- segments.push({ type: 'text', value: text.slice(lastIndex, tagStart) } as Text);
+ const lead: Text = { type: 'text', value: text.slice(lastIndex, tagStart) };
+ const pos = deriveFragmentPosition(source, child as Text, lastIndex, tagStart);
+ if (pos) lead.position = pos;
+ segments.push(lead);
}
const tagNode: TagMdast = { type: 'tag', value: tagValue };
+ const tagPos = deriveFragmentPosition(
+ source,
+ child as Text,
+ tagStart,
+ tagStart + 1 + tagValue.length,
+ );
+ if (tagPos) tagNode.position = tagPos;
segments.push(tagNode as unknown as Parent['children'][number]);
lastIndex = tagStart + 1 + tagValue.length; // 1 for `#`
@@ -85,7 +96,10 @@ export function promoteTagsInParent(parent: Parent, source: string = ''): void {
newChildren.push(child);
} else {
if (lastIndex < text.length) {
- segments.push({ type: 'text', value: text.slice(lastIndex) } as Text);
+ const tail: Text = { type: 'text', value: text.slice(lastIndex) };
+ const pos = deriveFragmentPosition(source, child as Text, lastIndex, text.length);
+ if (pos) tail.position = pos;
+ segments.push(tail);
}
newChildren.push(...segments);
}
diff --git a/packages/core/src/markdown/to-markdown-handlers.attention.test.ts b/packages/core/src/markdown/to-markdown-handlers.attention.test.ts
new file mode 100644
index 000000000..837c3e9d6
--- /dev/null
+++ b/packages/core/src/markdown/to-markdown-handlers.attention.test.ts
@@ -0,0 +1,190 @@
+import { describe, expect, test } from 'bun:test';
+import type { JSONContent } from '@tiptap/core';
+import type { Nodes } from 'mdast';
+import { sharedExtensions } from '../extensions/shared.ts';
+import { MarkdownManager } from './index.ts';
+
+const md = new MarkdownManager({ extensions: sharedExtensions });
+
+const em = (t: string, sourceDelimiter: '*' | '_' = '*'): JSONContent => ({
+ type: 'text',
+ marks: [{ type: 'emphasis', attrs: { sourceDelimiter } }],
+ text: t,
+});
+const strong = (t: string, sourceDelimiter: '**' | '__' = '**'): JSONContent => ({
+ type: 'text',
+ marks: [{ type: 'strong', attrs: { sourceDelimiter } }],
+ text: t,
+});
+const text = (t: string): JSONContent => ({ type: 'text', text: t });
+const paragraph = (...inline: JSONContent[]): JSONContent => ({
+ type: 'doc',
+ content: [{ type: 'paragraph', content: inline }],
+});
+
+function allNodeTypes(markdown: string): string[] {
+ const out: string[] = [];
+ const walk = (node: Nodes): void => {
+ out.push(node.type);
+ if ('children' in node) {
+ for (const child of node.children) {
+ walk(child);
+ }
+ }
+ };
+ walk(md.parseToMdast(markdown));
+ return out;
+}
+
+function expectByteStable(out: string): void {
+ expect(md.serialize(md.parse(out))).toBe(out);
+}
+
+describe('boundary-whitespace attention encode (WYSIWYG-shaped marks)', () => {
+ test('emphasis with trailing space re-parses as emphasis via char-ref', () => {
+ const out = md.serialize(paragraph(em('foo ')));
+ expect(out).toBe('*foo&#x20;*\n');
+ expect(allNodeTypes(out)).toContain('emphasis');
+ expectByteStable(out);
+ });
+
+ test('emphasis with leading space does not degrade to a bullet list', () => {
+ const out = md.serialize(paragraph(em(' foo')));
+ expect(out).toBe('*&#x20;foo*\n');
+ const types = allNodeTypes(out);
+ expect(types).toContain('emphasis');
+ expect(types).not.toContain('list');
+ expectByteStable(out);
+ });
+
+ test('strong with trailing space re-parses as strong', () => {
+ const out = md.serialize(paragraph(strong('foo ')));
+ expect(out).toBe('**foo&#x20;**\n');
+ expect(allNodeTypes(out)).toContain('strong');
+ expectByteStable(out);
+ });
+
+ test('strong with leading space re-parses as strong', () => {
+ const out = md.serialize(paragraph(strong(' foo')));
+ expect(out).toBe('**&#x20;foo**\n');
+ expect(allNodeTypes(out)).toContain('strong');
+ expectByteStable(out);
+ });
+
+ test('underscore emphasis with trailing space keeps its delimiter form', () => {
+ const out = md.serialize(paragraph(em('foo ', '_')));
+ expect(out).toBe('_foo&#x20;_\n');
+ expect(allNodeTypes(out)).toContain('emphasis');
+ expectByteStable(out);
+ });
+
+ test('underscore strong with trailing space keeps its delimiter form', () => {
+ const out = md.serialize(paragraph(strong('foo ', '__')));
+ expect(out).toBe('__foo&#x20;__\n');
+ expect(allNodeTypes(out)).toContain('strong');
+ expectByteStable(out);
+ });
+
+ test('NBSP inside an emphasis boundary downgrades to an encoded space', () => {
+ const out = md.serialize(paragraph(em('a\u00A0')));
+ expect(out).toBe('*a&#x20;*\n');
+ expect(allNodeTypes(out)).toContain('emphasis');
+ expectByteStable(out);
+ });
+
+ test('whitespace-only emphasis survives', () => {
+ const out = md.serialize(paragraph(em(' ')));
+ expect(out).toBe('*&#x20;*\n');
+ expect(allNodeTypes(out)).toContain('emphasis');
+ expectByteStable(out);
+ });
+
+ test('trailing space emphasis flush against following text survives', () => {
+ const out = md.serialize(paragraph(em('foo '), text('bar')));
+ expect(out).toBe('*foo&#x20;*bar\n');
+ expect(allNodeTypes(out)).toContain('emphasis');
+ expectByteStable(out);
+ });
+
+ test('leading space emphasis flush against preceding text survives', () => {
+ const out = md.serialize(paragraph(text('bar'), em(' foo')));
+ expect(out).toBe('bar*&#x20;foo*\n');
+ expect(allNodeTypes(out)).toContain('emphasis');
+ expectByteStable(out);
+ });
+
+ test('whitespace on both the run boundary and its neighbor needs no outside encode', () => {
+ const out = md.serialize(paragraph(text('a '), em(' x')));
+ expect(out).toBe('a *&#x20;x*\n');
+ expect(allNodeTypes(out)).toContain('emphasis');
+ expectByteStable(out);
+ });
+
+ test('nested strong+emphasis with boundary space survives structurally', () => {
+ const out = md.serialize(
+ paragraph({
+ type: 'text',
+ marks: [
+ { type: 'strong', attrs: { sourceDelimiter: '**' } },
+ { type: 'emphasis', attrs: { sourceDelimiter: '*' } },
+ ],
+ text: 'x ',
+ }),
+ );
+ expect(out).not.toMatch(/\\&/);
+ const types = allNodeTypes(out);
+ expect(types).toContain('strong');
+ expect(types).toContain('emphasis');
+ expectByteStable(out);
+ });
+});
+
+describe('guard-irreducible adjacencies stay byte-identical to before', () => {
+ test('letter before punctuation-fronted emphasis is left unencoded', () => {
+ const out = md.serialize(paragraph(text('x'), em('.y.')));
+ expect(out).toBe('x*.y.*\n');
+ expect(out).not.toContain('&#');
+ });
+
+ test('letter after punctuation-tailed emphasis is left unencoded', () => {
+ const out = md.serialize(paragraph(em('.y.'), text('z')));
+ expect(out).toBe('*.y.*z\n');
+ expect(out).not.toContain('&#');
+ });
+
+ test('intraword underscore emphasis is left unencoded', () => {
+ const out = md.serialize(paragraph(text('a'), em('b', '_'), text('c')));
+ expect(out).toBe('a_b_c\n');
+ expect(out).not.toContain('&#');
+ });
+
+ test('underscore run with boundary space and letter neighbor is left unencoded', () => {
+ const out = md.serialize(paragraph(em('foo ', '_'), text('bar')));
+ expect(out).toBe('_foo _bar\n');
+ expect(out).not.toContain('&#');
+ });
+});
+
+describe('zero regression on already-valid boundaries', () => {
+ test('plain emphasis and strong are byte-identical to before', () => {
+ expect(md.serialize(paragraph(em('foo')))).toBe('*foo*\n');
+ expect(md.serialize(paragraph(strong('foo')))).toBe('**foo**\n');
+ expect(md.serialize(paragraph(em('foo', '_')))).toBe('_foo_\n');
+ expect(md.serialize(paragraph(strong('foo', '__')))).toBe('__foo__\n');
+ });
+
+ test('parse-derived attention round-trips byte-identically', () => {
+ const sources = [
+ 'x *y* z\n',
+ '**a** b\n',
+ 'a _b_ c\n',
+ '*.padded.* x\n',
+ '.*foo*\n',
+ 'intra*word*em\n',
+ '*&#x41;foo*\n',
+ ];
+ for (const src of sources) {
+ expect(md.serialize(md.parse(src))).toBe(src);
+ }
+ });
+});
diff --git a/packages/core/src/markdown/to-markdown-handlers.charref-attention.test.ts b/packages/core/src/markdown/to-markdown-handlers.charref-attention.test.ts
new file mode 100644
index 000000000..81a9914de
--- /dev/null
+++ b/packages/core/src/markdown/to-markdown-handlers.charref-attention.test.ts
@@ -0,0 +1,83 @@
+import { describe, expect, test } from 'bun:test';
+import { sharedExtensions } from '../extensions/shared.ts';
+import { MarkdownManager } from './index.ts';
+
+const md = new MarkdownManager({ extensions: sharedExtensions });
+const rt = (s: string): string => md.serialize(md.parse(s));
+
+const expectStable = (src: string, out: string): void => {
+ const once = rt(src);
+ expect(once).toBe(out);
+ expect(rt(once)).toBe(out);
+};
+
+const stripTree = (n: unknown): unknown => {
+ if (Array.isArray(n)) return n.map(stripTree);
+ if (n && typeof n === 'object') {
+ const out: Record<string, unknown> = {};
+ for (const [k, v] of Object.entries(n as Record<string, unknown>)) {
+ if (k === 'position' || k === 'data') continue;
+ out[k] = stripTree(v);
+ }
+ return out;
+ }
+ return n;
+};
+
+const expectContentPreserved = (src: string): void => {
+ expect(stripTree(md.parseToMdast(rt(src)))).toEqual(stripTree(md.parseToMdast(src)));
+};
+
+describe('char-ref-inert escape churn (the entity is a bystander)', () => {
+ test('punct-inner asterisk run after a numeric char ref gains escapes, stays lossless', () => {
+ expectStable('a&#x41;*.y.*', 'a&#x41;\\*.y.\\*\n');
+ expectContentPreserved('a&#x41;*.y.*');
+ });
+
+ test('entity bytes survive verbatim through the escape churn', () => {
+ expect(rt('a&#x41;*.y.*')).toContain('a&#x41;');
+ });
+
+ test('the decoded twin escapes identically (the char ref does not alter the escape)', () => {
+ expectStable('aA*.y.*', 'aA\\*.y.\\*\n');
+ });
+
+ test('the no-ref control produces the same escape', () => {
+ expectStable('a*.y.*', 'a\\*.y.\\*\n');
+ });
+});
+
+describe('char-ref-causal PUA-flanking divergence (the char ref alters escaping)', () => {
+ test('underscore run after a numeric char ref escapes both delimiters, stays lossless', () => {
+ expectStable('a&#x41;_foo_', 'a&#x41;\\_foo\\_\n');
+ expectContentPreserved('a&#x41;_foo_');
+ expect(rt('a&#x41;_foo_')).toContain('a&#x41;');
+ });
+
+ test('the decoded twin diverges: intraword opener stays bare, only the closer escapes', () => {
+ expectStable('aA_foo_', 'aA_foo\\_\n');
+ });
+
+ test('punct-inner asterisk run after a named char ref escapes both delimiters, stays lossless', () => {
+ expectStable('a&amp;*.y.*', 'a&amp;\\*.y.\\*\n');
+ expectContentPreserved('a&amp;*.y.*');
+ expect(rt('a&amp;*.y.*')).toContain('a&amp;');
+ });
+
+ test('the decoded twin diverges: the bare ampersand form round-trips byte-exact with zero escapes', () => {
+ expectStable('a&*.y.*', 'a&*.y.*\n');
+ expect(rt('a&*.y.*')).not.toContain('\\');
+ });
+
+ test('the causal pairs differ in escape count, never in entity bytes or content', () => {
+ const refUnderscore = rt('a&#x41;_foo_');
+ const decodedUnderscore = rt('aA_foo_');
+ expect((refUnderscore.match(/\\/g) ?? []).length).toBe(2);
+ expect((decodedUnderscore.match(/\\/g) ?? []).length).toBe(1);
+
+ const refAmp = rt('a&amp;*.y.*');
+ const decodedAmp = rt('a&*.y.*');
+ expect((refAmp.match(/\\/g) ?? []).length).toBe(2);
+ expect((decodedAmp.match(/\\/g) ?? []).length).toBe(0);
+ });
+});
diff --git a/packages/core/src/markdown/to-markdown-handlers.inline-math.test.ts b/packages/core/src/markdown/to-markdown-handlers.inline-math.test.ts
new file mode 100644
index 000000000..98b9d2e44
--- /dev/null
+++ b/packages/core/src/markdown/to-markdown-handlers.inline-math.test.ts
@@ -0,0 +1,160 @@
+import { describe, expect, test } from 'bun:test';
+import type { JSONContent } from '@tiptap/core';
+import { sharedExtensions } from '../extensions/shared.ts';
+import { MarkdownManager } from './index.ts';
+
+const md = new MarkdownManager({ extensions: sharedExtensions });
+
+function findInJson(json: JSONContent, predicate: (n: JSONContent) => boolean): JSONContent | null {
+ if (predicate(json)) return json;
+ for (const child of json.content ?? []) {
+ const found = findInJson(child, predicate);
+ if (found) return found;
+ }
+ return null;
+}
+
+function countInJson(json: JSONContent, predicate: (n: JSONContent) => boolean): number {
+ let count = predicate(json) ? 1 : 0;
+ for (const child of json.content ?? []) {
+ count += countInJson(child, predicate);
+ }
+ return count;
+}
+
+const isMath = (n: JSONContent) => n.type === 'mathInline';
+
+function expectByteStable(source: string) {
+ const once = md.serialize(md.parse(source));
+ expect(once).toBe(source);
+ expect(md.serialize(md.parse(once))).toBe(once);
+}
+
+function expectReparsesAsMath(emitted: string, formula: string) {
+ const json = md.parse(emitted);
+ expect(countInJson(json, isMath)).toBe(1);
+ expect(findInJson(json, isMath)?.attrs?.formula).toBe(formula);
+}
+
+describe('single-dollar capture: `$x$` round-trips byte-identical', () => {
+ test('`$x$` mid-prose round-trips byte-identical', () => {
+ expectByteStable('A formula $x$ in prose.\n');
+ });
+
+ test('`$x^2$` round-trips byte-identical', () => {
+ expectByteStable('Result: $x^2$.\n');
+ });
+
+ test('the captured delimiter lands on the mathInline atom', () => {
+ const json = md.parse('A formula $x$ in prose.\n');
+ const node = findInJson(json, isMath);
+ expect(node?.attrs?.formula).toBe('x');
+ expect(node?.attrs?.sourceDelimiter).toBe('$');
+ });
+
+ test('`$$x$$` keeps the paired-double form', () => {
+ expectByteStable('Result: $$x^2$$.\n');
+ const node = findInJson(md.parse('Result: $$x^2$$.\n'), isMath);
+ expect(node?.attrs?.sourceDelimiter).toBe('$$');
+ });
+
+ test('mixed `$a$` and `$$b$$` in one paragraph each keep their form', () => {
+ expectByteStable('Both $a$ and $$b$$ here.\n');
+ });
+
+ test('adjacent single-dollar maths `$y$$x$` round-trip byte-identical', () => {
+ const source = '$y$$x$\n';
+ const json = md.parse(source);
+ expect(countInJson(json, isMath)).toBe(2);
+ expect(md.serialize(json)).toBe(source);
+ });
+
+ test('LaTeX command body `$\\alpha$` round-trips byte-identical', () => {
+ expectByteStable('Greek $\\alpha$ letter.\n');
+ });
+});
+
+describe('currency precision: the promoter boundary is unchanged', () => {
+ test('`Pay $5 to $10 dollars` stays prose, byte-identical', () => {
+ const source = 'Pay $5 to $10 dollars.\n';
+ expect(countInJson(md.parse(source), isMath)).toBe(0);
+ expectByteStable(source);
+ });
+
+ test('`$PATH and $HOME` stays prose, byte-identical', () => {
+ const source = '$PATH and $HOME\n';
+ expect(countInJson(md.parse(source), isMath)).toBe(0);
+ expectByteStable(source);
+ });
+
+ test('`$x$5` (digit after close) stays prose, byte-identical', () => {
+ const source = 'value $x$5 here\n';
+ expect(countInJson(md.parse(source), isMath)).toBe(0);
+ expectByteStable(source);
+ });
+
+ test('`Cost $5$ each` parses as math (documented edge) and round-trips', () => {
+ const source = 'Cost $5$ each\n';
+ expect(countInJson(md.parse(source), isMath)).toBe(1);
+ expectByteStable(source);
+ });
+});
+
+describe('demotion boundary: a stale single-$ capture never emits bytes the promoter rejects', () => {
+ function parsedCarrier(): JSONContent {
+ const json = md.parse('A $x$ b\n');
+ expect(findInJson(json, isMath)?.attrs?.sourceDelimiter).toBe('$');
+ return json;
+ }
+
+ test('digit typed right after the atom demotes to `$$…$$`', () => {
+ const json = parsedCarrier();
+ const para = json.content?.[0];
+ const after = para?.content?.[2];
+ expect(after?.text).toBe(' b');
+ if (after) after.text = '5 b';
+ const out = md.serialize(json);
+ expect(out).toBe('A $$x$$5 b\n');
+ expectReparsesAsMath(out, 'x');
+ });
+
+ test('formula edited to contain `$` demotes and grows the fence', () => {
+ const json = parsedCarrier();
+ const node = findInJson(json, isMath);
+ if (node?.attrs) node.attrs.formula = 'a$b';
+ const out = md.serialize(json);
+ expect(out).toBe('A $$a$b$$ b\n');
+ expectReparsesAsMath(out, 'a$b');
+ });
+
+ test('formula edited to start with whitespace demotes', () => {
+ const json = parsedCarrier();
+ const node = findInJson(json, isMath);
+ if (node?.attrs) node.attrs.formula = ' x';
+ const out = md.serialize(json);
+ expectReparsesAsMath(out, ' x');
+ });
+
+ test('formula edited to end with a backslash demotes (raw math text keeps it)', () => {
+ const json = parsedCarrier();
+ const node = findInJson(json, isMath);
+ if (node?.attrs) node.attrs.formula = 'x\\';
+ const out = md.serialize(json);
+ expectReparsesAsMath(out, 'x\\');
+ });
+
+ test('formula with an escape-active backslash sequence demotes (prose re-parse would eat it)', () => {
+ const json = parsedCarrier();
+ const node = findInJson(json, isMath);
+ if (node?.attrs) node.attrs.formula = 'a\\*b';
+ const out = md.serialize(json);
+ expectReparsesAsMath(out, 'a\\*b');
+ });
+
+ test('WYSIWYG-inserted math (no captured delimiter) keeps the `$$…$$` default', () => {
+ const json = parsedCarrier();
+ const node = findInJson(json, isMath);
+ if (node?.attrs) node.attrs.sourceDelimiter = null;
+ expect(md.serialize(json)).toBe('A $$x$$ b\n');
+ });
+});
diff --git a/packages/core/src/markdown/to-markdown-handlers.overescape.test.ts b/packages/core/src/markdown/to-markdown-handlers.overescape.test.ts
new file mode 100644
index 000000000..2caf792e8
--- /dev/null
+++ b/packages/core/src/markdown/to-markdown-handlers.overescape.test.ts
@@ -0,0 +1,201 @@
+import { describe, expect, test } from 'bun:test';
+import { sharedExtensions } from '../extensions/shared.ts';
+import { MarkdownManager } from './index.ts';
+
+const md = new MarkdownManager({ extensions: sharedExtensions });
+const rt = (s: string): string => md.serialize(md.parse(s));
+
+const expectByteExact = (src: string): void => {
+ const once = rt(src);
+ expect(once).toBe(src);
+ expect(rt(once)).toBe(src);
+};
+
+const expectStable = (src: string, out: string): void => {
+ const once = rt(src);
+ expect(once).toBe(out);
+ expect(rt(once)).toBe(out);
+};
+
+describe('inert delimiters stay bare (over-escape cleared)', () => {
+ test('whitespace-flanked asterisk round-trips byte-exact', () => {
+ expectByteExact('a * b\n');
+ });
+
+ test('whitespace-flanked asterisk run round-trips byte-exact', () => {
+ expectByteExact('a ** b\n');
+ });
+
+ test('intraword underscore round-trips byte-exact', () => {
+ expectByteExact('a_b_c\n');
+ });
+
+ test('under_score round-trips without gaining a backslash', () => {
+ expectByteExact('under_score\n');
+ });
+
+ test('snake_case identifiers in prose round-trip byte-exact', () => {
+ expectByteExact('call snake_case_name with under_score args\n');
+ });
+
+ test('whitespace-flanked tilde round-trips byte-exact', () => {
+ expectByteExact('word ~ word\n');
+ });
+
+ test('whitespace-flanked double tilde round-trips byte-exact', () => {
+ expectByteExact('a ~~ b\n');
+ });
+
+ test('whitespace-flanked asterisk inside an ATX heading round-trips byte-exact', () => {
+ expectByteExact('# a * b\n');
+ });
+
+ test('whitespace-flanked asterisk inside a blockquote round-trips byte-exact', () => {
+ expectByteExact('> a * b\n');
+ });
+
+ test('inert asterisk keeps parsing as plain text after the bare round-trip', () => {
+ const out = rt('a * b\n');
+ const para = md.parseToMdast(out).children[0];
+ expect(para?.type).toBe('paragraph');
+ const children = (para as { children: Array<{ type: string; value?: string }> }).children;
+ expect(children).toHaveLength(1);
+ expect(children[0]?.type).toBe('text');
+ expect(children[0]?.value).toBe('a * b');
+ });
+
+ test('intraword underscore inside an underscore-delimited emphasis stays bare', () => {
+ expectByteExact('_x_y_\n');
+ const para = md.parseToMdast('_x_y_\n').children[0];
+ const em = (para as { children: Array<{ type: string; children?: Array<{ value?: string }> }> })
+ .children[0];
+ expect(em?.type).toBe('emphasis');
+ expect(em?.children?.[0]?.value).toBe('x_y');
+ });
+});
+
+describe('line-start = escapes only for the setext-underline shape', () => {
+ test('padded highlight-like text does not gain a backslash', () => {
+ expectByteExact('== foo ==\n');
+ });
+
+ test('mid-line double equals stays bare', () => {
+ expectByteExact('x == y\n');
+ });
+
+ test('line-start = followed by prose stays bare', () => {
+ expectByteExact('= line of prose\n');
+ });
+
+ test('non-contiguous equals line stays bare (not a setext-underline shape)', () => {
+ expectByteExact('== ==\n');
+ });
+
+ test('lazy continuation line starting with = followed by prose stays bare', () => {
+ expectByteExact('foo\n== bar\n');
+ });
+
+ test('pure = paragraph stays escaped (setext-underline shape)', () => {
+ expectStable('==\n', '\\==\n');
+ });
+
+ test('escaped pure = line round-trips byte-exact', () => {
+ expectByteExact('\\==\n');
+ });
+
+ test('escaped setext-underline shape after a soft break round-trips byte-exact', () => {
+ expectByteExact('foo\n\\==\n');
+ });
+
+ test('escaped = before prose on a continuation line drops no bytes', () => {
+ expectByteExact('foo\n\\= bar\n');
+ });
+});
+
+describe('flanking-active delimiters still escape (precision sweep)', () => {
+ test('escaped emphasis-able pair round-trips byte-exact', () => {
+ expectByteExact('a \\*bc\\* d\n');
+ });
+
+ test('escaped full emphasis shape round-trips byte-exact', () => {
+ expectByteExact('\\*foo\\*\n');
+ });
+
+ test('escaped left-flanking single asterisk round-trips byte-exact', () => {
+ expectByteExact('a \\*b\n');
+ });
+
+ test('escaped underscore pair round-trips byte-exact', () => {
+ expectByteExact('a \\_b\\_ c\n');
+ });
+
+ test('escaped tilde pairs round-trip byte-exact', () => {
+ expectByteExact('a \\~\\~b\\~\\~ c\n');
+ });
+
+ test('line-start left-flanking asterisk still gains an escape', () => {
+ expectStable('*x\n', '\\*x\n');
+ });
+
+ test('left-flanking asterisk mid-phrasing still gains an escape', () => {
+ expectStable('a *b\n', 'a \\*b\n');
+ });
+
+ test('quote-flanked underscore still escapes (punctuation adjacency can open)', () => {
+ expectStable('a "_" b\n', 'a "\\_" b\n');
+ });
+
+ test('text ending in bare asterisks before emphasis keeps the boundary escaped', () => {
+ expectStable('foo***bar*\n', 'foo\\*\\**bar*\n');
+ const para = md.parseToMdast('foo\\*\\**bar*\n').children[0];
+ const children = (para as { children: Array<{ type: string; value?: string }> }).children;
+ expect(children.map((c) => c.type)).toEqual(['text', 'emphasis']);
+ expect(children[0]?.value).toBe('foo**');
+ });
+
+ test('text starting with bare asterisks after emphasis keeps the boundary escaped', () => {
+ expectStable('*foo***bar\n', '*foo*\\*\\*bar\n');
+ const para = md.parseToMdast('*foo*\\*\\*bar\n').children[0];
+ const children = (para as { children: Array<{ type: string; value?: string }> }).children;
+ expect(children.map((c) => c.type)).toEqual(['emphasis', 'text']);
+ expect(children[1]?.value).toBe('**bar');
+ });
+
+ test('rule-of-three blocked shapes stay all-literal with every run escaped', () => {
+ expectStable('foo**bar*\n', 'foo\\*\\*bar\\*\n');
+ expectStable('*foo**bar\n', '\\*foo\\*\\*bar\n');
+ });
+});
+
+describe('atBreak guards for * _ ~ are unchanged (list / thematic / fence threats)', () => {
+ test('escaped list-marker shape at line start round-trips byte-exact', () => {
+ expectByteExact('\\* foo\n');
+ });
+
+ test('line-start underscore before a space stays escaped (thematic-break guard)', () => {
+ expectStable('_ foo\n', '\\_ foo\n');
+ });
+
+ test('line-start tilde before a space stays escaped (fence guard)', () => {
+ expectStable('~ foo\n', '\\~ foo\n');
+ });
+});
+
+describe('genuine formation cases are untouched', () => {
+ test('emphasis, strong, strikethrough, and highlight round-trip byte-exact', () => {
+ expectByteExact('*em*\n');
+ expectByteExact('_em_\n');
+ expectByteExact('**st**\n');
+ expectByteExact('__st__\n');
+ expectByteExact('~~del~~\n');
+ expectByteExact('==hl==\n');
+ });
+
+ test('intraword strikethrough still parses and round-trips', () => {
+ expectByteExact('a~~b~~c\n');
+ });
+
+ test('emphasis adjacent to inert delimiters round-trips byte-exact', () => {
+ expectByteExact('a * b *em* c * d\n');
+ });
+});
diff --git a/packages/core/src/markdown/to-markdown-handlers.residual-axes.test.ts b/packages/core/src/markdown/to-markdown-handlers.residual-axes.test.ts
new file mode 100644
index 000000000..c2465b119
--- /dev/null
+++ b/packages/core/src/markdown/to-markdown-handlers.residual-axes.test.ts
@@ -0,0 +1,323 @@
+import { describe, expect, test } from 'bun:test';
+import { sharedExtensions } from '../extensions/shared.ts';
+import { MarkdownManager } from './index.ts';
+
+const md = new MarkdownManager({ extensions: sharedExtensions });
+const rt = (s: string): string => md.serialize(md.parse(s));
+
+const expectByteExact = (src: string): void => {
+ const expected = src.endsWith('\n') ? src : `${src}\n`;
+ const once = rt(src);
+ expect(once).toBe(expected);
+ expect(rt(once)).toBe(expected);
+};
+
+describe('gfm-table: outer-pipe presence + dash-row padding', () => {
+ test('pipe-less table round-trips byte-exact', () => {
+ expectByteExact('col|val\n-|-\nalpha|99');
+ });
+
+ test('unpadded delimiter row keeps zero padding', () => {
+ expectByteExact('|a|b|\n|-|-|\n|c|d|');
+ });
+
+ test('canonical padded table is unchanged', () => {
+ expectByteExact('| a | b |\n| - | - |\n| c | d |');
+ });
+
+ test('leading-pipe-only style round-trips', () => {
+ expectByteExact('|a|b\n|-|-\n|c|d');
+ });
+
+ test('alignment markers survive with zero padding', () => {
+ expectByteExact('|a|b|\n|:-|-:|\n|c|d|');
+ });
+
+ test('mixed outer-pipe styles fall back to canonical (non-uniform: no capture)', () => {
+ const out = rt('|a|b|\n-|-\nc|d');
+ expect(out).toBe('|a|b|\n|-|-|\n|c|d|\n');
+ expect(rt(out)).toBe(out);
+ });
+
+ test('stale outer-pipe capture with an emptied boundary cell falls back to piped form', () => {
+ const json = md.parse('col|val\n-|-\nalpha|99');
+ const firstRow = (
+ json as unknown as {
+ content: Array<{ type: string; content: Array<{ content: Array<unknown> }> }>;
+ }
+ ).content.find((n) => n.type === 'table');
+ const firstCell = firstRow?.content[0]?.content[0] as { content?: unknown[] } | undefined;
+ if (firstCell) firstCell.content = [];
+ const out = md.serialize(json);
+ expect(out.split('\n')[0]).toBe('||val|');
+ expect(rt(out)).toBe(out);
+ });
+});
+
+describe('list: marker spacing, ordinal replay, nested indent', () => {
+ test('double-space marker spacing round-trips', () => {
+ expectByteExact('- item');
+ });
+
+ test('quad-space marker spacing round-trips', () => {
+ expectByteExact('- item');
+ });
+
+ test('all-ones ordered numbering is not force-incremented', () => {
+ expectByteExact('1. a\n1. b');
+ });
+
+ test('descending-style numbering replays verbatim', () => {
+ expectByteExact('3. a\n1. b\n1. c');
+ });
+
+ test('canonical incrementing numbering is unchanged', () => {
+ expectByteExact('1. a\n2. b');
+ });
+
+ test('non-canonical nested-list indent round-trips', () => {
+ expectByteExact('- a\n - b');
+ });
+
+ test('canonical nested-list indent is unchanged', () => {
+ expectByteExact('- a\n - b');
+ });
+
+ test('three-level canonical nesting is idempotent (relative-indent capture)', () => {
+ expectByteExact('- L1\n - L2\n - L3');
+ });
+
+ test('non-canonical indent on a doubly-nested list round-trips', () => {
+ expectByteExact('- L1\n - L2\n - L3');
+ });
+
+ test('ordered marker spacing composes with ordinal replay', () => {
+ expectByteExact('1. a\n1. b');
+ });
+
+ test('marker spacing of 5+ stays canonical (indented-code boundary)', () => {
+ const out = rt('- item');
+ expect(rt(out)).toBe(out);
+ });
+});
+
+describe('gfm-task: checkbox case', () => {
+ test('uppercase [X] round-trips', () => {
+ expectByteExact('- [X] done');
+ });
+
+ test('lowercase [x] round-trips', () => {
+ expectByteExact('- [x] done');
+ });
+
+ test('unchecked task round-trips', () => {
+ expectByteExact('- [ ] todo');
+ });
+
+ test('mixed-case task list round-trips per item', () => {
+ expectByteExact('- [X] a\n- [x] b\n- [ ] c');
+ });
+});
+
+describe('blockquote: marker-spacing counts', () => {
+ test('double-space marker round-trips', () => {
+ expectByteExact('> quote');
+ });
+
+ test('triple-space marker round-trips', () => {
+ expectByteExact('> quote');
+ });
+
+ test('indented code inside a blockquote does not double its indent', () => {
+ expectByteExact('> code');
+ });
+
+ test('no-space marker still round-trips', () => {
+ expectByteExact('>quote');
+ });
+});
+
+describe('code-fence: closing length, leading indent, info padding', () => {
+ test('longer closing fence round-trips', () => {
+ expectByteExact('```\ncode\n````');
+ });
+
+ test('uniformly indented fence round-trips', () => {
+ expectByteExact(' ```\n code\n ```');
+ });
+
+ test('info-string padding round-trips', () => {
+ expectByteExact('``` js\ncode\n```');
+ });
+
+ test('single-space info padding round-trips', () => {
+ expectByteExact('``` js\ncode\n```');
+ });
+
+ test('indent + info padding + closing length compose', () => {
+ expectByteExact(' ``` ts\n code\n ````');
+ });
+
+ test('fence-recompute: synthesized value containing a closing run lengthens the fence', () => {
+ const json = JSON.parse(JSON.stringify(md.parse('```\nplaceholder\n```')));
+ const codeNode = (
+ json as { content: Array<{ type: string; content?: Array<{ text?: string }> }> }
+ ).content.find((n) => n.type === 'codeBlock');
+ expect(codeNode).toBeDefined();
+ if (codeNode?.content?.[0]) codeNode.content[0].text = 'a\n```\nb';
+ const out = md.serialize(json as ReturnType<typeof md.parse>);
+ expect(out).toBe('````\na\n```\nb\n````\n');
+ expect(rt(out)).toBe(out);
+ });
+});
+
+describe('atx heading: leading indent + interior space', () => {
+ test('leading indent round-trips', () => {
+ expectByteExact(' # Heading');
+ });
+
+ test('interior spacing round-trips', () => {
+ expectByteExact('# Heading');
+ });
+
+ test('indent + interior compose', () => {
+ expectByteExact(' # Heading');
+ });
+
+ test('interior spacing composes with a closing hash run', () => {
+ expectByteExact('# Heading ##');
+ });
+
+ test('heading inside a list item does not capture the container pad', () => {
+ expectByteExact('1. item\n\n # heading in item');
+ });
+});
+
+describe('gfm-strikethrough: delimiter run', () => {
+ test('single-tilde form round-trips', () => {
+ expectByteExact('~x~ y');
+ });
+
+ test('double-tilde form round-trips', () => {
+ expectByteExact('~~x~~ y');
+ });
+
+ test('both forms coexist per node', () => {
+ expectByteExact('~a~ and ~~b~~');
+ });
+});
+
+describe('wikiLink: authored padding survives the trim', () => {
+ test('padded target round-trips', () => {
+ expectByteExact('[[ Page ]]');
+ });
+
+ test('padded target + alias round-trip', () => {
+ expectByteExact('[[Page | alias ]]');
+ });
+
+ test('padded anchor round-trips', () => {
+ expectByteExact('[[Page# Anchor ]]');
+ });
+
+ test('unpadded forms are unchanged', () => {
+ expectByteExact('[[Page]]');
+ expectByteExact('[[Page|alias]]');
+ expectByteExact('[[Page#Anchor|alias]]');
+ });
+
+ test('stale raw segment is dropped after a target edit', () => {
+ const json = JSON.parse(JSON.stringify(md.parse('[[ Page ]]')));
+ const para = (json as { content: Array<{ content?: Array<Record<string, unknown>> }> })
+ .content[0];
+ const wiki = para.content?.find((n) => n.type === 'wikiLink') as
+ | { attrs: Record<string, unknown> }
+ | undefined;
+ expect(wiki).toBeDefined();
+ if (wiki) wiki.attrs.target = 'Renamed';
+ const out = md.serialize(json as ReturnType<typeof md.parse>);
+ expect(out).toBe('[[Renamed]]\n');
+ });
+});
+
+describe('inline-code: authored disambiguation pad', () => {
+ test('padded inline code round-trips', () => {
+ expectByteExact('a ` y ` b');
+ });
+
+ test('the intra-block multi-axis non-vacuity witness round-trips', () => {
+ expectByteExact('one \ntwo with ` y ` pad');
+ });
+
+ test('unpadded inline code is unchanged', () => {
+ expectByteExact('a `y` b');
+ });
+
+ test('structurally-required padding still applies without capture', () => {
+ expectByteExact('a `` `tick `` b');
+ });
+});
+
+describe('inline-code: pipe re-escape inside a table cell', () => {
+ test('escaped pipe inside inline code in a cell round-trips byte-exact', () => {
+ expectByteExact('| `a\\|b` |\n| - |');
+ });
+
+ test('the emitted bytes re-parse as a table, not a paragraph', () => {
+ const out = rt('| `a\\|b` |\n| - |\n');
+ expect(md.parseToMdast(out).children[0]?.type).toBe('table');
+ });
+
+ test('every pipe in a multi-pipe code span re-escapes', () => {
+ expectByteExact('| `a\\|b\\|c` |\n| - |');
+ });
+
+ test('fence disambiguation composes with the cell escape', () => {
+ expectByteExact('| ``a`\\|b`` |\n| - |');
+ });
+
+ test('inline code outside a table never gains the cell escape', () => {
+ expectByteExact('`a|b`');
+ expectByteExact('`a\\|b`');
+ });
+
+ test('plain-text cell pipe escape is unchanged', () => {
+ expectByteExact('| a\\|b |\n| - |');
+ });
+
+ test('padded code with a pipe keeps the escape; the pad capture is a separate open residual', () => {
+ expect(rt('| ` a\\|b ` |\n| - |\n')).toBe('| `a\\|b` |\n| - |\n');
+ });
+});
+
+describe('guard-sentinel literal collision (escape alphabet)', () => {
+ test('authored guard sentinels U+E000-E004 round-trip byte-exact', () => {
+ expectByteExact('literal \uE000\uE001\uE002\uE003\uE004 chars');
+ });
+
+ test('sentinel adjacent to guarded syntax round-trips', () => {
+ expectByteExact('x\uE000y and <https://example.com>');
+ });
+
+ test('sentinel inside a fence info string round-trips', () => {
+ expectByteExact('```l\uE000ng\ncode\n```');
+ });
+
+ test('escape alphabet U+E005-E009 is the documented reserved range', () => {
+ expect(rt('esc \uE005 char')).toBe('esc \uE000 char\n');
+ });
+});
+
+describe('already-byte-identical constructs stay untouched', () => {
+ test('%%-comment padding is unchanged', () => {
+ expectByteExact('%% padded comment %%');
+ });
+
+ test('callout case is unchanged', () => {
+ expectByteExact('> [!note]\n> body');
+ });
+
+ test('details-quote form is unchanged', () => {
+ expectByteExact('<details>\n\n<summary>t</summary>\n\n</details>');
+ });
+});
diff --git a/packages/core/src/markdown/to-markdown-handlers.test.ts b/packages/core/src/markdown/to-markdown-handlers.test.ts
index a7d0fa00a..aba70e1c8 100644
--- a/packages/core/src/markdown/to-markdown-handlers.test.ts
+++ b/packages/core/src/markdown/to-markdown-handlers.test.ts
@@ -198,8 +198,8 @@ describe('to-markdown: lone backtick in text NOT escaped (FR-1d)', () => {
});
describe('to-markdown: thematic break preservation', () => {
- test('doc-start --- normalizes to *** (NG10 serialize-side)', () => {
- expect(roundTrip('---\n')).toBe('***\n');
+ test('doc-start --- round-trips as --- (the *** override is retired)', () => {
+ expect(roundTrip('---\n')).toBe('---\n');
});
test('*** round-trips as ***', () => {
@@ -1406,7 +1406,7 @@ describe('to-markdown: blockquote marker spacing (FR-23)', () => {
const doc = schema.nodeFromJSON(json);
const blockquote = doc.firstChild;
expect(blockquote?.type.name).toBe('blockquote');
- expect(blockquote?.attrs.sourceMarkerSpacings).toEqual(['single', 'none']);
+ expect(blockquote?.attrs.sourceMarkerSpacings).toEqual([1, 0]);
});
test('PM blockquote attr is null for WYSIWYG-authored blockquote (no source)', () => {
diff --git a/packages/core/src/markdown/to-markdown-handlers.ts b/packages/core/src/markdown/to-markdown-handlers.ts
index 62542c87d..a5c985768 100644
--- a/packages/core/src/markdown/to-markdown-handlers.ts
+++ b/packages/core/src/markdown/to-markdown-handlers.ts
@@ -1,6 +1,7 @@
import type { Nodes, Parents } from 'mdast';
import type { MdxJsxAttribute, MdxJsxExpressionAttribute, MdxJsxFlowElement } from 'mdast-util-mdx';
import type { Info, State } from 'mdast-util-to-markdown';
+import { classifyCharacter } from 'micromark-util-classify-character';
import { isValidSourceLiteralRaw } from '../extensions/source-literal-mark.ts';
declare module 'mdast-util-to-markdown' {
@@ -8,6 +9,7 @@ declare module 'mdast-util-to-markdown' {
mark: 'mark';
comment: 'comment';
commentBlock: 'commentBlock';
+ strikethrough: 'strikethrough';
}
}
@@ -24,6 +26,10 @@ type MdastToMarkdownHandlers = {
mdxJsxFlowElement: (node: Nodes, parent: Parents | undefined, state: State, info: Info) => string;
};
+const inlineMathHandler = Object.assign(serializeInlineMath, { peek: () => '$' });
+
+const deleteHandler = Object.assign(serializeDelete, { peek: () => '~' });
+
export const toMarkdownHandlers = {
text(node, _parent, state, info) {
if (typeof node.data?.sourceRaw === 'string') {
@@ -39,13 +45,20 @@ export const toMarkdownHandlers = {
let lastIdx = 0;
for (const { offset, char } of escaped) {
if (offset > lastIdx) {
- result += safeText(state, value.slice(lastIdx, offset), info);
+ result += safeText(state, value.slice(lastIdx, offset), {
+ ...info,
+ before: info.before + result,
+ after: '\\',
+ });
}
result += `\\${char}`;
lastIdx = offset + 1;
}
if (lastIdx < value.length) {
- result += safeText(state, value.slice(lastIdx), info);
+ result += safeText(state, value.slice(lastIdx), {
+ ...info,
+ before: info.before + result,
+ });
}
return result;
}
@@ -58,11 +71,15 @@ export const toMarkdownHandlers = {
const tracker = state.createTracker(info);
const exit = state.enter('emphasis');
let value = tracker.move(delim);
- value += state.containerPhrasing(node, {
- before: value,
- after: delim,
- ...tracker.current(),
- });
+ value += encodeAttentionBoundaries(
+ state.containerPhrasing(node, {
+ before: value,
+ after: delim,
+ ...tracker.current(),
+ }),
+ info,
+ delim,
+ );
value += tracker.move(delim);
exit();
return value;
@@ -73,11 +90,15 @@ export const toMarkdownHandlers = {
const tracker = state.createTracker(info);
const exit = state.enter('strong');
let value = tracker.move(delim);
- value += state.containerPhrasing(node, {
- before: value,
- after: delim,
- ...tracker.current(),
- });
+ value += encodeAttentionBoundaries(
+ state.containerPhrasing(node, {
+ before: value,
+ after: delim,
+ ...tracker.current(),
+ }),
+ info,
+ delim,
+ );
value += tracker.move(delim);
exit();
return value;
@@ -203,16 +224,8 @@ export const toMarkdownHandlers = {
return value;
},
- thematicBreak(node, _parent, state) {
- const sourceRaw = node.data?.sourceRaw;
- const isDocStart =
- Array.isArray(state?.indexStack) &&
- state.indexStack.length === 1 &&
- state.indexStack[0] === 0;
- if (isDocStart && (!sourceRaw || /^-[-\s]*-\s*$/.test(sourceRaw))) {
- return '***';
- }
- return sourceRaw ?? '---';
+ thematicBreak(node) {
+ return node.data?.sourceRaw ?? '---';
},
break(node) {
@@ -223,22 +236,48 @@ export const toMarkdownHandlers = {
code(node) {
const value = node.value ?? '';
if (node.data?.sourceStyle === 'indented') {
- const indented = value
- .split('\n')
- .map((line) => (line.length > 0 ? ` ${line}` : line))
- .join('\n');
+ const lines = value.split('\n');
+ const indents = node.data.sourceIndents;
+ if (Array.isArray(indents) && indents.length === lines.length) {
+ return lines
+ .map((line, i) => `${indents[i] ?? (line.length > 0 ? ' ' : '')}${line}`)
+ .join('\n');
+ }
+ const indented = lines.map((line) => (line.length > 0 ? ` ${line}` : line)).join('\n');
return indented;
}
const fenceChar = node.data?.sourceFenceChar;
const char = fenceChar === '~' ? '~' : '`';
- const len = Math.max(3, node.data?.sourceFenceLength ?? 3);
+ let len = Math.max(3, node.data?.sourceFenceLength ?? 3);
+ const fenceRunRe = new RegExp(`^ {0,3}(\\${char}+)[ \\t]*$`);
+ for (const line of value.split('\n')) {
+ const run = fenceRunRe.exec(line);
+ if (run && run[1].length >= len) {
+ len = run[1].length + 1;
+ }
+ }
+ const closingCaptured = node.data?.sourceClosingFenceLength;
+ const closeLen =
+ typeof closingCaptured === 'number' && closingCaptured > len ? closingCaptured : len;
+ const indentCount = node.data?.sourceFenceIndent;
+ const indent =
+ typeof indentCount === 'number' && indentCount >= 1 && indentCount <= 3
+ ? ' '.repeat(indentCount)
+ : '';
+ const infoGap = node.data?.sourceInfoPadding;
const fence = char.repeat(len);
const lang = node.lang ?? '';
+ const langGap = lang && typeof infoGap === 'number' && infoGap >= 1 ? ' '.repeat(infoGap) : '';
const meta = node.meta ? ` ${node.meta}` : '';
- return `${fence}${lang}${meta}\n${value}\n${fence}`;
+ const body = `${fence}${langGap}${lang}${meta}\n${value}\n${char.repeat(closeLen)}`;
+ if (!indent) return body;
+ return body
+ .split('\n')
+ .map((l) => (l.length > 0 ? `${indent}${l}` : l))
+ .join('\n');
},
- inlineCode(node) {
+ inlineCode(node, _parent, state) {
let value = node.value ?? '';
const requestedChar = node.data?.sourceFenceChar;
const fenceChar = requestedChar === '~' ? '~' : '`';
@@ -256,16 +295,22 @@ export const toMarkdownHandlers = {
if (
/[^ \r\n]/.test(value) &&
- ((/^[ \r\n]/.test(value) && /[ \r\n]$/.test(value)) ||
+ (node.data?.sourcePadded === true ||
+ (/^[ \r\n]/.test(value) && /[ \r\n]$/.test(value)) ||
value.startsWith(fenceChar) ||
value.endsWith(fenceChar))
) {
value = ` ${value} `;
}
- return `${fence}${value}${fence}`;
+ const span = `${fence}${value}${fence}`;
+ return state.stack.includes('tableCell') ? span.replace(/\|/g, '\\$&') : span;
},
+ inlineMath: inlineMathHandler,
+
+ delete: deleteHandler,
+
heading(node, _parent, state, info) {
const style = node.data?.sourceStyle ?? 'atx';
const depth = node.depth;
@@ -277,6 +322,14 @@ export const toMarkdownHandlers = {
const underline = (depth === 1 ? '=' : '-').repeat(underlineLength);
return `${content}\n${underline}`;
}
+ const indentCount = node.data?.sourceLeadingIndent;
+ const indent =
+ typeof indentCount === 'number' && indentCount >= 1 && indentCount <= 3
+ ? ' '.repeat(indentCount)
+ : '';
+ const interiorCount = node.data?.sourceInteriorSpacing;
+ const interior =
+ typeof interiorCount === 'number' && interiorCount >= 2 ? ' '.repeat(interiorCount) : ' ';
const hashes = '#'.repeat(depth);
const content = state.containerPhrasing(node, {
...info,
@@ -286,11 +339,11 @@ export const toMarkdownHandlers = {
const trailingCount = node.data?.sourceTrailingHashes;
if (typeof trailingCount === 'number' && trailingCount > 0) {
const trailing = '#'.repeat(trailingCount);
- if (!content) return `${hashes} ${trailing}`;
- return `${hashes} ${content} ${trailing}`;
+ if (!content) return `${indent}${hashes} ${trailing}`;
+ return `${indent}${hashes}${interior}${content} ${trailing}`;
}
- if (!content) return hashes;
- return `${hashes} ${content}`;
+ if (!content) return `${indent}${hashes}`;
+ return `${indent}${hashes}${interior}${content}`;
},
list(node, _parent, state, info) {
@@ -313,15 +366,36 @@ export const toMarkdownHandlers = {
for (let i = 0; i < children.length; i++) {
const child = children[i];
- const baseMarker = ordered ? `${counter + i}${delim}` : (node.data?.bulletMarker ?? bullet);
+ const capturedOrdinal = child.data?.sourceOrdinal;
+ const ordinal =
+ typeof capturedOrdinal === 'number' && capturedOrdinal >= 0 ? capturedOrdinal : counter + i;
+ const baseMarker = ordered ? `${ordinal}${delim}` : (node.data?.bulletMarker ?? bullet);
+ const capturedSpacing = child.data?.sourceMarkerSpacing;
+ const spacing =
+ typeof capturedSpacing === 'number' && capturedSpacing >= 2 && capturedSpacing <= 4
+ ? capturedSpacing
+ : 1;
+ const markerGap = ' '.repeat(spacing);
let marker = baseMarker;
- if (child.checked === true) marker += ' [x]';
- else if (child.checked === false) marker += ' [ ]';
- const pad = ' '.repeat(baseMarker.length + 1);
+ if (child.checked === true) {
+ marker += `${markerGap}[${child.data?.sourceCheckboxChar === 'X' ? 'X' : 'x'}]`;
+ } else if (child.checked === false) {
+ marker += `${markerGap}[ ]`;
+ }
+ const capturedIndent = child.data?.sourceContinuationIndent;
+ const contentCol = baseMarker.length + spacing;
+ const padWidth =
+ typeof capturedIndent === 'number' &&
+ capturedIndent >= contentCol &&
+ capturedIndent <= contentCol + 3
+ ? capturedIndent
+ : contentCol;
+ const pad = ' '.repeat(padWidth);
+ const firstGap = child.checked === null || child.checked === undefined ? markerGap : ' ';
const itemContent = state.containerFlow(child, info);
const indented = itemContent
.split('\n')
- .map((l, idx) => (idx === 0 ? `${marker} ${l}` : l ? `${pad}${l}` : l))
+ .map((l, idx) => (idx === 0 ? `${marker}${firstGap}${l}` : l ? `${pad}${l}` : l))
.join('\n');
out.push(indented);
}
@@ -403,14 +477,35 @@ export const toMarkdownHandlers = {
const pad = paddings[i] ?? { left: 1, right: 1 };
out.push(' '.repeat(pad.left) + content + ' '.repeat(pad.right));
}
- return `|${out.join('|')}|`;
+ const leading = outerPipesSafe && !outerPipesSafe.leading ? '' : '|';
+ const trailing = outerPipesSafe && !outerPipesSafe.trailing ? '' : '|';
+ return `${leading}${out.join('|')}${trailing}`;
};
+ const capturedAlignPadding = node.data?.sourceAlignmentPadding;
const alignmentPaddings: Array<{ left: number; right: number } | null> = Array.from(
{ length: mostCellsPerRow },
- () => null,
+ (_, col) => {
+ const pad = capturedAlignPadding?.[col];
+ return pad && typeof pad.left === 'number' && typeof pad.right === 'number' ? pad : null;
+ },
);
+ const outerPipes = node.data?.sourceOuterPipes;
+ const outerPipesSafe =
+ outerPipes &&
+ mostCellsPerRow >= 2 &&
+ matrix.every((cells) => {
+ const first = (cells[0] ?? '').trim();
+ const last = (cells[mostCellsPerRow - 1] ?? '').trim();
+ return (
+ (outerPipes.leading || first.length > 0) &&
+ (outerPipes.trailing || (last.length > 0 && !last.endsWith('\\')))
+ );
+ })
+ ? outerPipes
+ : null;
+
if (matrix.length === 0) {
return formatRow(alignmentCells, alignmentPaddings);
}
@@ -553,7 +648,10 @@ export const toMarkdownHandlers = {
return state.indentLines(inner, (line, _lineNumber, blank) => {
if (blank) return '>';
const spacing = spacings?.[nonBlankIdx++];
- if (spacing === 'none') return `>${line}`;
+ if (spacing === 'none' || spacing === 0) return `>${line}`;
+ if (typeof spacing === 'number' && spacing >= 2 && !line.startsWith(' ')) {
+ return `>${' '.repeat(spacing)}${line}`;
+ }
return `> ${line}`;
});
},
@@ -649,6 +747,13 @@ function safeText(state: State, value: string, info: Info): string {
if (u.character === '[') return false;
if (u.character === '(') return false;
if (u.character === '`') return false;
+ if (
+ (u.character === '*' || u.character === '_' || u.character === '~') &&
+ u.inConstruct === 'phrasing'
+ ) {
+ return false;
+ }
+ if (u.character === '=' && u.atBreak === true) return false;
return true;
});
let result: string;
@@ -657,5 +762,178 @@ function safeText(state: State, value: string, info: Info): string {
} finally {
state.unsafe = originalUnsafe;
}
- return escapeEntityAmpersands(result);
+ return escapeEntityAmpersands(escapeActiveDelimiterRuns(result, info));
+}
+
+const CHARACTER_GROUP_WHITESPACE = 1;
+
+const CHARACTER_GROUP_PUNCTUATION = 2;
+
+type FlankClass = 'ws' | 'punct' | 'other';
+
+function classifyFlank(code: number): FlankClass {
+ if (Number.isNaN(code)) return 'ws';
+ const group = classifyCharacter(code);
+ if (group === CHARACTER_GROUP_WHITESPACE) return 'ws';
+ if (group === CHARACTER_GROUP_PUNCTUATION) return 'punct';
+ return 'other';
+}
+
+function isActiveDelimiterRun(
+ marker: '*' | '_' | '~',
+ beforeCode: number,
+ afterCode: number,
+): boolean {
+ const before = classifyFlank(beforeCode);
+ const after = classifyFlank(afterCode);
+ const leftFlanking = after !== 'ws' && (after === 'other' || before !== 'other');
+ const rightFlanking = before !== 'ws' && (before === 'other' || after !== 'other');
+ if (marker === '_') {
+ const canOpen = leftFlanking && (!rightFlanking || before === 'punct');
+ const canClose = rightFlanking && (!leftFlanking || after === 'punct');
+ return canOpen || canClose;
+ }
+ return leftFlanking || rightFlanking;
+}
+
+function isSetextUnderlineShape(value: string, runEnd: number, info: Info): boolean {
+ let k = runEnd;
+ while (k < value.length && (value[k] === ' ' || value[k] === '\t')) k++;
+ if (k < value.length) return value[k] === '\n' || value[k] === '\r';
+ const next = info.after.charCodeAt(0);
+ return Number.isNaN(next) || next === 0x0a || next === 0x0d;
+}
+
+function escapeActiveDelimiterRuns(value: string, info: Info): string {
+ let result = '';
+ let i = 0;
+ while (i < value.length) {
+ const ch = value.charAt(i);
+ if (ch === '\\') {
+ result += value.slice(i, i + 2);
+ i += 2;
+ continue;
+ }
+ if (ch === '*' || ch === '_' || ch === '~' || ch === '=') {
+ let j = i + 1;
+ while (j < value.length && value.charAt(j) === ch) j++;
+ const beforeCode =
+ i === 0 ? info.before.charCodeAt(info.before.length - 1) : value.charCodeAt(i - 1);
+ if (ch === '=') {
+ const atLineStart = Number.isNaN(beforeCode) || beforeCode === 0x0a || beforeCode === 0x0d;
+ result +=
+ atLineStart && isSetextUnderlineShape(value, j, info)
+ ? `\\${value.slice(i, j)}`
+ : value.slice(i, j);
+ } else {
+ const afterCode = j < value.length ? value.charCodeAt(j) : info.after.charCodeAt(0);
+ result += isActiveDelimiterRun(ch, beforeCode, afterCode)
+ ? `\\${ch}`.repeat(j - i)
+ : value.slice(i, j);
+ }
+ i = j;
+ continue;
+ }
+ result += ch;
+ i += 1;
+ }
+ return result;
+}
+
+function encodeAttentionBoundaries(between: string, info: Info, delim: string): string {
+ const marker = delim.startsWith('_') ? '_' : '*';
+ let result = between;
+ const head = result.charCodeAt(0);
+ if (shouldEncodeAttentionBoundary(info.before.charCodeAt(info.before.length - 1), head, marker)) {
+ result = encodeCharacterReference(head) + result.slice(1);
+ }
+ const tail = result.charCodeAt(result.length - 1);
+ if (shouldEncodeAttentionBoundary(info.after.charCodeAt(0), tail, marker)) {
+ result = result.slice(0, -1) + encodeCharacterReference(tail);
+ }
+ return result;
+}
+
+function shouldEncodeAttentionBoundary(
+ outside: number,
+ inside: number,
+ marker: '*' | '_',
+): boolean {
+ if (classifyCharacter(inside) !== CHARACTER_GROUP_WHITESPACE) return false;
+ if (marker === '_' && classifyCharacter(outside) === undefined) return false;
+ return true;
+}
+
+function encodeCharacterReference(code: number): string {
+ return `&#x${code.toString(16).toUpperCase()};`;
+}
+
+function serializeInlineMath(
+ node: Extract<Nodes, { type: 'inlineMath' }>,
+ _parent: Parents | undefined,
+ state: State,
+ info: Info,
+): string {
+ let value = node.value ?? '';
+ const captured = node.data?.sourceDelimiter;
+ let size = typeof captured === 'string' && /^\$+$/.test(captured) ? captured.length : 2;
+ if (size === 1 && !singleDollarMathReparses(value, info)) {
+ size = 2;
+ }
+ while (new RegExp(`(^|[^$])${'\\$'.repeat(size)}([^$]|$)`).test(value)) {
+ size++;
+ }
+ const sequence = '$'.repeat(size);
+ if (
+ /[^ \r\n]/.test(value) &&
+ ((/^[ \r\n]/.test(value) && /[ \r\n]$/.test(value)) || /^\$|\$$/.test(value))
+ ) {
+ value = ` ${value} `;
+ }
+ let index = -1;
+ while (++index < state.unsafe.length) {
+ const pattern = state.unsafe[index];
+ if (!pattern.atBreak) continue;
+ const expression = state.compilePattern(pattern);
+ let match: RegExpExecArray | null;
+ // biome-ignore lint/suspicious/noAssignInExpressions: idiomatic regex iteration
+ while ((match = expression.exec(value)) !== null) {
+ let position = match.index;
+ if (value.codePointAt(position) === 10 && value.codePointAt(position - 1) === 13) {
+ position--;
+ }
+ value = `${value.slice(0, position)} ${value.slice(match.index + 1)}`;
+ }
+ }
+ return sequence + value + sequence;
+}
+
+function serializeDelete(
+ node: Extract<Nodes, { type: 'delete' }>,
+ _parent: Parents | undefined,
+ state: State,
+ info: Info,
+): string {
+ const delim = node.data?.sourceDelimiter === '~' ? '~' : '~~';
+ const tracker = state.createTracker(info);
+ const exit = state.enter('strikethrough');
+ let value = tracker.move(delim);
+ value += state.containerPhrasing(node, {
+ ...tracker.current(),
+ before: value,
+ after: '~',
+ });
+ value += tracker.move(delim);
+ exit();
+ return value;
+}
+
+function singleDollarMathReparses(value: string, info: Info): boolean {
+ if (value.length === 0) return false;
+ if (!/^\S/.test(value) || !/\S$/.test(value)) return false;
+ if (/[$\r\n]/.test(value)) return false;
+ if (value.endsWith('\\') || /\\[!-/:-@[-`{-~]/.test(value)) return false;
+ if (/^\d/.test(info.after)) return false;
+ if (info.before.endsWith('\\')) return false;
+ return true;
}
diff --git a/packages/core/src/markdown/wiki-embed-opaque.test.ts b/packages/core/src/markdown/wiki-embed-opaque.test.ts
new file mode 100644
index 000000000..b5c4ef28a
--- /dev/null
+++ b/packages/core/src/markdown/wiki-embed-opaque.test.ts
@@ -0,0 +1,181 @@
+import { describe, expect, test } from 'bun:test';
+import type { JSONContent } from '@tiptap/core';
+import { sharedExtensions } from '../extensions/shared.ts';
+import { MarkdownManager } from './index.ts';
+
+const mdManager = new MarkdownManager({ extensions: sharedExtensions });
+
+function roundTrip(md: string): string {
+ return mdManager.serialize(mdManager.parse(md));
+}
+
+function expectByteStable(md: string): void {
+ const once = roundTrip(md);
+ expect(once).toBe(md);
+ expect(roundTrip(once)).toBe(once);
+}
+
+function findNodes(json: JSONContent, type: string): JSONContent[] {
+ const out: JSONContent[] = [];
+ const visit = (n: JSONContent) => {
+ if (n.type === type) out.push(n);
+ for (const child of n.content ?? []) visit(child);
+ };
+ visit(json);
+ return out;
+}
+
+describe('opaque-extension wiki-embed — byte-exact PM round-trip', () => {
+ test('block-context opaque embed round-trips byte-identical', () => {
+ expectByteStable('![[notes.foo]]\n');
+ });
+
+ test('inline mid-prose opaque embed round-trips byte-identical', () => {
+ expectByteStable('see ![[notes.foo]] here\n');
+ });
+
+ test('extension-less embed round-trips byte-identical', () => {
+ expectByteStable('![[plainname]]\n');
+ });
+
+ test('serveable-but-not-inline-renderable extension round-trips byte-identical', () => {
+ expectByteStable('![[diagram.avi]]\n');
+ });
+
+ test('opaque embed with alias round-trips byte-identical', () => {
+ expectByteStable('![[notes.foo|My Alias]]\n');
+ });
+
+ test('opaque embed with anchor round-trips byte-identical', () => {
+ expectByteStable('![[notes.foo#section]]\n');
+ });
+
+ test('opaque embed with anchor and alias round-trips byte-identical', () => {
+ expectByteStable('![[notes.foo#section|Custom]]\n');
+ });
+
+ test('opaque embed coexisting with wikiLink and allowlisted embed round-trips', () => {
+ expectByteStable('See [[Index]], ![[diagram.png]] and ![[archive.xyz]] together.\n');
+ });
+});
+
+describe('opaque-extension wiki-embed — WYSIWYG label edit is honored, never reverted', () => {
+ function editEmbedLabel(json: JSONContent, newText: string): JSONContent {
+ let edited = 0;
+ const visit = (n: JSONContent) => {
+ if (
+ n.type === 'text' &&
+ n.marks?.some((m) => m.type === 'link' && m.attrs?.sourceForm === 'wikiembed')
+ ) {
+ n.text = newText;
+ edited++;
+ }
+ for (const child of n.content ?? []) visit(child);
+ };
+ visit(json);
+ expect(edited).toBe(1);
+ return json;
+ }
+
+ test('editing an aliased embed label replaces the alias', () => {
+ const json = editEmbedLabel(mdManager.parse('![[notes.foo|My Label]]\n'), 'Edited Label');
+ expect(mdManager.serialize(json)).toBe('![[notes.foo|Edited Label]]\n');
+ });
+
+ test('editing an alias-less embed label adds an alias (target preserved)', () => {
+ const json = editEmbedLabel(mdManager.parse('![[notes.foo]]\n'), 'My New Text');
+ expect(mdManager.serialize(json)).toBe('![[notes.foo|My New Text]]\n');
+ });
+
+ test('editing an anchored embed label adds an alias (target#anchor preserved)', () => {
+ const json = editEmbedLabel(mdManager.parse('![[notes.foo#section]]\n'), 'Renamed');
+ expect(mdManager.serialize(json)).toBe('![[notes.foo#section|Renamed]]\n');
+ });
+
+ test('the edited form is byte-stable on subsequent round-trips', () => {
+ const json = editEmbedLabel(mdManager.parse('![[notes.foo|My Label]]\n'), 'Edited Label');
+ const once = mdManager.serialize(json);
+ expect(roundTrip(once)).toBe(once);
+ });
+
+ test('an unedited label still round-trips byte-identical through the attrs path', () => {
+ const json = mdManager.parse('![[notes.foo|My Label]]\n');
+ expect(mdManager.serialize(json)).toBe('![[notes.foo|My Label]]\n');
+ });
+
+ test('a whitespace-only label edit does not produce an empty alias', () => {
+ const json = editEmbedLabel(mdManager.parse('![[notes.foo|My Label]]\n'), ' ');
+ expect(mdManager.serialize(json)).toBe('![[notes.foo|My Label]]\n');
+ });
+
+ test('a label edit containing `]]` keeps the attrs label — the closing delimiter is unrepresentable in the alias slot', () => {
+ const json = editEmbedLabel(mdManager.parse('![[notes.foo|My Label]]\n'), 'My ]] Label');
+ const once = mdManager.serialize(json);
+ expect(once).toBe('![[notes.foo|My Label]]\n');
+ expect(roundTrip(once)).toBe(once);
+ });
+});
+
+describe('opaque-extension wiki-embed — PM shape contract', () => {
+ test('opaque embed lands on text+link-mark tagged wikiembed, never a PM wikiLinkEmbed node', () => {
+ const json = mdManager.parse('![[notes.foo]]\n');
+ expect(findNodes(json, 'wikiLinkEmbed')).toHaveLength(0);
+ const texts = findNodes(json, 'text');
+ expect(texts).toHaveLength(1);
+ const linkMark = texts[0]?.marks?.find((mk) => mk.type === 'link');
+ expect(linkMark).toBeDefined();
+ expect(linkMark?.attrs?.sourceForm).toBe('wikiembed');
+ expect(linkMark?.attrs?.target).toBe('notes.foo');
+ expect(linkMark?.attrs?.anchor).toBeNull();
+ expect(linkMark?.attrs?.alias).toBeNull();
+ });
+
+ test('alias and anchor are carried on the mark, label shows the alias', () => {
+ const json = mdManager.parse('![[notes.foo#sec|My Alias]]\n');
+ const texts = findNodes(json, 'text');
+ expect(texts[0]?.text).toBe('My Alias');
+ const linkMark = texts[0]?.marks?.find((mk) => mk.type === 'link');
+ expect(linkMark?.attrs?.target).toBe('notes.foo');
+ expect(linkMark?.attrs?.anchor).toBe('sec');
+ expect(linkMark?.attrs?.alias).toBe('My Alias');
+ });
+
+ test('resolver-remapped href does not perturb the byte round-trip', () => {
+ const json = mdManager.parse('![[notes.foo]]\n', {
+ resolveEmbed: (target) => (target === 'notes.foo' ? 'attachments/notes.foo' : null),
+ sourcePath: 'docs/meeting.md',
+ });
+ const texts = findNodes(json, 'text');
+ const linkMark = texts[0]?.marks?.find((mk) => mk.type === 'link');
+ expect(linkMark?.attrs?.href).toBe('/attachments/notes.foo');
+ expect(linkMark?.attrs?.target).toBe('notes.foo');
+ expect(mdManager.serialize(json)).toBe('![[notes.foo]]\n');
+ });
+});
+
+describe('opaque-extension widening — no regression on neighbors', () => {
+ test('plain markdown link keeps a null sourceForm and round-trips unchanged', () => {
+ const json = mdManager.parse('[label](dest.foo)\n');
+ const texts = findNodes(json, 'text');
+ const linkMark = texts[0]?.marks?.find((mk) => mk.type === 'link');
+ expect(linkMark).toBeDefined();
+ expect(linkMark?.attrs?.sourceForm).toBeNull();
+ expect(mdManager.serialize(json)).toBe('[label](dest.foo)\n');
+ });
+
+ test('allowlisted block-context embed still promotes to jsxComponent and round-trips', () => {
+ const json = mdManager.parse('![[photo.png]]\n');
+ const node = json.content?.[0];
+ expect(node?.type).toBe('jsxComponent');
+ expect(node?.attrs?.componentName).toBe('WikiEmbedImage');
+ expect(mdManager.serialize(json)).toBe('![[photo.png]]\n');
+ });
+
+ test('allowlisted inline embed keeps the wikiembed chip and round-trips', () => {
+ expectByteStable('text ![[photo.png]] more text\n');
+ });
+
+ test('wikiLink sibling is untouched', () => {
+ expectByteStable('[[Page]]\n');
+ });
+});
diff --git a/packages/core/src/markdown/wiki-link-micromark.ts b/packages/core/src/markdown/wiki-link-micromark.ts
index 17dc71614..7aaf1099c 100644
--- a/packages/core/src/markdown/wiki-link-micromark.ts
+++ b/packages/core/src/markdown/wiki-link-micromark.ts
@@ -290,19 +290,31 @@ function topNode<T>(ctx: CompileContext): T {
function exitTarget(this: CompileContext, token: Token) {
const node = topNode<WikiLinkMdast | WikiLinkEmbedMdast>(this);
- node.data.target = this.sliceSerialize(token).trim();
+ const raw = this.sliceSerialize(token);
+ node.data.target = raw.trim();
+ if (raw !== node.data.target) {
+ node.data.sourceTarget = raw;
+ }
}
function exitAnchor(this: CompileContext, token: Token) {
const node = topNode<WikiLinkMdast | WikiLinkEmbedMdast>(this);
- const raw = this.sliceSerialize(token).trim();
- node.data.anchor = raw.length ? raw : null;
+ const raw = this.sliceSerialize(token);
+ const trimmed = raw.trim();
+ node.data.anchor = trimmed.length ? trimmed : null;
+ if (node.data.anchor !== null && raw !== trimmed) {
+ node.data.sourceAnchor = raw;
+ }
}
function exitAlias(this: CompileContext, token: Token) {
const node = topNode<WikiLinkMdast | WikiLinkEmbedMdast>(this);
- const raw = this.sliceSerialize(token).trim();
- node.data.alias = raw.length ? raw : null;
+ const raw = this.sliceSerialize(token);
+ const trimmed = raw.trim();
+ node.data.alias = trimmed.length ? trimmed : null;
+ if (node.data.alias !== null && raw !== trimmed) {
+ node.data.sourceAlias = raw;
+ }
}
function finalizeLabel(node: WikiLinkMdast | WikiLinkEmbedMdast): void {
@@ -336,14 +348,18 @@ export const wikiLinkFromMarkdown: FromMarkdownExtension = {
},
};
+function rawSegmentOr(raw: string | null | undefined, current: string): string {
+ return typeof raw === 'string' && raw.trim() === current ? raw : current;
+}
+
const wikiLinkHandler: ToMarkdownHandle = (node) => {
const wiki = node as unknown as WikiLinkMdast;
const target = wiki.data?.target ?? '';
const anchor = wiki.data?.anchor;
const alias = wiki.data?.alias;
- let out = `[[${target}`;
- if (anchor) out += `#${anchor}`;
- if (alias) out += `|${alias}`;
+ let out = `[[${rawSegmentOr(wiki.data?.sourceTarget, target)}`;
+ if (anchor) out += `#${rawSegmentOr(wiki.data?.sourceAnchor, anchor)}`;
+ if (alias) out += `|${rawSegmentOr(wiki.data?.sourceAlias, alias)}`;
return `${out}]]`;
};
diff --git a/packages/core/src/schema-snapshot.json b/packages/core/src/schema-snapshot.json
index 3239a366f..4c0619bbe 100644
--- a/packages/core/src/schema-snapshot.json
+++ b/packages/core/src/schema-snapshot.json
@@ -14,6 +14,18 @@
},
"spread": {
"hasDefault": true
+ },
+ "sourceMarkerSpacing": {
+ "hasDefault": true
+ },
+ "sourceOrdinal": {
+ "hasDefault": true
+ },
+ "sourceCheckboxChar": {
+ "hasDefault": true
+ },
+ "sourceContinuationIndent": {
+ "hasDefault": true
}
},
"content": "paragraph block*",
@@ -22,7 +34,11 @@
"atom": false
},
"doc": {
- "attrs": {},
+ "attrs": {
+ "sourceDocBoundary": {
+ "hasDefault": true
+ }
+ },
"content": "block+",
"group": "",
"inline": false,
@@ -99,6 +115,9 @@
},
"language": {
"hasDefault": true
+ },
+ "sourceDelimiter": {
+ "hasDefault": true
}
},
"content": "",
@@ -119,6 +138,15 @@
},
"resolved": {
"hasDefault": true
+ },
+ "sourceTarget": {
+ "hasDefault": true
+ },
+ "sourceAnchor": {
+ "hasDefault": true
+ },
+ "sourceAlias": {
+ "hasDefault": true
}
},
"content": "",
@@ -196,6 +224,18 @@
},
"sourceStyle": {
"hasDefault": true
+ },
+ "sourceClosingFenceLength": {
+ "hasDefault": true
+ },
+ "sourceFenceIndent": {
+ "hasDefault": true
+ },
+ "sourceInfoPadding": {
+ "hasDefault": true
+ },
+ "sourceIndents": {
+ "hasDefault": true
}
},
"content": "text*",
@@ -230,6 +270,12 @@
},
"sourceContiguousNext": {
"hasDefault": true
+ },
+ "sourceLeadingIndent": {
+ "hasDefault": true
+ },
+ "sourceInteriorSpacing": {
+ "hasDefault": true
}
},
"content": "inline*",
@@ -297,6 +343,12 @@
"attrs": {
"sourceDashCounts": {
"hasDefault": true
+ },
+ "sourceOuterPipes": {
+ "hasDefault": true
+ },
+ "sourceAlignmentPadding": {
+ "hasDefault": true
}
},
"content": "tableRow+",
@@ -447,6 +499,9 @@
},
"sourceFenceLength": {
"hasDefault": true
+ },
+ "sourcePadded": {
+ "hasDefault": true
}
},
"excludes": "",
@@ -454,13 +509,6 @@
"inclusive": true,
"spanning": true
},
- "strike": {
- "attrs": {},
- "excludes": "strike",
- "group": "",
- "inclusive": true,
- "spanning": true
- },
"underline": {
"attrs": {},
"excludes": "underline",
@@ -497,6 +545,17 @@
"inclusive": true,
"spanning": true
},
+ "strike": {
+ "attrs": {
+ "sourceDelimiter": {
+ "hasDefault": true
+ }
+ },
+ "excludes": "strike",
+ "group": "",
+ "inclusive": true,
+ "spanning": true
+ },
"link": {
"attrs": {
"href": {
@@ -583,6 +642,7 @@
"listItem",
"emphasis",
"strong",
+ "strike",
"code",
"codeBlock",
"blockquote",
@@ -593,6 +653,7 @@
"linkRefDef",
"hardBreak",
"escapeMark",
+ "docFidelity",
"sourceLiteral",
"starterKit",
"table",
diff --git a/packages/server/package.json b/packages/server/package.json
index 0f153b79a..733354aba 100644
--- a/packages/server/package.json
+++ b/packages/server/package.json
@@ -62,6 +62,7 @@
"test": "bun test"
},
"devDependencies": {
+ "@types/mdast": "^4.0.4",
"@types/shell-quote": "^1.7.5",
"lib0": "^0.2.117",
"@types/ws": "^8.18.1",
diff --git a/packages/server/src/bridge-watchdog.test.ts b/packages/server/src/bridge-watchdog.test.ts
index 6305e5563..0dbec4a74 100644
--- a/packages/server/src/bridge-watchdog.test.ts
+++ b/packages/server/src/bridge-watchdog.test.ts
@@ -101,6 +101,25 @@ describe('assertBridgeInvariant — no-op for tolerance-equivalent inputs', () =
}).not.toThrow();
expect(getMetrics().bridgeInvariantViolations).toBe(0);
});
+
+ test('table-row trailing-pipe divergence tolerated (row-no-trailing-pipe)', () => {
+ expect(() => {
+ assertBridgeInvariant('| a | b\n| - | -\n| 1 | 2\n', '| a | b|\n| - | -|\n| 1 | 2|\n', {
+ site: 'observer-b',
+ });
+ }).not.toThrow();
+ expect(getMetrics().bridgeInvariantViolations).toBe(0);
+ });
+
+ test('touched-cell table divergence is NOT absorbed by the trailing-pipe tolerance', () => {
+ expect(() => {
+ assertBridgeInvariant(
+ '| a | b |\n| - | - |\n| 1 | 2 |\n',
+ '| a | b |\n| - | - |\n| 1 | 99 |\n',
+ { site: 'observer-b' },
+ );
+ }).toThrow(BridgeInvariantViolationError);
+ });
});
describe('assertBridgeInvariant — throws under NODE_ENV=test (default for bun test)', () => {
diff --git a/packages/server/src/bridge-watchdog.ts b/packages/server/src/bridge-watchdog.ts
index 74af45c29..6cf3483b4 100644
--- a/packages/server/src/bridge-watchdog.ts
+++ b/packages/server/src/bridge-watchdog.ts
@@ -4,6 +4,7 @@ import {
BridgeInvariantViolationError,
type BridgeToleranceClass,
detectAppliedToleranceClasses,
+ emitToleranceFire,
normalizeBridge,
toBridgeInvariantLog,
} from '@inkeep/open-knowledge-core';
@@ -238,6 +239,7 @@ export function assertBridgeInvariant(
if (ytextNorm === fragNorm) {
if (ytextSnapshot !== fragmentMdSnapshot) {
const classes = detectAppliedToleranceClasses(ytextSnapshot, fragmentMdSnapshot);
+ emitToleranceFire(classes, ytextSnapshot, fragmentMdSnapshot, opts.docName);
for (const cls of classes) {
if (shouldEmitBridgeToleranceApplied(opts.site, cls, opts.nowMs)) {
incrementBridgeToleranceApplied(cls);
diff --git a/packages/server/src/index.ts b/packages/server/src/index.ts
index f99444644..a743dfc78 100644
--- a/packages/server/src/index.ts
+++ b/packages/server/src/index.ts
@@ -481,6 +481,11 @@ export {
spansCurrentPath,
spansPreviousPath,
} from './telemetry-file-sink.ts';
+export {
+ initToleranceTelemetryWriter,
+ isToleranceTelemetryEnabled,
+ teardownToleranceTelemetryWriter,
+} from './tolerance-telemetry-writer.ts';
export {
acquireUiLock,
readUiLock,
diff --git a/packages/server/src/map-driven-observer-a.test.ts b/packages/server/src/map-driven-observer-a.test.ts
new file mode 100644
index 000000000..22604edf9
--- /dev/null
+++ b/packages/server/src/map-driven-observer-a.test.ts
@@ -0,0 +1,354 @@
+import { describe, expect, test } from 'bun:test';
+import { MarkdownManager, sharedExtensions } from '@inkeep/open-knowledge-core';
+import { getSchema } from '@tiptap/core';
+import { updateYFragment } from '@tiptap/y-tiptap';
+import * as Y from 'yjs';
+import { AGENT_WRITE_ORIGIN } from './agent-sessions.ts';
+import { composeAndWriteRawBody } from './bridge-intake.ts';
+import { computeMapDrivenBodySplice } from './map-driven-splice.ts';
+import { getMetrics } from './metrics.ts';
+import { OBSERVER_SYNC_ORIGIN, setupServerObservers } from './server-observers.ts';
+
+const mdManager = new MarkdownManager({ extensions: sharedExtensions });
+const schema = getSchema(sharedExtensions);
+
+function createTestDoc() {
+ const doc = new Y.Doc();
+ const xmlFragment = doc.getXmlFragment('default');
+ const ytext = doc.getText('source');
+ return { doc, xmlFragment, ytext };
+}
+
+function populateFragment(doc: Y.Doc, xmlFragment: Y.XmlFragment, md: string): void {
+ const json = mdManager.parse(md);
+ const pmNode = schema.nodeFromJSON(json);
+ const meta = { mapping: new Map(), isOMark: new Map() };
+ updateYFragment(doc, xmlFragment, pmNode, meta);
+}
+
+interface CapturedDelta {
+ readonly origin: unknown;
+ readonly ops: ReadonlyArray<{ retain?: number; insert?: string | unknown[]; delete?: number }>;
+}
+
+function captureYTextDeltas(ytext: Y.Text): CapturedDelta[] {
+ const captured: CapturedDelta[] = [];
+ const handler = (event: Y.YTextEvent, transaction: Y.Transaction): void => {
+ captured.push({ origin: transaction.origin, ops: event.changes.delta });
+ };
+ ytext.observe(handler);
+ return captured;
+}
+
+describe('map-driven Observer A — default Path A behavior', () => {
+ test('(a) single-block edit produces narrow splice covering only the edited block', () => {
+ const { doc, xmlFragment, ytext } = createTestDoc();
+ populateFragment(doc, xmlFragment, '# Heading\n\nFirst paragraph.\n\nSecond paragraph.\n');
+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });
+
+ const before = ytext.toString();
+ const deltas = captureYTextDeltas(ytext);
+
+ populateFragment(
+ doc,
+ xmlFragment,
+ '# Heading\n\nFirst paragraph EDITED.\n\nSecond paragraph.\n',
+ );
+
+ const observerWrites = deltas.filter((d) => d.origin === OBSERVER_SYNC_ORIGIN);
+ expect(observerWrites.length).toBeGreaterThanOrEqual(1);
+ const mapDrivenWrite = observerWrites[observerWrites.length - 1];
+
+ const retainOps = mapDrivenWrite.ops.filter((op) => op.retain !== undefined);
+ const insertOps = mapDrivenWrite.ops.filter((op) => op.insert !== undefined);
+ const deleteOps = mapDrivenWrite.ops.filter((op) => op.delete !== undefined);
+
+ expect(retainOps.length).toBeLessThanOrEqual(2);
+ expect(insertOps.length + deleteOps.length).toBeGreaterThanOrEqual(1);
+
+ const headingEnd = before.indexOf('# Heading') + '# Heading'.length;
+ const secondParaStart = before.indexOf('Second paragraph');
+
+ const leadingRetain = mapDrivenWrite.ops[0]?.retain ?? 0;
+ expect(leadingRetain).toBeGreaterThanOrEqual(headingEnd);
+ expect(leadingRetain).toBeLessThanOrEqual(before.indexOf('First paragraph') + 1);
+
+ let cursorAfterWrite = leadingRetain;
+ for (const op of mapDrivenWrite.ops.slice(1)) {
+ if (op.delete !== undefined) cursorAfterWrite += op.delete;
+ }
+ expect(cursorAfterWrite).toBeLessThanOrEqual(secondParaStart);
+
+ cleanup();
+ });
+
+ test('(b) untouched bytes outside the splice are byte-identical pre→post (AC1)', () => {
+ const { doc, xmlFragment, ytext } = createTestDoc();
+ populateFragment(doc, xmlFragment, '# Heading\n\nFirst.\n\nUntouched bytes here.\n');
+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });
+
+ const before = ytext.toString();
+ const untouchedSnippet = 'Untouched bytes here.';
+ const untouchedStartBefore = before.indexOf(untouchedSnippet);
+ expect(untouchedStartBefore).toBeGreaterThanOrEqual(0);
+ const tailBefore = before.slice(untouchedStartBefore);
+
+ populateFragment(doc, xmlFragment, '# Heading\n\nEDITED first.\n\nUntouched bytes here.\n');
+
+ const after = ytext.toString();
+ const untouchedStartAfter = after.indexOf(untouchedSnippet);
+ expect(untouchedStartAfter).toBeGreaterThanOrEqual(0);
+ expect(after.slice(untouchedStartAfter)).toBe(tailBefore);
+
+ cleanup();
+ });
+
+ test('(c) contiguous multi-block edit produces splice union covering both edited blocks', () => {
+ const { doc, xmlFragment, ytext } = createTestDoc();
+ populateFragment(doc, xmlFragment, 'first.\n\nsecond.\n\nthird.\n');
+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });
+
+ const before = ytext.toString();
+ const deltas = captureYTextDeltas(ytext);
+
+ populateFragment(doc, xmlFragment, 'first EDITED.\n\nsecond EDITED.\n\nthird.\n');
+
+ const observerWrites = deltas.filter((d) => d.origin === OBSERVER_SYNC_ORIGIN);
+ const mapDrivenWrite = observerWrites[observerWrites.length - 1];
+
+ const leadingRetain = mapDrivenWrite.ops[0]?.retain ?? 0;
+ let cursorAfterDeletes = leadingRetain;
+ for (const op of mapDrivenWrite.ops.slice(1)) {
+ if (op.delete !== undefined) cursorAfterDeletes += op.delete;
+ }
+ const thirdStart = before.indexOf('third.');
+ expect(cursorAfterDeletes).toBeLessThanOrEqual(thirdStart);
+
+ const after = ytext.toString();
+ expect(after).toContain('first EDITED.');
+ expect(after).toContain('second EDITED.');
+ expect(after.slice(after.indexOf('third.'))).toBe(before.slice(thirdStart));
+
+ cleanup();
+ });
+
+ test('(d) synthetic-doc name short-circuits to fallback path (no map-driven splice attempted)', () => {
+ const { doc, xmlFragment, ytext } = createTestDoc();
+ populateFragment(doc, xmlFragment, 'A.\n\nB.\n');
+ const cleanup = setupServerObservers({
+ doc,
+ xmlFragment,
+ ytext,
+ mdManager,
+ schema,
+ docName: '__system__',
+ });
+
+ populateFragment(doc, xmlFragment, 'A EDITED.\n\nB.\n');
+
+ expect(ytext.toString()).toContain('A EDITED.');
+ expect(ytext.toString()).toContain('B.');
+
+ cleanup();
+ });
+
+ test('(e) edit in paragraph containing ==highlight== degrades to block granularity (documented sub-block limitation)', () => {
+ const { doc, xmlFragment, ytext } = createTestDoc();
+ populateFragment(doc, xmlFragment, 'Para with ==highlight== inside.\n\nUntouched after.\n');
+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });
+
+ const before = ytext.toString();
+ const untouchedSnippet = 'Untouched after.';
+ const tailBefore = before.slice(before.indexOf(untouchedSnippet));
+
+ populateFragment(
+ doc,
+ xmlFragment,
+ 'Para with ==highlight== inside EDITED.\n\nUntouched after.\n',
+ );
+
+ const after = ytext.toString();
+ const untouchedStartAfter = after.indexOf(untouchedSnippet);
+ expect(untouchedStartAfter).toBeGreaterThanOrEqual(0);
+ expect(after.slice(untouchedStartAfter)).toBe(tailBefore);
+
+ cleanup();
+ });
+
+ test('(f) map-driven splice is the default — active with no env configuration', () => {
+ expect(process.env.OK_MAP_DRIVEN_OBSERVER_A).toBeUndefined();
+
+ const raw = '# Notes\n\n| a | b |\n| - | - |\n| 1 | 2\n';
+ const { doc, xmlFragment, ytext } = createTestDoc();
+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });
+ doc.transact(() => {
+ composeAndWriteRawBody(doc, raw, 'agent');
+ }, AGENT_WRITE_ORIGIN);
+ expect(ytext.toString()).toBe(raw);
+
+ populateFragment(doc, xmlFragment, raw.replace('# Notes', '# Notes EDITED'));
+
+ expect(ytext.toString()).toContain('# Notes EDITED');
+ expect(ytext.toString()).toContain('| 1 | 2\n');
+
+ cleanup();
+ });
+
+ test('(g) fallback: an offset-less block (comment block) falls back to applyIncrementalDiff and still converges', () => {
+ const { doc, xmlFragment, ytext } = createTestDoc();
+ populateFragment(doc, xmlFragment, '<!-- note -->\n\nOriginal.\n');
+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });
+
+ populateFragment(doc, xmlFragment, '<!-- note -->\n\nOriginal.\n\nAdded.\n');
+
+ expect(ytext.toString()).toContain('Original.');
+ expect(ytext.toString()).toContain('Added.');
+
+ cleanup();
+ });
+
+ describe('dash-count tripwire — concurrent source-form table edits are detected + spliced, never silently dropped', () => {
+ const narrowDashBody = 'before\n\n| a | b |\n| - | - |\n| 1 | 2 |\n\nafter\n';
+ const wideDashBody = 'before\n\n| a | b |\n| --- | --- |\n| 1 | 2 |\n\nafter\n';
+
+ test('computeMapDrivenBodySplice detects a dash-count-only change and emits the splice', () => {
+ const splice = computeMapDrivenBodySplice(
+ narrowDashBody,
+ mdManager.parse(wideDashBody),
+ mdManager,
+ );
+
+ expect(splice).not.toBeNull();
+ if (!splice) throw new Error('unreachable');
+ const applied =
+ narrowDashBody.slice(0, splice.spliceStart) +
+ splice.newSlice +
+ narrowDashBody.slice(splice.spliceEnd);
+ expect(applied).toBe(wideDashBody);
+ });
+
+ test('Observer A applies a dash-count-only fragment change to Y.Text (detected + spliced, blocks outside untouched)', () => {
+ const { doc, xmlFragment, ytext } = createTestDoc();
+ populateFragment(doc, xmlFragment, narrowDashBody);
+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });
+ expect(ytext.toString()).toBe(narrowDashBody);
+
+ populateFragment(doc, xmlFragment, wideDashBody);
+
+ expect(ytext.toString()).toBe(wideDashBody);
+
+ cleanup();
+ });
+ });
+
+ describe('splice-path observability — applied vs fallback(reason) counters', () => {
+ function fallbackTotal(m: ReturnType<typeof getMetrics>): number {
+ return Object.values(m.mapDrivenSpliceFallback).reduce((a, b) => a + (b ?? 0), 0);
+ }
+
+ test('a successful map-driven splice increments mapDrivenSpliceApplied (no fallback)', () => {
+ const raw = '# Heading\n\nFirst.\n\nSecond.\n';
+ const { doc, xmlFragment, ytext } = createTestDoc();
+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });
+ doc.transact(() => {
+ composeAndWriteRawBody(doc, raw, 'agent');
+ }, AGENT_WRITE_ORIGIN);
+ expect(ytext.toString()).toBe(raw);
+
+ const before = getMetrics();
+ populateFragment(doc, xmlFragment, raw.replace('First.', 'First EDITED.'));
+ const after = getMetrics();
+
+ expect(after.mapDrivenSpliceApplied - before.mapDrivenSpliceApplied).toBe(1);
+ expect(fallbackTotal(after) - fallbackTotal(before)).toBe(0);
+
+ cleanup();
+ });
+
+ test('a synthetic-doc drain increments fallback reason synthetic-doc, not applied', () => {
+ const raw = 'A.\n\nB.\n';
+ const { doc, xmlFragment, ytext } = createTestDoc();
+ const cleanup = setupServerObservers({
+ doc,
+ xmlFragment,
+ ytext,
+ mdManager,
+ schema,
+ docName: '__system__',
+ });
+ doc.transact(() => {
+ composeAndWriteRawBody(doc, raw, 'agent');
+ }, AGENT_WRITE_ORIGIN);
+
+ const before = getMetrics();
+ populateFragment(doc, xmlFragment, 'A EDITED.\n\nB.\n');
+ const after = getMetrics();
+
+ expect(
+ (after.mapDrivenSpliceFallback['synthetic-doc'] ?? 0) -
+ (before.mapDrivenSpliceFallback['synthetic-doc'] ?? 0),
+ ).toBe(1);
+ expect(after.mapDrivenSpliceApplied - before.mapDrivenSpliceApplied).toBe(0);
+
+ cleanup();
+ });
+
+ test('an offset-less block drain increments fallback reason missing-position', () => {
+ const raw = '<!-- note -->\n\nOriginal.\n';
+ const { doc, xmlFragment, ytext } = createTestDoc();
+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });
+ doc.transact(() => {
+ composeAndWriteRawBody(doc, raw, 'agent');
+ }, AGENT_WRITE_ORIGIN);
+
+ const before = getMetrics();
+ populateFragment(doc, xmlFragment, '<!-- note -->\n\nOriginal.\n\nAdded.\n');
+ const after = getMetrics();
+
+ expect(
+ (after.mapDrivenSpliceFallback['missing-position'] ?? 0) -
+ (before.mapDrivenSpliceFallback['missing-position'] ?? 0),
+ ).toBe(1);
+ expect(after.mapDrivenSpliceApplied - before.mapDrivenSpliceApplied).toBe(0);
+
+ cleanup();
+ });
+
+ test('a parse/serialize throw inside the splice reports parse-error instead of vanishing', () => {
+ const throwingManager = {
+ parseToMdast: () => {
+ throw new Error('synthetic parser regression');
+ },
+ serialize: () => '',
+ } as unknown as MarkdownManager;
+ const reasons: string[] = [];
+
+ const splice = computeMapDrivenBodySplice(
+ 'A.\n',
+ mdManager.parse('A.\n'),
+ throwingManager,
+ (reason) => {
+ reasons.push(reason);
+ },
+ );
+
+ expect(splice).toBeNull();
+ expect(reasons).toEqual(['parse-error']);
+ });
+
+ test('an offset-less block reports missing-position through the pure computer', () => {
+ const reasons: string[] = [];
+ const splice = computeMapDrivenBodySplice(
+ '<!-- note -->\n',
+ mdManager.parse('<!-- note -->\n\nX.\n'),
+ mdManager,
+ (reason) => {
+ reasons.push(reason);
+ },
+ );
+
+ expect(splice).toBeNull();
+ expect(reasons).toEqual(['missing-position']);
+ });
+ });
+});
diff --git a/packages/server/src/map-driven-splice.test.ts b/packages/server/src/map-driven-splice.test.ts
new file mode 100644
index 000000000..d674c46df
--- /dev/null
+++ b/packages/server/src/map-driven-splice.test.ts
@@ -0,0 +1,206 @@
+import { describe, expect, test } from 'bun:test';
+import { MarkdownManager, sharedExtensions } from '@inkeep/open-knowledge-core';
+import type { JSONContent } from '@tiptap/core';
+import { computeMapDrivenBodySplice } from './map-driven-splice.ts';
+
+const mdManager = new MarkdownManager({ extensions: sharedExtensions });
+
+function applySplice(
+ oldBody: string,
+ splice: { spliceStart: number; spliceEnd: number; newSlice: string },
+): string {
+ return oldBody.slice(0, splice.spliceStart) + splice.newSlice + oldBody.slice(splice.spliceEnd);
+}
+
+function pmFromMd(md: string): JSONContent {
+ return mdManager.parse(md);
+}
+
+describe('computeMapDrivenBodySplice', () => {
+ describe('byte preservation outside the splice', () => {
+ test('single-block edit produces splice covering only the edited block', () => {
+ const oldBody = '# Heading\n\nFirst paragraph.\n\nSecond paragraph.\n';
+ const newBody = '# Heading\n\nFirst paragraph EDITED.\n\nSecond paragraph.\n';
+
+ const splice = computeMapDrivenBodySplice(oldBody, pmFromMd(newBody), mdManager);
+ expect(splice).not.toBeNull();
+ if (!splice) return;
+
+ const headingEnd = oldBody.indexOf('# Heading') + '# Heading'.length;
+ expect(splice.spliceStart).toBeGreaterThanOrEqual(headingEnd);
+ const secondParaStart = oldBody.indexOf('Second paragraph');
+ expect(splice.spliceEnd).toBeLessThanOrEqual(secondParaStart);
+
+ expect(oldBody.slice(0, splice.spliceStart)).toBe(
+ applySplice(oldBody, splice).slice(0, splice.spliceStart),
+ );
+ const reconstructed = applySplice(oldBody, splice);
+ expect(reconstructed.slice(splice.spliceStart + splice.newSlice.length)).toBe(
+ oldBody.slice(splice.spliceEnd),
+ );
+ });
+
+ test('result of applying splice equals the canonical newBody serialization', () => {
+ const oldBody = '# Heading\n\nFirst.\n\nSecond.\n';
+ const newPm = pmFromMd('# Heading\n\nFirst CHANGED.\n\nSecond.\n');
+ const splice = computeMapDrivenBodySplice(oldBody, newPm, mdManager);
+ expect(splice).not.toBeNull();
+ if (!splice) return;
+
+ const reconstructed = applySplice(oldBody, splice);
+ const canonicalNew = mdManager.serialize(newPm);
+ const reconstructedMdast = mdManager.parseToMdast(reconstructed);
+ const canonicalMdast = mdManager.parseToMdast(canonicalNew);
+ expect(reconstructedMdast.children.length).toBe(canonicalMdast.children.length);
+ });
+ });
+
+ describe('source-form preservation through structural equality', () => {
+ test('an untouched block whose canonical form would canonicalize bytes is excluded from splice', () => {
+ const oldBody = '*italic one*\n\nuntouched two\n';
+ const newPmJson = pmFromMd('*italic one* EDIT\n\nuntouched two\n');
+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);
+ expect(splice).not.toBeNull();
+ if (!splice) return;
+
+ const result = applySplice(oldBody, splice);
+ expect(result).toContain('untouched two');
+ const oldUntouched = oldBody.slice(oldBody.indexOf('untouched two'));
+ const newUntouched = result.slice(result.indexOf('untouched two'));
+ expect(newUntouched).toBe(oldUntouched);
+ });
+
+ test('block matching the structural shape but canonicalized in newBody is NOT spliced', () => {
+ const oldBody = '*italic*\n\nplain\n';
+ const newPmJson = pmFromMd('*italic*\n\nplain CHANGED\n');
+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);
+ expect(splice).not.toBeNull();
+ if (!splice) return;
+
+ const result = applySplice(oldBody, splice);
+ expect(result.startsWith('*italic*\n\n')).toBe(true);
+ });
+ });
+
+ describe('insertions and deletions at boundaries', () => {
+ test('append a new paragraph at end', () => {
+ const oldBody = 'First.\n';
+ const newPmJson = pmFromMd('First.\n\nSecond.\n');
+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);
+ expect(splice).not.toBeNull();
+ if (!splice) return;
+
+ const result = applySplice(oldBody, splice);
+ expect(result).toContain('First.');
+ expect(result).toContain('Second.');
+ expect(result.indexOf('First.')).toBe(0);
+ });
+
+ test('prepend a new paragraph at start', () => {
+ const oldBody = 'Second.\n';
+ const newPmJson = pmFromMd('First.\n\nSecond.\n');
+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);
+ expect(splice).not.toBeNull();
+ if (!splice) return;
+
+ const result = applySplice(oldBody, splice);
+ expect(result).toContain('First.');
+ expect(result).toContain('Second.');
+ expect(result.indexOf('First.')).toBeLessThan(result.indexOf('Second.'));
+ });
+
+ test('insert a paragraph in the middle preserves surrounding blocks byte-identically', () => {
+ const oldBody = '*Pre*\n\nPost.\n';
+ const newPmJson = pmFromMd('*Pre*\n\nMiddle.\n\nPost.\n');
+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);
+ expect(splice).not.toBeNull();
+ if (!splice) return;
+
+ const result = applySplice(oldBody, splice);
+ expect(result.startsWith('*Pre*')).toBe(true);
+ expect(result).toContain('Middle.');
+ expect(result.endsWith('Post.\n')).toBe(true);
+ });
+
+ test('delete a middle block', () => {
+ const oldBody = 'A.\n\nB.\n\nC.\n';
+ const newPmJson = pmFromMd('A.\n\nC.\n');
+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);
+ expect(splice).not.toBeNull();
+ if (!splice) return;
+
+ const result = applySplice(oldBody, splice);
+ expect(result).toContain('A.');
+ expect(result).toContain('C.');
+ expect(result).not.toContain('B.');
+ });
+ });
+
+ describe('synthetic / empty inputs', () => {
+ test('empty oldBody + new content produces splice that yields the new content', () => {
+ const oldBody = '';
+ const newPmJson = pmFromMd('A new paragraph.\n');
+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);
+ expect(splice).not.toBeNull();
+ if (!splice) return;
+
+ const result = applySplice(oldBody, splice);
+ expect(result).toContain('A new paragraph.');
+ });
+
+ test('no-change input produces no-op splice', () => {
+ const oldBody = 'A.\n\nB.\n';
+ const newPmJson = pmFromMd(oldBody);
+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);
+ expect(splice).not.toBeNull();
+ if (!splice) return;
+
+ const result = applySplice(oldBody, splice);
+ const oldChildren = mdManager.parseToMdast(oldBody).children;
+ const resultChildren = mdManager.parseToMdast(result).children;
+ expect(resultChildren.length).toBe(oldChildren.length);
+ });
+ });
+
+ describe('contiguous multi-block edits', () => {
+ test('editing two adjacent blocks unions their splice ranges', () => {
+ const oldBody = 'first.\n\nsecond.\n\nthird.\n';
+ const newPmJson = pmFromMd('first EDITED.\n\nsecond EDITED.\n\nthird.\n');
+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);
+ expect(splice).not.toBeNull();
+ if (!splice) return;
+
+ const result = applySplice(oldBody, splice);
+ expect(result.endsWith('third.\n')).toBe(true);
+
+ const thirdStartOld = oldBody.indexOf('third.');
+ expect(splice.spliceEnd).toBeLessThanOrEqual(thirdStartOld);
+ });
+
+ test('non-contiguous multi-block edits collapse into one over-wide splice (documented AC2 degradation)', () => {
+ const oldBody = 'first.\n\nmiddle.\n\nthird.\n';
+ const newPmJson = pmFromMd('first EDITED.\n\nmiddle.\n\nthird EDITED.\n');
+ const splice = computeMapDrivenBodySplice(oldBody, newPmJson, mdManager);
+ expect(splice).not.toBeNull();
+ if (!splice) return;
+
+ const middleStart = oldBody.indexOf('middle.');
+ expect(splice.spliceStart).toBeLessThanOrEqual(middleStart);
+ expect(splice.spliceEnd).toBeGreaterThanOrEqual(middleStart + 'middle.'.length);
+
+ const result = applySplice(oldBody, splice);
+ expect(result).toContain('first EDITED.');
+ expect(result).toContain('middle.');
+ expect(result).toContain('third EDITED.');
+ });
+ });
+
+ describe('robustness to parse failure', () => {
+ test('returns null when serialize throws on schema-rejected JSON', () => {
+ const oldBody = 'A.\n';
+ const malformed = { type: 'not-a-real-node-type' } as JSONContent;
+ const splice = computeMapDrivenBodySplice(oldBody, malformed, mdManager);
+ expect(splice).toBeNull();
+ });
+ });
+});
diff --git a/packages/server/src/map-driven-splice.ts b/packages/server/src/map-driven-splice.ts
new file mode 100644
index 000000000..5c9d0ce41
--- /dev/null
+++ b/packages/server/src/map-driven-splice.ts
@@ -0,0 +1,111 @@
+import type { MarkdownManager } from '@inkeep/open-knowledge-core';
+import type { JSONContent } from '@tiptap/core';
+import type { RootContent } from 'mdast';
+
+export interface MapDrivenSplice {
+ readonly spliceStart: number;
+ readonly spliceEnd: number;
+ readonly newSlice: string;
+}
+
+export function computeMapDrivenBodySplice(
+ oldBody: string,
+ newPmJson: JSONContent,
+ mdManager: MarkdownManager,
+ onFallback?: (reason: 'parse-error' | 'missing-position') => void,
+): MapDrivenSplice | null {
+ let oldChildren: readonly RootContent[];
+ let newBody: string;
+ let newChildren: readonly RootContent[];
+ try {
+ oldChildren = mdManager.parseToMdast(oldBody).children;
+ newBody = mdManager.serialize(newPmJson);
+ newChildren = mdManager.parseToMdast(newBody).children;
+ } catch {
+ onFallback?.('parse-error');
+ return null;
+ }
+
+ if (!allBlocksCarryPositions(oldChildren) || !allBlocksCarryPositions(newChildren)) {
+ onFallback?.('missing-position');
+ return null;
+ }
+
+ let prefixLen = 0;
+ while (
+ prefixLen < oldChildren.length &&
+ prefixLen < newChildren.length &&
+ structurallyEqual(oldChildren[prefixLen], newChildren[prefixLen])
+ ) {
+ prefixLen++;
+ }
+
+ let suffixLen = 0;
+ while (
+ suffixLen < oldChildren.length - prefixLen &&
+ suffixLen < newChildren.length - prefixLen &&
+ structurallyEqual(
+ oldChildren[oldChildren.length - 1 - suffixLen],
+ newChildren[newChildren.length - 1 - suffixLen],
+ )
+ ) {
+ suffixLen++;
+ }
+
+ const spliceStart =
+ prefixLen > 0
+ ? blockEndOffset(oldChildren[prefixLen - 1])
+ : oldChildren.length > 0
+ ? blockStartOffset(oldChildren[0])
+ : 0;
+ const spliceEnd =
+ suffixLen > 0 ? blockStartOffset(oldChildren[oldChildren.length - suffixLen]) : oldBody.length;
+
+ const newSliceStart =
+ prefixLen > 0
+ ? blockEndOffset(newChildren[prefixLen - 1])
+ : newChildren.length > 0
+ ? blockStartOffset(newChildren[0])
+ : 0;
+ const newSliceEnd =
+ suffixLen > 0 ? blockStartOffset(newChildren[newChildren.length - suffixLen]) : newBody.length;
+
+ return {
+ spliceStart,
+ spliceEnd,
+ newSlice: newBody.slice(newSliceStart, newSliceEnd),
+ };
+}
+
+function allBlocksCarryPositions(children: readonly RootContent[]): boolean {
+ for (const child of children) {
+ const start = child.position?.start?.offset;
+ const end = child.position?.end?.offset;
+ if (typeof start !== 'number' || typeof end !== 'number') return false;
+ }
+ return true;
+}
+
+function blockStartOffset(node: RootContent): number {
+ const offset = node.position?.start?.offset;
+ if (typeof offset !== 'number') {
+ throw new Error('mdast node missing position.start.offset');
+ }
+ return offset;
+}
+
+function blockEndOffset(node: RootContent): number {
+ const offset = node.position?.end?.offset;
+ if (typeof offset !== 'number') {
+ throw new Error('mdast node missing position.end.offset');
+ }
+ return offset;
+}
+
+function structurallyEqual(a: RootContent, b: RootContent): boolean {
+ return stringifyIgnorePosition(a) === stringifyIgnorePosition(b);
+}
+
+function stringifyIgnorePosition(node: unknown): string {
+ return JSON.stringify(node, (key, value) => (key === 'position' ? undefined : value));
+}
diff --git a/packages/server/src/metrics.test.ts b/packages/server/src/metrics.test.ts
index ac6b809c2..4026bcfe5 100644
--- a/packages/server/src/metrics.test.ts
+++ b/packages/server/src/metrics.test.ts
@@ -8,6 +8,8 @@ import {
incrementBridgeMergeContentLoss,
incrementCollabSocketFilteredError,
incrementConflict,
+ incrementMapDrivenSpliceApplied,
+ incrementMapDrivenSpliceFallback,
incrementPark,
incrementReconcile,
incrementRescueBuffer,
@@ -66,6 +68,25 @@ describe('reconciliation metrics', () => {
expect(m.parkCount).toBe(2);
});
+ test('map-driven splice counters: applied increments and fallback is keyed by reason', () => {
+ resetMetrics();
+ incrementMapDrivenSpliceApplied();
+ incrementMapDrivenSpliceApplied();
+ incrementMapDrivenSpliceFallback('parse-error');
+ incrementMapDrivenSpliceFallback('parse-error');
+ incrementMapDrivenSpliceFallback('text-mismatch');
+ incrementMapDrivenSpliceFallback('synthetic-doc');
+ incrementMapDrivenSpliceFallback('missing-position');
+ const m = getMetrics();
+ expect(m.mapDrivenSpliceApplied).toBe(2);
+ expect(m.mapDrivenSpliceFallback).toEqual({
+ 'parse-error': 2,
+ 'text-mismatch': 1,
+ 'synthetic-doc': 1,
+ 'missing-position': 1,
+ });
+ });
+
test('getMetrics returns a snapshot (not a reference)', () => {
resetMetrics();
incrementReconcile();
@@ -84,7 +105,11 @@ describe('reconciliation metrics', () => {
resetMetrics();
const m = getMetrics();
for (const [key, value] of Object.entries(m)) {
- if (key === 'cc1LastSeq' || key === 'bridgeToleranceApplied') {
+ if (
+ key === 'cc1LastSeq' ||
+ key === 'bridgeToleranceApplied' ||
+ key === 'mapDrivenSpliceFallback'
+ ) {
expect(value).toEqual({});
} else {
expect(value).toBe(0);
diff --git a/packages/server/src/metrics.ts b/packages/server/src/metrics.ts
index 22ecca7e6..200916587 100644
--- a/packages/server/src/metrics.ts
+++ b/packages/server/src/metrics.ts
@@ -1,5 +1,11 @@
import type { BridgeToleranceClass, CC1Channel } from '@inkeep/open-knowledge-core';
+export type MapDrivenSpliceFallbackReason =
+ | 'text-mismatch'
+ | 'synthetic-doc'
+ | 'parse-error'
+ | 'missing-position';
+
export interface ReconciliationMetrics {
reconcileCount: number;
conflictCount: number;
@@ -159,6 +165,25 @@ export interface ReconciliationMetrics {
* Path-B rate via `actual_rate = observerAPathBFires +
* observerAPathBFiresSuppressed`. */
observerAPathBFiresSuppressed: number;
+ /** Count of Observer A drains where the map-driven block-aligned splice
+ * computed and applied — the byte-preserving default path that keeps
+ * untouched blocks byte-identical in Y.Text. The companion
+ * `mapDrivenSpliceFallback` records every drain the splice could NOT
+ * serve, keyed by reason; `splice_rate = applied / (applied + Σfallback)`
+ * is the health signal for the default fidelity path. */
+ mapDrivenSpliceApplied: number;
+ /** Companion to `mapDrivenSpliceApplied` — count of Observer A drains
+ * that fell back off the map-driven splice, keyed by reason:
+ * `text-mismatch` (Y.Text diverged from baseline → Path B; expected
+ * under concurrent edits), `synthetic-doc` (system/config docs never
+ * use the splice; expected), `parse-error` (parse/serialize threw
+ * inside the splice computation — a sustained non-zero here means a
+ * serializer/parser regression is silently routing every edit through
+ * the lossier incremental-diff fallback), and `missing-position` (a
+ * top-level mdast block lacked byte offsets). Keys are a closed
+ * 4-value union, mirroring `bridgeToleranceApplied`'s compile-time
+ * bounded-cardinality guard. */
+ mapDrivenSpliceFallback: Partial<Record<MapDrivenSpliceFallbackReason, number>>;
/** Y.Text-is-truth contract — count of Observer A in-sync canonical-base
* residual merges: the byte-preserving slow path on docs whose settled
* bytes sit beyond `normalizeBridge` tolerance of the canonical witness.
@@ -347,6 +372,8 @@ const counters: ReconciliationMetrics = {
bridgeToleranceApplied: {},
observerAPathBFires: 0,
observerAPathBFiresSuppressed: 0,
+ mapDrivenSpliceApplied: 0,
+ mapDrivenSpliceFallback: {},
observerAResidualMergeRuns: 0,
bridgeSplitBrainRederives: 0,
bridgeSplitBrainRederivesSuppressed: 0,
@@ -479,6 +506,14 @@ export function incrementObserverAPathBFiresSuppressed(): void {
counters.observerAPathBFiresSuppressed++;
}
+export function incrementMapDrivenSpliceApplied(): void {
+ counters.mapDrivenSpliceApplied++;
+}
+
+export function incrementMapDrivenSpliceFallback(reason: MapDrivenSpliceFallbackReason): void {
+ counters.mapDrivenSpliceFallback[reason] = (counters.mapDrivenSpliceFallback[reason] ?? 0) + 1;
+}
+
export function incrementObserverAResidualMergeRuns(): void {
counters.observerAResidualMergeRuns++;
}
@@ -577,6 +612,7 @@ export function getMetrics(): ReconciliationMetrics {
...counters,
cc1LastSeq: { ...counters.cc1LastSeq },
bridgeToleranceApplied: { ...counters.bridgeToleranceApplied },
+ mapDrivenSpliceFallback: { ...counters.mapDrivenSpliceFallback },
};
}
@@ -618,6 +654,8 @@ export function resetMetrics(): void {
counters.bridgeToleranceApplied = {};
counters.observerAPathBFires = 0;
counters.observerAPathBFiresSuppressed = 0;
+ counters.mapDrivenSpliceApplied = 0;
+ counters.mapDrivenSpliceFallback = {};
counters.observerAResidualMergeRuns = 0;
counters.bridgeSplitBrainRederives = 0;
counters.bridgeSplitBrainRederivesSuppressed = 0;
diff --git a/packages/server/src/path-conditional-map-driven-origin.test.ts b/packages/server/src/path-conditional-map-driven-origin.test.ts
new file mode 100644
index 000000000..cdc11c727
--- /dev/null
+++ b/packages/server/src/path-conditional-map-driven-origin.test.ts
@@ -0,0 +1,45 @@
+import { describe, expect, test } from 'bun:test';
+import { spawnSync } from 'node:child_process';
+import { join } from 'node:path';
+
+const REPO_ROOT = join(__dirname, '..', '..', '..');
+const FIXTURE_REL = 'biome-plugins/__fixtures__/path-conditional-map-driven-origin.fixture.tsx';
+
+describe('path-conditional-map-driven-origin GritQL plugin', () => {
+ test('fires on exactly 7 positive cases (and on no negative case)', () => {
+ const result = spawnSync('bunx', ['biome', 'check', FIXTURE_REL], {
+ cwd: REPO_ROOT,
+ encoding: 'utf-8',
+ });
+ expect(result.status).not.toBe(0);
+ const output = `${result.stdout}\n${result.stderr}`;
+ const fires = (output.match(/Observer-side transact call missing sanctioned origin/g) ?? [])
+ .length;
+ expect(fires).toBe(7);
+ expect(output).toContain('Pass `OBSERVER_SYNC_ORIGIN` as the second argument');
+ expect(output).toMatch(/https?:\/\/[^\s]+/);
+ expect(output).toContain('biome-plugins/README.md#path-conditional-map-driven-origingrit');
+ });
+
+ test('plugin is registered in biome.jsonc via overrides (not root plugins)', () => {
+ const config = require(join(REPO_ROOT, 'biome.jsonc')) as {
+ plugins?: string[];
+ overrides?: Array<{ includes?: string[]; plugins?: string[] }>;
+ };
+
+ const rootPlugins = config.plugins ?? [];
+ expect(rootPlugins).not.toContain('./biome-plugins/path-conditional-map-driven-origin.grit');
+
+ const overrides = config.overrides ?? [];
+ const matchingOverride = overrides.find((entry) =>
+ (entry.plugins ?? []).includes('./biome-plugins/path-conditional-map-driven-origin.grit'),
+ );
+ expect(matchingOverride).toBeDefined();
+
+ const includes = matchingOverride?.includes ?? [];
+ expect(includes).toContain('packages/server/src/server-observers.ts');
+ expect(includes).toContain(
+ 'biome-plugins/__fixtures__/path-conditional-map-driven-origin.fixture.tsx',
+ );
+ });
+});
diff --git a/packages/server/src/prd-6654-e2e-repro.test.ts b/packages/server/src/prd-6654-e2e-repro.test.ts
new file mode 100644
index 000000000..417339fee
--- /dev/null
+++ b/packages/server/src/prd-6654-e2e-repro.test.ts
@@ -0,0 +1,215 @@
+import { describe, expect, test } from 'bun:test';
+import { mkdirSync, mkdtempSync, rmSync } from 'node:fs';
+import type { IncomingMessage, ServerResponse } from 'node:http';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { Readable } from 'node:stream';
+import { Hocuspocus } from '@hocuspocus/server';
+import { MarkdownManager, sharedExtensions } from '@inkeep/open-knowledge-core';
+import { getSchema } from '@tiptap/core';
+import * as Y from 'yjs';
+import { AGENT_WRITE_ORIGIN, AgentSessionManager } from './agent-sessions.ts';
+import { createApiExtension } from './api-extension.ts';
+import { composeAndWriteRawBody } from './bridge-intake.ts';
+import { createServerObserverExtension } from './server-observer-extension.ts';
+
+const mdManager = new MarkdownManager({ extensions: sharedExtensions });
+const schema = getSchema(sharedExtensions);
+
+interface CapturedResponse {
+ status: number;
+ body: string;
+}
+
+function makeJsonPostReq(body: unknown): IncomingMessage {
+ const readable = Readable.from(Buffer.from(JSON.stringify(body))) as unknown as IncomingMessage;
+ readable.method = 'POST';
+ readable.url = '/api/agent-patch';
+ readable.headers = {
+ host: 'localhost',
+ 'content-type': 'application/json',
+ };
+ return readable;
+}
+
+function makeRes(): { res: ServerResponse; captured: CapturedResponse } {
+ const captured: CapturedResponse = { status: 0, body: '' };
+ const res = {
+ writeHead(status: number) {
+ captured.status = status;
+ },
+ end(body?: string) {
+ captured.body = body ?? '';
+ },
+ } as unknown as ServerResponse;
+ return { res, captured };
+}
+
+async function callAgentPatch(
+ hocuspocus: Hocuspocus,
+ sessionManager: AgentSessionManager,
+ contentDir: string,
+ body: unknown,
+): Promise<CapturedResponse> {
+ const ext = createApiExtension({
+ hocuspocus,
+ sessionManager,
+ contentDir,
+ serverInstanceId: 'prd-6654-e2e-test-instance',
+ getFileIndex: () => new Map(),
+ });
+ const req = makeJsonPostReq(body);
+ const { res, captured } = makeRes();
+ await (
+ ext as {
+ onRequest: (ctx: { request: IncomingMessage; response: ServerResponse }) => Promise<void>;
+ }
+ ).onRequest({ request: req, response: res });
+ return captured;
+}
+
+function setup() {
+ const projectDir = mkdtempSync(join(tmpdir(), 'ok-prd-6654-e2e-'));
+ const contentDir = join(projectDir, 'content');
+ mkdirSync(contentDir, { recursive: true });
+ const hocuspocus = new Hocuspocus({
+ quiet: true,
+ extensions: [createServerObserverExtension({ mdManager, schema })],
+ });
+ const sessionManager = new AgentSessionManager(hocuspocus);
+ return {
+ projectDir,
+ contentDir,
+ hocuspocus,
+ sessionManager,
+ cleanup: async () => {
+ await sessionManager.closeAll();
+ rmSync(projectDir, { recursive: true, force: true });
+ },
+ };
+}
+
+const USER_TYPING_ORIGIN = {
+ source: 'local' as const,
+ context: { origin: 'user-typing' },
+};
+
+const SEED_ORIGIN = {
+ source: 'local' as const,
+ skipStoreHooks: false,
+ context: { origin: 'test-seed', paired: true as const },
+};
+
+describe('PRD-6654 — full end-to-end gate (handleAgentPatch + WYSIWYG touch)', () => {
+ test('row-without-trailing-pipe shape: the second find succeeds (200, no "Text not found")', async () => {
+ const env = setup();
+ try {
+ const session = await env.sessionManager.getSession('test-doc');
+ const ytext = session.dc.document.getText('source');
+ const xmlFragment = session.dc.document.getXmlFragment('default');
+
+ session.dc.document.transact(() => {
+ ytext.delete(0, ytext.length);
+ ytext.insert(0, '# Notes\n\nplaceholder\n');
+ }, SEED_ORIGIN);
+
+ const r0 = await callAgentPatch(env.hocuspocus, env.sessionManager, env.contentDir, {
+ docName: 'test-doc',
+ find: 'placeholder',
+ replace: '| a | b |\n| - | - |\n| 1 | 2',
+ });
+ expect(r0.status).toBe(200);
+ expect(ytext.toString().includes('| 1 | 2\n')).toBe(true);
+
+ session.dc.document.transact(() => {
+ const para = new Y.XmlElement('paragraph');
+ para.insert(0, [new Y.XmlText('hi')]);
+ xmlFragment.insert(xmlFragment.length, [para]);
+ }, USER_TYPING_ORIGIN);
+
+ const r2 = await callAgentPatch(env.hocuspocus, env.sessionManager, env.contentDir, {
+ docName: 'test-doc',
+ find: '| 1 | 2\n',
+ replace: '| 1 | 2\n| 3 | 4\n',
+ });
+ expect(r2.status).toBe(200);
+ expect(ytext.toString().includes('| 3 | 4')).toBe(true);
+ } finally {
+ await env.cleanup();
+ }
+ });
+
+ test('leading-blank-line shape: the second find succeeds (200)', async () => {
+ const env = setup();
+ try {
+ const session = await env.sessionManager.getSession('test-doc');
+ const ytext = session.dc.document.getText('source');
+ const xmlFragment = session.dc.document.getXmlFragment('default');
+
+ session.dc.document.transact(() => {
+ composeAndWriteRawBody(session.dc.document, '\n\nhello\n', 'agent');
+ }, AGENT_WRITE_ORIGIN);
+
+ session.dc.document.transact(() => {
+ const para = new Y.XmlElement('paragraph');
+ para.insert(0, [new Y.XmlText('z')]);
+ xmlFragment.insert(xmlFragment.length, [para]);
+ }, USER_TYPING_ORIGIN);
+
+ const r2 = await callAgentPatch(env.hocuspocus, env.sessionManager, env.contentDir, {
+ docName: 'test-doc',
+ find: '\n\nhello',
+ replace: '\n\nGOODBYE',
+ });
+ expect(r2.status).toBe(200);
+ expect(ytext.toString().includes('GOODBYE')).toBe(true);
+ } finally {
+ await env.cleanup();
+ }
+ });
+
+ test('reporter scenario: 12 rapid sequential edits with interleaved WYSIWYG noise all succeed', async () => {
+ const env = setup();
+ try {
+ const session = await env.sessionManager.getSession('test-doc');
+ const ytext = session.dc.document.getText('source');
+ const xmlFragment = session.dc.document.getXmlFragment('default');
+
+ const seed = '# Trace\n\n| step | note |\n| ---- | ---- |\n| 0 | start |\n';
+ session.dc.document.transact(() => {
+ ytext.delete(0, ytext.length);
+ ytext.insert(0, seed);
+ }, SEED_ORIGIN);
+
+ let prevRow = '| 0 | start |';
+ let successes = 0;
+ const failures: string[] = [];
+ for (let i = 1; i <= 12; i++) {
+ const newRow = `| ${i} | iter${i} |`;
+ const r = await callAgentPatch(env.hocuspocus, env.sessionManager, env.contentDir, {
+ docName: 'test-doc',
+ find: prevRow,
+ replace: `${prevRow}\n${newRow}`,
+ });
+ if (r.status === 200) {
+ successes++;
+ prevRow = newRow;
+ } else {
+ failures.push(`iter ${i}: status=${r.status} body=${r.body.slice(0, 120)}`);
+ }
+ if (i % 2 === 0) {
+ session.dc.document.transact(() => {
+ const para = new Y.XmlElement('paragraph');
+ para.insert(0, [new Y.XmlText(`n${i}`)]);
+ xmlFragment.insert(xmlFragment.length, [para]);
+ }, USER_TYPING_ORIGIN);
+ }
+ }
+
+ expect(failures).toEqual([]);
+ expect(successes).toBe(12);
+ } finally {
+ await env.cleanup();
+ }
+ });
+});
diff --git a/packages/server/src/prd-6654-multi-client-repro.test.ts b/packages/server/src/prd-6654-multi-client-repro.test.ts
new file mode 100644
index 000000000..844b379f9
--- /dev/null
+++ b/packages/server/src/prd-6654-multi-client-repro.test.ts
@@ -0,0 +1,220 @@
+import { describe, expect, test } from 'bun:test';
+import { MarkdownManager, sharedExtensions } from '@inkeep/open-knowledge-core';
+import { getSchema } from '@tiptap/core';
+import { updateYFragment } from '@tiptap/y-tiptap';
+import * as Y from 'yjs';
+import { AGENT_WRITE_ORIGIN } from './agent-sessions.ts';
+import { composeAndWriteRawBody } from './bridge-intake.ts';
+import { setupServerObservers } from './server-observers.ts';
+
+const mdManager = new MarkdownManager({ extensions: sharedExtensions });
+const schema = getSchema(sharedExtensions);
+
+const USER_TYPING_ORIGIN = {
+ source: 'connection' as const,
+ context: { origin: 'user-typing' },
+};
+
+interface AgentThenWysiwygResult {
+ readonly afterAgent: string;
+ readonly afterWysiwyg: string;
+}
+
+function agentWriteThenWysiwygTouch(raw: string): AgentThenWysiwygResult {
+ const doc = new Y.Doc();
+ const xmlFragment = doc.getXmlFragment('default');
+ const ytext = doc.getText('source');
+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });
+
+ doc.transact(() => {
+ composeAndWriteRawBody(doc, raw, 'agent');
+ }, AGENT_WRITE_ORIGIN);
+ const afterAgent = ytext.toString();
+
+ doc.transact(() => {
+ const para = new Y.XmlElement('paragraph');
+ para.insert(0, [new Y.XmlText('x')]);
+ xmlFragment.insert(xmlFragment.length, [para]);
+ }, USER_TYPING_ORIGIN);
+ const afterWysiwyg = ytext.toString();
+
+ cleanup();
+ return { afterAgent, afterWysiwyg };
+}
+
+describe('PRD-6654 — agent bytes survive a concurrent non-paired WYSIWYG edit', () => {
+ describe('canonical-form controls (serializer retention; pass on either Path-A variant)', () => {
+ test('under-padded table row in a padded table round-trips byte-exact through serialize(parse)', () => {
+ const rawTable = [
+ '# Trace',
+ '',
+ '| Step | Time | Note | Status |',
+ '| ---- | ---- | ----------------------------- | --------- |',
+ '| 5 | T+5s | Your browser edits survived! | CONFIRMED |',
+ '',
+ ].join('\n');
+ expect(mdManager.serialize(mdManager.parse(rawTable))).toBe(rawTable);
+ });
+
+ test('narrow row appended to a wide-padded table round-trips byte-exact through serialize(parse)', () => {
+ const composed =
+ '| step | description | status |\n' +
+ '| ---- | ------------------------------------------ | ------ |\n' +
+ '| 1 | This is a really really long row of stuff | DONE |\n' +
+ '| 3 | short row | OK |\n';
+ expect(mdManager.serialize(mdManager.parse(composed))).toBe(composed);
+ });
+
+ test('agent table row with under-padded columns stays verbatim after a WYSIWYG touch', () => {
+ const agentRow = '| 5 | T+5s | Your browser edits survived! | CONFIRMED |';
+ const rawTable = [
+ '# Trace',
+ '',
+ '| Step | Time | Note | Status |',
+ '| ---- | ---- | ----------------------------- | --------- |',
+ agentRow,
+ '',
+ ].join('\n');
+
+ const { afterAgent, afterWysiwyg } = agentWriteThenWysiwygTouch(rawTable);
+ expect(afterAgent).toBe(rawTable);
+ expect(afterWysiwyg.includes(agentRow)).toBe(true);
+ });
+
+ test('end-to-end find flow: indexOf of the agent-composed row succeeds after a WYSIWYG touch', () => {
+ const findTarget = '| 5 | T+5s | Your browser edits survived! | CONFIRMED |';
+ const initial = [
+ '# Trace',
+ '',
+ '| Step | Time | Note | Status |',
+ '| ---- | ---- | ----------------------------- | --------- |',
+ findTarget,
+ '',
+ ].join('\n');
+
+ const { afterWysiwyg } = agentWriteThenWysiwygTouch(initial);
+ expect(afterWysiwyg.indexOf(findTarget)).not.toBe(-1);
+ });
+ });
+
+ describe('splice discriminators (require the map-driven Path A; fail under whole-body line diff)', () => {
+ test('table row WITHOUT trailing pipe stays verbatim when the WYSIWYG edit is in another block', () => {
+ const raw = '# Notes\n\n| a | b |\n| - | - |\n| 1 | 2\n';
+ const { afterAgent, afterWysiwyg } = agentWriteThenWysiwygTouch(raw);
+
+ expect(afterAgent).toBe(raw);
+ expect(mdManager.serialize(mdManager.parse(raw))).not.toBe(raw);
+ expect(afterWysiwyg.indexOf('| 1 | 2\n')).not.toBe(-1);
+ });
+
+ test('multi-blank-line block separator stays verbatim when the WYSIWYG edit is in another block', () => {
+ const raw = '# H\n\na\n\n\n\nb\n';
+ const { afterAgent, afterWysiwyg } = agentWriteThenWysiwygTouch(raw);
+
+ expect(afterAgent).toBe(raw);
+ expect(afterWysiwyg.indexOf('a\n\n\n\nb')).not.toBe(-1);
+ });
+
+ test('leading blank lines stay verbatim when the WYSIWYG edit is in another block', () => {
+ const raw = '\n\n## Section\n\nLorem ipsum.\n';
+ const { afterAgent, afterWysiwyg } = agentWriteThenWysiwygTouch(raw);
+
+ expect(afterAgent).toBe(raw);
+ expect(afterWysiwyg.startsWith('\n\n## Section')).toBe(true);
+ });
+ });
+
+ describe('row-no-trailing-pipe tolerance (Observer A already-in-sync gate + watchdog)', () => {
+ const UNIFORM = '# Notes\n\n| a | b\n| - | -\n| 1 | 2\n';
+
+ function stripTableCaptureAttrs(json: unknown): void {
+ if (!json || typeof json !== 'object') return;
+ const node = json as {
+ type?: string;
+ attrs?: Record<string, unknown>;
+ content?: unknown[];
+ };
+ if (node.type === 'table' && node.attrs) {
+ node.attrs.sourceOuterPipes = null;
+ }
+ for (const child of node.content ?? []) stripTableCaptureAttrs(child);
+ }
+
+ function rebuildFragmentFrom(
+ doc: Y.Doc,
+ xmlFragment: Y.XmlFragment,
+ json: Record<string, unknown>,
+ ): void {
+ const pmNode = schema.nodeFromJSON(json);
+ doc.transact(() => {
+ const meta = { mapping: new Map(), isOMark: new Map() };
+ updateYFragment(doc, xmlFragment, pmNode, meta);
+ }, USER_TYPING_ORIGIN);
+ }
+
+ test('attr-only fragment churn (capture attrs lost, zero content change) leaves Y.Text byte-identical', () => {
+ const doc = new Y.Doc();
+ const xmlFragment = doc.getXmlFragment('default');
+ const ytext = doc.getText('source');
+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });
+
+ doc.transact(() => {
+ composeAndWriteRawBody(doc, UNIFORM, 'agent');
+ }, AGENT_WRITE_ORIGIN);
+ expect(ytext.toString()).toBe(UNIFORM);
+
+ const stripped = mdManager.parse(UNIFORM) as unknown as Record<string, unknown>;
+ stripTableCaptureAttrs(stripped);
+ rebuildFragmentFrom(doc, xmlFragment, stripped);
+
+ expect(ytext.toString()).toBe(UNIFORM);
+ expect(ytext.toString().indexOf('| 1 | 2\n')).not.toBe(-1);
+
+ cleanup();
+ });
+
+ test('source-mode typed no-trailing-pipe table settles without a bridge-invariant violation', () => {
+ const doc = new Y.Doc();
+ const xmlFragment = doc.getXmlFragment('default');
+ const ytext = doc.getText('source');
+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });
+
+ const raw = '# Notes\n\n| a | b |\n| - | - |\n| 1 | 2\n';
+ doc.transact(
+ () => {
+ ytext.insert(0, raw);
+ },
+ {
+ source: 'connection' as const,
+ context: { origin: 'source-typing' },
+ },
+ );
+
+ expect(ytext.toString()).toBe(raw);
+
+ cleanup();
+ });
+
+ test('genuine cell edit through the same attr-churn path still syncs (not absorbed by the tolerance)', () => {
+ const doc = new Y.Doc();
+ const xmlFragment = doc.getXmlFragment('default');
+ const ytext = doc.getText('source');
+ const cleanup = setupServerObservers({ doc, xmlFragment, ytext, mdManager, schema });
+
+ doc.transact(() => {
+ composeAndWriteRawBody(doc, UNIFORM, 'agent');
+ }, AGENT_WRITE_ORIGIN);
+
+ const edited = mdManager.parse(UNIFORM.replace('| 1 | 2', '| 1 | 99')) as unknown as Record<
+ string,
+ unknown
+ >;
+ stripTableCaptureAttrs(edited);
+ rebuildFragmentFrom(doc, xmlFragment, edited);
+
+ expect(ytext.toString()).toContain('99');
+
+ cleanup();
+ });
+ });
+});
diff --git a/packages/server/src/server-observers.test.ts b/packages/server/src/server-observers.test.ts
index f206b1447..8dabe5377 100644
--- a/packages/server/src/server-observers.test.ts
+++ b/packages/server/src/server-observers.test.ts
@@ -890,7 +890,7 @@ describe('Server Observer B — error recovery paths', () => {
resetMetrics();
const ngRaw =
- '---\ntitle: NG recovery fixture\n---\n\n# Hello\n\nInline math: $a + b$ stays.\n\nBody text stays.\n';
+ '---\ntitle: NG recovery fixture\n---\n\n# Hello\n\n> Lazy quote\nstays lazy.\n\nBody text stays.\n';
const doc = new Y.Doc();
const xmlFragment = doc.getXmlFragment('default');
const ytext = doc.getText('source');
@@ -1039,7 +1039,7 @@ describe('Server Observer B — Y.Text-is-truth contract (FR-31)', () => {
});
describe('Observer A routing — Path B fires iff Y.Text holds unabsorbed changes (FR-3)', () => {
- const RESIDUAL_RAW = '---\ntitle: Routing fixture\n---\n\n# Hello\n\nBody text stays.\n';
+ const RESIDUAL_RAW = '---\ntitle: Routing fixture\n---\n\n# Hello \n\nBody text stays.\n';
function canonicalOf(raw: string): string {
const { frontmatter, body } = stripFrontmatter(raw);
@@ -1283,7 +1283,7 @@ describe('Observer A routing — Path B fires iff Y.Text holds unabsorbed change
});
const NG_RAW =
- '---\ntitle: NG routing fixture\n---\n\n# Hello\n\nInline math: $a + b$ stays.\n\nBody text stays.\n';
+ '---\ntitle: NG routing fixture\n---\n\n# Hello\n\n> Lazy quote\nstays lazy.\n\nBody text stays.\n';
test('in-sync doc with beyond-tolerance residual: fragment change preserves NG bytes without a Path B fire', () => {
__resetBridgeWatchdogForTests();
@@ -1310,8 +1310,8 @@ describe('Observer A routing — Path B fires iff Y.Text holds unabsorbed change
const finalText = ytext.toString();
expect(finalText).toContain('User WYSIWYG edit.');
- expect(finalText).toContain('Inline math: $a + b$ stays.');
- expect(finalText).not.toContain('$$a + b$$');
+ expect(finalText).toContain('> Lazy quote\nstays lazy.');
+ expect(finalText).not.toContain('> stays lazy.');
expect(finalText).toContain('Body text stays.');
cleanup();
@@ -1364,8 +1364,8 @@ describe('Observer A routing — Path B fires iff Y.Text holds unabsorbed change
});
expect(events1).toHaveLength(0);
expect(getMetrics().observerAResidualMergeRuns).toBe(1);
- expect(ytext.toString()).toContain('Inline math: $a + b$ stays.');
- expect(ytext.toString()).not.toContain('$$a + b$$');
+ expect(ytext.toString()).toContain('> Lazy quote\nstays lazy.');
+ expect(ytext.toString()).not.toContain('> stays lazy.');
const events2 = capturePathBEvents(() => {
populateFragment(doc, xmlFragment, `${serializeFragmentBody(xmlFragment)}\nSecond edit.\n`);
@@ -1377,8 +1377,8 @@ describe('Observer A routing — Path B fires iff Y.Text holds unabsorbed change
const finalText = ytext.toString();
expect(finalText).toContain('First edit.');
expect(finalText).toContain('Second edit.');
- expect(finalText).toContain('Inline math: $a + b$ stays.');
- expect(finalText).not.toContain('$$a + b$$');
+ expect(finalText).toContain('> Lazy quote\nstays lazy.');
+ expect(finalText).not.toContain('> stays lazy.');
cleanup();
});
@@ -1394,7 +1394,7 @@ describe('Observer A routing — Path B fires iff Y.Text holds unabsorbed change
expect(ytext.toString()).toBe(NG_RAW);
const pairedRaw =
- '---\ntitle: NG routing fixture\n---\n\n# Hello\n\nInline math: $a + b$ stays.\n\nPaired body.\n';
+ '---\ntitle: NG routing fixture\n---\n\n# Hello\n\n> Lazy quote\nstays lazy.\n\nPaired body.\n';
expect(normalizeBridge(canonicalOf(pairedRaw))).not.toBe(normalizeBridge(pairedRaw));
const pairedJson = mdManager.parse(stripFrontmatter(pairedRaw).body);
const pairedNode = schema.nodeFromJSON(pairedJson);
diff --git a/packages/server/src/server-observers.ts b/packages/server/src/server-observers.ts
index 623981441..043076f7a 100644
--- a/packages/server/src/server-observers.ts
+++ b/packages/server/src/server-observers.ts
@@ -20,11 +20,15 @@ import {
emitBridgeSplitBrainRederive,
emitObserverAPathBFired,
} from './bridge-watchdog.ts';
+import { isConfigDoc, isSystemDoc } from './cc1-broadcast.ts';
import { recordFrontmatterEditSurface } from './frontmatter-telemetry.ts';
+import { computeMapDrivenBodySplice } from './map-driven-splice.ts';
import {
incrementBridgeMergeCheckpointCreated,
incrementBridgeMergeContentLoss,
incrementBridgeSplitBrainRederives,
+ incrementMapDrivenSpliceApplied,
+ incrementMapDrivenSpliceFallback,
incrementObserverAPathBFires,
incrementObserverAResidualMergeRuns,
incrementServerObserverError,
@@ -56,6 +60,56 @@ export function shouldRethrowBridgeMergeLoss(env: NodeJS.ProcessEnv = process.en
return env.NODE_ENV === 'test' || env.OK_RETHROW_BRIDGE_LOSS === '1';
}
+interface YTextMapDrivenSplice {
+ readonly spliceStart: number;
+ readonly spliceEnd: number;
+ readonly newSlice: string;
+}
+
+interface TryComputeMapDrivenSpliceArgs {
+ readonly currentText: string;
+ readonly lastSyncedXmlMd: string;
+ readonly json: unknown;
+ readonly mdManager: MarkdownManager;
+ readonly docName: string | undefined;
+}
+
+function tryComputeMapDrivenSplice(
+ args: TryComputeMapDrivenSpliceArgs,
+): YTextMapDrivenSplice | null {
+ const { currentText, lastSyncedXmlMd, json, mdManager, docName } = args;
+ if (currentText !== lastSyncedXmlMd) {
+ incrementMapDrivenSpliceFallback('text-mismatch');
+ return null;
+ }
+ if (docName !== undefined && (isSystemDoc(docName) || isConfigDoc(docName))) {
+ incrementMapDrivenSpliceFallback('synthetic-doc');
+ return null;
+ }
+
+ const { body: oldBody } = stripFrontmatter(currentText);
+ const bodyOffset = currentText.length - oldBody.length;
+ const splice = computeMapDrivenBodySplice(
+ oldBody,
+ json as Parameters<typeof computeMapDrivenBodySplice>[1],
+ mdManager,
+ incrementMapDrivenSpliceFallback,
+ );
+ if (!splice) return null;
+
+ return {
+ spliceStart: bodyOffset + splice.spliceStart,
+ spliceEnd: bodyOffset + splice.spliceEnd,
+ newSlice: splice.newSlice,
+ };
+}
+
+function applyMapDrivenSplice(ytext: Y.Text, splice: YTextMapDrivenSplice): void {
+ const deleteLength = splice.spliceEnd - splice.spliceStart;
+ if (deleteLength > 0) ytext.delete(splice.spliceStart, deleteLength);
+ if (splice.newSlice.length > 0) ytext.insert(splice.spliceStart, splice.newSlice);
+}
+
type ShadowAccessor = () => ShadowHandle | undefined;
type BranchAccessor = () => string;
@@ -206,7 +260,8 @@ export function setupServerObservers(opts: SetupServerObserversOpts): () => void
const ytextCanonicalBody = mdManager.serialize(
mdManager.parseWithFallback(stripFrontmatter(ytext.toString()).body, initParseOpts),
);
- if (ytextCanonicalBody === initialBody) {
+ const stripDocBoundary = (b: string): string => b.replace(/^\n+/, '').replace(/\n+$/, '');
+ if (stripDocBoundary(ytextCanonicalBody) === stripDocBoundary(initialBody)) {
recordSettledBaselines(canonicalInit);
} else {
recordDivergedAttachBaselines(canonicalInit);
@@ -261,8 +316,25 @@ export function setupServerObservers(opts: SetupServerObserversOpts): () => void
: 'path-b',
});
const pathBState: { mergedText: string | null } = { mergedText: null };
+
+ const mapDrivenSplice =
+ ytextInSync && residualMergeEligible
+ ? null
+ : tryComputeMapDrivenSplice({
+ currentText,
+ lastSyncedXmlMd: lastSyncedYTextBytes,
+ json,
+ mdManager,
+ docName: opts.docName,
+ });
+ if (mapDrivenSplice) {
+ setActiveSpanAttributes({ 'observer.a.path': 'map-driven-splice' });
+ }
+
doc.transact(() => {
- if (ytextInSync && !residualMergeEligible) {
+ if (mapDrivenSplice) {
+ applyMapDrivenSplice(ytext, mapDrivenSplice);
+ } else if (ytextInSync && !residualMergeEligible) {
applyIncrementalDiff(ytext, currentText, md);
} else {
const mergeBase = ytextInSync ? lastSyncedCanonicalMd : preMergeBaseline;
@@ -280,6 +352,8 @@ export function setupServerObservers(opts: SetupServerObserversOpts): () => void
}
}, OBSERVER_SYNC_ORIGIN);
+ if (mapDrivenSplice) incrementMapDrivenSpliceApplied();
+
if (pathBState.mergedText !== null && !ytextInSync) {
if (emitObserverAPathBFired(opts.docName)) {
incrementObserverAPathBFires();
diff --git a/packages/server/src/tolerance-telemetry-writer.test.ts b/packages/server/src/tolerance-telemetry-writer.test.ts
new file mode 100644
index 000000000..0c878ffa1
--- /dev/null
+++ b/packages/server/src/tolerance-telemetry-writer.test.ts
@@ -0,0 +1,129 @@
+import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
+import { randomUUID } from 'node:crypto';
+import { existsSync, mkdirSync, readFileSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { resolve } from 'node:path';
+
+import { emitToleranceFire } from '@inkeep/open-knowledge-core';
+
+import {
+ initToleranceTelemetryWriter,
+ isToleranceTelemetryEnabled,
+ teardownToleranceTelemetryWriter,
+} from './tolerance-telemetry-writer.ts';
+
+let tmpProjectDir: string;
+
+beforeEach(() => {
+ tmpProjectDir = resolve(tmpdir(), `ok-telemetry-test-${randomUUID()}`);
+ mkdirSync(tmpProjectDir, { recursive: true });
+ delete process.env.OK_BRIDGE_TOLERANCE_TELEMETRY;
+});
+
+afterEach(() => {
+ teardownToleranceTelemetryWriter();
+ try {
+ rmSync(tmpProjectDir, { recursive: true, force: true });
+ } catch {}
+});
+
+describe('isToleranceTelemetryEnabled', () => {
+ test('returns false by default', () => {
+ expect(isToleranceTelemetryEnabled({})).toBe(false);
+ });
+
+ test('returns true when OK_BRIDGE_TOLERANCE_TELEMETRY=1', () => {
+ expect(isToleranceTelemetryEnabled({ OK_BRIDGE_TOLERANCE_TELEMETRY: '1' })).toBe(true);
+ });
+
+ test('returns false for other values', () => {
+ expect(isToleranceTelemetryEnabled({ OK_BRIDGE_TOLERANCE_TELEMETRY: '0' })).toBe(false);
+ expect(isToleranceTelemetryEnabled({ OK_BRIDGE_TOLERANCE_TELEMETRY: 'true' })).toBe(false);
+ });
+});
+
+describe('initToleranceTelemetryWriter', () => {
+ test('does nothing when flag is OFF', () => {
+ process.env.OK_BRIDGE_TOLERANCE_TELEMETRY = '0';
+ initToleranceTelemetryWriter(tmpProjectDir);
+
+ emitToleranceFire(['crlf'], 'a\r\n', 'a\n', 'test-doc');
+
+ const logPath = resolve(tmpProjectDir, '.ok', 'local', 'tolerance-telemetry.jsonl');
+ expect(existsSync(logPath)).toBe(false);
+ });
+
+ test('writes JSONL when flag is ON', () => {
+ process.env.OK_BRIDGE_TOLERANCE_TELEMETRY = '1';
+ initToleranceTelemetryWriter(tmpProjectDir);
+
+ emitToleranceFire(['crlf'], 'a\r\n', 'a\n', 'notes/meeting');
+
+ const logPath = resolve(tmpProjectDir, '.ok', 'local', 'tolerance-telemetry.jsonl');
+ expect(existsSync(logPath)).toBe(true);
+
+ const content = readFileSync(logPath, 'utf-8');
+ const lines = content.trim().split('\n');
+ expect(lines).toHaveLength(1);
+
+ const record = JSON.parse(lines[0] ?? '');
+ expect(record.event).toBe('bridge-tolerance-fire');
+ expect(record.class).toBe('crlf');
+ expect(record.document).toBe('notes/meeting');
+ expect(record.codeUnitPosition).toBe(1);
+ expect(record.severity).toBe('parser-caused');
+ expect(record.timestamp).toMatch(/^\d{4}-\d{2}-\d{2}T/);
+ });
+
+ test('appends multiple fires to same file', () => {
+ process.env.OK_BRIDGE_TOLERANCE_TELEMETRY = '1';
+ initToleranceTelemetryWriter(tmpProjectDir);
+
+ emitToleranceFire(['crlf'], 'a\r\n', 'a\n', 'doc-1');
+ emitToleranceFire(['bom'], 'hello', 'hello', 'doc-2');
+
+ const logPath = resolve(tmpProjectDir, '.ok', 'local', 'tolerance-telemetry.jsonl');
+ const lines = readFileSync(logPath, 'utf-8').trim().split('\n');
+ expect(lines).toHaveLength(2);
+ expect(JSON.parse(lines[0] ?? '').class).toBe('crlf');
+ expect(JSON.parse(lines[1] ?? '').class).toBe('bom');
+ });
+
+ test('multi-class fire produces one line per class', () => {
+ process.env.OK_BRIDGE_TOLERANCE_TELEMETRY = '1';
+ initToleranceTelemetryWriter(tmpProjectDir);
+
+ emitToleranceFire(['bom', 'crlf', 'trailing-whitespace'], 'a \r\n', 'a\n', 'doc');
+
+ const logPath = resolve(tmpProjectDir, '.ok', 'local', 'tolerance-telemetry.jsonl');
+ const lines = readFileSync(logPath, 'utf-8').trim().split('\n');
+ expect(lines).toHaveLength(3);
+
+ const classes = lines.map((l) => JSON.parse(l).class);
+ expect(classes).toEqual(['bom', 'crlf', 'trailing-whitespace']);
+ });
+
+ test('null document name serializes as null', () => {
+ process.env.OK_BRIDGE_TOLERANCE_TELEMETRY = '1';
+ initToleranceTelemetryWriter(tmpProjectDir);
+
+ emitToleranceFire(['trailing-newline'], 'hello\n', 'hello', undefined);
+
+ const logPath = resolve(tmpProjectDir, '.ok', 'local', 'tolerance-telemetry.jsonl');
+ const record = JSON.parse(readFileSync(logPath, 'utf-8').trim());
+ expect(record.document).toBeNull();
+ });
+
+ test('teardown stops further writes', () => {
+ process.env.OK_BRIDGE_TOLERANCE_TELEMETRY = '1';
+ initToleranceTelemetryWriter(tmpProjectDir);
+
+ emitToleranceFire(['crlf'], 'a\r', 'a', 'doc');
+ teardownToleranceTelemetryWriter();
+ emitToleranceFire(['crlf'], 'b\r', 'b', 'doc');
+
+ const logPath = resolve(tmpProjectDir, '.ok', 'local', 'tolerance-telemetry.jsonl');
+ const lines = readFileSync(logPath, 'utf-8').trim().split('\n');
+ expect(lines).toHaveLength(1);
+ });
+});
diff --git a/packages/server/src/tolerance-telemetry-writer.ts b/packages/server/src/tolerance-telemetry-writer.ts
new file mode 100644
index 000000000..69f137759
--- /dev/null
+++ b/packages/server/src/tolerance-telemetry-writer.ts
@@ -0,0 +1,40 @@
+import { appendFileSync, mkdirSync } from 'node:fs';
+import { resolve } from 'node:path';
+
+import { setToleranceTelemetryHook, type ToleranceFireRecord } from '@inkeep/open-knowledge-core';
+
+import { getLocalDir } from './config/paths.ts';
+
+const TELEMETRY_FILENAME = 'tolerance-telemetry.jsonl';
+
+export function isToleranceTelemetryEnabled(
+ env: Record<string, string | undefined> = process.env,
+): boolean {
+ return env.OK_BRIDGE_TOLERANCE_TELEMETRY === '1';
+}
+
+export function initToleranceTelemetryWriter(projectDir: string): void {
+ if (!isToleranceTelemetryEnabled()) return;
+
+ const localDir = getLocalDir(projectDir);
+ mkdirSync(localDir, { recursive: true });
+ const logPath = resolve(localDir, TELEMETRY_FILENAME);
+
+ setToleranceTelemetryHook((record: ToleranceFireRecord) => {
+ const line = JSON.stringify({
+ event: 'bridge-tolerance-fire',
+ timestamp: record.timestamp,
+ class: record.className,
+ document: record.documentName ?? null,
+ codeUnitPosition: record.codeUnitPosition,
+ severity: record.severity,
+ });
+ try {
+ appendFileSync(logPath, `${line}\n`);
+ } catch {}
+ });
+}
+
+export function teardownToleranceTelemetryWriter(): void {
+ setToleranceTelemetryHook(null);
+}