Beautiful, AI-native markdown IDE and LLM wiki
TypeScript 2nd-brainagent-skillsclaudecodexcompany-braindocsknowledge-knowledge-baseknowledge-managementllmllm-wikillm-wiki-karpathymarkdownmarkdown-editormdmdxnotespkmsecond-brainskills
Guard the ymark CRDT mark-encoding against silent re-port drift (#2986)
Nick Gomez728bc6efd51b
diff --git a/packages/core/src/y-prosemirror-patch.test.ts b/packages/core/src/y-prosemirror-patch.test.tsindex da4e60b76..a5aecf1a1 100644--- a/packages/core/src/y-prosemirror-patch.test.ts+++ b/packages/core/src/y-prosemirror-patch.test.ts@@ -5,7 +5,7 @@** ## Why this test exists*- * The R13 patch is applied via `bun patch` at install time in two places+ * The R13 patch is applied via pnpm's `patchedDependencies` at install time in two places* because two different `@tiptap/*` extensions import different packages:* (a) `patches/[email protected]` — `@tiptap/extension-collaboration-cursor`* imports `yCursorPlugin` from this package.@@ -19,16 +19,20 @@** If either patch silently fails to apply (e.g., upstream drift, corrupted* lockfile, missing patchedDependencies entry), the destructive- * `el._item.delete(transaction)` path returns, which is catastrophic —+ * `(el._item).delete(transaction)` path returns, which is catastrophic —* schema-throw silently destroys peer data across the CRDT.** This test reads each installed bundle and asserts:* 1. The patch marker comment `R13 patch:` is present at both throw sites- * 2. The destructive `_item.delete(transaction)` call is absent from those sites+ * 2. The destructive `(el._item).delete(transaction)` call is absent from+ * those sites* 3. The `rawMdxFallback` substitution + `globalThis.__okYpsCounters` increments are present- * 4. The `patchedDependencies` entries are registered in package.json+ * 4. The `ymark:` mark-encoding helpers (`YMARK_PREFIX` / `splitYAttrs` /+ * `equalYMarks`) survive, so ProseMirror marks on inline element/leaf+ * nodes still cross the Y bridge+ * 5. The `patchedDependencies` entries are registered in pnpm-workspace.yaml*- * If this test fails on a clean `bun install`, the fix is to investigate+ * If this test fails on a clean `pnpm install`, the fix is to investigate* the patch file (it may need re-porting to a new package version).** End-to-end verification of the patch actually firing on a live Y.Doc@@ -46,25 +50,39 @@* 1. **Diff upstream** — compare the old patched bundle against the new* version. Focus on the two `catch (e) {` blocks inside* `createNodeFromYElement` and `createTextNodesFromYText`. If upstream- * moved or replaced the destructive `_item.delete(transaction)` call,+ * moved or replaced the destructive `(el._item).delete(transaction)` call,* re-port to the new call sites. Patch invariants to preserve:- * - NO `_item.delete(transaction)` anywhere in the bundle+ * - NO `(el._item).delete(transaction)` anywhere in the bundle* - `rawMdxFallback` substitution in block-context `schema.node()` catch* - `globalThis.__okYpsCounters.{block,inline}++` at every catch site* - Structured `console.warn('[y-prosemirror] ...')` retained (the log* prefix is a stable identifier tests/ops filter on — keep it even* when the host package is `@tiptap/y-tiptap`)+ * - The `ymark:`-prefixed reserved-attribute encoding (`YMARK_PREFIX` /+ * `splitYAttrs` / `equalYMarks`) must be re-ported alongside the+ * delete-fix — it is what keeps ProseMirror marks on inline+ * element/leaf nodes alive across the Y bridge. `equalYMarks` depends+ * on upstream `yattr2markname` to reverse hash-suffixed attribute+ * keys back to mark names; do NOT replace it with direct key+ * comparison or hash-suffixed mark lookup breaks.+ * - EVERY artifact the `exports` map can serve must be re-ported, not+ * just the default one. `y-prosemirror` serves `dist/y-prosemirror.cjs`+ * to `require` and `src/plugins/sync-plugin.js` (via+ * `src/y-prosemirror.js`) to `import`; the browser build takes the+ * latter, so a `dist`-only re-port ships an unpatched editor.*- * 2. **Regenerate via `bun patch`**:- * `bun patch <pkg>@N.N.N`- * edit both `dist/*.cjs` AND `dist/*.js` if the package ships both,- * then `bun patch --commit node_modules/<pkg>`. Bun writes the patch- * file under `patches/` and updates `package.json`.+ * 2. **Regenerate via `pnpm patch`**:+ * `pnpm patch <pkg>@N.N.N`+ * edit EVERY file the package's `PATCHED_BUNDLES` entries below point at+ * (that array is the authoritative list — do not infer the layout, since+ * the ESM artifact is not always under `dist/`), then+ * `pnpm patch-commit <dir>`. pnpm writes the patch file under `patches/`+ * and updates `patchedDependencies` in `pnpm-workspace.yaml`.** 3. **Update the `PATCHED_BUNDLES` array below** to reflect new paths if* the bundle layout changed.*- * 4. **Run the full gate**: `bun run check` PLUS the live-fire regression+ * 4. **Run the full gate**: `pnpm check` PLUS the live-fire regression* at `packages/app/tests/integration/y-tiptap-schema-throw-substitution.test.ts`.** If upstream ever adds a non-destructive hook (e.g., `onSchemaError`@@ -72,8 +90,9 @@* at https://github.com/yjs/y-prosemirror and https://github.com/ueberdosis/y-tiptap.*/+import type { Dirent } from 'node:fs';import { readdirSync, readFileSync, realpathSync, statSync } from 'node:fs';-import { dirname, join } from 'node:path';+import { basename, dirname, join } from 'node:path';import { fileURLToPath } from 'node:url';import { describe, expect, test } from 'vitest';@@ -84,6 +103,59 @@ function findRepoRoot(): string {const REPO_ROOT = findRepoRoot();+/**+ * The upstream destructive-delete call the R13 patch removes.+ *+ * Upstream does not write it as a bare `_item.delete(transaction)` — it casts+ * first, so the emitted source is `(el._item).delete(transaction)` with the+ * cast's closing paren sitting between `_item` and `.delete`. A matcher that+ * looks for the bare substring finds nothing in a fully stock bundle, which+ * makes every assertion built on it pass against unpatched code. The optional+ * `\)?` is what gives this matcher its discriminating power; the canary test+ * below pins that behavior so the vacuous form cannot come back.+ */+const DESTRUCTIVE_DELETE = /_item\s*\)?\s*\.delete\(transaction\)/;++/**+ * Directories under a package that can hold shipped code. `dist` alone is not+ * enough: `y-prosemirror` serves its `import` condition straight out of `src/`,+ * so a `dist`-only sweep misses the copy the browser build actually loads.+ */+const ARTIFACT_ROOTS = ['dist', 'src', 'lib', 'build', 'esm', 'cjs'] as const;++/** Depth cap so a pathological symlink cycle cannot hang the sweep. */+const MAX_ARTIFACT_DEPTH = 6;++/**+ * Every `node_modules` directory the dep-tree walk must start from.+ *+ * pnpm installs with an isolated layout: the repo-root `node_modules/` holds+ * only the workspace ROOT package's own dependencies, and each workspace+ * package's dependencies live in the dot-prefixed `.pnpm/` virtual store+ * (surfaced to packages via symlinks). A walk that starts at the repo root and+ * skips dot-directories therefore never reaches a single workspace-package+ * dependency — including the two packages these patches exist to guard. The+ * store is enumerated explicitly for that reason; `walkInstalledPackageDirs`+ * de-dupes by realpath, so hoisted layouts (Bun/npm) still work through the+ * repo-root entry alone and cost nothing extra here.+ */+function installedPackageRoots(): string[] {+ const rootNodeModules = join(REPO_ROOT, 'node_modules');+ const roots = [rootNodeModules];++ const store = join(rootNodeModules, '.pnpm');+ let storeEntries: string[];+ try {+ storeEntries = readdirSync(store);+ } catch {+ return roots; // Not a pnpm layout — the repo-root walk already covers it.+ }+ for (const entry of storeEntries) {+ roots.push(join(store, entry, 'node_modules'));+ }+ return roots;+}+function resolveFileFromSpecifier(specifier: string): string {const resolved = import.meta.resolve(specifier);return resolved.startsWith('file:') ? fileURLToPath(resolved) : resolved;@@ -150,11 +222,21 @@ function walkInstalledPackageDirs(}/**- * Every bundle in our dep tree that ships its own copy of the destructive-- * delete code. Both are real production paths: `@tiptap/extension-collaboration-cursor`- * imports from `y-prosemirror`; `@tiptap/extension-collaboration` and our 27+- * direct imports go through `@tiptap/y-tiptap`. Patching only one leaves a- * live CRDT data-loss bug in the other.+ * Every patched artifact in our dep tree that ships its own copy of the+ * destructive-delete code. All are real production paths:+ * `@tiptap/extension-collaboration-cursor` imports from `y-prosemirror`;+ * `@tiptap/extension-collaboration` and our 27+ direct imports go through+ * `@tiptap/y-tiptap`. Patching only one leaves a live CRDT data-loss bug in+ * the other.+ *+ * `y-prosemirror` needs TWO entries because it ships two copies of the same+ * code and its `exports["."]` map serves them by condition: `require` gets+ * `dist/y-prosemirror.cjs`, `import` gets `src/y-prosemirror.js`, which+ * re-exports `ySyncPlugin` / `updateYFragment` from `src/plugins/sync-plugin.js`.+ * The Vite build of `packages/app` takes the `import` condition, so `src/` is+ * the copy that actually reaches the browser and asserting only against+ * `dist/` would leave the shipped path unguarded. `@tiptap/y-tiptap` needs no+ * equivalent entry — its `import` condition also resolves into `dist/`.*/const PATCHED_BUNDLES = [{@@ -162,6 +244,11 @@ const PATCHED_BUNDLES = [packageName: 'y-prosemirror',relativePath: ['dist', 'y-prosemirror.cjs'],},+ {+ label: 'y-prosemirror ESM src',+ packageName: 'y-prosemirror',+ relativePath: ['src', 'plugins', 'sync-plugin.js'],+ },{label: '@tiptap/y-tiptap CJS',packageName: '@tiptap/y-tiptap',@@ -179,6 +266,31 @@ function resolvePatchedBundlePath(bundle: (typeof PATCHED_BUNDLES)[number]): str}describe('R13 patch verification (y-prosemirror + @tiptap/y-tiptap)', () => {+ /**+ * Canary for the matcher every destructive-delete assertion in this file is+ * built on. Those assertions are all negative (`not.toMatch`, "no offending+ * file found"), so a matcher that matches nothing makes all of them pass —+ * against a patched bundle AND against a fully stock one. Nothing else in+ * this file can catch that, because a vacuous negative assertion looks+ * exactly like a satisfied one.+ *+ * The two fixtures are the verbatim upstream call sites from stock+ * `[email protected]` `src/plugins/sync-plugin.js` (the block-context site+ * in `createNodeFromYElement` and the inline-context site in+ * `createTextNodesFromYText`).+ */+ test('destructive-delete matcher matches the upstream shape it guards against', () => {+ const upstreamBlockContext = '/** @type {Y.Item} */ (el._item).delete(transaction)';+ const upstreamInlineContext = '/** @type {Y.Item} */ (text._item).delete(transaction)';++ expect(upstreamBlockContext).toMatch(DESTRUCTIVE_DELETE);+ expect(upstreamInlineContext).toMatch(DESTRUCTIVE_DELETE);++ // And it must not fire on the patched replacement, or every bundle would+ // report as unpatched.+ expect("schema.node('rawMdxFallback', { value: raw })").not.toMatch(DESTRUCTIVE_DELETE);+ });+test('both patches are registered in pnpm-workspace.yaml patchedDependencies', () => {// pnpm declares patches in pnpm-workspace.yaml (not package.json, where Bun// kept them). Parse the block with a line matcher to avoid a YAML dependency.@@ -226,39 +338,104 @@ describe('R13 patch verification (y-prosemirror + @tiptap/y-tiptap)', () => {expect(src).toMatch(/\[y-prosemirror\] schema\.text\(/);});+ test('contains the ymark: mark-encoding (not upstream mark-drop)', () => {+ const src = readFileSync(resolvePatchedBundlePath(bundle), 'utf8');++ // These identifiers carry ProseMirror marks on inline element/leaf+ // nodes across the Y bridge, encoded as `ymark:`-prefixed reserved+ // attributes. They are a SEPARATE concern from the R13 delete-fix and+ // are the exact surface that vanishes if a version bump re-ports the+ // delete-fix but drops the mark encoding.+ //+ // On the y-prosemirror artifacts this is the SOLE line of defense:+ // every functional mark test imports through @tiptap/y-tiptap, so a+ // y-prosemirror-only regression is silent mark loss on the+ // collaboration-cursor import path with nothing else to catch it. On+ // the @tiptap/y-tiptap artifacts it is deliberate defense-in-depth —+ // functional tests do cover that path, but only one of its two dist+ // artifacts at a time and only with a server running, so a re-port+ // that drops the encoding from just one of them still fails here, at+ // unit-test time. A failure on a y-tiptap bundle is therefore real,+ // not redundant.+ //+ // Thresholds, not exact counts, so a re-bundle that inlines or+ // duplicates a call site does not fail spuriously.+ const prefixMarkers = src.match(/YMARK_PREFIX/g);+ expect(prefixMarkers).not.toBeNull();+ // At minimum: the constant declaration + one encode + one decode site.+ expect(prefixMarkers?.length).toBeGreaterThanOrEqual(3);++ // Patch-introduced helpers: declaration + at least one call site. Both+ // are absent from a stock bundle, so this is the assertion that fails+ // on a dropped re-port.+ for (const helper of ['splitYAttrs', 'equalYMarks']) {+ const helperMarkers = src.match(new RegExp(helper, 'g'));+ expect(helperMarkers, `${helper} missing from ${bundle.label}`).not.toBeNull();+ expect(helperMarkers?.length).toBeGreaterThanOrEqual(2);+ }++ // `yattr2markname` is UPSTREAM code — the patch only adds a call site,+ // so it is asserted as a dependency pin rather than a patch marker: a+ // version bump that drops the upstream helper breaks `equalYMarks`,+ // and this fails at test time instead of silently at re-port time.+ // Deliberately no occurrence threshold — any floor a stock bundle+ // already clears would prove nothing.+ expect(src).toContain('yattr2markname');++ // The reserved attribute-key prefix itself — the wire format that both+ // the encode and decode halves agree on.+ expect(src).toContain('ymark:');+ });+test('patched throw sites do NOT retain upstream destructive _item.delete calls', () => {const src = readFileSync(resolvePatchedBundlePath(bundle), 'utf8');// Split on 'R13 patch:' and for each hunk, verify the patch body does- // NOT contain `_item.delete(transaction)` — that's the upstream- // destructive path the patch replaced.+ // NOT contain the destructive delete the patch replaced.const hunks = src.split(/R13 patch:/);// The first hunk is everything BEFORE the first patch marker — skip it.for (let i = 1; i < hunks.length; i++) {const hunk = hunks[i].slice(0, 4000);- expect(hunk).not.toMatch(/_item\.delete\(transaction\)/);+ expect(hunk).not.toMatch(DESTRUCTIVE_DELETE);}- // Stronger check: NO `_item.delete(transaction)` anywhere in the- // bundle. Both call sites (block + text) lived only in the patched- // catch blocks; after patching, neither bundle should reference it- // at all.- expect(src).not.toMatch(/_item\.delete\(transaction\)/);+ // Stronger check: no destructive delete anywhere in the bundle. Both+ // call sites (block + text) lived only in the patched catch blocks;+ // after patching, neither bundle should reference it at all.+ expect(src).not.toMatch(DESTRUCTIVE_DELETE);});});}- test('y-prosemirror patch file exists on disk and references y-prosemirror', () => {+ /**+ * The ymark encoding must survive a re-port in the PATCH FILE, not just in+ * the installed bundle. The per-bundle assertions above only run after+ * `pnpm install` has applied the patch; asserting the same identifiers+ * against the patch source fails a dropped-encoding re-port at patch-review+ * time, which is where it is cheapest to fix.+ */+ function expectYmarkEncoding(patchContent: string) {+ for (const identifier of ['YMARK_PREFIX', 'splitYAttrs', 'equalYMarks', 'yattr2markname']) {+ expect(patchContent, `${identifier} missing from patch file`).toContain(identifier);+ }+ }++ test('y-prosemirror patch file exists on disk and references both artifacts', () => {const patchPath = join(REPO_ROOT, 'patches', '[email protected]');const patchContent = readFileSync(patchPath, 'utf8');expect(patchContent).toContain('y-prosemirror.cjs');+ // The `import` condition resolves here, so a patch that touches only the+ // CJS artifact ships an unpatched editor. Asserting it against the patch+ // file catches an incomplete re-port before `pnpm install` applies it.+ expect(patchContent).toContain('src/plugins/sync-plugin.js');expect(patchContent).toContain('R13 patch:');expect(patchContent).toContain('rawMdxFallback');expect(patchContent).toContain('__okYpsCounters');+ expectYmarkEncoding(patchContent);});test('@tiptap/y-tiptap patch file exists on disk and references both bundles', () => {- // Bun encodes `/` as `%2F` in scoped-package patch filenames.+ // Scoped-package patch filenames encode `/` as `%2F`.const patchPath = join(REPO_ROOT, 'patches', '@tiptap%[email protected]');const patchContent = readFileSync(patchPath, 'utf8');expect(patchContent).toContain('dist/y-tiptap.cjs');@@ -266,12 +443,13 @@ describe('R13 patch verification (y-prosemirror + @tiptap/y-tiptap)', () => {expect(patchContent).toContain('R13 patch:');expect(patchContent).toContain('rawMdxFallback');expect(patchContent).toContain('__okYpsCounters');+ expectYmarkEncoding(patchContent);});/*** Dep-tree invariant: NO shipped bundle anywhere in node_modules retains the- * upstream destructive-delete pattern `_item.delete(transaction)`. This is- * the future-proof gate — if a new dependency ships another vendored copy+ * upstream destructive-delete pattern `(el._item).delete(transaction)`. This+ * is the future-proof gate — if a new dependency ships another vendored copy* of the same code, this test fails in CI and points at the exact file, so* the patch surface is extended before the regression can ship.*@@ -284,56 +462,86 @@ describe('R13 patch verification (y-prosemirror + @tiptap/y-tiptap)', () => {* makes it trivially easy to miss the next one; checking the invariant* mechanically cannot.*- * Scoping: skips dot-prefixed package-manager internals and source maps;- * scans `.js` / `.cjs` under each reachable package's `dist/`. The walk- * starts at the repo root `node_modules/` and follows nested `node_modules/`- * directories with realpath de-dupe, so it works across Bun/npm/pnpm- * layouts without assuming a private store path.+ * Scoping: scans `.js` / `.cjs` / `.mjs` recursively under each reachable+ * package's ARTIFACT_ROOTS, skipping nested `node_modules/` (visited as+ * packages in their own right) and dot-directories. The walk starts from+ * every root in `installedPackageRoots()` with shared realpath de-dupe, so+ * it covers hoisted (Bun/npm) and isolated (pnpm) layouts alike.+ *+ * The sweep asserts reachability BEFORE it reports offenders: a walk that+ * reaches nothing finds zero offending files and is indistinguishable from a+ * clean tree, which is exactly how this invariant silently stopped covering+ * anything when the repo moved off Bun's hoisted layout.*/- // Filesystem-heavy walk over the entire node_modules tree — scans every- // `.js`/`.cjs` under each package's `dist/`. Wall time varies with FS-cache+ // Filesystem-heavy walk over the entire node_modules tree — reads every+ // `.js`/`.cjs`/`.mjs` under each package's artifact roots. Wall time is+ // dominated by bytes read (roughly 400 MB here) and so varies with FS-cache// state (cold runner vs. warm cache) and total dep count; the default 5s// timeout is too tight and produces spurious flakes (observed 9.7s on a// cold CI runner). 30s leaves comfortable headroom while still failing// loud on a genuine hang.- test('dep-tree invariant: no destructive _item.delete(transaction) in any dist bundle', () => {+ test('dep-tree invariant: no destructive delete in any shipped bundle', () => {const offending: Array<{ path: string; line: number }> = [];+ const reachedPackages = new Set<string>();- function scanDistDir(distDir: string) {- let entries: string[];+ function scanArtifactDir(dir: string, depth: number) {+ if (depth > MAX_ARTIFACT_DEPTH) return;+ let entries: Dirent[];try {- entries = readdirSync(distDir);+ entries = readdirSync(dir, { withFileTypes: true });} catch {return;}for (const entry of entries) {- if (!entry.endsWith('.js') && !entry.endsWith('.cjs')) continue;- const full = join(distDir, entry);+ if (entry.name === 'node_modules' || entry.name.startsWith('.')) continue;+ const full = join(dir, entry.name);+ if (entry.isDirectory()) {+ scanArtifactDir(full, depth + 1);+ continue;+ }+ if (!/\.(?:js|cjs|mjs)$/.test(entry.name)) continue;let src: string;try {src = readFileSync(full, 'utf8');} catch {continue;}- if (!src.includes('_item.delete(transaction)')) continue;+ if (!DESTRUCTIVE_DELETE.test(src)) continue;// Report first offending line for actionable failure output.const lines = src.split('\n');- const lineIdx = lines.findIndex((l) => l.includes('_item.delete(transaction)'));+ const lineIdx = lines.findIndex((l) => DESTRUCTIVE_DELETE.test(l));offending.push({ path: full, line: lineIdx + 1 });}}function scanPackageDir(pkgDir: string) {- scanDistDir(join(pkgDir, 'dist'));+ reachedPackages.add(basename(pkgDir));+ for (const artifactRoot of ARTIFACT_ROOTS) {+ scanArtifactDir(join(pkgDir, artifactRoot), 0);+ }}- walkInstalledPackageDirs(join(REPO_ROOT, 'node_modules'), scanPackageDir);+ const visited = new Set<string>();+ for (const root of installedPackageRoots()) {+ walkInstalledPackageDirs(root, scanPackageDir, visited);+ }++ // Reachability canary — see the header comment. A zero-offender result is+ // only meaningful if the packages this patch guards were actually visited.+ for (const guarded of ['y-prosemirror', 'y-tiptap']) {+ expect(+ reachedPackages.has(guarded),+ `dep-tree walk never reached "${guarded}" (visited ${reachedPackages.size} packages), ` ++ `so a zero-offender result proves nothing. The node_modules layout has probably ` ++ `changed — extend installedPackageRoots() to cover it.`,+ ).toBe(true);+ }if (offending.length > 0) {const details = offending.map(({ path, line }) => ` ${path}:${line}`).join('\n');throw new Error(`Found ${offending.length} bundle(s) with the upstream destructive-delete pattern ` +- `\`_item.delete(transaction)\`. Every such bundle must be patched via \`bun patch\` ` ++ `\`(el._item).delete(transaction)\`. Every such bundle must be patched via \`pnpm patch\` ` +`to substitute rawMdxFallback (block-context) or log+skip (inline-context); ` +`otherwise a schema.node()/schema.text() throw will tombstone Y.Items and ` +`broadcast the delete to all peers (see PRECEDENTS.md precedent #9):\n${details}`,