Skip to content

Beautiful, AI-native markdown IDE and LLM wiki

fix(ok): stop reading raw source as lost text (PRD-8286) (#3917)

Andrew Mikofalvy6bef11078fa1
diff --git a/.changeset/producer-guard-raw-source-false-positive.md b/.changeset/producer-guard-raw-source-false-positive.md
new file mode 100644
index 000000000..a29bbf765
--- /dev/null
+++ b/.changeset/producer-guard-raw-source-false-positive.md
@@ -0,0 +1,13 @@
+---
+"@inkeep/open-knowledge": patch
+---
+
+A document holding a component the editor does not recognize inside another component was reported as losing content when nothing had been lost.
+
+When the editor meets a component it does not recognize, it shows that component as its raw source text. Before persisting a document the server compares the version it is about to write against a fresh reading of those same bytes, so that a write which drops your writing gets caught. That comparison could not tell raw source text apart from the structure the very same bytes read back as, so it counted the tags and the markdown inside a raw view as writing that had gone missing. Documents in that shape were flagged, and left rows in their own history announcing a recovered loss that never happened.
+
+The comparison now reads a raw-source block as source rather than as writing, so it no longer expects that block's markup to survive being parsed. Documents in that shape are no longer flagged for it.
+
+What this gives up: this check gets less sensitive, not just more accurate. Writing that vanishes from inside a raw-source block is no longer visible to it. Neither, in a document holding a raw-source block, is some writing that vanishes from outside one: the check compares two whole documents as a single run of letters with no way to tell which letters came from which block, so once a raw-source block is set aside its letters can stand in for writing lost nearby. What that costs is this check's warning and the restore point it saves before applying such a write. What bounds it is where the check started: before this change it flagged every save of a document holding an unrecognized component, so what it reported for those documents was already unusable.
+
+Unchanged: when a wrapper the editor does recognize, such as a callout, an accordion or a set of tabs, holds a component it does not recognize, blank lines you wrote around that wrapper are still rewritten. That is tracked separately.
diff --git a/packages/core/src/bridge/pm-structural-equivalence.ts b/packages/core/src/bridge/pm-structural-equivalence.ts
index 754dc5cc2..bb53e7c7c 100644
--- a/packages/core/src/bridge/pm-structural-equivalence.ts
+++ b/packages/core/src/bridge/pm-structural-equivalence.ts
@@ -125,6 +125,7 @@ export type StructuralEquivalenceResult =
export interface ComparePmStructuralOptions {
ignoreAttrs?: (attrKey: string) => boolean;
+ rawSourceSide?: 'expected';
}
/** Rebuild a node with `fn` applied to each child; identity when childless.
@@ -207,10 +208,14 @@ function stableStringify(value: unknown): string {
/** Concatenated, whitespace-stripped text of a tree — the byte stream a
* faithful round-trip must not LOSE. Compared as a subsequence so a degrade
- * that INSERTS tokens (a `<br/>` literal) never reads as loss. */
-function textSkeleton(node: PmStructuralNode): string {
+ * that INSERTS tokens (a `<br/>` literal) never reads as loss.
+ *
+ * `omitRawSource` skips `rawMdxFallback` subtrees. L1a decides when to pass it
+ * from the caller's `rawSourceSide`; that call site owns the rationale. */
+function textSkeleton(node: PmStructuralNode, omitRawSource = false): string {
let acc = '';
const walk = (n: PmStructuralNode): void => {
+ if (omitRawSource && n.type === 'rawMdxFallback') return;
if (n.text !== undefined) acc += n.text;
if (n.content) for (const child of n.content) walk(child);
};
@@ -257,7 +262,12 @@ export function comparePmStructural(
const a = applyDegrades(actual);
const appliedDegrades = [...new Set([...e.fired, ...a.fired])];
- if (!isSubsequence(textSkeleton(e.normalized), textSkeleton(a.normalized))) {
+ const intactStrict = isSubsequence(textSkeleton(e.normalized), textSkeleton(a.normalized));
+ const exemptExonerates =
+ !intactStrict &&
+ opts.rawSourceSide === 'expected' &&
+ isSubsequence(textSkeleton(e.normalized, true), textSkeleton(a.normalized, true));
+ if (!intactStrict && !exemptExonerates) {
return {
equivalent: false,
level: 'L1',
diff --git a/packages/server/src/server-observers.producer-guard.test.ts b/packages/server/src/server-observers.producer-guard.test.ts
index 9e2a522c1..e0ccbb052 100644
--- a/packages/server/src/server-observers.producer-guard.test.ts
+++ b/packages/server/src/server-observers.producer-guard.test.ts
@@ -64,13 +64,51 @@ function createDoc() {
/** Seed the fragment in one null-origin drain so Observer A sees xmlDirty. A
* guard throw inside the settlement handler propagates out of this call. */
-function seedFragment(doc: Y.Doc, xmlFragment: Y.XmlFragment, md: string): void {
- const pmNode = schema.nodeFromJSON(mdManager.parse(md));
+function seedFragmentJson(doc: Y.Doc, xmlFragment: Y.XmlFragment, json: unknown): void {
+ const pmNode = schema.nodeFromJSON(json);
doc.transact(() => {
updateYFragment(doc, xmlFragment, pmNode, { mapping: new Map(), isOMark: new Map() });
}, null);
}
+function seedFragment(doc: Y.Doc, xmlFragment: Y.XmlFragment, md: string): void {
+ seedFragmentJson(doc, xmlFragment, mdManager.parse(md));
+}
+
+/** A RECOGNIZED container holding an unrecognized child as a `rawMdxFallback` —
+ * the tree a client NodeView leaves behind after swapping that child for its
+ * own source. The wrapper is recognized on purpose: for an unrecognized one the
+ * NodeView replaces the whole subtree with a single top-level fallback, so
+ * container-holding-fallback is only a transient state there, whereas a
+ * recognized wrapper never converts and the shape is durable. `reason` carries
+ * the real prefix the NodeView stamps, which `severity.ts` classifies on. No
+ * markdown parse yields this shape, so it is built as PM JSON. The container is
+ * `sourceDirty`, so serialize re-derives from the children rather than
+ * replaying `sourceRaw`, and the fallback's text is what reaches the guard. */
+const CONTAINER_WITH_FALLBACK_CHILD = {
+ type: 'doc',
+ content: [
+ {
+ type: 'jsxComponent',
+ attrs: {
+ componentName: 'Callout',
+ kind: 'element',
+ attributes: [],
+ sourceRaw: '<Callout>\n\n<Step>\n\n**bold** step\n\n</Step>\n\n</Callout>',
+ sourceDirty: true,
+ props: { type: 'info' },
+ },
+ content: [
+ {
+ type: 'rawMdxFallback',
+ attrs: { reason: 'Unregistered component: Step' },
+ content: [{ type: 'text', text: '<Step>\n\n**bold** step\n\n</Step>' }],
+ },
+ ],
+ },
+ ],
+};
+
/** MarkdownManager whose `serialize` drops every `dropText` run from its output
* — a serializer that silently loses content, faulted at the system boundary. */
function makeContentLosingManager(dropText: string): MarkdownManager {
@@ -177,6 +215,29 @@ describe('Producer guard (FR6) — dev/test posture throws (M2)', () => {
}
});
+ test('a faithful serialize of a container holding a rawMdxFallback does NOT fire', () => {
+ // No faulted serializer: the real MarkdownManager emits the fallback's own
+ // bytes and a fresh parse rebuilds them as a `<Step>` component. Nothing is
+ // lost, but the live tree holds those tags as TEXT while the reparse holds
+ // them as STRUCTURE. L1a compared the two skeletons and reported the markup
+ // the parse legitimately consumed as dropped content, so the guard fired —
+ // and, in the packaged posture, checkpointed — on every drain of a document
+ // in this shape.
+ // No `console.warn` assertion here: in this posture a fire THROWS before
+ // `reportProducerGuardViolation` runs, so a warn spy could never see one and
+ // would read as coverage of the packaged log path it cannot reach. The
+ // packaged case below asserts that path through the artifact users see.
+ const { doc, xmlFragment, ytext } = createDoc();
+ const cleanup = setupServerObservers(
+ baseOpts({ doc, xmlFragment, ytext, docName: 'fallback.md' }),
+ );
+ try {
+ expect(() => seedFragmentJson(doc, xmlFragment, CONTAINER_WITH_FALLBACK_CHILD)).not.toThrow();
+ } finally {
+ cleanup();
+ }
+ });
+
test('a container-shatter (text preserved, container gone) does NOT fire — silent on shatter', () => {
// Pins "fire on content-loss, silent on shatter": the serializer drops the
// Callout wrapper but keeps its text, so a fresh parse shatters the container
@@ -285,6 +346,43 @@ describe('Producer guard (FR6) — packaged posture logs + checkpoints, never th
}
});
+ test('a container holding a rawMdxFallback logs nothing and leaves no surfaced checkpoint', async () => {
+ // The packaged posture is the only one that reaches the log and the
+ // checkpoint, and the checkpoint is the harm the changeset names: its
+ // registry entry is `visibility: 'surfaced'`, so a bogus one shows up as a
+ // restore row in the user's own version history. Asserted through that
+ // artifact rather than through a spy that the dev/test posture forecloses.
+ const { doc, xmlFragment, ytext } = createDoc();
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const cleanup = setupServerObservers(
+ baseOpts({
+ doc,
+ xmlFragment,
+ ytext,
+ docName: 'fallback.md',
+ shadow: () => shadow,
+ contentRoot: 'content',
+ getBranch: () => 'main',
+ }),
+ );
+ try {
+ expect(() => seedFragmentJson(doc, xmlFragment, CONTAINER_WITH_FALLBACK_CHILD)).not.toThrow();
+ const fired = warn.mock.calls
+ .map((call) => String(call[0]))
+ .some((line) => line.includes('producer-guard-violation'));
+ expect(fired).toBe(false);
+ // Inverted use of a helper built for the positive case: it polls to its
+ // deadline and returns empty, which is the answer we want here. An
+ // explicit shorter bound, because this one has to expire every run and a
+ // real checkpoint on this path commits well inside it (the same
+ // assertion under a mutated guard returns one ref in about a second).
+ expect(await waitForCheckpointRefs(shadow, 1500)).toEqual([]);
+ } finally {
+ warn.mockRestore();
+ cleanup();
+ }
+ });
+
test('two distinct losses in the cooldown: one log suppressed, BOTH checkpointed, next emit carries the suppressed count', async () => {
const { doc, xmlFragment, ytext } = createDoc();
const losing = makeContentLosingManager(LOSS_SENTINEL);
diff --git a/packages/server/src/server-observers.ts b/packages/server/src/server-observers.ts
index fc5340cab..952892c06 100644
--- a/packages/server/src/server-observers.ts
+++ b/packages/server/src/server-observers.ts
@@ -1789,11 +1789,21 @@ export function setupServerObservers(opts: SetupServerObserversOpts): () => void
* serialize). A fresh parse of the bytes we are about to persist must
* reconstruct the same authored CONTENT: markdown never legitimately drops
* text on a round-trip, so a content-loss verdict means the serializer emitted
- * corrupt bytes that only a fresh parser sees. Container-shatter is
- * deliberately NOT a fire condition — some shatters are inherent CommonMark
- * round-trip limits (a blockquote nested in a Callout re-merges on parse), a
- * fidelity gap the offline I22 property test owns; firing on them here would
- * cry wolf on legal-but-lossy nestings. Dev/test throw loud; packaged reports.
+ * corrupt bytes that only a fresh parser sees.
+ *
+ * One carve-out beyond the shatter one below: the guard declares
+ * `rawSourceSide: 'expected'`, without which it fired on every drain of any
+ * document holding an unrecognized component, which is no signal at all. The
+ * cost is sensitivity, not just precision: a loss inside a `rawMdxFallback`,
+ * and — in a document whose fallback the reparse consumed — a loss outside
+ * one, are both rejected by the comparator at another level, which this guard
+ * does not fire on. What that does and does not cover lives at the L1a site.
+ *
+ * Container-shatter is deliberately NOT a fire condition — some shatters are
+ * inherent CommonMark round-trip limits (a blockquote nested in a Callout
+ * re-merges on parse), a fidelity gap the offline I22 property test owns;
+ * firing on them here would cry wolf on legal-but-lossy nestings. Dev/test
+ * throw loud; packaged reports.
*/
const runProducerGuard = (json: PmStructuralNode, body: string): void => {
if (body === lastGuardedBody) return;
@@ -1801,7 +1811,10 @@ export function setupServerObservers(opts: SetupServerObserversOpts): () => void
if (!fragmentContainsDangerSpace(json)) return;
const reparsed = mdManager.parseWithFallback(body, observerParseOpts) as PmStructuralNode;
- const verdict = comparePmStructural(json, reparsed);
+ // `json` is the live fragment. `reparsed` can hold a `rawMdxFallback` too —
+ // `parseWithFallback` mints them — so this declares an orientation, it does
+ // not assert that either side is fallback-free.
+ const verdict = comparePmStructural(json, reparsed, { rawSourceSide: 'expected' });
// Narrow to the failure branch, then to the one reason the guard fires on.
// The union makes `reason`/`detail` reachable only here, and `detail`
// required — no optional-fallback crutch.