An open source design system that's fully customizable and agent ready
TypeScript
feat(markdown): expose server-safe AST parsers (#6382)
Cindy Zhang970b59262e3d
diff --git a/.changeset/markdown-plugin-core.md b/.changeset/markdown-plugin-core.mdindex aabfc63700..1668749cfa 100644--- a/.changeset/markdown-plugin-core.md+++ b/.changeset/markdown-plugin-core.md@@ -5,8 +5,10 @@[feat] Markdown: add the core plugin protocolUse `createMarkdownPlugin()` and Markdown's `plugins` prop to compose bounded-source syntax, immutable typed AST transforms, and extension renderers. The same-ordered plugins work with parser entry points and Markdown-derived Outline-items, while omitted or empty plugin lists preserve existing behavior.+source syntax, immutable typed AST transforms, and extension renderers. Import+`parseMarkdownAst()` or `parseInlineAst()` from+`@astryxdesign/core/Markdown/parser` when server code needs the canonical tree.+The same ordered plugins work with parser entry points and Markdown-derived+Outline items, while omitted or empty plugin lists preserve existing behavior.@cixzhangdiff --git a/docs/specs/AST-036/spec.md b/docs/specs/AST-036/spec.mdindex 7d9c9bbeb7..3cf2352296 100644--- a/docs/specs/AST-036/spec.md+++ b/docs/specs/AST-036/spec.md@@ -79,7 +79,7 @@ A plugin entry is an opaque value created by Markdown's public `createMarkdownPlThe public construction surface is `createMarkdownPlugin()`, `createMarkdownTextTransform()`, `createMarkdownFenceTransform()`, and `createMarkdownSourceDecoration()`. Limited Remark compatibility is a separate tree-shakeable `createMarkdownRemarkTransform()` export from `@astryxdesign/core/Markdown/remark`. These helpers return or annotate the one `transform` capability; they never accept React render callbacks and are not additional lifecycle phases. In particular, `createMarkdownFenceTransform()` maps eligible built-in code nodes to an owning plugin's typed extension data, while that plugin's standard `renderers` map owns presentation and `toText`. Detailed TypeScript signatures and examples live in the Markdown consumer docs, while these names and responsibilities are part of this contract.-The canonical document is an immutable, discriminated, Unist-shaped tree aligned with the supported MDAST subset below. **MDAST is a public community specification maintained by the syntax-tree/unified ecosystem, not a browser or standards-body API.** Astryx's public contract is its own exported, versioned TypeScript node union: it uses the listed MDAST node names and field meanings where they align, documents every deviation, and does not promise nominal or structural interchangeability with arbitrary `mdast`, Unified, or third-party plugin versions. Parsing, transforms, Markdown rendering, and Outline projection observe the same node meanings. Released parser entrypoints retain their current result types and values through a compatibility boundary; adopting plugins requires no migration.+The canonical document is an immutable, discriminated, Unist-shaped tree aligned with the supported MDAST subset below. **MDAST is a public community specification maintained by the syntax-tree/unified ecosystem, not a browser or standards-body API.** Astryx's public contract is its own exported, versioned TypeScript node union: it uses the listed MDAST node names and field meanings where they align, documents every deviation, and does not promise nominal or structural interchangeability with arbitrary `mdast`, Unified, or third-party plugin versions. `parseMarkdownAst()` and `parseInlineAst()` expose that tree additively from the server-safe `@astryxdesign/core/Markdown/parser` subpath; the component barrel also re-exports them for client code. Parsing, transforms, Markdown rendering, and Outline projection observe the same node meanings. Released parser entrypoints retain their current result types and values through a compatibility boundary; adopting plugins requires no migration.The Markdown owner exposes stable node unions and node-kind-narrowed traversal. Plugin-defined extension kinds augment the generic union without widening unrelated built-in callbacks. A transform receives readonly input and returns the original root or a replacement root. Equivalent validation and immutability implementations are permitted.@@ -160,7 +160,7 @@ Compatibility is explicit per plugin and tested version. “Remark-compatible”- **FR6 — Fixed order.** Built-in and extension syntax parse first; transforms then run in plugin-array order; rendering runs last. A transform observes the validated result of every earlier transform.- **FR7 — Built-in lexical shields win during syntax.** Escapes, inline and fenced code, links, images, citations, math, and accepted opaque syntax remain protected according to their current owners. Block extension syntax remains top-level unless a current owner explicitly broadens it.- **FR8 — Ordered syntax claims.** Duplicate plugin names fail validation. Syntax contributions resolve in plugin and declaration order; first match wins, defer reserves the bounded candidate, and terminal parsing resolves all pending source.-- **FR9 — Canonical AST is MDAST-aligned and strictly typed.** Parsing, transforms, rendering, and Outline projection use one stable Unist-shaped subset with complete available source positions. Public node-kind traversal narrows callbacks by discriminant, including generic plugin extensions. Released parser functions retain their existing result types and values until a separately approved migration.+- **FR9 — Canonical AST is MDAST-aligned, strictly typed, and reachable.** Parsing, transforms, rendering, and Outline projection use one stable Unist-shaped subset with complete available source positions. `parseMarkdownAst()` and `parseInlineAst()` return that canonical tree from the server-safe `@astryxdesign/core/Markdown/parser` subpath and the Markdown component barrel. Public node-kind traversal narrows callbacks by discriminant, including generic plugin extensions. Released parser functions retain their existing result types and values until a separately approved migration.- **FR10 — Transform input is immutable.** A transform returns the same root or a replacement root. Attempted mutation cannot change any observed input node, array, data object, source position, prior snapshot, or later plugin input.- **FR11 — Transform output is representable and validated.** Output is finite, acyclic, typed data that satisfies both matrices above. Core rejects raw HTML, React values, DOM nodes, functions in node data, invalid built-in structure, foreign extension ownership, authored/shifted provenance, and block-shaped output from inline entrypoints.- **FR12 — Transform failure is local to the plugin.** A throw, promise, rejected thenable, invalid result, or unsupported adapter output reports one source-free diagnostic and passes the last valid root to later plugins and rendering. Core attaches a rejection sink before refusing an asynchronous tokenizer or transform so a rejected promise cannot escape through the host's unhandled-rejection channel. Source-backed nodes preserve authored-source fallback; synthetic extension leaves use their required perceivable-text projection; extension containers preserve their validated children.@@ -184,7 +184,7 @@ Compatibility is explicit per plugin and tested version. “Remark-compatible”- **FR21 — Unclaimed work is bounded.** A helper callback runs only for an actual claim; unmatched source cannot invoke it. Stable and recreated semantically equivalent plugin lists must both satisfy FR23, without requiring any particular preparation or dispatch implementation.- **FR22 — Existing parser budgets remain the floor.** The canonical [parser performance fixture](../../../packages/core/src/Markdown/parser.perf.test.ts) remains green: full parse stays below 20/50/100/400/1000 ms at 10/50/200/500/2000 sections; full and incremental 50-character streaming stay below 5000 ms at 50 sections and 30000 ms at 500; incremental time stays at most 1.1× full reparse; and its exact settled-tail work, object-reuse, source-range, median/worst-block, and tracked whole-prefix assertions remain unchanged.- **FR23 — Focused overhead budgets cover both transform paths.** Under the evidence protocol below, each budget names the path it measures. First-party helper transforms: five zero-work helpers add at most 15 percent and the representative three-helper set adds at most 25 percent, at both fixture sizes. Plugin-authored transforms, which pay full immutability and validation guards: five zero-work transforms add at most 30 percent and the representative three-transform set adds at most 50 percent, at both fixture sizes. Streaming the representative plugin-authored set stays within FR22's streaming budgets and at most 1.25× the same stream with an empty pipeline. Per-transform guard work is bounded by one traversal of that transform's output; whole-document invariant collection happens once per pipeline run, not once per transform. Core may skip a guard for a helper-authored transform only where the helper's own construction already guarantees the property, and the published budget must say which path a number describes.-- **FR24 — Optional work stays optional.** Remark adapters, the conformance kit, and heavy renderers are tree-shakeable and absent from parser-only/server bundles unless imported.+- **FR24 — Optional work stays optional.** The server-safe `@astryxdesign/core/Markdown/parser` subpath contains no client directive or component dependency. Remark adapters, the conformance kit, and heavy renderers are tree-shakeable and absent from parser-only/server bundles unless imported.### Extension containers, ownership, and node data@@ -233,7 +233,7 @@ Compatibility is explicit per plugin and tested version. “Remark-compatible”## Public parser integration-Block, inline, incremental, canonical-AST, Markdown, and Outline entrypoints accept the same ordered plugin configuration and infer the same extension-node union while retaining their released signatures and projected result shapes. The canonical entrypoints return the canonical tree directly; released entrypoints keep their projection. The server-safe parser entry exposes the same behavior without a client boundary. Inline transforms must satisfy the one-paragraph phrasing constraint. Transform-only updates cannot increase syntax tokenizer calls, mutate prior snapshots, or remount unchanged settled output beyond current behavior.+Block, inline, incremental, canonical-AST, Markdown, and Outline entrypoints accept the same ordered plugin configuration and infer the same extension-node union while retaining their released signatures and projected result shapes. The canonical entrypoints return the canonical tree directly; released entrypoints keep their projection. The server-safe parser entry exposes the same behavior without a client boundary, and the Markdown component barrel re-exports it for client code. Inline transforms must satisfy the one-paragraph phrasing constraint. Transform-only updates cannot increase syntax tokenizer calls, mutate prior snapshots, or remount unchanged settled output beyond current behavior.### Public API admission evidencediff --git a/packages/core/package.json b/packages/core/package.jsonindex e5c4031067..37b58640ac 100644--- a/packages/core/package.json+++ b/packages/core/package.json@@ -624,6 +624,11 @@"types": "./dist/Markdown/plugins/index.d.ts","default": "./dist/Markdown/plugins/index.js"},+ "./Markdown/parser": {+ "source": "./src/Markdown/parser/index.ts",+ "types": "./dist/Markdown/parser/index.d.ts",+ "default": "./dist/Markdown/parser/index.js"+ },"./Calendar/utils": {"source": "./src/Calendar/utils.ts","types": "./dist/Calendar/utils.d.ts",diff --git a/packages/core/src/Markdown/Markdown.doc.mjs b/packages/core/src/Markdown/Markdown.doc.mjsindex 6910564295..7357f8eb71 100644--- a/packages/core/src/Markdown/Markdown.doc.mjs+++ b/packages/core/src/Markdown/Markdown.doc.mjs@@ -267,6 +267,11 @@ export const docs = {description:'Use createMarkdownSourceDecoration to attach non-visual metadata — search hits, review annotations — to the blocks a source range touches, and getMarkdownSourceDecorations to read it back in a later plugin. Decorations appear on the settled document rather than on partial streaming chunks, and never change rendering, copyable text, accessible names, ids, focus order, or navigation.',},+ {+ guidance: true,+ description:+ 'Import parseMarkdownAst or parseInlineAst from @astryxdesign/core/Markdown/parser when server or React Server Component code needs to run plugins against the canonical readonly tree. The parser and plugin subpaths have no use-client boundary. The Markdown component remains client-owned, so function-bearing plugin entries must not be passed across an RSC serialization boundary.',+ },{guidance: true,description:@@ -722,6 +727,11 @@ export const docsZh = {description:'Use createMarkdownSourceDecoration to attach non-visual metadata — search hits, review annotations — to the blocks a source range touches, and getMarkdownSourceDecorations to read it back in a later plugin. Decorations appear on the settled document rather than on partial streaming chunks, and never change rendering, copyable text, accessible names, ids, focus order, or navigation.',},+ {+ guidance: true,+ description:+ 'Import parseMarkdownAst or parseInlineAst from @astryxdesign/core/Markdown/parser when server or React Server Component code needs to run plugins against the canonical readonly tree. The parser and plugin subpaths have no use-client boundary. The Markdown component remains client-owned, so function-bearing plugin entries must not be passed across an RSC serialization boundary.',+ },{guidance: true,description:@@ -799,6 +809,11 @@ export const docsDense = {description:'Use createMarkdownSourceDecoration to attach non-visual metadata — search hits, review annotations — to the blocks a source range touches, and getMarkdownSourceDecorations to read it back in a later plugin. Decorations appear on the settled document rather than on partial streaming chunks, and never change rendering, copyable text, accessible names, ids, focus order, or navigation.',},+ {+ guidance: true,+ description:+ 'Import parseMarkdownAst or parseInlineAst from @astryxdesign/core/Markdown/parser when server or React Server Component code needs to run plugins against the canonical readonly tree. The parser and plugin subpaths have no use-client boundary. The Markdown component remains client-owned, so function-bearing plugin entries must not be passed across an RSC serialization boundary.',+ },{guidance: true,description:diff --git a/packages/core/src/Markdown/Markdown.spec.md b/packages/core/src/Markdown/Markdown.spec.mdindex d08065276e..cc2fd2bb2d 100644--- a/packages/core/src/Markdown/Markdown.spec.md+++ b/packages/core/src/Markdown/Markdown.spec.md@@ -106,7 +106,12 @@ Consumer migration instructions belong in consumer docs and release notes.`createMarkdownPlugin()`. A plugin declares only the `syntax`, `transform`, and`renderers` capabilities it uses. Syntax-bearing entries supply stable parseidentity. Parsing, transforms, rendering, and Outline use one stable, strictly typed,-MDAST-aligned canonical tree; `MarkdownAstNodeMap` and `visitMarkdownNodes` provide+MDAST-aligned canonical tree; `parseMarkdownAst()` and `parseInlineAst()` expose+that tree from the server-safe `@astryxdesign/core/Markdown/parser` subpath while+`@astryxdesign/core/Markdown/plugins` remains server-safe for constructing and+running plugins in server or RSC code. The rendered `Markdown` component remains a+client entry: function-bearing plugin entries are not serializable props and cannot+cross an RSC boundary. `MarkdownAstNodeMap` and `visitMarkdownNodes` providenode-kind narrowing. Released parser functions preserve their existing result shapethrough a compatibility projection. Transforms return validated replacement rootswithout entering parse identity. Every extension node introduced@@ -168,6 +173,7 @@ unions. Enabled calls return the explicit `InlineNodeWithMath` and| FR22 | An extension renderer may opt into the Markdown-owned extension theme target so themes reach plugin output. Opting out leaves output untargeted. The target adds no default styling or anatomy beyond the block spacing and content width Core already applies. || FR23 | Markdown owns an explicit supported dialect rather than claiming full CommonMark or GFM conformance. Adjacent compatible ordered or unordered items remain one list regardless of task-marker presence; each item independently preserves its checked state or ordinary list-item semantics, including at nested levels. The default grammar keeps its released task-list and table support, while `autolink: 'gfm'` adds only the documented autolink behavior and does not toggle any other syntax. || FR24 | `createMarkdownFrontmatter()` recognizes only a leading `---` block of unique `key: value` lines, decodes it through the caller's typed parser, stores finite JSON-like metadata on the canonical document, and removes the syntax from rendered content. An unfinished leading block yields no visible Markdown while streaming; malformed or unfinished final input remains ordinary Markdown. Frontmatter has no renderer and requires no Remark compatibility. |+| FR25 | The canonical parser and plugin-construction subpaths remain server-safe and can run function-bearing plugins entirely within server or RSC code. The client-owned `Markdown` component supports traditional and streaming SSR, but plugin entries containing functions cannot be serialized from a Server Component into that client boundary; direct RSC rendering requires a future additive server renderer rather than weakening the plugin protocol. |### Allowed variation@@ -326,19 +332,20 @@ and this change preserves the existing spelling exactly.## Verification map-| Contract | Verification | Representative states | Failure signal |-| ---------------------- | -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |-| FR1–FR5 | `Markdown.test.tsx`, theme-target tests, and `scripts/check-knowledge.mjs` | Default block/inline output and all current targets | Existing DOM, target, spacing, or renderer behavior changes. |-| FR6–FR10 | `parser.test.ts` and `Markdown.test.tsx` | Opt-out, inline/display math, escapes, currency, code, links, plugins | A delimiter is claimed without opt-in, TeX is formatted as Markdown, or opaque contexts leak. |-| FR11 | `incremental.test.ts` and `Markdown.test.tsx` | Every-character top-level/list/task-list/blockquote splits, CRLF, source ranges | Streaming diverges from a full parse, shows partial syntax, mistakes a nested closer for an opener, or crosses a closed container. |-| FR12–FR16 | plugin, transform, adapter, performance, and Outline parser tests | omitted/empty lists, immutable transforms, invalid outputs, live updates, one compatible Remark plugin, duplicate headings | Empty behavior forks, input mutates, invalid structure escapes, transforms reparse, adapter loses content, budgets fail, or heading targets diverge. |-| FR17–FR18 | container parse/validation, ownership, and dependency tests | leaf/container declarations, nested containers, foreign read/remove/mint/edit, unmet/misordered dependencies | Plugin-built parsed children, invalid content, lost fallback children, foreign mint/edit, or generic ownership codes. |-| FR19 | diagnostic-channel tests in development and production | every phase, advisory reports, rate suppression, no handler, malformed list, duplicate Core, version skew | A silent production failure, document content in a diagnostic, or one entrypoint throwing where another recovers. |-| FR20–FR22 | canonical/server imports, inference, streaming, preparation, theming tests | server imports, no explicit type args, chunk boundaries, recreated lists, themed/unthemed extensions | Client references, explicit-type workarounds, oscillation, per-render re-preparation, or unreachable opted-in output. |-| FR23 | parser, renderer, nesting, and public option tests | task-only, plain-only, and mixed ordered/unordered lists at top level and nested; autolink omitted/enabled | A mixed list splits or loses order/state, a plain item becomes a checkbox, or `autolink: 'gfm'` changes non-autolink syntax. |-| FR24 | `plugins/frontmatter.test.tsx`, Storybook, and server rendering | complete, malformed, non-leading, LF/CRLF, unfinished streaming, full plugin stack | Metadata syntax renders after completion, unfinished syntax leaks while streaming, typing is lost, or later plugins stop composing. |-| Public syntax/types | `Markdown.public.test.ts`, core typecheck, and `Markdown.doc.mjs` | Legacy exhaustive switches, math opt-ins, inferred extension-node unions | A released union widens, an enabled union loses nodes, or docs drift from declarations. |-| Security/accessibility | `parser.test.ts`, `Markdown.test.tsx`, and renderer guidance | Inert expression strings and renderer-owned semantics | Astryx executes math as HTML or silently claims renderer-owned accessibility. |+| Contract | Verification | Representative states | Failure signal |+| ---------------------- | -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |+| FR1–FR5 | `Markdown.test.tsx`, theme-target tests, and `scripts/check-knowledge.mjs` | Default block/inline output and all current targets | Existing DOM, target, spacing, or renderer behavior changes. |+| FR6–FR10 | `parser.test.ts` and `Markdown.test.tsx` | Opt-out, inline/display math, escapes, currency, code, links, plugins | A delimiter is claimed without opt-in, TeX is formatted as Markdown, or opaque contexts leak. |+| FR11 | `incremental.test.ts` and `Markdown.test.tsx` | Every-character top-level/list/task-list/blockquote splits, CRLF, source ranges | Streaming diverges from a full parse, shows partial syntax, mistakes a nested closer for an opener, or crosses a closed container. |+| FR12–FR16 | plugin, transform, adapter, performance, and Outline parser tests | omitted/empty lists, immutable transforms, invalid outputs, live updates, one compatible Remark plugin, duplicate headings | Empty behavior forks, input mutates, invalid structure escapes, transforms reparse, adapter loses content, budgets fail, or heading targets diverge. |+| FR17–FR18 | container parse/validation, ownership, and dependency tests | leaf/container declarations, nested containers, foreign read/remove/mint/edit, unmet/misordered dependencies | Plugin-built parsed children, invalid content, lost fallback children, foreign mint/edit, or generic ownership codes. |+| FR19 | diagnostic-channel tests in development and production | every phase, advisory reports, rate suppression, no handler, malformed list, duplicate Core, version skew | A silent production failure, document content in a diagnostic, or one entrypoint throwing where another recovers. |+| FR20–FR22 | canonical/server imports, inference, streaming, preparation, theming tests | server imports, no explicit type args, chunk boundaries, recreated lists, themed/unthemed extensions | Client references, explicit-type workarounds, oscillation, per-render re-preparation, or unreachable opted-in output. |+| FR23 | parser, renderer, nesting, and public option tests | task-only, plain-only, and mixed ordered/unordered lists at top level and nested; autolink omitted/enabled | A mixed list splits or loses order/state, a plain item becomes a checkbox, or `autolink: 'gfm'` changes non-autolink syntax. |+| FR24 | `plugins/frontmatter.test.tsx`, Storybook, and server rendering | complete, malformed, non-leading, LF/CRLF, unfinished streaming, full plugin stack | Metadata syntax renders after completion, unfinished syntax leaks while streaming, typing is lost, or later plugins stop composing. |+| FR25 | `parser.public.test.ts`, plugin SSR tests, and package export checks | server/RSC parsing with plugins, synchronous SSR, suspending renderer streaming boundary | A server import gains `use client`, plugin execution needs serialization, SSR loses fallback, or direct RSC rendering is misrepresented as supported. |+| Public syntax/types | `Markdown.public.test.ts`, core typecheck, and `Markdown.doc.mjs` | Legacy exhaustive switches, math opt-ins, inferred extension-node unions | A released union widens, an enabled union loses nodes, or docs drift from declarations. |+| Security/accessibility | `parser.test.ts`, `Markdown.test.tsx`, and renderer guidance | Inert expression strings and renderer-owned semantics | Astryx executes math as HTML or silently claims renderer-owned accessibility. |Focused tests continue to pin all nine current target names and default blockplacement. Math intentionally adds no target and no default anatomy.diff --git a/packages/core/src/Markdown/ast.ts b/packages/core/src/Markdown/ast.tsindex 39e19a5289..bde9528e33 100644--- a/packages/core/src/Markdown/ast.ts+++ b/packages/core/src/Markdown/ast.ts@@ -4,7 +4,7 @@* @file ast.ts* @input Parsed Markdown structure and optional Astryx extension nodes* @output Canonical immutable MDAST-aligned Markdown node types and text projection- * @position Internal canonical Markdown tree shared by parsing, rendering, and Outline+ * @position Public canonical Markdown tree shared by parsing, transforms, rendering, and Outline*/export type MarkdownAstDataValue =diff --git a/packages/core/src/Markdown/index.ts b/packages/core/src/Markdown/index.tsindex 3824628479..38aae88ec4 100644--- a/packages/core/src/Markdown/index.ts+++ b/packages/core/src/Markdown/index.ts@@ -4,7 +4,7 @@/*** @file index.ts- * @output Exports Markdown component, parser functions, and types+ * @output Exports Markdown component, canonical and compatibility parser functions, and types* @position Component entry point; re-exported by /packages/core/src/index.ts*/@@ -50,9 +50,11 @@ export type {export {parseMarkdown,+ parseMarkdownAst,parseMarkdownIncremental,createIncrementalState,parseInline,+ parseInlineAst,} from './parser';export type {BlockNode,diff --git a/packages/core/src/Markdown/parser.public.test.ts b/packages/core/src/Markdown/parser.public.test.tsnew file mode 100644index 0000000000..5c914774a7--- /dev/null+++ b/packages/core/src/Markdown/parser.public.test.ts@@ -0,0 +1,103 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++/**+ * @file parser.public.test.ts+ * @input Imports the documented Markdown parser-only package subpath+ * @output Runtime and type evidence for its canonical server-safe contract+ * @position Public package-contract test for @astryxdesign/core/Markdown/parser+ */++import {readFileSync} from 'node:fs';+import {join} from 'node:path';+import {describe, expect, expectTypeOf, it} from 'vitest';+import {+ createMarkdownFrontmatter,+ createMarkdownPlugin,+} from '@astryxdesign/core/Markdown/plugins';+import {+ createIncrementalState,+ parseInline,+ parseInlineAst,+ parseMarkdown,+ parseMarkdownAst,+ parseMarkdownIncremental,+} from '@astryxdesign/core/Markdown/parser';+import type {+ BlockNode,+ InlineNode,+ MarkdownAstPhrasingContent,+ MarkdownAstRoot,+} from '@astryxdesign/core/Markdown/parser';++describe('@astryxdesign/core/Markdown/parser', () => {+ it('is a generated public package subpath', () => {+ const packageJson = JSON.parse(+ readFileSync(join(process.cwd(), 'packages/core/package.json'), 'utf8'),+ ) as {+ exports: Record<string, unknown>;+ };+ const entrySources = [+ 'packages/core/src/Markdown/parser/index.ts',+ 'packages/core/src/Markdown/parser.ts',+ 'packages/core/src/Markdown/plugins/index.ts',+ 'packages/core/src/Markdown/plugins/protocol.ts',+ 'packages/core/src/Markdown/plugins/frontmatter.ts',+ ].map(path => readFileSync(join(process.cwd(), path), 'utf8'));++ expect(packageJson.exports['./Markdown/parser']).toEqual({+ source: './src/Markdown/parser/index.ts',+ types: './dist/Markdown/parser/index.d.ts',+ default: './dist/Markdown/parser/index.js',+ });+ for (const entrySource of entrySources) {+ expect(entrySource).not.toMatch(/^\s*['"]use client['"]/m);+ }+ });++ it('exposes legacy and canonical parser results from a server-only entry point', () => {+ const source = '# Heading\n\nParagraph';+ const blockAst = parseMarkdownAst(source);+ const inlineAst = parseInlineAst('**bold**');++ expect(blockAst).toMatchObject({+ type: 'root',+ children: [{type: 'heading', depth: 1}, {type: 'paragraph'}],+ });+ expect(inlineAst).toMatchObject([{type: 'strong'}]);+ expect(parseMarkdown(source)[0]).toMatchObject({type: 'heading', level: 1});+ expect(parseInline('**bold**')[0]).toMatchObject({type: 'bold'});+ expect(parseMarkdownIncremental(source, createIncrementalState())).toEqual(+ parseMarkdown(source),+ );++ const canonicalBlockAst: MarkdownAstRoot = blockAst;+ const canonicalInlineAst: ReadonlyArray<MarkdownAstPhrasingContent> =+ inlineAst;+ expect(canonicalBlockAst).toBe(blockAst);+ expect(canonicalInlineAst).toBe(inlineAst);+ expectTypeOf(parseMarkdown(source)).toEqualTypeOf<BlockNode[]>();+ expectTypeOf(parseInline('text')).toEqualTypeOf<InlineNode[]>();+ });++ it('runs typed plugins through the server-safe parser boundary', () => {+ const frontmatter = createMarkdownFrontmatter({+ name: 'server-document-metadata',+ parse: fields => ({title: fields.title ?? 'Untitled'}),+ });+ let title: string | undefined;+ const observer = createMarkdownPlugin({+ name: 'server-metadata-observer',+ apiVersion: 1,+ transform(root) {+ title = frontmatter.getMetadata(root)?.title;+ return root;+ },+ });+ const root = parseMarkdownAst('---\ntitle: Server proof\n---\n# Body', {+ plugins: [frontmatter.plugin, observer],+ });++ expect(title).toBe('Server proof');+ expect(root.children).toMatchObject([{type: 'heading', depth: 1}]);+ });+});diff --git a/packages/core/src/Markdown/parser.ts b/packages/core/src/Markdown/parser.tsindex dc31ceb7cd..418c30bdd6 100644--- a/packages/core/src/Markdown/parser.ts+++ b/packages/core/src/Markdown/parser.ts@@ -3,9 +3,10 @@/*** @file parser.ts* @input Markdown string, released parse options, and optional ordered plugins- * @output Canonical MDAST-aligned nodes for internal consumers plus unchanged+ * @output Canonical MDAST-aligned nodes for public/server consumers plus unchanged* released parser-node projections; shared heading slug helpers- * @position Core parser and compatibility boundary; consumed by Markdown and Outline+ * @position Core parser and compatibility boundary; consumed by public parser entry,+ * Markdown, and Outline*/import {@@ -1356,11 +1357,43 @@ export function parseInline(text: string,arg?: ReadonlySet<string> | RuntimeParseOptions,): RuntimeInlineNode[] {- return projectInlineNodes(parseInlineAst(text, arg), wantsLegacyRanges(arg));+ return projectInlineNodes(+ parseInlineAstRuntime(text, arg),+ wantsLegacyRanges(arg),+ );}-/** @internal Canonical inline parse used by Markdown rendering. */+/**+ * Parse inline Markdown into the canonical immutable Astryx AST.+ *+ * Unlike `parseInline`, this returns MDAST-aligned node names and fields. Math+ * nodes are present only when `math: true`, and plugin extension nodes are+ * inferred from the ordered `plugins` tuple.+ */export function parseInlineAst(+ text: string,+ sourceIds?: ReadonlySet<string>,+): MarkdownAstPhrasingContent[];+export function parseInlineAst(+ text: string,+ options: ParseOptionsWithoutPlugins | MathParseOptionsWithoutPlugins,+): MarkdownAstPhrasingContent[];+export function parseInlineAst<+ const Plugins extends ReadonlyArray<MarkdownPluginEntry>,+>(+ text: string,+ options: (ParseOptionsWithoutPlugins | MathParseOptionsWithoutPlugins) & {+ plugins: Plugins;+ },+): MarkdownAstPhrasingContent<MarkdownExtensionsOf<Plugins>>[];+export function parseInlineAst(+ text: string,+ arg?: ReadonlySet<string> | RuntimeParseOptions,+): MarkdownAstPhrasingContent<RuntimeExtensionNode>[] {+ return parseInlineAstRuntime(text, arg);+}++function parseInlineAstRuntime(text: string,arg?: ReadonlySet<string> | RuntimeParseOptions,): MarkdownAstPhrasingContent<RuntimeExtensionNode>[] {@@ -2320,16 +2353,46 @@ export function parseMarkdown(arg?: ReadonlySet<string> | RuntimeParseOptions,): RuntimeBlockNode[] {return projectMarkdownRoot(- parseMarkdownAst(input, arg),+ parseMarkdownAstInternal(input, arg, true),wantsLegacyRanges(arg),);}-/** @internal Canonical block parse used by Markdown and Outline rendering. */+/**+ * Parse block Markdown into the canonical immutable Astryx AST.+ *+ * Unlike `parseMarkdown`, this returns the MDAST-aligned root used by Markdown,+ * transforms, and Outline. Math nodes are present only when `math: true`, and+ * plugin extension nodes are inferred from the ordered `plugins` tuple.+ */+export function parseMarkdownAst(+ input: string,+ sourceIds?: ReadonlySet<string>,+): MarkdownAstRoot;+export function parseMarkdownAst(+ input: string,+ options: ParseOptionsWithoutPlugins | MathParseOptionsWithoutPlugins,+): MarkdownAstRoot;+export function parseMarkdownAst<+ const Plugins extends ReadonlyArray<MarkdownPluginEntry>,+>(+ input: string,+ options: (ParseOptionsWithoutPlugins | MathParseOptionsWithoutPlugins) & {+ plugins: Plugins;+ },+): MarkdownAstRoot<MarkdownExtensionsOf<Plugins>>;export function parseMarkdownAst(input: string,arg?: ReadonlySet<string> | RuntimeParseOptions,- isFinal = true,+): MarkdownAstRoot<RuntimeExtensionNode> {+ return parseMarkdownAstInternal(input, arg, true);+}++/** @internal Full-parse entry point with streaming finality for Outline. */+export function parseMarkdownAstInternal(+ input: string,+ arg: ReadonlySet<string> | RuntimeParseOptions | undefined,+ isFinal: boolean,): MarkdownAstRoot<RuntimeExtensionNode> {const resolved = resolveOptions(arg);const opts = isFinal ? resolved : {...resolved, isFinal: false};diff --git a/packages/core/src/Markdown/parser/index.ts b/packages/core/src/Markdown/parser/index.tsnew file mode 100644index 0000000000..630cf2c21d--- /dev/null+++ b/packages/core/src/Markdown/parser/index.ts@@ -0,0 +1,64 @@+// Copyright (c) Meta Platforms, Inc. and affiliates.++/**+ * @file index.ts+ * @input Markdown source and public parser options+ * @output Server-safe legacy and canonical Markdown parser entry points and types+ * @position Public subpath entry point: `@astryxdesign/core/Markdown/parser`+ */++export {+ createIncrementalState,+ parseInline,+ parseInlineAst,+ parseMarkdown,+ parseMarkdownAst,+ parseMarkdownIncremental,+} from '../parser';+export type {+ BlockNode,+ BlockNodeWithMath,+ MathBlockNode,+ InlineNode,+ InlineNodeWithMath,+ MathInlineNode,+ SourceRange,+ ListItemNode,+ TableCellNode,+ TableAlignment,+ ParseOptions,+ MathParseOptions,+ IncrementalParseOptions,+ IncrementalMathParseOptions,+ IncrementalState as IncrementalParseState,+} from '../parser';+export type {+ MarkdownAstPoint,+ MarkdownAstDataValue,+ MarkdownAstPosition,+ MarkdownAstNodeBase,+ MarkdownAstExtensionNode,+ MarkdownAstText,+ MarkdownAstInlineCode,+ MarkdownAstInlineMath,+ MarkdownAstLink,+ MarkdownAstImage,+ MarkdownAstCitation,+ MarkdownAstBreak,+ MarkdownAstPhrasingContent,+ MarkdownAstHeading,+ MarkdownAstParagraph,+ MarkdownAstCode,+ MarkdownAstMath,+ MarkdownAstBlockquote,+ MarkdownAstList,+ MarkdownAstListItem,+ MarkdownAstTable,+ MarkdownAstTableRow,+ MarkdownAstTableCell,+ MarkdownAstThematicBreak,+ MarkdownAstBlockContent,+ MarkdownAstRoot,+ MarkdownAstNodeMap,+ MarkdownAstNode,+} from '../ast';diff --git a/packages/core/src/Markdown/plugins/frontmatter.test.tsx b/packages/core/src/Markdown/plugins/frontmatter.test.tsxindex 2cbcd4f6d0..2bdb7d6596 100644--- a/packages/core/src/Markdown/plugins/frontmatter.test.tsx+++ b/packages/core/src/Markdown/plugins/frontmatter.test.tsx@@ -10,7 +10,7 @@import {renderToString} from 'react-dom/server';import {describe, expect, expectTypeOf, it} from 'vitest';import {Markdown} from '../Markdown';-import {parseMarkdownAst} from '../parser';+import {parseMarkdownAst, parseMarkdownAstInternal} from '../parser';import {createMarkdownPlugin, type MarkdownTransform} from './protocol';import {createMarkdownFrontmatter} from './frontmatter';@@ -53,7 +53,7 @@ describe('createMarkdownFrontmatter', () => {});it('hides unfinished streaming frontmatter and exposes it once closed', () => {- const partial = parseMarkdownAst(+ const partial = parseMarkdownAstInternal('---\ntitle: Pending',{plugins: [documentMetadata.plugin]},false,@@ -61,7 +61,7 @@ describe('createMarkdownFrontmatter', () => {expect(documentMetadata.getMetadata(partial)).toBeUndefined();expect(partial.children).toEqual([]);- const settled = parseMarkdownAst(+ const settled = parseMarkdownAstInternal(SOURCE,{plugins: [documentMetadata.plugin]},false,diff --git a/packages/core/src/Markdown/plugins/protocol.test.tsx b/packages/core/src/Markdown/plugins/protocol.test.tsxindex 0b5ed9b4d3..dbf72a53dd 100644--- a/packages/core/src/Markdown/plugins/protocol.test.tsx+++ b/packages/core/src/Markdown/plugins/protocol.test.tsx@@ -7,7 +7,8 @@* @position Focused acceptance tests for the core Markdown plugin protocol*/-import {renderToString} from 'react-dom/server';+import {PassThrough} from 'node:stream';+import {renderToPipeableStream, renderToString} from 'react-dom/server';import {act, render, screen} from '@testing-library/react';import {describe, expect, expectTypeOf, it, vi} from 'vitest';import {Markdown} from '../Markdown';@@ -328,6 +329,75 @@ describe('Markdown plugin protocol', () => {expect(screen.getByText('after.', {exact: false})).toBeInTheDocument();});+ it('streams readable node fallback and siblings before a renderer resolves', async () => {+ let resolved = false;+ let resolve: (() => void) | undefined;+ const pending = new Promise<void>(done => {+ resolve = () => {+ resolved = true;+ done();+ };+ });++ function StreamingMention({label}: {readonly label: string}) {+ if (!resolved) {+ // React Suspense uses a thrown thenable as control flow.+ // eslint-disable-next-line @typescript-eslint/only-throw-error+ throw pending;+ }+ return <span>@{label} streamed</span>;+ }++ const plugin = createMarkdownPlugin<'mentions', MentionNode>({+ ...mentionDefinition,+ renderers: {+ mention: {+ render: ({node}) => <StreamingMention label={node.data.label} />,+ toText: node => `@${node.data.label}`,+ },+ },+ });+ const destination = new PassThrough();+ destination.setEncoding('utf8');+ const chunks: string[] = [];+ destination.on('data', chunk => chunks.push(String(chunk)));+ const finished = new Promise<void>((done, reject) => {+ destination.on('end', done);+ destination.on('error', reject);+ });+ let rejectShell: (error: unknown) => void = () => {};+ const shellReady = new Promise<void>((done, reject) => {+ rejectShell = reject;+ const {pipe} = renderToPipeableStream(+ <Markdown plugins={[plugin]}>{'Before @{Ada} after.'}</Markdown>,+ {+ onShellReady() {+ pipe(destination);+ done();+ },+ onShellError(error) {+ reject(error);+ },+ },+ );+ });+ destination.on('error', error => rejectShell(error));++ await shellReady;+ await new Promise<void>(done => setImmediate(done));+ const shell = chunks.join('');+ expect(shell).toContain('Before ');+ expect(shell).toContain('@{Ada}');+ expect(shell).toContain(' after.');+ expect(shell).not.toContain('@Ada streamed');++ resolve?.();+ await finished;+ expect(chunks.join('').replaceAll('<!-- -->', '')).toContain(+ '@Ada streamed',+ );+ });+it('rejects duplicate entries and contains invalid tokenizer output', () => {expect(() =>parseMarkdown('plain', {plugins: [mentionPlugin, mentionPlugin]}),diff --git a/packages/core/src/Outline/parseOutlineFromMarkdown.ts b/packages/core/src/Outline/parseOutlineFromMarkdown.tsindex ec3b9a04e4..4e1f12b9c5 100644--- a/packages/core/src/Outline/parseOutlineFromMarkdown.ts+++ b/packages/core/src/Outline/parseOutlineFromMarkdown.ts@@ -11,7 +11,11 @@* - /packages/core/src/Outline/index.ts*/-import {parseMarkdownAst, slugify, uniqueSlug} from '../Markdown/parser';+import {+ parseMarkdownAstInternal,+ slugify,+ uniqueSlug,+} from '../Markdown/parser';import {markdownAstText} from '../Markdown/ast';import {markdownExtensionText,@@ -50,7 +54,7 @@ export function parseOutlineFromMarkdown<? undefined: prepareMarkdownPlugins(options.plugins);const counts = new Map<string, number>();- return parseMarkdownAst(+ return parseMarkdownAstInternal(markdown,{plugins: options?.plugins},options?.isFinal ?? true,diff --git a/scripts/sync-exports.js b/scripts/sync-exports.jsindex ffc10330c2..255e624512 100644--- a/scripts/sync-exports.js+++ b/scripts/sync-exports.js@@ -97,17 +97,11 @@ const STATIC_EXPORTS = {'./locales/*.json': './locales/*.json',};-/**- * Server-safe utility subpath exports.- *- * These re-export pure functions from component directories without- * the `'use client'` directive, making them importable from React- * Server Components. Each entry points to a `utils.ts` file that- * re-exports only the server-safe subset of a component's utilities.- *- * See: https://github.com/facebook/astryx/issues/1977- */-const DIRECTORY_MODULE_SUBPATH_EXPORTS = ['Markdown/plugins'];+/** Nested modules backed by an index.ts entry point. */+const DIRECTORY_MODULE_SUBPATH_EXPORTS = [+ 'Markdown/plugins',+ 'Markdown/parser',+];const UTIL_SUBPATH_DIRS = ['Calendar',