Build your personal knowledge base with Trilium Notes
TypeScript electronelectron-appknowledge-baseknowledge-graphknowledge-managementknowledge-management-graphlocal-firstnote-managmentnote-takernote-takingnotebooknotesnotes-apppersonal-knowledge-basepersonal-wikiscriptableself-hostedself-hostingwiki
chore(skills): add a skill for the Mermaid syntax highlighting
Elian Doran1bcc9bd20e20
diff --git a/.claude/skills/developing-mermaid-highlighting/SKILL.md b/.claude/skills/developing-mermaid-highlighting/SKILL.mdnew file mode 100644index 0000000000..58e33ad0bf--- /dev/null+++ b/.claude/skills/developing-mermaid-highlighting/SKILL.md@@ -0,0 +1,166 @@+---+name: developing-mermaid-highlighting+description: Use when changing how Mermaid diagram source is highlighted in Trilium's code editor — the Lezer grammars vendored in `packages/codemirror/src/languages/mermaid/` (`parsers/<diagram>/<diagram>.grammar`, `tokens.ts`, `highlight.ts`, `tags/`, the `mermaid` dispatcher grammar, `index.ts`), adding highlighting for a diagram type it does not know yet (radar, treemap, Venn, railroad, …), fixing a diagram that highlights wrongly, the `test/<diagram>.txt` fixtures, `scripts/build_mermaid_grammars.mts`, or the `text/vnd.mermaid` entry in `syntax_highlighting.ts`. Covers regenerating the committed parsers, the checklist for a new diagram type, the Lezer traps already hit (precedence beats length, external-tokenizer order, CRLF), and how to prove a grammar change (fixtures first, real Mermaid as the referee, tree comparison over a generated corpus). Includes tree.mts, fixtures.mts and compare.mts — don't write new parse-tree probes.+---++# Developing Mermaid syntax highlighting++A Mermaid note is highlighted by a Lezer language in `packages/codemirror/src/languages/mermaid/`,+loaded by `syntax_highlighting.ts` for the `text/vnd.mermaid` and `text/mermaid` MIME types. It was+vendored from [Edge-Tools/codemirror-lang-mermaid](https://github.com/Edge-Tools/codemirror-lang-mermaid)+at `03bcd7d`, itself a fork of inspirnathan's `codemirror-lang-mermaid`. Both upstreams are dormant:+**the code is ours now**. Never re-vendor over it; `git log -- packages/codemirror/src/languages/mermaid`+shows what changed after the verbatim copy and why.++## Layout++```+languages/mermaid/+ index.ts mermaid(), mermaidLanguage, DIAGRAM_PARSERS; re-exports foldByIndent()+ extensions/index.ts foldByIndent() — folds a line with the deeper-indented lines after it+ tags/index.ts highlight tags, one set per diagram+ parsers/mermaid/ the dispatcher: detects the diagram type from the first meaningful line+ parsers/<diagram>/ <diagram>.grammar, tokens.ts (external tokenizers, optional),+ highlight.ts (styleTags), parser.ts + parser.terms.ts (GENERATED)+ test/<diagram>.txt Lezer fixtures, one file per parsers/ directory, same name+ mermaid.spec.ts runs every fixture file, checks generated parsers are current+ LICENSE upstream MIT licence — keep it+```++How a document parses: `mermaid.grammar` skips the preamble (front matter, `%%{init}%%`, blank+lines) as `preDiagramLine`s, then its `diagramText` tokenizer (`parsers/mermaid/tokens.ts`) matches+the first other line against `diagramMap` and emits one node for the rest of the document, such as+`FlowchartDiagram`. `parseMixed()` in `index.ts` looks that node's name up in `DIAGRAM_PARSERS` and+mounts the diagram's own parser over it. A diagram with no entry there, or a keyword missing from+`diagramMap`, gets no highlighting at all.++## Rules++- **Never edit `parser.ts` or `parser.terms.ts`.** After any `.grammar` change, regenerate:+ `pnpm --filter @triliumnext/codemirror build:mermaid-grammars`. The spec fails while a committed+ parser differs from what its grammar generates. The parsers are committed because the client+ consumes `packages/codemirror` as source, and no Vite/Vitest config compiles `.grammar` files.+- **Relative imports end in `.js`** (`nodenext`), in TypeScript and in a grammar's+ `@external … from "./tokens.js"` alike.+- **Run the narrow suite**: `pnpm --filter @triliumnext/codemirror test src/languages/mermaid`+ (add `syntax_highlighting` when touching the wiring). Then `pnpm typecheck` once.+- **The package gates on 100% coverage** in CI (`pnpm run --filter=codemirror test --coverage`),+ and vendored code counts. Measure just this folder:+ ```bash+ cd packages/codemirror && pnpm exec vitest run src/languages/mermaid \+ --coverage --coverage.include='src/languages/mermaid/**/*.ts' --coverage.reporter=json-summary+ ```+ then read `test-output/vitest/coverage/coverage-summary.json`. The text table hides rows that are+ at 100%, so read totals rather than inferring them from missing output. Generated parsers are+ excluded from coverage (`vite.config.ts`) and from the formatter (`eslint.format.config.mjs`).+- **Format check** only the folder:+ `pnpm exec eslint -c eslint.format.config.mjs packages/codemirror/src/languages/mermaid`.++## Adding a diagram type++Twelve built-in Mermaid types have no grammar yet: radar, treemap, tree view, event modeling,+Ishikawa, Venn, use case, Wardley, Cynefin, railroad (four dialects), swimlane, agent flow. The+official VS Code extension (Mermaid-Chart/vscode-mermaid-chart, MIT, `syntaxes/`) has a TextMate+grammar for radar; the rest have none anywhere, so write them from Mermaid's syntax docs.++1. **`parsers/<diagram>/<diagram>.grammar`.** Its `@top` node is named `<Something>Diagram`. Start+ from the closest existing grammar: line-oriented diagrams (pie, packet, sankey) are small;+ indentation-based ones (mindmap, kanban) need a context tracker.+2. **`highlight.ts`** maps node names to tags with `styleTags`, using `@external propSource … from+ "./highlight.js"` in the grammar. **`tokens.ts`** only if the grammar needs external tokenizers.+3. **Tags** go in `tags/index.ts`. Give each one a standard parent+ (`Tag.define(t.keyword)`): themes style standard tags only, so a parentless tag renders unstyled+ (mindmap's `lineText1..5` are, deliberately or not).+4. **Teach the dispatcher.** In `parsers/mermaid/mermaid.grammar`, add the node to both the `@top`+ alternatives and the `@external tokens diagramText` list; in `parsers/mermaid/tokens.ts` add each+ keyword the diagram starts with to `diagramMap` (`radar-beta`, …). `diagramMap` matches with+ `startsWith` and takes the first hit, so list a longer key before a shorter key it starts with.+5. **Wire it** into `DIAGRAM_PARSERS` in `index.ts`, keyed by the node name. The spec fails if a+ parser directory is missing from the map.+6. **Fixtures**: `test/<diagram>.txt`, named exactly like the directory. Start with Trilium's own+ sample for the type (`apps/client/src/widgets/type_widgets/mermaid/sample_diagrams.ts`), then one+ case per construct. Add a detection case to `test/mermaid.txt`.+7. Regenerate, run the suite, measure coverage.++## Lezer traps already hit++- **`@precedence` beats length.** A higher-precedence token wins even where a lower one would match+ more. Flowchart's `--` label opener ranked below the link tokens, so `-- text -->` always lexed as+ a link: the edge-label rule could never match, and upstream fixtures recorded the broken tree.+ The fix made the opener longer (`"--" $[ \t]+`) so no link token overlaps it, then ranked it first.+- **External tokenizers run in declaration order; the first that accepts wins.** Kanban's+ `columnTitleTokens` claims every line without a `[`, so `nodeIdTokens` only ever sees card lines.+ Check what an earlier tokenizer swallows before deciding a later one's branch is reachable.+- **`acceptToken(term, offset)`** with a negative offset ends the token before what was read —+ kanban uses it to keep the spaces in `id1 [Task]` out of the id. Fixture trees do not show token+ ranges; assert those in the spec, or look with `tree.mts --spans`.+- **Line endings.** CodeMirror normalizes `\r\n` and `\r` to `\n` before parsing, but the grammars+ still meet CRLF in specs and anywhere else they run. Git normalizes CRLF in `.txt` fixtures, so a+ CRLF case belongs in `mermaid.spec.ts` (`parses Windows line endings like Unix ones`).+- **Error recovery is behaviour too.** A fixture whose tree has `⚠` pins how the grammar recovers.+ That is fine for input Mermaid rejects; for input Mermaid accepts it records a limitation, and the+ fixture name must say so (`… (unsupported, parses as empty lines)`).++## Proving a change++**Fixtures first.** Write or correct the fixture, run it red against the current grammar, then fix.+`fixtures.mts` generates fixture blocks from the current parser — that is for recording behaviour+you have already judged correct, never for deciding what correct is.++**Real Mermaid is the referee** on what syntax is valid. Before pinning a tree for an odd input, ask+Mermaid, from a throwaway client spec (delete it afterwards):++```ts+// apps/client/src/zz_probe.spec.ts+import mermaid from "mermaid";+import { it } from "vitest";+it("probe", async () => {+ for (const code of [ "kanban\n todo\n id1 [Task]", "mindmap root" ]) {+ try { await mermaid.parse(code); console.log("OK ", code); }+ catch (e) { console.log("ERR", code, (e as Error).message.split("\n")[0]); }+ }+});+```++run with `cd apps/client && pnpm exec vitest run src/zz_probe.spec.ts`. The upstream fixtures were+written against the parser, not against Mermaid, and some recorded bugs as expected output.++**A refactor that claims "no behaviour change" needs proof, not reasoning.** Record the trees,+change the code, check:++```bash+K=.claude/skills/developing-mermaid-highlighting+npx tsx $K/compare.mts record mindmap /tmp/before.json+# … change, regenerate if a grammar changed …+npx tsx $K/compare.mts check mindmap /tmp/before.json+```++The corpus recombines the lines of the diagram's fixtures (2000 documents, LF and CRLF) and+compares every node with its range. Then **break the changed line on purpose once** and confirm+`check` reports differences — a corpus that never reaches the code passes everything. Both halves+have caught real mistakes: a mindmap check that a handful of hand-picked documents never reached+(logging suggested it was dead; 98 corpus documents changed without it), and a token-range change+that an earlier name-only comparison missed.++**Before deleting a branch as unreachable**, log what reaches it across the corpus, not across a+few documents you wrote. If the corpus reaches it, it needs a fixture; if neither the corpus nor+any document Mermaid accepts reaches it, remove it, and say how you know in the commit.++## Tools++| | does |+|---|---|+| `tree.mts <diagram> '<input>' [--spans]` | prints the parse tree; `--spans` adds each node's range and text. `mermaid` parses with the top-level language |+| `fixtures.mts <cases.json> [--append]` | `[diagram, name, input][]` → fixture blocks from the current parser; `--append` writes them to `test/<diagram>.txt` |+| `compare.mts record\|check <diagram> <file>` | before/after tree comparison over a generated corpus |++All run with `npx tsx .claude/skills/developing-mermaid-highlighting/<script>` from the repository+root; `\n`, `\r` and `\t` in an input are unescaped.++## Known limitations++- `mindmap root` (root on the `mindmap` line, valid Mermaid) parses as empty lines.+- A bare word under a kanban column parses as another column, not a card.+- Dotted edge labels (`A -. text .-> B`) parse as two links.+- No autocompletion exists for Mermaid in any editor library; it would be an+ `@codemirror/autocomplete` source written here.diff --git a/.claude/skills/developing-mermaid-highlighting/compare.mts b/.claude/skills/developing-mermaid-highlighting/compare.mtsnew file mode 100644index 0000000000..fcb9aaecd2--- /dev/null+++ b/.claude/skills/developing-mermaid-highlighting/compare.mts@@ -0,0 +1,77 @@+/**+ * Checks that a change leaves a grammar's parse trees alone, over a generated corpus.+ *+ * npx tsx .claude/skills/developing-mermaid-highlighting/compare.mts record <diagram> <trees.json>+ * (make the change, regenerate the parser if a grammar changed)+ * npx tsx .claude/skills/developing-mermaid-highlighting/compare.mts check <diagram> <trees.json>+ *+ * The corpus recombines the lines of every input in test/<diagram>.txt: 2000 documents of up to+ * eight lines, deterministic, each also with CRLF line endings. A tree is compared node by node+ * with the range each covers, so a token that grows or shrinks counts as a change even when the+ * tree prints the same. `check` lists the documents whose tree changed. A clean result only means+ * something if the corpus reaches the code: break the changed line on purpose once and confirm+ * `check` reports differences.+ */+import { readFileSync, writeFileSync } from "node:fs";+import { join } from "node:path";++const ROOT = join(import.meta.dirname, "../../../packages/codemirror/src/languages/mermaid");++const [ mode, diagram, treesPath ] = process.argv.slice(2);+if ((mode !== "record" && mode !== "check") || !diagram || !treesPath) {+ console.error("usage: compare.mts record|check <diagram> <trees.json>");+ process.exit(1);+}++const { parser } = diagram === "mermaid"+ ? { parser: (await import(join(ROOT, "index.ts"))).mermaidLanguage.parser }+ : await import(join(ROOT, "parsers", diagram, "parser.ts"));++const docs = buildCorpus(readFileSync(join(ROOT, "test", `${diagram}.txt`), "utf8"));+const trees = docs.map((doc) => serialize(parser.parse(doc)));++if (mode === "record") {+ writeFileSync(treesPath, JSON.stringify({ docs, trees }));+ console.log(`recorded ${trees.length} trees (${new Set(trees).size} distinct) to ${treesPath}`);+} else {+ const before: { docs: string[]; trees: string[] } = JSON.parse(readFileSync(treesPath, "utf8"));+ let changed = 0;+ for (const [ index, tree ] of trees.entries()) {+ if (tree === before.trees[index]) continue;+ changed++;+ if (changed <= 8) {+ console.log(`${JSON.stringify(docs[index])}\n before: ${before.trees[index]}\n after: ${tree}`);+ }+ }+ console.log(changed ? `${changed} of ${trees.length} trees changed` : `all ${trees.length} trees identical`);+ process.exitCode = changed ? 1 : 0;+}++function serialize(tree: { iterate(spec: { enter(node: Node): void }): void }) {+ const nodes: string[] = [];+ tree.iterate({ enter: (node) => { nodes.push(`${node.name}@${node.from}-${node.to}`); } });+ return nodes.join(" ");+}++interface Node { name: string; from: number; to: number }++function buildCorpus(fixtures: string) {+ const inputs = fixtures.split(/^# .*$/m)+ .map((block) => block.split(/^==>$/m)[0].trim())+ .filter(Boolean);+ const header = inputs[0].split("\n")[0];+ const lines = [ ...new Set(inputs.flatMap((input) => input.split("\n").slice(1))) ];+ lines.push("", " ", "\t");++ let seed = 2654435761;+ const random = () => (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648;+ const pick = () => lines[Math.floor(random() * lines.length)];++ const docs: string[] = [ ...inputs ];+ while (docs.length < 2000) {+ const body = Array.from({ length: 1 + Math.floor(random() * 8) }, pick);+ const doc = [ header, ...body ].join("\n") + (random() < 0.5 ? "\n" : "");+ docs.push(doc, doc.replaceAll("\n", "\r\n"));+ }+ return docs;+}diff --git a/.claude/skills/developing-mermaid-highlighting/fixtures.mts b/.claude/skills/developing-mermaid-highlighting/fixtures.mtsnew file mode 100644index 0000000000..e3d98c754c--- /dev/null+++ b/.claude/skills/developing-mermaid-highlighting/fixtures.mts@@ -0,0 +1,40 @@+/**+ * Turns test cases into Lezer fixture blocks, with the expected tree taken from the current parser.+ *+ * npx tsx .claude/skills/developing-mermaid-highlighting/fixtures.mts <cases.json> [--append]+ *+ * <cases.json> holds `[diagram, name, input][]`, e.g.+ * [["kanban", "Card id with a hyphen", "kanban\n todo\n my-card[Task]"]]+ * Without `--append` the blocks are printed for review; with it they are appended to+ * test/<diagram>.txt. Review every tree before appending: the parser's current output is what the+ * fixture will then require, wrong or not.+ */+import { appendFileSync, readFileSync } from "node:fs";+import { join } from "node:path";++const ROOT = join(import.meta.dirname, "../../../packages/codemirror/src/languages/mermaid");++const [ casesPath, flag ] = process.argv.slice(2);+if (!casesPath) {+ console.error("usage: fixtures.mts <cases.json> [--append]");+ process.exit(1);+}++const cases: [string, string, string][] = JSON.parse(readFileSync(casesPath, "utf8"));+for (const [ diagram, name, input ] of cases) {+ if (input.includes("\r")) {+ console.error(`skipped "${name}": Git normalizes CRLF in fixtures, test it in mermaid.spec.ts`);+ continue;+ }+ const { parser } = diagram === "mermaid"+ ? { parser: (await import(join(ROOT, "index.ts"))).mermaidLanguage.parser }+ : await import(join(ROOT, "parsers", diagram, "parser.ts"));+ const block = `# ${name}\n\n${input}\n\n==>\n\n${parser.parse(input).toString()}\n`;++ if (flag === "--append") {+ appendFileSync(join(ROOT, "test", `${diagram}.txt`), `\n${block}`);+ console.log(`appended to test/${diagram}.txt: ${name}`);+ } else {+ console.log(`--- test/${diagram}.txt\n${block}`);+ }+}diff --git a/.claude/skills/developing-mermaid-highlighting/tree.mts b/.claude/skills/developing-mermaid-highlighting/tree.mtsnew file mode 100644index 0000000000..25d4f40b3a--- /dev/null+++ b/.claude/skills/developing-mermaid-highlighting/tree.mts@@ -0,0 +1,35 @@+/**+ * Prints the Lezer parse tree of a Mermaid document, as one diagram's grammar sees it.+ *+ * npx tsx .claude/skills/developing-mermaid-highlighting/tree.mts <diagram> '<input>' [--spans]+ *+ * <diagram> is a directory under packages/codemirror/src/languages/mermaid/parsers/ (`flowchart`,+ * `kanban`, …); `mermaid` parses with the top-level language, which mounts the diagram's grammar+ * the way the editor does. `\n`, `\r` and `\t` in <input> are unescaped. `--spans` also prints+ * every node with the text it covers, which a fixture's tree cannot show.+ */+import { join } from "node:path";++const PARSERS = join(import.meta.dirname, "../../../packages/codemirror/src/languages/mermaid");++const [ diagram, raw, flag ] = process.argv.slice(2);+if (!diagram || raw === undefined) {+ console.error("usage: tree.mts <diagram> '<input>' [--spans]");+ process.exit(1);+}++const input = raw.replace(/\\n/g, "\n").replace(/\\r/g, "\r").replace(/\\t/g, "\t");+const { parser } = diagram === "mermaid"+ ? { parser: (await import(join(PARSERS, "index.ts"))).mermaidLanguage.parser }+ : await import(join(PARSERS, "parsers", diagram, "parser.ts"));++const tree = parser.parse(input);+console.log(tree.toString());++if (flag === "--spans") {+ tree.iterate({+ enter(node: { name: string; from: number; to: number; depth?: number }) {+ console.log(` ${node.name.padEnd(24)} ${node.from}-${node.to} ${JSON.stringify(input.slice(node.from, node.to))}`);+ }+ });+}