Memos is an open-source, self-hosted note-taking app that brings your everyday thoughts together in a personal timeline. Your data stays yours, and you choose what to keep private or share.
Go dockerfossgomarkdownmemomicroblognote-takingnotesopen-sourceown-your-datareactself-hostedsqlitetimeline
perf(editor): replace highlight.js with Highlights tokens
johnnyjoygh24cce5ca27b4
diff --git a/web/package.json b/web/package.jsonindex c5d620c12..4f7e3f842 100644--- a/web/package.json+++ b/web/package.json@@ -22,6 +22,7 @@"@connectrpc/connect-web": "^2.1.1","@github/relative-time-element": "^5.0.0","@maplibre/maplibre-gl-leaflet": "^0.1.4",+ "@pierre/highlights": "0.2.0","@tailwindcss/vite": "^4.2.4","@tanstack/react-query": "^5.100.9","@tanstack/react-query-devtools": "^5.100.9",@@ -31,7 +32,6 @@"copy-to-clipboard": "^4.0.2","dayjs": "^1.11.20","exifr": "^7.1.3",- "highlight.js": "^11.11.1","html-to-image": "^1.11.13","i18next": "^26.3.6","katex": "^0.16.45",diff --git a/web/pnpm-lock.yaml b/web/pnpm-lock.yamlindex 1ba25b30b..e73f68dfe 100644--- a/web/pnpm-lock.yaml+++ b/web/pnpm-lock.yaml@@ -26,6 +26,9 @@ importers:'@maplibre/maplibre-gl-leaflet':specifier: ^0.1.4+ '@pierre/highlights':+ specifier: 0.2.0+ version: 0.2.0'@tailwindcss/vite':specifier: ^4.2.4@@ -53,9 +56,6 @@ importers:exifr:specifier: ^7.1.3version: 7.1.3- highlight.js:- specifier: ^11.11.1- version: 11.11.1html-to-image:specifier: ^1.11.13version: 1.11.13@@ -555,6 +555,9 @@ packages:'@oxc-project/[email protected]':resolution: {integrity: sha512-huv1Y/LzBJkBVHt3OlC7u0zHBW9qXf1FdD7sGmc1rXc2P1mTwHssYv7jyGx5KAACSCH+9B3Bhn6Z9luHRvf7pQ==}+ '@pierre/[email protected]':+ resolution: {integrity: sha512-hTpcOU1EzDAP4bYtEhxmMu2jwMvtkUydRUBUpWMSHTs3dw0IaC3Ld5ansMy18L0GE3b6Vfl2OGkLK+pWWpnBmw==}+'@react-leaflet/[email protected]':resolution: {integrity: sha512-3EWmekh4Nz+pGcr+xjf0KNyYfC3U2JjnkWsh0zcqaexYqmmB5ZhH37kz41JXGmKzpaMZCnPofBBm64i+YrEvGQ==}peerDependencies:@@ -1272,10 +1275,6 @@ packages:resolution: {integrity: sha512-g7df9rMFX/SPi34tyGCyUBREQoKkapwdY/T04Qn9TDWfHhAYt4/I0gMVirzK5wEzeUqIjEB+LXC/ypb7Aqno5w==}- resolution: {integrity: sha512-Xwwo44whKBVCYoliBQwaPvtd/2tYFkRQtXDWj1nackaV2JPXx3L0+Jvd8/qCJ2p+ML0/XVkJ2q+Mr+UVdpJK5w==}- engines: {node: '>=12.0.0'}-resolution: {integrity: sha512-CV9TW3Y3f8/wT0BRFc1/KAVQ3TUHiXmaAb6VW9vtiMFf7SLoMd1PdAc4W3KFOFETBJUb90KatHqlsZMWV+R9Gg==}engines: {node: ^20.19.0 || ^22.12.0 || >=24.0.0}@@ -2372,6 +2371,8 @@ snapshots:'@oxc-project/[email protected]': {}+ '@pierre/[email protected]': {}+'@react-leaflet/[email protected]([email protected])([email protected]([email protected]))([email protected])':dependencies:leaflet: 1.9.4@@ -2945,8 +2946,6 @@ snapshots:property-information: 7.1.0space-separated-tokens: 2.0.2- [email protected]: {}-dependencies:'@exodus/bytes': 1.15.0diff --git a/web/src/components/MemoEditor/README.md b/web/src/components/MemoEditor/README.mdindex 2c8ca39b9..8caac4985 100644--- a/web/src/components/MemoEditor/README.md+++ b/web/src/components/MemoEditor/README.md@@ -27,6 +27,8 @@ MemoEditor memo level: name | draft seed, mode, data, draft stor- **`view/`** is the read-only surface over `MemoMarkdown`: tags and mentions navigate, in-memo anchors scroll, a paragraph that is one bare link becomes a preview card through the editor's block view (`linkCards.ts`), the page's search matches are painted, and heading ids match the outline. Tag and mention clicks arrive through the editor's `onClick` callbacks, so nothing in memos matches the editor's class names or mutates its DOM. Previews that have Markdown but no memo query (the link-memo search, the share image) use `MemoContentView` directly.- **`edit/`** is the editing surface: `EditSession` renders the chrome (timestamp, attachments, suggestions, toolbar, focus mode, audio recording) over the draft store `MemoEditor` provides, persists the draft while open, and saves through the memo mutations. `Editor.tsx` is the composer's editor: `MemoMarkdown` editable with the composer's own extensions (`#tag` and `:emoji` autocomplete, file paste and drop), behind the `EditorController` contract in `controller.ts`.+Code blocks use lazy-loaded `@pierre/highlights` tokens as editor decorations, without an HTML conversion. GitHub Light and GitHub Dark Dimmed colors follow the document's color scheme; ordinary text and structural punctuation keep the editor's text color. Unsupported languages, unavailable WebAssembly, or browsers without `light-dark()` colors leave the code as plain text. Highlighting never changes the saved Markdown.+## Data- **Reads are by name.** `useMemoByName` (`hooks/useMemoQueries.ts`) returns the freshest copy of the memo any query holds (a list, a comment page, the calendar month, a detail, a shared memo) and fetches the detail only when nothing holds it. A card in a list therefore costs no request beyond the list's, follows the list's refetches, and never refetches on its own.diff --git a/web/src/lib/code-highlight.ts b/web/src/lib/code-highlight.tsindex 799890d94..8026d9ddf 100644--- a/web/src/lib/code-highlight.ts+++ b/web/src/lib/code-highlight.ts@@ -1,147 +1,71 @@-import type { HLJSApi, LanguageFn } from "highlight.js";-import { escape as escapeHTML } from "lodash-es";--type LanguageModule = { default: LanguageFn };-type LanguageLoader = () => Promise<LanguageModule>;--// Keep the languages most commonly used in memos cheap. Less common languages-// fall back to the complete build, preserving existing language support without-// making every visitor download all language definitions.-const commonLanguageLoaders = {- bash: () => import("highlight.js/lib/languages/bash"),- c: () => import("highlight.js/lib/languages/c"),- cpp: () => import("highlight.js/lib/languages/cpp"),- csharp: () => import("highlight.js/lib/languages/csharp"),- css: () => import("highlight.js/lib/languages/css"),- diff: () => import("highlight.js/lib/languages/diff"),- dockerfile: () => import("highlight.js/lib/languages/dockerfile"),- go: () => import("highlight.js/lib/languages/go"),- graphql: () => import("highlight.js/lib/languages/graphql"),- java: () => import("highlight.js/lib/languages/java"),- javascript: () => import("highlight.js/lib/languages/javascript"),- json: () => import("highlight.js/lib/languages/json"),- kotlin: () => import("highlight.js/lib/languages/kotlin"),- markdown: () => import("highlight.js/lib/languages/markdown"),- php: () => import("highlight.js/lib/languages/php"),- python: () => import("highlight.js/lib/languages/python"),- ruby: () => import("highlight.js/lib/languages/ruby"),- rust: () => import("highlight.js/lib/languages/rust"),- sql: () => import("highlight.js/lib/languages/sql"),- swift: () => import("highlight.js/lib/languages/swift"),- typescript: () => import("highlight.js/lib/languages/typescript"),- xml: () => import("highlight.js/lib/languages/xml"),- yaml: () => import("highlight.js/lib/languages/yaml"),-} satisfies Record<string, LanguageLoader>;--type CommonLanguage = keyof typeof commonLanguageLoaders;--const languageAliases: Record<string, CommonLanguage> = {- cs: "csharp",- cxx: "cpp",- golang: "go",- htm: "xml",- html: "xml",- js: "javascript",- jsx: "javascript",- md: "markdown",- py: "python",- rb: "ruby",- sh: "bash",- shell: "bash",- ts: "typescript",- tsx: "typescript",- vue: "xml",- yml: "yaml",- zsh: "bash",-};+import type { Theme } from "@pierre/highlights";+import type { Decoration } from "@usememos/editor";+type CodeHighlighter = (code: string) => Decoration[];+const readyHighlighters = new Map<string, CodeHighlighter>();const plainTextLanguages = new Set(["", "plain", "plaintext", "text", "txt"]);-const registeredLanguagePromises = new Map<CommonLanguage, Promise<HLJSApi>>();-let corePromise: Promise<HLJSApi> | undefined;-let fullBuildPromise: Promise<HLJSApi> | undefined;-let injectedThemeKey: string | undefined;--const isPlainTextLanguage = (language: string): boolean => plainTextLanguages.has(language.trim().toLowerCase());-// The hljs theme stylesheet is a document-global singleton, so it is managed here-// (keyed by theme) rather than by each rendered code block.-export const ensureHighlightTheme = async (isDarkTheme: boolean): Promise<void> => {- const themeKey = isDarkTheme ? "dark" : "light";- if (injectedThemeKey === themeKey) {- return;- }- injectedThemeKey = themeKey;-- try {- const cssModule = isDarkTheme- ? await import("highlight.js/styles/github-dark-dimmed.css?inline")- : await import("highlight.js/styles/github.css?inline");- if (injectedThemeKey !== themeKey) {- return;+/** Keep structural punctuation plain without adding decoration spans. */+const codeTheme = (theme: Theme): Theme => {+ const foreground = theme.style["editor.foreground"] ?? theme.style.text;+ const syntax = { ...theme.style.syntax };+ for (const name of Object.keys(syntax)) {+ if (name === "operator" || name === "keyword.operator" || name === "punctuation" || name.startsWith("punctuation.")) {+ syntax[name] = { color: foreground };}-- document.querySelector("style[data-hljs-theme]")?.remove();- const style = document.createElement("style");- style.textContent = cssModule.default;- style.setAttribute("data-hljs-theme", themeKey);- document.head.appendChild(style);- } catch (error) {- injectedThemeKey = undefined;- console.warn("Failed to load highlight.js theme:", error);}+ return { ...theme, style: { ...theme.style, syntax } };};-const loadCore = (): Promise<HLJSApi> => {- corePromise ??= import("highlight.js/lib/core").then((module) => module.default);- return corePromise;-};--const loadCommonLanguage = (language: CommonLanguage): Promise<HLJSApi> => {- const existingPromise = registeredLanguagePromises.get(language);- if (existingPromise) {- return existingPromise;+/** Load the Wasm engine and just the two themes we use, only when a code block needs them. */+const loadHighlights = () =>+ Promise.all([+ import("@pierre/highlights"),+ import("@pierre/highlights/themes/github-light"),+ import("@pierre/highlights/themes/github-dark-dimmed"),+ ]).then(([{ codeToTokens, isSupportedLanguage }, light, dark]) => ({+ codeToTokens,+ isSupportedLanguage,+ themes: { light: codeTheme(light.default), dark: codeTheme(dark.default) },+ }));++let highlightsPromise: ReturnType<typeof loadHighlights> | undefined;++/** Loads a token highlighter. Unsupported languages or unavailable Wasm remain plain text. */+export const loadHighlighter = async (language: string): Promise<void> => {+ const key = language.trim().toLowerCase();+ if (plainTextLanguages.has(key) || readyHighlighters.has(key)) return;++ // The editor follows the document's color-scheme. Older browsers keep plain code.+ const supportsThemeColors = typeof CSS === "undefined" || !CSS.supports || CSS.supports("color", "light-dark(black, white)");+ const highlights = supportsThemeColors ? await (highlightsPromise ??= loadHighlights()).catch(() => null) : null;+ if (highlights?.isSupportedLanguage(key)) {+ readyHighlighters.set(key, (code) => {+ const { tokens, fg } = highlights.codeToTokens(code, { lang: key, themes: highlights.themes, defaultColor: "light-dark()" });+ return tokens.flatMap((line) =>+ line+ .filter((token) => token.htmlStyle && !(Object.keys(token.htmlStyle).length === 1 && token.htmlStyle.color === fg))+ .map((token) => ({+ from: token.offset,+ to: token.offset + token.content.length,+ class: "memos-code-token",+ attributes: {+ style: Object.entries(token.htmlStyle ?? {})+ .map(([name, value]) => `${name}:${value}`)+ .join(";"),+ },+ })),+ );+ });+ return;}- const languagePromise = Promise.all([loadCore(), commonLanguageLoaders[language]()]).then(([hljs, languageModule]) => {- if (!hljs.getLanguage(language)) {- hljs.registerLanguage(language, languageModule.default);- }- return hljs;- });- registeredLanguagePromises.set(language, languagePromise);- return languagePromise;-};--const loadFullBuild = (): Promise<HLJSApi> => {- fullBuildPromise ??= import("highlight.js").then((module) => module.default);- return fullBuildPromise;-};--/** Highlighters ready for use, by normalized language; null when the language is unknown to highlight.js. */-const readyHighlighters = new Map<string, HLJSApi | null>();--const canonicalLanguageOf = (normalizedLanguage: string): CommonLanguage | undefined =>- languageAliases[normalizedLanguage] ??- (Object.hasOwn(commonLanguageLoaders, normalizedLanguage) ? (normalizedLanguage as CommonLanguage) : undefined);--/** Loads what highlighting `language` needs. Afterwards, highlightLoadedCode returns a result for it. */-export const loadHighlighter = async (language: string): Promise<void> => {- const normalizedLanguage = language.trim().toLowerCase();- if (isPlainTextLanguage(normalizedLanguage) || readyHighlighters.has(normalizedLanguage)) return;- const canonicalLanguage = canonicalLanguageOf(normalizedLanguage);- const hljs = canonicalLanguage ? await loadCommonLanguage(canonicalLanguage) : await loadFullBuild();- readyHighlighters.set(normalizedLanguage, hljs.getLanguage(canonicalLanguage ?? normalizedLanguage) ? hljs : null);+ readyHighlighters.set(key, () => []);};-/**- * Highlighted HTML for code whose language is loaded, the escaped code for plain text- * and unknown languages, or undefined while the language still has to load.- */-export const highlightLoadedCode = (code: string, language: string): string | undefined => {- const normalizedLanguage = language.trim().toLowerCase();- if (isPlainTextLanguage(normalizedLanguage)) return escapeHTML(code);- if (!readyHighlighters.has(normalizedLanguage)) return undefined;- const hljs = readyHighlighters.get(normalizedLanguage);- if (!hljs) return escapeHTML(code);- return hljs.highlight(code, { language: canonicalLanguageOf(normalizedLanguage) ?? normalizedLanguage }).value;+/** Text decorations, or undefined until the language has loaded. Plain text needs no decorations. */+export const highlightLoadedCode = (code: string, language: string): Decoration[] | undefined => {+ const key = language.trim().toLowerCase();+ if (!code || plainTextLanguages.has(key)) return [];+ return readyHighlighters.get(key)?.(code);};diff --git a/web/src/lib/memo-markdown.ts b/web/src/lib/memo-markdown.tsindex 87600f1ee..bfa20e6fa 100644--- a/web/src/lib/memo-markdown.ts+++ b/web/src/lib/memo-markdown.ts@@ -21,7 +21,7 @@ import {type Theme,tags,} from "@usememos/editor";-import { ensureHighlightTheme, highlightLoadedCode, loadHighlighter } from "@/lib/code-highlight";+import { highlightLoadedCode, loadHighlighter } from "@/lib/code-highlight";import { colorToHex } from "@/lib/color";import { headingClasses, markdownStyles, mentionStyles, tagStyles } from "@/lib/markdownStyles";import { findTagMetadata } from "@/lib/tag";@@ -65,29 +65,8 @@ const highlightedEditors = new Set<import("@usememos/editor").Editor>();/** Languages loading now, so each load redecorates the editors once, however many blocks wait for it. */const loadingLanguages = new Set<string>();-/** highlight.js output as decorations: one range per highlighted span, classes and all. */-function highlightRanges(html: string): { from: number; to: number; class: string }[] {- const template = document.createElement("template");- template.innerHTML = html;- const ranges: { from: number; to: number; class: string }[] = [];- let offset = 0;- const walk = (node: Node) => {- for (const child of node.childNodes) {- if (child.nodeType === Node.TEXT_NODE) {- offset += child.textContent?.length ?? 0;- } else if (child instanceof Element) {- const from = offset;- walk(child);- if (child.className && offset > from) ranges.push({ from, to: offset, class: child.className });- }- }- };- walk(template.content);- return ranges;-}-/**- * Syntax highlighting for code blocks with highlight.js, while editing and when read-only.+ * Token-based syntax highlighting for code blocks, while editing and when read-only.* A language loads the first time it is needed; the editors then highlight again.*/const codeHighlight = (): Extension => ({@@ -98,9 +77,8 @@ const codeHighlight = (): Extension => ({},codeDecorations: ({ text, language }) => {if (!language.trim() || !text) return [];- void ensureHighlightTheme(document.documentElement.style.colorScheme === "dark");- const html = highlightLoadedCode(text, language);- if (html === undefined) {+ const ranges = highlightLoadedCode(text, language);+ if (ranges === undefined) {const key = language.trim().toLowerCase();if (!loadingLanguages.has(key)) {loadingLanguages.add(key);@@ -113,7 +91,7 @@ const codeHighlight = (): Extension => ({}return [];}- return highlightRanges(html);+ return ranges;},});diff --git a/web/tests/code-highlight.test.ts b/web/tests/code-highlight.test.tsnew file mode 100644index 000000000..64cca51c4--- /dev/null+++ b/web/tests/code-highlight.test.ts@@ -0,0 +1,67 @@+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";++beforeEach(() => {+ vi.resetModules();+ vi.doUnmock("@pierre/highlights");+});++afterEach(() => vi.unstubAllGlobals());++describe("code highlighting", () => {+ it("leaves plain text undecorated and loads normalized language aliases on demand", async () => {+ const { highlightLoadedCode, loadHighlighter } = await import("@/lib/code-highlight");+ expect(highlightLoadedCode("const value = 1;", "js")).toBeUndefined();+ for (const language of ["", "plain", "plaintext", "text", "txt"]) {+ expect(highlightLoadedCode("<script>&</script>", language)).toEqual([]);+ }+ await loadHighlighter(" JS ");+ const ranges = highlightLoadedCode("const value = 1;", "js");+ expect(ranges?.[0]).toMatchObject({ from: 0, class: "memos-code-token" });+ expect(ranges?.[0].attributes?.style).toContain("light-dark(");+ });++ it("uses source UTF-16 offsets across Unicode, CRLF and multiline tokens", async () => {+ const { highlightLoadedCode, loadHighlighter } = await import("@/lib/code-highlight");+ await loadHighlighter("typescript");+ const code = 'const emoji = "πδΈζ";\r\n/* first\nsecond */\nconst tail = 2;';+ const ranges = highlightLoadedCode(code, "typescript") ?? [];+ const slices = ranges.map(({ from, to }) => code.slice(from, to));+ expect(slices).toContain('"πδΈζ"');+ expect(slices).toContain("/* first");+ expect(slices).toContain("second */");+ expect(ranges.find(({ from }) => from === code.lastIndexOf("const"))?.class).toBe("memos-code-token");+ expect(ranges.every(({ from, to }) => from >= 0 && to > from && to <= code.length)).toBe(true);+ });++ it("leaves ordinary text and structural punctuation to the editor's foreground", async () => {+ const { highlightLoadedCode, loadHighlighter } = await import("@/lib/code-highlight");+ await loadHighlighter("js");+ const code = 'const message = "hello";';+ const ranges = highlightLoadedCode(code, "js") ?? [];+ expect(ranges.map(({ from, to }) => code.slice(from, to))).toEqual(["const ", '"hello"']);+ });++ it("leaves unsupported languages plain", async () => {+ const { highlightLoadedCode, loadHighlighter } = await import("@/lib/code-highlight");+ await loadHighlighter("brainfuck");+ expect(highlightLoadedCode("++[>++<-]", "brainfuck")).toEqual([]);+ await loadHighlighter("not-a-language");+ expect(highlightLoadedCode("<script>&</script>", "not-a-language")).toEqual([]);+ });++ it("keeps plain code when the Wasm module cannot initialize", async () => {+ vi.doMock("@pierre/highlights", () => {+ throw new Error("WebAssembly unavailable");+ });+ const { highlightLoadedCode, loadHighlighter } = await import("@/lib/code-highlight");+ await loadHighlighter("js");+ expect(highlightLoadedCode("const value = 1;", "js")).toEqual([]);+ });++ it("keeps plain code when light-dark colors are unavailable", async () => {+ vi.stubGlobal("CSS", { supports: () => false });+ const { highlightLoadedCode, loadHighlighter } = await import("@/lib/code-highlight");+ await loadHighlighter("js");+ expect(highlightLoadedCode("const value = 1;", "js")).toEqual([]);+ });+});diff --git a/web/tests/memo-content-view.test.tsx b/web/tests/memo-content-view.test.tsxindex 9339f2e43..f59d2599d 100644--- a/web/tests/memo-content-view.test.tsx+++ b/web/tests/memo-content-view.test.tsx@@ -1,8 +1,10 @@import { act, fireEvent, render, screen, waitFor } from "@testing-library/react";+import type { Editor } from "@usememos/editor";+import { createRef } from "react";import { MemoryRouter } from "react-router-dom";import { beforeEach, describe, expect, it, vi } from "vitest";+import { MemoMarkdown } from "@/components/MemoEditor/MemoMarkdown";import { MemoContentView } from "@/components/MemoEditor/view/MemoContentView";-import { highlightLoadedCode, loadHighlighter } from "@/lib/code-highlight";const state = vi.hoisted(() => ({linkPreview: true,@@ -126,13 +128,33 @@ describe("code blocks", () => {expect(container.querySelector(".mbe-code .mbe-code-copy")).toBeInTheDocument();expect(container.querySelector<HTMLInputElement>(".mbe-code-lang")?.value).toBe("js");expect(container.querySelector(".mbe-code code")).toHaveTextContent("const value = 1;");- await waitFor(() => expect(container.querySelector(".mbe-code code .hljs-keyword")).toHaveTextContent("const"));+ await waitFor(() => expect(container.querySelector(".mbe-code code .memos-code-token")).toHaveTextContent("const"));});- it("escapes plain code and highlights common languages", async () => {- expect(highlightLoadedCode('<script data-test="x">&</script>', "")).toBe("<script data-test="x">&</script>");- await loadHighlighter("js");- expect(highlightLoadedCode("const value = 1;", "js")).toContain("hljs-keyword");+ it("keeps HTML literal in highlighted and plain code", async () => {+ const code = '<script data-test="x">&</script>';+ const content = ["html", "", "not-a-language"].map((language) => `\`\`\`${language}\n${code}\n\`\`\``).join("\n\n");+ const { container } = renderMemo(content);+ await waitFor(() => expect(container.querySelector(".memos-code-token")).not.toBeNull());+ expect(container.querySelectorAll(".mbe-code code")).toHaveLength(3);+ for (const block of container.querySelectorAll(".mbe-code code")) expect(block.textContent).toBe(code);+ expect(container.querySelector("script")).toBeNull();+ expect(container.querySelector("[data-test]")).toBeNull();+ });++ it("highlights changed code while editing without changing the stored Markdown", async () => {+ const editorRef = createRef<Editor>();+ const onChange = vi.fn();+ const content = '```ts\nconst message = "πδΈζ";\n```';+ const { container, rerender } = render(<MemoMarkdown value={content} editable editorRef={editorRef} onChange={onChange} />);+ await waitFor(() => expect(container.querySelector(".memos-code-token")).not.toBeNull());+ expect(editorRef.current?.getMarkdown()).toBe(content);++ const updated = content.replace("message", "greeting");+ rerender(<MemoMarkdown value={updated} editable editorRef={editorRef} onChange={onChange} />);+ expect(container.querySelector(".mbe-code code")).toHaveTextContent('const greeting = "πδΈζ";');+ expect(editorRef.current?.getMarkdown()).toBe(updated);+ expect(onChange).not.toHaveBeenCalled();});});