Skip to content

Beautiful, AI-native markdown IDE and LLM wiki

Keep letter-forming marks in link and heading slugs (PRD-9406) (#6416)

Andrew Mikofalvy34888384dc4a
diff --git a/.changeset/slug-keeps-letter-marks.md b/.changeset/slug-keeps-letter-marks.md
new file mode 100644
index 000000000..d963c7a3c
--- /dev/null
+++ b/.changeset/slug-keeps-letter-marks.md
@@ -0,0 +1,5 @@
+---
+"@inkeep/open-knowledge": patch
+---
+
+Links and heading anchors in Japanese, Hindi, Thai and other scripts whose marks change a letter keep those marks, so `[[ガラス]]` no longer opens a page named カラス. Links the editor saved earlier without the marks, such as `#カラス` for `## ガラス`, need the marks restored.
diff --git a/docs/content/features/editor.mdx b/docs/content/features/editor.mdx
index 26a65073e..7bb0ceea5 100644
--- a/docs/content/features/editor.mdx
+++ b/docs/content/features/editor.mdx
@@ -98,7 +98,7 @@ Links work the way they do in the editors you already know:
- **Type a URL, then press space or enter.** A full URL (`https://…`, `www.…`, or an email address) becomes a link when you press space or enter after it. Plain words and filenames like `AGENTS.md` are left alone, and one `Cmd+Z` turns a conversion back into plain text.
- **Paste or drag in a URL.** Pasted (or dropped) on its own, it becomes a link. Pasted over selected text, it links that text instead — the text stays, the URL becomes its destination. `Cmd+Shift+V` pastes plain, with no linking.
- **Type `[text](url)`.** Closing the parenthesis converts the markdown shorthand into a real link; `Cmd+Z` restores the literal text.
-- **Link to a section.** Append `#` and the heading's slug to a doc link — `[text](./spec.md#data-model-v2)` — and the WYSIWYG editor opens the doc scrolled to that heading (the source editor ignores the fragment and opens at the top). The slug is the heading text in lowercase with accents dropped, every run of characters that aren't letters or digits replaced by one hyphen, and hyphens trimmed from both ends: `## Data model: v2` becomes `data-model-v2`, `## FAQ:` becomes `faq`, and a second identical heading in the same doc gets `-1`. Agents write the same form in chat, so a reply about one part of a long spec can take you straight to it.
+- **Link to a section.** Append `#` and the heading's slug to a doc link — `[text](./spec.md#data-model-v2)` — and the WYSIWYG editor opens the doc scrolled to that heading (the source editor ignores the fragment and opens at the top). The slug is the heading text in lowercase with accents on Latin, Greek, Cyrillic, Arabic and Hebrew letters dropped (other scripts keep their marks, so `## ガラス` stays `ガラス`), every run of characters that aren't letters or digits replaced by one hyphen, and hyphens trimmed from both ends: `## Data model: v2` becomes `data-model-v2`, `## FAQ:` becomes `faq`, and a second identical heading in the same doc gets `-1`. Agents write the same form in chat, so a reply about one part of a long spec can take you straight to it.
- **Press `Cmd+K` with text selected.** The link popover opens with the URL field focused, pre-filled from your clipboard when it holds a URL. With the cursor inside an existing link, `Cmd+K` opens that link for editing. With no selection, `Cmd+K` keeps opening the command palette, as everywhere else. To reach the command palette while text is selected, use `Cmd+P` (`Ctrl+P` on Windows / Linux), which always opens it.
For legacy wiki links and ambiguous names, see [how links resolve and what to review after upgrading](../reference/core-concepts.md#review-links-after-upgrading).
diff --git a/docs/content/reference/core-concepts.md b/docs/content/reference/core-concepts.md
index c2c69ec09..34c548785 100644
--- a/docs/content/reference/core-concepts.md
+++ b/docs/content/reference/core-concepts.md
@@ -94,6 +94,8 @@ A wiki link such as `[[notes/beta.md]]` first checks for a document whose path i
After the original exact-path check, OpenKnowledge fully resolves the suffix-stripped spelling: exact path, case- and punctuation-insensitive slug, folder `index` page (or its same-named page), then matching basename when the link has no slash. Only if that finds nothing does it try the original spelling's fuzzy, folder-index, and basename fallbacks. Without a Markdown suffix, those checks run once on the original spelling. If multiple paths have the same fuzzy slug or basename, the path smallest in UTF-16 code-unit order wins. For example, with `reports/index.md` and `reports-md.md`, `[[reports.md]]` opens `reports/index` before trying the fuzzy match for `reports-md`. An exact, case-matching path avoids a fuzzy tie.
+Slugs keep the combining marks that change a letter outside Latin, Greek, Cyrillic, Arabic and Hebrew text, such as Japanese voicing marks and Hindi vowel signs. Earlier versions dropped them, and the editor saved that form into some links: a section link to `## ガラス` picked from the heading list became `#カラス`, and a link typed as `ガラス` before that page existed became `[[カラス|ガラス]]`. Such links now open the document at the top or show as missing, or, when a heading or page with the unmarked spelling exists, open that one instead; put the marks back in the link.
+
Every exact-path check, in every link form except wiki asset links and embeds (see below), compares names with one identity rule. Document and folder names match case-sensitively. File names (images, PDFs, and other non-markdown files) match case-insensitively. Names that differ only in Unicode normalization are the same name for every kind: `é` stored as one character (NFC) or as `e` plus a combining accent (NFD) matches either spelling, so a link typed on Linux opens a file whose name macOS stored decomposed. Characters that only compatibility normalization would merge, such as the `fi` ligature and `fi`, stay distinct. When two stored names are equivalent, an exact spelling wins; otherwise the name smallest in UTF-16 code-unit order wins.
Markdown and wiki asset links treat folder case differently. A Markdown asset link such as `![cat](./Images/cat.png)` matches each parent folder case-sensitively and only the file name case-insensitively, so it no longer finds `images/cat.png`. A wiki embed such as `![[Images/cat.png]]` still matches its whole path case-insensitively. On macOS and Windows, whose file systems ignore case, a Markdown asset link whose folder case differs from the folder on disk can show as broken after upgrading; correct the folder spelling in the link.
diff --git a/knip.config.ts b/knip.config.ts
index eb9727af8..e43118957 100644
--- a/knip.config.ts
+++ b/knip.config.ts
@@ -155,15 +155,11 @@ export default {
'src/shadow-fixture-launch.test-helper.mjs',
],
project: 'src/**',
- ignoreDependencies: [
- '@types/shell-quote',
- ],
+ ignoreDependencies: ['@types/shell-quote'],
},
'packages/cli': {
entry: ['src/**/*.test.ts', 'scripts/*.ts', 'tests/**/*.ts', 'src/parse-worker.ts'],
- ignoreDependencies: [
- 'yjs',
- ],
+ ignoreDependencies: ['yjs'],
},
'packages/desktop': {
entry: [
diff --git a/packages/app/src/editor/TiptapEditor.tsx b/packages/app/src/editor/TiptapEditor.tsx
index e80cef98e..b22290be2 100644
--- a/packages/app/src/editor/TiptapEditor.tsx
+++ b/packages/app/src/editor/TiptapEditor.tsx
@@ -52,6 +52,7 @@ import { useSingleFileMode } from '@/lib/single-file-mode';
import { useIdentity } from '../presence/identity';
import { registerEditor, unregisterEditor } from './active-editor';
import { changedRangeIsOnScreen } from './agent-follow-scroll';
+import { findAnchorTarget } from './anchor-target';
import { applyLintFixes } from './apply-lint-fix.ts';
import { getAwarenessHeartbeat } from './awareness-heartbeat-runtime';
import { buildAwarenessUser } from './awareness-user';
@@ -983,18 +984,9 @@ const TiptapEditorChrome: FC<TiptapEditorChromeProps> = ({
attempts = 0;
}
- function findAnchorTarget(anchor: string): HTMLElement | null {
- const realView = getEditorView(editor);
- if (!realView) return null;
- const escapedAnchor = CSS.escape(anchor);
- return (
- realView.dom.querySelector<HTMLElement>(`#${escapedAnchor}`) ??
- realView.dom.querySelector<HTMLElement>(`[data-mirror-source-id="${escapedAnchor}"]`)
- );
- }
-
function scrollAnchorIntoView(anchor: string): boolean {
- const el = findAnchorTarget(anchor);
+ const realView = getEditorView(editor);
+ const el = realView ? findAnchorTarget(realView.dom, anchor) : null;
if (!el) return false;
return runScrollNavigation(docName, 'deep-link', () => {
el.scrollIntoView({ behavior: 'smooth', block: 'start' });
diff --git a/packages/app/src/editor/anchor-target.dom.test.ts b/packages/app/src/editor/anchor-target.dom.test.ts
new file mode 100644
index 000000000..26a890660
--- /dev/null
+++ b/packages/app/src/editor/anchor-target.dom.test.ts
@@ -0,0 +1,53 @@
+import { afterAll, afterEach, beforeAll, describe, expect, test, vi } from 'vitest';
+import { findAnchorTarget } from './anchor-target';
+
+function mount(html: string): HTMLElement {
+ const root = document.createElement('div');
+ root.innerHTML = html;
+ document.body.append(root);
+ return root;
+}
+
+beforeAll(() => {
+ if (typeof globalThis.CSS?.escape !== 'function') {
+ vi.stubGlobal('CSS', { escape: (value: string) => value.replace(/[^\w-]/g, (c) => `\\${c}`) });
+ }
+});
+
+afterAll(() => {
+ vi.unstubAllGlobals();
+});
+
+afterEach(() => {
+ document.body.replaceChildren();
+});
+
+describe('findAnchorTarget', () => {
+ test('finds a heading by its exact id', () => {
+ const root = mount('<h2 id="data-model-v2">Data model: v2</h2>');
+ expect(findAnchorTarget(root, 'data-model-v2')?.textContent).toBe('Data model: v2');
+ });
+
+ test('finds a mirror source by its source id', () => {
+ const root = mount('<div data-mirror-source-id="block-1">Mirrored</div>');
+ expect(findAnchorTarget(root, 'block-1')?.textContent).toBe('Mirrored');
+ });
+
+ test('finds a Korean heading from an anchor written as decomposed jamo', () => {
+ const root = mount('<h2 id="한국어">한국어</h2>');
+ expect(findAnchorTarget(root, '한국어'.normalize('NFD'))?.textContent).toBe('한국어');
+ });
+
+ test('finds a heading from an anchor that spells its text instead of its slug', () => {
+ const root = mount('<h2 id="data-model-v2">Data model: v2</h2><h2 id="ガラス">ガラス</h2>');
+ expect(findAnchorTarget(root, 'Data model: v2')?.textContent).toBe('Data model: v2');
+ expect(findAnchorTarget(root, 'ガラス'.normalize('NFD'))?.textContent).toBe('ガラス');
+ });
+
+ test('an exact id wins over a slug match, and a missing anchor finds nothing', () => {
+ const root = mount('<h2 id="Notes">Exact</h2><h2 id="notes">Slugged</h2>');
+ expect(findAnchorTarget(root, 'Notes')?.textContent).toBe('Exact');
+ expect(findAnchorTarget(root, 'missing')).toBeNull();
+ expect(findAnchorTarget(root, '!!!')).toBeNull();
+ });
+});
diff --git a/packages/app/src/editor/anchor-target.ts b/packages/app/src/editor/anchor-target.ts
new file mode 100644
index 000000000..3cd4bfbb4
--- /dev/null
+++ b/packages/app/src/editor/anchor-target.ts
@@ -0,0 +1,14 @@
+import { toWikiLinkSlug } from '@inkeep/open-knowledge-core/utils/slug';
+
+function findById(root: ParentNode, id: string): HTMLElement | null {
+ return root.querySelector<HTMLElement>(`#${CSS.escape(id)}`);
+}
+
+export function findAnchorTarget(root: ParentNode, anchor: string): HTMLElement | null {
+ const exact =
+ findById(root, anchor) ??
+ root.querySelector<HTMLElement>(`[data-mirror-source-id="${CSS.escape(anchor)}"]`);
+ if (exact) return exact;
+ const slug = toWikiLinkSlug(anchor);
+ return slug && slug !== anchor ? findById(root, slug) : null;
+}
diff --git a/packages/app/src/editor/internal-link-helpers.anchor.dom.test.ts b/packages/app/src/editor/internal-link-helpers.anchor.dom.test.ts
new file mode 100644
index 000000000..1a7f392a0
--- /dev/null
+++ b/packages/app/src/editor/internal-link-helpers.anchor.dom.test.ts
@@ -0,0 +1,40 @@
+import { afterAll, afterEach, beforeAll, describe, expect, test, vi } from 'vitest';
+import { navigateToMarkdownTarget } from './internal-link-helpers';
+
+beforeAll(() => {
+ if (typeof globalThis.CSS?.escape !== 'function') {
+ vi.stubGlobal('CSS', { escape: (value: string) => value.replace(/[^\w-]/g, (c) => `\\${c}`) });
+ }
+});
+
+afterAll(() => {
+ vi.unstubAllGlobals();
+});
+
+afterEach(() => {
+ document.body.replaceChildren();
+ vi.restoreAllMocks();
+});
+
+function mountHeading(id: string): ReturnType<typeof vi.fn> {
+ const heading = document.createElement('h2');
+ heading.id = id;
+ const scrollIntoView = vi.fn();
+ heading.scrollIntoView = scrollIntoView;
+ document.body.append(heading);
+ return scrollIntoView;
+}
+
+describe('navigateToMarkdownTarget — in-document anchors', () => {
+ test('scrolls to the heading whose id is the anchor', () => {
+ const scrollIntoView = mountHeading('install');
+ navigateToMarkdownTarget({ kind: 'anchor', anchor: 'install' }, '#/docs/guide');
+ expect(scrollIntoView).toHaveBeenCalledOnce();
+ });
+
+ test('scrolls to a Korean heading from an anchor saved as decomposed jamo', () => {
+ const scrollIntoView = mountHeading('한국어');
+ navigateToMarkdownTarget({ kind: 'anchor', anchor: '한국어'.normalize('NFD') }, '#/docs/guide');
+ expect(scrollIntoView).toHaveBeenCalledOnce();
+ });
+});
diff --git a/packages/app/src/editor/internal-link-helpers.ts b/packages/app/src/editor/internal-link-helpers.ts
index 5c39bc01a..ba1440d1f 100644
--- a/packages/app/src/editor/internal-link-helpers.ts
+++ b/packages/app/src/editor/internal-link-helpers.ts
@@ -10,6 +10,7 @@ import {
} from '@inkeep/open-knowledge-core/utils/link-targets';
import { hashFromAssetPath, hashFromDocName } from '../lib/doc-hash';
import { openExternalUrl } from '../lib/external-link';
+import { findAnchorTarget } from './anchor-target';
import { dispatchAssetClick } from './asset-dispatch';
import { isSafeNavigationUrl } from './safe-navigation-url';
@@ -110,7 +111,7 @@ function navigateToAnchorHref(anchor: string, locationHash = window.location.has
const currentDocName = getCurrentDocNameFromHash(locationHash);
if (!currentDocName) return;
- const element = document.getElementById(anchor);
+ const element = findAnchorTarget(document, anchor);
if (element) {
element.scrollIntoView({ behavior: 'smooth', block: 'start' });
}
diff --git a/packages/app/tests/integration/bug3-source-mode-writeback.test.ts b/packages/app/tests/integration/bug3-source-mode-writeback.test.ts
index 4549c74b3..a50b00123 100644
--- a/packages/app/tests/integration/bug3-source-mode-writeback.test.ts
+++ b/packages/app/tests/integration/bug3-source-mode-writeback.test.ts
@@ -1,4 +1,3 @@
-
import { readFileSync } from 'node:fs';
import { join } from 'node:path';
import { setTimeout as wait } from 'node:timers/promises';
diff --git a/packages/app/tests/integration/indented-jsx-byte-sacred.test.ts b/packages/app/tests/integration/indented-jsx-byte-sacred.test.ts
index 4fc2c11f9..6fcdc23f2 100644
--- a/packages/app/tests/integration/indented-jsx-byte-sacred.test.ts
+++ b/packages/app/tests/integration/indented-jsx-byte-sacred.test.ts
@@ -1,4 +1,3 @@
-
import { setTimeout as wait } from 'node:timers/promises';
import { normalizeBridge } from '@inkeep/open-knowledge-core';
import { updateYFragment } from '@tiptap/y-tiptap';
diff --git a/packages/app/tests/integration/interior-edit-freshness.test.ts b/packages/app/tests/integration/interior-edit-freshness.test.ts
index 9f2f6feb6..40b5cdfd6 100644
--- a/packages/app/tests/integration/interior-edit-freshness.test.ts
+++ b/packages/app/tests/integration/interior-edit-freshness.test.ts
@@ -1,4 +1,3 @@
-
import { setTimeout as wait } from 'node:timers/promises';
import { updateYFragment, yXmlFragmentToProseMirrorRootNode } from '@tiptap/y-tiptap';
import { afterAll, beforeAll, describe, expect, test } from 'vitest';
diff --git a/packages/app/tests/integration/keystroke-cadence-danger-space.test.ts b/packages/app/tests/integration/keystroke-cadence-danger-space.test.ts
index 2a085742b..93666af70 100644
--- a/packages/app/tests/integration/keystroke-cadence-danger-space.test.ts
+++ b/packages/app/tests/integration/keystroke-cadence-danger-space.test.ts
@@ -1,4 +1,3 @@
-
import { setTimeout as wait } from 'node:timers/promises';
import { updateYFragment, yXmlFragmentToProseMirrorRootNode } from '@tiptap/y-tiptap';
import { afterAll, beforeAll, describe, expect, test } from 'vitest';
diff --git a/packages/app/tests/integration/producer-guard-wiring.test.ts b/packages/app/tests/integration/producer-guard-wiring.test.ts
index 878d5196b..867598d34 100644
--- a/packages/app/tests/integration/producer-guard-wiring.test.ts
+++ b/packages/app/tests/integration/producer-guard-wiring.test.ts
@@ -1,4 +1,3 @@
-
import { setTimeout as wait } from 'node:timers/promises';
import { updateYFragment, yXmlFragmentToProseMirrorRootNode } from '@tiptap/y-tiptap';
import { afterAll, beforeAll, describe, expect, test } from 'vitest';
diff --git a/packages/app/tests/integration/qa-039-reconciliation-counters-http.test.ts b/packages/app/tests/integration/qa-039-reconciliation-counters-http.test.ts
index 4ada3cf26..0970ac12f 100644
--- a/packages/app/tests/integration/qa-039-reconciliation-counters-http.test.ts
+++ b/packages/app/tests/integration/qa-039-reconciliation-counters-http.test.ts
@@ -1,4 +1,3 @@
-
import { randomUUID } from 'node:crypto';
import { afterEach, describe, expect, test } from 'vitest';
import { agentWriteMd, createTestServer, type TestServer } from './test-harness';
diff --git a/packages/app/tests/integration/qa-045-restore-after-gc.test.ts b/packages/app/tests/integration/qa-045-restore-after-gc.test.ts
index cbe2b705f..b632185e1 100644
--- a/packages/app/tests/integration/qa-045-restore-after-gc.test.ts
+++ b/packages/app/tests/integration/qa-045-restore-after-gc.test.ts
@@ -1,4 +1,3 @@
-
import { execFileSync } from 'node:child_process';
import { randomUUID } from 'node:crypto';
import { join } from 'node:path';
diff --git a/packages/app/tests/integration/qa-050-rollback-undo-shipped-path.test.ts b/packages/app/tests/integration/qa-050-rollback-undo-shipped-path.test.ts
index 89bc9b4d9..c530ce4ba 100644
--- a/packages/app/tests/integration/qa-050-rollback-undo-shipped-path.test.ts
+++ b/packages/app/tests/integration/qa-050-rollback-undo-shipped-path.test.ts
@@ -1,4 +1,3 @@
-
import { randomUUID } from 'node:crypto';
import { afterAll, afterEach, beforeAll, describe, expect, test } from 'vitest';
import type * as Y from 'yjs';
diff --git a/packages/app/tests/integration/undo-after-rollback.test.ts b/packages/app/tests/integration/undo-after-rollback.test.ts
index a78aeb0e4..d3767adc9 100644
--- a/packages/app/tests/integration/undo-after-rollback.test.ts
+++ b/packages/app/tests/integration/undo-after-rollback.test.ts
@@ -1,4 +1,3 @@
-
import { ROLLBACK_ORIGIN } from '@inkeep/open-knowledge-server';
import { afterAll, beforeAll, describe, expect, test } from 'vitest';
import * as Y from 'yjs';
diff --git a/packages/app/tests/integration/undo-recycle-reset.test.ts b/packages/app/tests/integration/undo-recycle-reset.test.ts
index e7cc69c3f..b076f955c 100644
--- a/packages/app/tests/integration/undo-recycle-reset.test.ts
+++ b/packages/app/tests/integration/undo-recycle-reset.test.ts
@@ -1,4 +1,3 @@
-
import { writeFileSync } from 'node:fs';
import { join } from 'node:path';
import { setTimeout as wait } from 'node:timers/promises';
diff --git a/packages/app/tests/integration/undo-rederive-survival.test.ts b/packages/app/tests/integration/undo-rederive-survival.test.ts
index 9ddcd0730..7935e35f8 100644
--- a/packages/app/tests/integration/undo-rederive-survival.test.ts
+++ b/packages/app/tests/integration/undo-rederive-survival.test.ts
@@ -1,4 +1,3 @@
-
import { prependFrontmatter } from '@inkeep/open-knowledge-core';
import { Plugin, PluginKey } from '@tiptap/pm/state';
import { updateYFragment, yXmlFragmentToProseMirrorRootNode } from '@tiptap/y-tiptap';
diff --git a/packages/core/src/bridge/compose.ts b/packages/core/src/bridge/compose.ts
index 2a597ec3d..f4cc8a286 100644
--- a/packages/core/src/bridge/compose.ts
+++ b/packages/core/src/bridge/compose.ts
@@ -1,4 +1,3 @@
-
import { prependFrontmatter, stripFrontmatter } from '../extensions/frontmatter.ts';
export type ComposeAdjustment = 'none' | 'body' | 'frontmatter' | 'unresolved';
diff --git a/packages/core/src/bridge/doc-boundary-space.ts b/packages/core/src/bridge/doc-boundary-space.ts
index ffd8fce06..e7e63f43d 100644
--- a/packages/core/src/bridge/doc-boundary-space.ts
+++ b/packages/core/src/bridge/doc-boundary-space.ts
@@ -1,4 +1,3 @@
-
import { FRONTMATTER_RE, stripFrontmatter } from '../extensions/frontmatter.ts';
import { carriedEdgeEmpties, type DocEdgeEmpties } from '../markdown/doc-edge-blank-runs.ts';
diff --git a/packages/core/src/markdown/comment-promoter.ts b/packages/core/src/markdown/comment-promoter.ts
index fb0ba5a7c..fc267cbea 100644
--- a/packages/core/src/markdown/comment-promoter.ts
+++ b/packages/core/src/markdown/comment-promoter.ts
@@ -1,4 +1,3 @@
-
import type { Nodes, Paragraph, PhrasingContent, Root, RootContent, Text } from 'mdast';
import { SKIP, visit } from 'unist-util-visit';
import type { VFile } from 'vfile';
diff --git a/packages/core/src/markdown/merged-walker.ts b/packages/core/src/markdown/merged-walker.ts
index ae8cb390b..861003fb4 100644
--- a/packages/core/src/markdown/merged-walker.ts
+++ b/packages/core/src/markdown/merged-walker.ts
@@ -1,4 +1,3 @@
-
import type { Nodes, Parent, Root } from 'mdast';
import { SKIP, visit } from 'unist-util-visit';
import type { VFile } from 'vfile';
diff --git a/packages/core/src/markdown/safe-url.ts b/packages/core/src/markdown/safe-url.ts
index 8be632ba4..4d3250d3a 100644
--- a/packages/core/src/markdown/safe-url.ts
+++ b/packages/core/src/markdown/safe-url.ts
@@ -1,4 +1,3 @@
-
export const SAFE_URL_SCHEMES = ['https', 'http', 'mailto', 'tel', 'ftp', 'sms'] as const;
const SCHEME_ALT = SAFE_URL_SCHEMES.map((s) => `${s}:`).join('|');
diff --git a/packages/core/src/utils/slug.test.ts b/packages/core/src/utils/slug.test.ts
index 69b825bd9..354d080b3 100644
--- a/packages/core/src/utils/slug.test.ts
+++ b/packages/core/src/utils/slug.test.ts
@@ -15,8 +15,60 @@ describe('toWikiLinkSlug', () => {
expect(toWikiLinkSlug('مرحبا بالعالم')).toBe('مرحبا-بالعالم');
});
+ test('keeps the marks that distinguish letters outside Latin, Greek, Cyrillic, Arabic and Hebrew', () => {
+ expect(toWikiLinkSlug('ガラス')).toBe('ガラス');
+ expect(toWikiLinkSlug('カラス')).toBe('カラス');
+ expect(toWikiLinkSlug('パン')).toBe('パン');
+ expect(toWikiLinkSlug('がっこう')).toBe('がっこう');
+ expect(toWikiLinkSlug('किताब')).toBe('किताब');
+ expect(toWikiLinkSlug('क्या हाल')).toBe('क्या-हाल');
+ expect(toWikiLinkSlug('বাংলা')).toBe('বাংলা');
+ expect(toWikiLinkSlug('ที่นี่')).toBe('ที่นี่');
+ });
+
+ test('returns composed text for scripts whose letters decompose', () => {
+ expect(toWikiLinkSlug('\uff76\uff9e\uff97\uff7d')).toBe('ガラス');
+ expect(toWikiLinkSlug('\u30ab\u3099\u30e9\u30b9')).toBe('ガラス');
+ expect(toWikiLinkSlug('한국어 문서')).toBe('한국어-문서');
+ });
+
+ test('keeps folding accents in Latin, Greek, Cyrillic, Arabic and Hebrew', () => {
+ expect(toWikiLinkSlug('Crème Brûlée')).toBe('creme-brulee');
+ expect(toWikiLinkSlug('Tiếng Việt')).toBe('tieng-viet');
+ expect(toWikiLinkSlug('Άλφα Ωμέγα')).toBe('αλφα-ωμεγα');
+ expect(toWikiLinkSlug('Ёлка')).toBe('елка');
+ expect(toWikiLinkSlug('مَرْحَبًا')).toBe('مرحبا');
+ expect(toWikiLinkSlug('שָׁלוֹם')).toBe('שלום');
+ expect(toWikiLinkSlug('مـَـرْحَبًا')).toBe('مــرحبا');
+ expect(toWikiLinkSlug('الرحمـٰن')).toBe('الرحمـن');
+ });
+
+ test('drops marks that do not sit on a letter', () => {
+ expect(toWikiLinkSlug('1\ufe0f\u20e3 Intro')).toBe('1-intro');
+ expect(toWikiLinkSlug('Party \u{1f389}\ufe0f')).toBe('party');
+ expect(toWikiLinkSlug('\u0301Café')).toBe('cafe');
+ expect(toWikiLinkSlug('葛\u{e0100}飾区')).toBe('葛飾区');
+ });
+
+ test('takes linear time on a long run of marks after a letter that keeps them', () => {
+ const input = `葛${'\u0301'.repeat(20_000)}`;
+ const started = performance.now();
+ const slug = toWikiLinkSlug(input);
+ expect(performance.now() - started).toBeLessThan(1500);
+ expect(slug).toBe(input);
+ });
+
test('is idempotent once a slug has been produced', () => {
- const samples = ['nonexistent-page', 'cafe-menu', '東京-2026', 'привет-мир', 'مرحبا-بالعالم'];
+ const samples = [
+ 'nonexistent-page',
+ 'cafe-menu',
+ '東京-2026',
+ 'привет-мир',
+ 'مرحبا-بالعالم',
+ 'ガラス',
+ 'किताब',
+ '한국어-문서',
+ ];
for (const sample of samples) {
expect(toWikiLinkSlug(toWikiLinkSlug(sample))).toBe(sample);
}
@@ -38,6 +90,13 @@ describe('getHeadingSlug', () => {
expect(getHeadingSlug('Café', slugCounts)).toBe('cafe');
expect(getHeadingSlug('Café', slugCounts)).toBe('cafe-1');
});
+
+ test('headings that differ only by a voicing mark get distinct anchors', () => {
+ const slugCounts = new Map<string, number>();
+ expect(getHeadingSlug('ガラス', slugCounts)).toBe('ガラス');
+ expect(getHeadingSlug('カラス', slugCounts)).toBe('カラス');
+ expect(getHeadingSlug('ガラス', slugCounts)).toBe('ガラス-1');
+ });
});
describe('wikiLinkHref', () => {
diff --git a/packages/core/src/utils/slug.ts b/packages/core/src/utils/slug.ts
index cc9c4fdd8..ec5b4e5ff 100644
--- a/packages/core/src/utils/slug.ts
+++ b/packages/core/src/utils/slug.ts
@@ -4,17 +4,25 @@ export interface HeadingEntry {
slug: string;
}
-const COMBINING_MARK_RE = /\p{M}+/gu;
-const NON_LETTER_OR_NUMBER_RE = /[^\p{L}\p{N}]+/gu;
+const MARK_RUN_RE = /(^|\P{M})(\p{M}+)/gu;
+const MARK_KEEPING_BASE_RE =
+ /^(?![\p{Script_Extensions=Latin}\p{Script_Extensions=Greek}\p{Script_Extensions=Cyrillic}\p{Script_Extensions=Arabic}\p{Script_Extensions=Hebrew}])\p{L}$/u;
+const DEFAULT_IGNORABLE_RE = /\p{DI}/gu;
+const NON_SLUG_CHARACTER_RE = /[^\p{L}\p{M}\p{N}]+/gu;
const EDGE_HYPHENS_RE = /^-+|-+$/g;
+function foldMarkRun(_run: string, base: string, marks: string): string {
+ return MARK_KEEPING_BASE_RE.test(base) ? base + marks.replace(DEFAULT_IGNORABLE_RE, '') : base;
+}
+
export function toWikiLinkSlug(text: string): string {
return text
.trim()
.normalize('NFKD')
- .replace(COMBINING_MARK_RE, '')
+ .replace(MARK_RUN_RE, foldMarkRun)
+ .normalize('NFC')
.toLowerCase()
- .replace(NON_LETTER_OR_NUMBER_RE, '-')
+ .replace(NON_SLUG_CHARACTER_RE, '-')
.replace(EDGE_HYPHENS_RE, '');
}
diff --git a/packages/core/src/utils/wiki-link-resolve.test.ts b/packages/core/src/utils/wiki-link-resolve.test.ts
index 05466d677..79b28e0b9 100644
--- a/packages/core/src/utils/wiki-link-resolve.test.ts
+++ b/packages/core/src/utils/wiki-link-resolve.test.ts
@@ -308,6 +308,27 @@ describe('resolution chain boundaries', () => {
expect(resolveWikiLinkTargetDocName('sub/foo', pages)).toBeUndefined();
});
+ test('a voiced kana target does not fall back to the unvoiced page', () => {
+ const pages = new Set(['カラス', 'notes/ハン']);
+ const index: WikiLinkLookupIndex = {
+ pages,
+ pagesBySlug: buildPagesBySlugIndex(pages, toWikiLinkSlug),
+ pagesByBasename: buildPagesByBasenameIndex(pages, toWikiLinkSlug),
+ };
+ for (const input of [pages, index]) {
+ expect(resolveWikiLinkTargetDocName('ガラス', input)).toBeUndefined();
+ expect(resolveWikiLinkTargetDocName('パン', input)).toBeUndefined();
+ expect(resolveWikiLinkTargetDocName('カラス', input)).toBe('カラス');
+ expect(resolveWikiLinkTargetDocName('ハン', input)).toBe('notes/ハン');
+ }
+ });
+
+ test('a Devanagari target does not fall back to a page missing its vowel signs', () => {
+ const pages = new Set(['कतब', 'किताब']);
+ expect(resolveWikiLinkTargetDocName('किताब', new Set(['कतब']))).toBeUndefined();
+ expect(resolveWikiLinkTargetDocName('किताब', pages)).toBe('किताब');
+ });
+
test('an asset that shares a name with a document resolves as the asset', () => {
const pages = new Set(['meeting.pdf']);
const assets = new Set(['meeting.pdf']);
diff --git a/packages/server/assets/skills/project/references/linking.md b/packages/server/assets/skills/project/references/linking.md
index a95ad644b..8e0501463 100644
--- a/packages/server/assets/skills/project/references/linking.md
+++ b/packages/server/assets/skills/project/references/linking.md
@@ -6,7 +6,7 @@
- **Never glue `./` onto a content-root path.** `./wiki/modules/tasks`, written from a doc already inside `wiki/`, resolves to the doubled garbage `wiki/wiki/modules/tasks` — correct resolution of a malformed path, i.e. a silently broken link. Pick ONE form: a relative path from your doc (`./tasks.md`, `../modules/tasks.md`), or a `/`-rooted absolute path (`/wiki/modules/tasks.md`). The `./` prefix and a content-root path never combine.
- **External web sources are NOT inline body links.** Per the Grounding rule, web URLs live in the `source_url:` frontmatter of an ingested doc under `external-sources/` (or the project's equivalent raw-sources folder); the body cites the local path: `[source name](./external-sources/source-slug.md)`. A raw `[source](https://...)` inline in the body is a TODO, not a citation — see Grounding for the closed-loop contract.
- **Internal cross-refs between OK docs** → `[text](./other-doc.md)` — link liberally to aid navigation.
-- **Link the section when you mean one part of a long document** — `[text](./spec.md#heading-slug)`, in docs and in chat replies alike, so the reader lands on the heading instead of scrolling a long file (in the WYSIWYG editor; the source editor ignores the fragment and opens at the top). In a chat reply there is no document for a relative path to start from: write the path from the content root, as `./specs/spec.md#heading-slug` or `/specs/spec.md#heading-slug`, never with `../`, which the chat cannot resolve. The slug is the heading's text lowercased, accents dropped, with each run of characters that are not letters or digits replaced by one `-` and edge hyphens removed (`## Data model: v2` → `#data-model-v2`); the second identical heading in a doc gets `-1`, the third `-2`. Copy the heading text rather than guessing: `brokenLinks` and `audit` check only the path before `#`, so a wrong slug still resolves as a link to the top of the doc.
+- **Link the section when you mean one part of a long document** — `[text](./spec.md#heading-slug)`, in docs and in chat replies alike, so the reader lands on the heading instead of scrolling a long file (in the WYSIWYG editor; the source editor ignores the fragment and opens at the top). In a chat reply there is no document for a relative path to start from: write the path from the content root, as `./specs/spec.md#heading-slug` or `/specs/spec.md#heading-slug`, never with `../`, which the chat cannot resolve. The slug is the heading's text lowercased, accents on Latin, Greek, Cyrillic, Arabic and Hebrew letters dropped (other scripts keep their marks, so `## ガラス` → `#ガラス`), with each run of characters that are not letters or digits replaced by one `-` and edge hyphens removed (`## Data model: v2` → `#data-model-v2`); the second identical heading in a doc gets `-1`, the third `-2`. Copy the heading text rather than guessing: `brokenLinks` and `audit` check only the path before `#`, so a wrong slug still resolves as a link to the top of the doc.
- **Every link must resolve to a doc that exists by the time you're done.** Within a single multi-doc authoring pass, linking a page you'll create later in the same pass is fine — `brokenLinks` reports it as `no-such-doc` until the target lands, an expected transient forward-reference rather than a wrong-path error. What's not acceptable is *leaving* a dead link behind: create the target in the same pass, or record it as a tracked task (`TaskCreate`, or your host's task tool — if the host has none, tell the user) and leave the mention as plain prose.
- **Never wrap a link in backticks.** `` `[text](./foo.md)` `` is a bug — the backticks make it render as literal code rather than a link.
- **Never use HTML anchors** (`<a href="...">`). Markdown link syntax only.