Skip to content

Beautiful, AI-native markdown IDE and LLM wiki

fix(ok): stop the generated index emitting a duplicate Index heading (PRD-8338) (#3915)

serafin-garcia4bcec3f8cbf1
diff --git a/.changeset/generated-index-index-type-heading.md b/.changeset/generated-index-index-type-heading.md
new file mode 100644
index 000000000..ab2afb454
--- /dev/null
+++ b/.changeset/generated-index-index-type-heading.md
@@ -0,0 +1,13 @@
+---
+'@inkeep/open-knowledge': patch
+---
+
+Generated `index.md` files no longer produce a markdownlint MD024 warning when two of a folder's headings reduce to the same text under MD024's own comparison, which drops inline HTML and an ATX closing sequence. The generator titles every index `# Index`, so a document whose frontmatter `type` is `Index` produced a second `## Index` heading with the same content. Projects that had enabled both the `markdownlint` and `okf` plugins along with `okf.generate.index` saw the duplicate flagged in `ok lint`, in the Problems panel, and in inline diagnostics. All three are off by default, so a project using the defaults was never affected.
+
+A document typed `Index` is now listed directly under the index's own title with no `##` heading of its own. That differs from a document typed `Subdirectories`, which keeps its visible section.
+
+The same merge applies between two ordinary types, so a project that has never used `type: Index` can still see a change. `Flow` beside `Flow #`, and two types that render to no text at all, each now share a single section. Its heading is one of the colliding `type` strings, chosen deterministically rather than by file order, so a folder holding both `Flow` and `Flow #` lists everything under one of the two. Headings the generator owns, meaning `Index`, `Other` and `Subdirectories`, keep their own spelling when a `type` collides with them.
+
+One further case merges that markdownlint itself would not flag. Two Unicode spellings of the same grapheme cluster, which look identical in an editor, are treated as one heading. That is the generator normalizing beyond the rule, so that the file it rewrites settles to stable bytes instead of changing with the order the folder is read in.
+
+Every document and every subdirectory link still appears, and indexes in affected folders are rewritten once on the next regeneration.
diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts
index 76cc7837e..763aeaa2b 100644
--- a/packages/core/src/index.ts
+++ b/packages/core/src/index.ts
@@ -656,6 +656,7 @@ export {
} from './i18n/resolve-locale.ts';
export { selectFenceChar, widenFenceLength } from './markdown/code-fence.ts';
export { MIN_CARRIED_EDGE_EMPTIES } from './markdown/doc-edge-blank-runs.ts';
+export { headingContentIdentity } from './markdown/heading-content-identity.ts';
export {
HTML_MAX_BYTES,
HtmlPayloadTooLargeError,
diff --git a/packages/core/src/markdown/heading-content-identity.ts b/packages/core/src/markdown/heading-content-identity.ts
new file mode 100644
index 000000000..dd4433204
--- /dev/null
+++ b/packages/core/src/markdown/heading-content-identity.ts
@@ -0,0 +1,10 @@
+import { fromMarkdown } from 'mdast-util-from-markdown';
+import { toString as mdastToString } from 'mdast-util-to-string';
+
+export function headingContentIdentity(heading: string): string {
+ const singleLine = heading.replace(/\s+/g, ' ').trim();
+ const tree = fromMarkdown(`## ${singleLine}`);
+ const node = tree.children[0];
+ if (node === undefined || node.type !== 'heading') return '';
+ return mdastToString(node, { includeHtml: false }).replace(/\s+/g, ' ').trim().normalize('NFC');
+}
diff --git a/packages/server/src/content/generate-index-conformance.test.ts b/packages/server/src/content/generate-index-conformance.test.ts
index 47684cc44..77b074c78 100644
--- a/packages/server/src/content/generate-index-conformance.test.ts
+++ b/packages/server/src/content/generate-index-conformance.test.ts
@@ -163,6 +163,62 @@ describe('generated index — default lint profile', () => {
expect(await findingCodes(markdown, 'concepts/index.md')).toEqual([]);
});
+ test('an Index document type is clean under markdownlint defaults', async () => {
+ // `Index` is an ordinary OKF type — a folder README that indexes its folder
+ // declares it — and it lands in the same heading namespace as the
+ // generator's own title. MD024 compares heading CONTENT across levels, so
+ // the title and a `type`-derived section collide even at different levels.
+ const readmeOnly = buildIndexMarkdown(
+ [{ path: 'testing/README.md', title: 'Testing', type: 'Index' }],
+ { isRoot: false, directory: 'testing' },
+ );
+
+ expect(await findingCodes(readmeOnly, 'testing/index.md')).toEqual([]);
+ });
+
+ test('an Index document type alongside other typed sections is clean under markdownlint defaults', async () => {
+ // The collision is not a property of README-only directories. A directory
+ // carrying several typed sections plus one `Index`-typed document collides
+ // just the same, so a fix that only holds for the single-entry shape
+ // under-covers the contract.
+ const multiSection = buildIndexMarkdown(
+ [
+ { path: 'auth/README.md', title: 'Auth', type: 'Index' },
+ { path: 'auth/login-flow.md', title: 'Login flow', type: 'Flow' },
+ { path: 'auth/sso.md', title: 'SSO', type: 'Feature Doc' },
+ ],
+ { isRoot: false, directory: 'auth' },
+ );
+
+ expect(await findingCodes(multiSection, 'auth/index.md')).toEqual([]);
+ });
+
+ test('a root index with an Index document type is clean under markdownlint defaults', async () => {
+ // The root branch renders the same headings behind frontmatter, so it
+ // carries the collision independently of the non-root branch.
+ const markdown = buildIndexMarkdown(
+ [
+ { path: 'README.md', title: 'Knowledge base', type: 'Index' },
+ { path: 'welcome.md', title: 'Welcome', type: 'note' },
+ ],
+ { isRoot: true },
+ );
+
+ expect(await findingCodes(markdown, 'index.md')).toEqual([]);
+ });
+
+ test('a duplicate heading would be caught — the guard is not vacuous', async () => {
+ // The third leg of the squeeze, alongside MD025 and MD041 below. Proving
+ // MD024 fires on the shape the generator must avoid is what keeps the
+ // assertions above from passing under a linter that silently stopped
+ // running markdownlint, or a generator that emits no sections at all.
+ const duplicateHeading = ['# Index', '', '## Index', '', '* [Testing](./README.md)', ''];
+
+ expect(await findingCodes(duplicateHeading.join('\n'), 'testing/index.md')).toEqual([
+ expect.stringContaining('MD024'),
+ ]);
+ });
+
test('level-one sections would be caught — the guard is not vacuous', async () => {
// The shape this generator used to emit. Proving it still fires is what
// keeps the assertions above from passing under a linter that silently
@@ -198,3 +254,185 @@ describe('generated index — default lint profile', () => {
]);
});
});
+
+// The generator decides which sections share a bucket by mirroring the identity
+// MD024 compares headings by. That mirror is a coupling to another tool's
+// internals, so it is pinned from the outside rather than trusted: this suite
+// runs `type` values through the real generator and the real linter and fails if
+// the two ever disagree about whether a collision exists.
+describe('generated index — heading-identity agreement with MD024', () => {
+ const mdOnly = (codes: string[]): string[] => codes.filter((code) => code.includes('MD024'));
+
+ const withType = (type: string): string =>
+ buildIndexMarkdown(
+ [
+ { path: 'readme.md', title: 'Overview', description: undefined, type },
+ { path: 'flow.md', title: 'Login flow', description: undefined, type: 'Flow' },
+ ],
+ { isRoot: false, directory: '' },
+ );
+
+ // Values that reduce to the title's content without being the title's string.
+ // Each is conformant frontmatter, since `type` is constrained only to be
+ // non-blank.
+ test.each([
+ ['Index'],
+ ['Index #'],
+ ['Index ##'],
+ ['<b></b>Index'],
+ ['Index<!--x-->'],
+ // Raw HTML may legally contain `>` inside a comment or a quoted attribute
+ // value (CommonMark §6.6). A `<[^>]*>` strip stops at the first one and
+ // leaves behind text the rule discards — narrowing, which reopens the
+ // duplicate rather than merely costing a section.
+ ['Index<!-- a>b -->'],
+ ['Index<a href="x>y">'],
+ ])('a type reducing to the title merges into it: %j', async (type) => {
+ const markdown = withType(type);
+
+ expect(mdOnly(await findingCodes(markdown, 'index.md')), `emitted:\n${markdown}`).toEqual([]);
+ expect(markdown, `emitted:\n${markdown}`).toContain('](./readme.md)');
+ });
+
+ // `index` and `INDEX` matter because MD024's comparison is case sensitive, so
+ // folding them would merge documents the linter is content to see side by side.
+ //
+ // The two HTML-wrapped `#` values belong here rather than above: the `#` sits
+ // inside an element, so it is never a closing sequence, and the heading text
+ // keeps it. They read as near-misses and are not — which is why they are worth
+ // pinning on this side, where over-merging would be the failure.
+ test.each([
+ ['index'],
+ ['INDEX'],
+ ['Indexes'],
+ ['Index Notes'],
+ ['Flow'],
+ ['Index <b>#</b>'],
+ ['Index<b> #</b>'],
+ ])('a type distinct from the title keeps its own section: %j', async (type) => {
+ const markdown = withType(type);
+
+ expect(markdown, `emitted:\n${markdown}`).toContain(`## ${type}`);
+ expect(mdOnly(await findingCodes(markdown, 'index.md')), `emitted:\n${markdown}`).toEqual([]);
+ });
+
+ test.each([
+ [['#', '##']],
+ [['#', '<b></b>']],
+ [['<b></b>', '<i></i>']],
+ ])('types whose rendered heading text is empty merge with each other: %j', async (pair) => {
+ // A heading of only hashes or only HTML has no text, and the rule compares
+ // those to each other and calls them duplicates. Giving them a bucket each
+ // emits two headings it flags, so the empty identity has to collapse like
+ // any other. Nothing is invented by merging: the spelling is chosen by the
+ // same reduction every bucket uses.
+ const markdown = buildIndexMarkdown(
+ [
+ { path: 'a.md', title: 'A', description: undefined, type: pair[0] },
+ { path: 'b.md', title: 'B', description: undefined, type: pair[1] },
+ ],
+ { isRoot: false, directory: '' },
+ );
+
+ expect(mdOnly(await findingCodes(markdown, 'index.md')), `emitted:\n${markdown}`).toEqual([]);
+ expect(markdown).toContain('](./a.md)');
+ expect(markdown).toContain('](./b.md)');
+
+ // The heading has to be one of the colliding types. Without this the rows
+ // pass against a generator that never merges and instead routes an
+ // empty-reducing `type` to `Other` — a natural-looking simplification whose
+ // outcome is two documents filed as untyped when neither is, in a file the
+ // author cannot hand-edit. The sibling suite pins the other direction, that
+ // a blank `type` IS absent, so both classifications stay pinned.
+ const sections = (markdown.match(/^## .+$/gm) ?? []).map((line) => line.slice(3));
+ expect(sections, `emitted:\n${markdown}`).toHaveLength(1);
+ expect(pair, `emitted:\n${markdown}`).toContain(sections[0]);
+ });
+
+ test('two derived sections reducing to the same content merge with each other', async () => {
+ const markdown = buildIndexMarkdown(
+ [
+ { path: 'a.md', title: 'A', description: undefined, type: 'Flow' },
+ { path: 'b.md', title: 'B', description: undefined, type: 'Flow #' },
+ ],
+ { isRoot: false, directory: '' },
+ );
+
+ expect(mdOnly(await findingCodes(markdown, 'index.md')), `emitted:\n${markdown}`).toEqual([]);
+ expect(markdown).toContain('](./a.md)');
+ expect(markdown).toContain('](./b.md)');
+ });
+
+ test('two empty headings would be caught — the guard is not vacuous', async () => {
+ // The third reduction needs its own leg for the same reason the other two
+ // do. After the merge, an empty-reducing pair emits a single heading and the
+ // rule has nothing to compare, so those rows hold whether or not it treats
+ // two empty headings as duplicates. Only a hand-built pair keeps both to the
+ // comparison, which is what makes the merge a fix rather than a coincidence.
+ const handBuilt = [
+ '# Index',
+ '',
+ '* [Overview](./readme.md)',
+ '',
+ '## #',
+ '',
+ '* [A](./a.md)',
+ '',
+ '## <b></b>',
+ '',
+ '* [B](./b.md)',
+ '',
+ ].join('\n');
+
+ expect(mdOnly(await findingCodes(handBuilt, 'index.md')), `emitted:\n${handBuilt}`).toEqual([
+ expect.stringContaining('MD024'),
+ ]);
+ });
+
+ test('a duplicate reached through inline HTML would be caught — the guard is not vacuous', async () => {
+ // The identity makes two reductions and each needs its own leg. The corpus
+ // rows cannot supply this one: the generator merges an HTML-bearing type
+ // into the title's bucket before the linter sees it, so the emitted file has
+ // a single heading and MD024 has nothing to compare — the row then passes
+ // whether the merge was necessary or not. Hand-building both headings is
+ // what proves the linter really does drop `htmlText` when comparing, and so
+ // that the merge closes a live failure rather than an imagined one.
+ const handBuilt = [
+ '# Index',
+ '',
+ '* [Overview](./readme.md)',
+ '',
+ '## <b></b>Index',
+ '',
+ '* [Other](./other.md)',
+ '',
+ ].join('\n');
+
+ const codes = await findingCodes(handBuilt, 'index.md');
+
+ expect(mdOnly(codes), `emitted:\n${handBuilt}`).toEqual([expect.stringContaining('MD024')]);
+ // MD033 rides along on the same line, which is the class tracked separately.
+ // Asserting it here keeps this leg honest about what the fixture really is.
+ expect(codes.some((code) => code.includes('MD033'))).toBe(true);
+ });
+
+ test('a duplicate reached through an ATX closer would be caught — the guard is not vacuous', async () => {
+ // Proves the linter really does reduce `## Index #` to `Index` and flag it,
+ // so the merges asserted above close a live failure rather than one this
+ // profile never reported in the first place.
+ const handBuilt = [
+ '# Index',
+ '',
+ '* [Overview](./readme.md)',
+ '',
+ '## Index #',
+ '',
+ '* [Other](./other.md)',
+ '',
+ ].join('\n');
+
+ expect(mdOnly(await findingCodes(handBuilt, 'index.md'))).toEqual([
+ expect.stringContaining('MD024'),
+ ]);
+ });
+});
diff --git a/packages/server/src/content/generate-index.test.ts b/packages/server/src/content/generate-index.test.ts
index b5a682df1..7cbf0b47a 100644
--- a/packages/server/src/content/generate-index.test.ts
+++ b/packages/server/src/content/generate-index.test.ts
@@ -1,6 +1,6 @@
import { resolveInternalHref } from '@inkeep/open-knowledge-core';
import { describe, expect, test } from 'vitest';
-import { buildIndexMarkdown, type IndexEntry } from './generate-index.ts';
+import { buildIndexMarkdown, GENERATOR_OWNED_HEADINGS, type IndexEntry } from './generate-index.ts';
function entry(overrides: Partial<IndexEntry> & Pick<IndexEntry, 'path' | 'title'>): IndexEntry {
return { description: undefined, type: undefined, ...overrides };
@@ -145,6 +145,136 @@ describe('buildIndexMarkdown', () => {
expect(buildIndexMarkdown([], { isRoot: false })).toBe('# Index\n');
});
+ test('a bucket whose members spell the heading differently renders the same either way', () => {
+ // Distinct source strings can share one heading identity, so which spelling
+ // renders must be a pure function of the bucket's members. `entries` arrives
+ // in live file-index order, which differs between a cold boot and an
+ // incremental rebuild, so a first-seen-wins rule would make the bytes depend
+ // on traversal order and the write guard would never settle.
+ const mixed = [
+ entry({ path: 'a.md', title: 'Alpha', type: 'Flow' }),
+ entry({ path: 'b.md', title: 'Beta', type: 'Flow #' }),
+ entry({ path: 'c.md', title: 'Gamma', type: 'Café' }),
+ entry({ path: 'd.md', title: 'Delta', type: 'Café' }),
+ ];
+
+ for (const isRoot of [true, false]) {
+ const forward = buildIndexMarkdown(mixed, { isRoot });
+ const reversed = buildIndexMarkdown(mixed.toReversed(), { isRoot });
+
+ expect(reversed, `isRoot=${isRoot}`).toBe(forward);
+ // Not vacuous: every member still has to reach the output, so a generator
+ // that dropped the losing spelling's documents cannot satisfy this.
+ for (const href of ['](./a.md)', '](./b.md)', '](./c.md)', '](./d.md)']) {
+ expect(forward, `isRoot=${isRoot} ${href}`).toContain(href);
+ }
+ }
+ });
+
+ test('a generator-owned heading keeps its own spelling against a colliding type', () => {
+ // `Subdirectories` is contributed after the entry loop, so a document typed
+ // `Subdirectories #` can reach the bucket first. It must not rename the
+ // section the generator writes its own child-index links under.
+ const out = buildIndexMarkdown(
+ [entry({ path: 'a.md', title: 'Alpha', type: 'Subdirectories #' })],
+ { isRoot: false, subdirectories: [{ path: 'nested/index.md', title: 'nested' }] },
+ );
+
+ expect(out).toContain('## Subdirectories\n');
+ expect(out).not.toContain('## Subdirectories #');
+ expect(out).toContain('](./a.md)');
+ expect(out).toContain('](./nested/index.md)');
+ });
+
+ test('every heading the generator owns keeps its own spelling against a colliding type', () => {
+ // Both live pin routes at once: `Subdirectories` via the post-loop
+ // contribution, `Other` via an untyped document inside the entry loop. A
+ // `type` that reduces to the same identity must not rename either.
+ // `<b></b>` is chosen because U+003C sorts below every letter, so it would
+ // win the spelling reduction outright if the bucket were not pinned.
+ //
+ // The `<b></b>Index` entry is here for coverage of the render path, not as a
+ // third pin route. It never pins anything: nothing in this fixture declares
+ // `type: 'Index'`, so that bucket is created unpinned, and the title is
+ // filtered out of the section blocks and emitted from the constant instead.
+ const out = buildIndexMarkdown(
+ [
+ entry({ path: 'untyped.md', title: 'Untyped' }),
+ entry({ path: 'a.md', title: 'Alpha', type: '<b></b>Other' }),
+ entry({ path: 'b.md', title: 'Beta', type: '<b></b>Subdirectories' }),
+ entry({ path: 'c.md', title: 'Gamma', type: '<b></b>Index' }),
+ ],
+ { isRoot: false, subdirectories: [{ path: 'nested/index.md', title: 'nested' }] },
+ );
+
+ expect(out).toContain('# Index\n');
+ expect(out).toContain('## Other\n');
+ expect(out).toContain('## Subdirectories\n');
+ expect(out).not.toContain('<b>');
+
+ // Not vacuous: pinning must not drop the documents whose spelling lost.
+ for (const href of ['](./untyped.md)', '](./a.md)', '](./b.md)', '](./c.md)']) {
+ expect(out, href).toContain(href);
+ }
+ });
+
+ test.each([
+ ...GENERATOR_OWNED_HEADINGS,
+ ])('a colliding type cannot rename the generator-owned heading %j, reached from the entry loop alone', (owned) => {
+ // Every owned heading, reached through a document's `type` with nothing else
+ // supplied — no subdirectories, no untyped document. The strength of the leg
+ // differs by row and that is deliberate: `Other` and `Subdirectories` reach
+ // the bucket only by this route here, so the pin is what decides them, while
+ // the title is filtered out of the section blocks and rendered from the
+ // constant either way. The `Index` row is a consistency check rather than a
+ // discriminating one.
+ //
+ // Stated as the general rule over the exported set so a heading added later
+ // is covered without editing this test.
+ for (const order of [
+ [
+ entry({ path: 'a.md', title: 'Alpha', type: owned }),
+ entry({ path: 'b.md', title: 'Beta', type: `<b></b>${owned}` }),
+ ],
+ [
+ entry({ path: 'b.md', title: 'Beta', type: `<b></b>${owned}` }),
+ entry({ path: 'a.md', title: 'Alpha', type: owned }),
+ ],
+ ]) {
+ const label = `${owned} order=${order.map((e) => e.type).join(',')}`;
+ const out = buildIndexMarkdown(order, { isRoot: false });
+
+ // Positive half: the heading is present, on its own line, under its own
+ // spelling. Without it a mutant that dropped the heading prefix entirely
+ // would satisfy the negative assertion below. Compared as whole-line
+ // heading TEXT rather than a substring of the output, which both avoids
+ // naming a constant to pick the level and avoids `# Index` being satisfied
+ // by an emitted `## Index`.
+ expect(headingContents(out), label).toContain(owned);
+ expect(out, label).not.toContain('<b>');
+ expect(out, label).toContain('](./a.md)');
+ expect(out, label).toContain('](./b.md)');
+ }
+ });
+
+ test('a document sharing the title heading renders under it, byte for byte', () => {
+ // The one shape this file did not already pin exactly. These bytes are the
+ // write guard's fixed point, so a change to the separator between the title
+ // and its bullets would rewrite every generated index in every project on
+ // upgrade while every behavioural assertion still passed.
+ const out = buildIndexMarkdown(
+ [
+ entry({ path: 'readme.md', title: 'Overview', type: 'Index' }),
+ entry({ path: 'login-flow.md', title: 'Login flow', type: 'Flow' }),
+ ],
+ { isRoot: false },
+ );
+
+ expect(out).toBe(
+ '# Index\n\n* [Overview](./readme.md)\n\n## Flow\n\n* [Login flow](./login-flow.md)\n',
+ );
+ });
+
test('the same entries in any input order produce identical bytes', () => {
const entries = [
entry({ path: 'b.md', title: 'Beta', type: 'note', description: 'second' }),
@@ -321,6 +451,57 @@ describe('buildIndexMarkdown', () => {
expect(reversed).toBe(forward);
});
+ test('several documents sharing a heading the generator owns produce identical bytes in any input order', () => {
+ // A heading the generator writes from a fixed string collects members from
+ // two sources at once: the documents whose `type` is that string, and the
+ // generator's own contribution. Ordering inside such a shared bucket is
+ // only observable once it holds more than one member, and the write guard
+ // compares bytes, so a merge that depended on read order would rewrite the
+ // file on every rebuild.
+ const entries = [
+ entry({ path: 'concepts/overview.md', title: 'Overview', type: 'Index' }),
+ entry({ path: 'concepts/catalog.md', title: 'Catalog', type: 'Index' }),
+ entry({ path: 'concepts/login-flow.md', title: 'Login flow', type: 'Flow' }),
+ entry({ path: 'concepts/handbook.md', title: 'Handbook', type: 'Subdirectories' }),
+ entry({ path: 'concepts/atlas.md', title: 'Atlas', type: 'Subdirectories' }),
+ entry({ path: 'concepts/aggregate.md', title: 'Aggregate', type: 'concept' }),
+ ];
+ const subdirectories = [
+ { path: 'concepts/zeta/index.md', title: 'zeta' },
+ { path: 'concepts/alpha/index.md', title: 'alpha' },
+ ];
+
+ for (const isRoot of [true, false]) {
+ const label = `isRoot=${isRoot}`;
+ const forward = buildIndexMarkdown(entries, {
+ isRoot,
+ directory: 'concepts',
+ subdirectories,
+ });
+ const reversed = buildIndexMarkdown([...entries].reverse(), {
+ isRoot,
+ directory: 'concepts',
+ subdirectories: [...subdirectories].reverse(),
+ });
+
+ expect(reversed, label).toBe(forward);
+
+ // Identical bytes are also what an output that dropped the merged
+ // members would produce, so every source of both shared buckets has to
+ // still be listed.
+ for (const href of [
+ './overview.md',
+ './catalog.md',
+ './handbook.md',
+ './atlas.md',
+ './zeta/index.md',
+ './alpha/index.md',
+ ]) {
+ expect(forward, `${label}: ${href}`).toContain(`](${href})`);
+ }
+ }
+ });
+
test('emitted hrefs round-trip through the canonical resolver to the entry they came from', () => {
// The generator's percent-encoding is the only CommonMark-valid way to link
// these names (a literal space does not parse as an unbracketed link
@@ -348,4 +529,83 @@ describe('buildIndexMarkdown', () => {
);
expect(resolved.toSorted()).toEqual(names.map((name) => `blogs/drafts/${name}`).toSorted());
});
+
+ test('no heading the generator owns collides with a section derived from a document type', () => {
+ // `type` is free-form by design, so every fixed heading string the
+ // generator writes shares a namespace with a heading it derives from user
+ // input. Duplicate heading CONTENT is MD024 regardless of level, which
+ // makes "the emitted heading set has no duplicates" the post-condition —
+ // not "these two particular strings were deconflicted".
+ //
+ // The reserved set is harvested from the generator's own output rather
+ // than listed here, so a fixed heading added later is covered without this
+ // test being updated to name it.
+ //
+ // This pins the SOURCE-level set. The linter compares a heading identity
+ // that also drops inline HTML and the ATX closing sequence, so several
+ // distinct source strings can still collide under it; agreement with that
+ // identity is pinned against the real linter in the conformance suite.
+ // Checked against the exported set rather than a restated list, so the
+ // declaration and what the generator actually emits cannot drift: a constant
+ // added to one and not the other fails here.
+ const reserved = generatorOwnedHeadings();
+ expect(reserved.toSorted()).toEqual([...GENERATOR_OWNED_HEADINGS].toSorted());
+
+ for (const type of reserved) {
+ for (const isRoot of [true, false]) {
+ const label = `type=${type} isRoot=${isRoot}`;
+ const out = buildIndexMarkdown(
+ [
+ entry({ path: 'readme.md', title: 'Readme', type }),
+ entry({ path: 'login-flow.md', title: 'Login flow', type: 'Flow' }),
+ ],
+ { isRoot, subdirectories: [{ path: 'nested/index.md', title: 'nested' }] },
+ );
+
+ // Every input still has to reach the output. Without this a generator
+ // that emitted the title alone would satisfy the duplicate check.
+ expect(out, label).toContain('](./readme.md)');
+ expect(out, label).toContain('](./login-flow.md)');
+ expect(out, label).toContain('](./nested/index.md)');
+ expect(out.match(/^# .+$/gm), label).toHaveLength(1);
+
+ const headings = headingContents(out);
+ expect(new Set(headings).size, `${label}: ${headings.join(' | ')}`).toBe(headings.length);
+ }
+ }
+ });
});
+
+/**
+ * Heading source text, which is deliberately NOT the identity `MD024` compares
+ * by — that one also drops inline HTML and the ATX closing sequence, so this is
+ * strictly weaker. Source-level uniqueness is what this suite pins; agreement
+ * with the linter's own identity is pinned against the real linter in
+ * `generate-index-conformance.test.ts`, which is where a claim about `MD024`
+ * belongs.
+ */
+function headingContents(markdown: string): string[] {
+ return (markdown.match(/^#{1,6} .+$/gm) ?? []).map((line) => line.replace(/^#+ /, ''));
+}
+
+/**
+ * The headings the generator writes from its own fixed strings, read back off
+ * output produced from inputs that declare no `type` at all — so nothing a
+ * caller supplied can be mistaken for one.
+ *
+ * Both `isRoot` branches are probed: the root branch renders the same headings
+ * behind frontmatter, and harvesting only one of them would leave a heading
+ * introduced on the other invisible to every caller of this helper.
+ */
+function generatorOwnedHeadings(): string[] {
+ const probes = [false, true].flatMap((isRoot) => [
+ buildIndexMarkdown([], { isRoot }),
+ buildIndexMarkdown([entry({ path: 'a.md', title: 'A' })], { isRoot }),
+ buildIndexMarkdown([], {
+ isRoot,
+ subdirectories: [{ path: 'nested/index.md', title: 'nested' }],
+ }),
+ ]);
+
+ return [...new Set(probes.flatMap(headingContents))];
+}
diff --git a/packages/server/src/content/generate-index.ts b/packages/server/src/content/generate-index.ts
index 21620aec7..43e6a9f24 100644
--- a/packages/server/src/content/generate-index.ts
+++ b/packages/server/src/content/generate-index.ts
@@ -23,7 +23,7 @@
* without a server, a disk, or a Y.Doc.
*/
-import { encodeHrefPath } from '@inkeep/open-knowledge-core';
+import { encodeHrefPath, headingContentIdentity } from '@inkeep/open-knowledge-core';
/** One document as the index renders it. `path` is content-root-relative. */
export interface IndexEntry {
@@ -91,6 +91,14 @@ const SUBDIRECTORY_SECTION = 'Subdirectories';
* One title plus level-two sections satisfies both, and is the shape the
* hand-written index this generator replaced already used.
*
+ * `MD024` is the third default rule over this namespace, and it compares heading
+ * content across levels, so this string is not the generator's alone: it shares a
+ * namespace with the free-form `type` values that become section headings. So a
+ * document whose `type` reduces to this string joins the title's own key in the
+ * section collection of `buildIndexMarkdown` and lists under the title, rather
+ * than opening a second heading with the same content. Absent such a document
+ * the key never exists, and the title is emitted from this constant.
+ *
* Not localized, and not merely because this is file content rather than UI: the
* bytes are the fixed point the write guard compares against, so a title that
* varied by host locale would make every rebuild a write on a differently
@@ -98,6 +106,25 @@ const SUBDIRECTORY_SECTION = 'Subdirectories';
*/
const INDEX_TITLE = 'Index';
+/**
+ * Every heading the generator writes from a fixed string of its own.
+ *
+ * `type` is free-form, so any of these can also arrive as one, and the two are
+ * then indistinguishable by the time they reach a bucket. Naming the set once is
+ * what keeps that a membership question: enumerating the constants at each site
+ * is how `SUBDIRECTORY_SECTION` got a reservation while `INDEX_TITLE` did not,
+ * and adding a fourth string here is the only step a later heading needs.
+ *
+ * Exported so the suite can assert this set is exactly what the generator emits
+ * from inputs that declare no `type` at all, rather than restating the strings
+ * and drifting from them.
+ */
+export const GENERATOR_OWNED_HEADINGS: ReadonlySet<string> = new Set([
+ INDEX_TITLE,
+ UNTYPED_SECTION,
+ SUBDIRECTORY_SECTION,
+]);
+
/** The shared shape a link line renders from — a document or a subdirectory. */
interface RenderableLink {
path: string;
@@ -138,6 +165,18 @@ function compareSections(left: string, right: string): number {
);
}
+/**
+ * Whether `candidate` is the spelling a bucket should render under, given the
+ * one it currently holds. `compareSections` cannot decide this alone: it folds
+ * case and normalizes BOTH sides to NFC, so two spellings of one grapheme
+ * cluster compare equal and the answer would fall through to whichever arrived
+ * first. Raw code points break that tie, which is what keeps the rendered
+ * heading a function of the bucket's members rather than of traversal order.
+ */
+function prefersSpelling(candidate: string, current: string): boolean {
+ return (compareSections(candidate, current) || compareCodePoints(candidate, current)) < 0;
+}
+
function compareLinks(left: RenderableLink, right: RenderableLink): number {
return (
compareCodePoints(normalizedSortKey(left.title), normalizedSortKey(right.title)) ||
@@ -187,10 +226,41 @@ function renderLink(link: RenderableLink, directory: string): string {
return description ? `${anchor} - ${description}` : anchor;
}
+/**
+ * The bullet list under one heading. Every bucket renders through here, the
+ * title's included, so within-bucket order stays a pure function of the entry
+ * set no matter which heading owns it.
+ */
+function renderBody(links: readonly RenderableLink[], directory: string): string {
+ return links
+ .slice()
+ .sort(compareLinks)
+ .map((link) => renderLink(link, directory))
+ .join('\n');
+}
+
/**
* Group into `## <type>` sections (plus a `## Subdirectories` section when child
* directories are supplied) under the index title, and render.
*
+ * Every section heading this emits is a key of one collection, keyed by
+ * `headingContentIdentity` rather than by source text. That mirrors the rule's
+ * own reduction rather than sharing it, and it is case SENSITIVE, unlike the
+ * case-folding comparators further up this file, which order sections rather
+ * than key them. The title joins that collection only when a document's `type`
+ * reduces to its identity, which is the whole point. For every other directory
+ * no such key exists and the title is emitted from the constant, so the `?? []`
+ * read below is the ordinary path rather than a guard.
+ *
+ * A `Map` cannot hold two entries under one key, so a heading whose content
+ * already belongs to the generator merges into that bucket instead of opening a
+ * second heading with the same content. `type` is free-form, so any
+ * of the generator's own heading strings can arrive as one, and `MD024` compares
+ * heading content across levels: a heading kept outside the collection would
+ * collide. The title renders at level one and every other bucket at level two,
+ * which is what `MD025` and `MD041` require, so the title is selected by key
+ * rather than sorted with the rest.
+ *
* Sections sort alphabetically by heading and entries sort by title
* case-insensitively, both matching the reference generator. Ordering is what
* makes the output a fixed point: two runs over the same content must produce
@@ -204,39 +274,92 @@ export function buildIndexMarkdown(
const directory = options.directory ?? '';
const subdirectories = options.subdirectories ?? [];
- const grouped = new Map<string, RenderableLink[]>();
+ // Each bucket carries the heading it will render under, because the key is an
+ // identity several distinct source strings can share. Which spelling wins has
+ // to be a pure function of the bucket's members: `entries` arrives in live
+ // file-index order, which differs between a cold boot and an incremental
+ // rebuild, so picking the first one seen would make the bytes depend on
+ // traversal order and the write guard would never settle.
+ //
+ // `pinned` marks a heading the generator owns. Those keep their own spelling
+ // even when a document's `type` reaches the bucket first, which is what stops
+ // a colliding `<b></b>Subdirectories` from renaming the section the generator
+ // writes its own child-index links under.
+ interface Bucket {
+ heading: string;
+ pinned: boolean;
+ links: RenderableLink[];
+ }
+
+ const titleKey = headingContentIdentity(INDEX_TITLE);
+ const grouped = new Map<string, Bucket>();
+
+ // Whether a heading is pinned is derived here, not supplied by the caller.
+ // Every route into this function resolves a string that is either one the
+ // generator owns or one a document declared, and membership is the whole
+ // question — so a caller cannot state it wrongly, and a route added later
+ // cannot forget to. That is what makes the promise on the set's declaration
+ // true: a fourth heading needs one line there and nothing at any call site.
+ //
+ // Pinning matters because the spelling reduction otherwise hands the heading
+ // to a colliding `<b></b>` variant, whose U+003C sorts below every letter, and
+ // the generator ends up writing its own content under a heading carrying
+ // inline HTML. Both live collision routes need it: `Other` beside
+ // `<b></b>Other` with no untyped document, and `Subdirectories` beside
+ // `<b></b>Subdirectories` with no child indexes.
+ const bucketFor = (heading: string): RenderableLink[] => {
+ const key = headingContentIdentity(heading);
+ const pinned = GENERATOR_OWNED_HEADINGS.has(heading);
+ const existing = grouped.get(key);
+ if (!existing) {
+ const created: Bucket = { heading, pinned, links: [] };
+ grouped.set(key, created);
+ return created.links;
+ }
+ if (pinned && !existing.pinned) {
+ existing.heading = heading;
+ existing.pinned = true;
+ } else if (!pinned && !existing.pinned && prefersSpelling(heading, existing.heading)) {
+ existing.heading = heading;
+ }
+ return existing.links;
+ };
+
for (const entry of entries) {
- const section = sectionOf(entry);
- const bucket = grouped.get(section);
- if (bucket) bucket.push(entry);
- else grouped.set(section, [entry]);
+ bucketFor(sectionOf(entry)).push(entry);
}
if (subdirectories.length > 0) {
// `Subdirectories` is also a valid document type. Sharing its bucket keeps
// the semantic heading unique when documents and child indexes coexist.
- const bucket = grouped.get(SUBDIRECTORY_SECTION);
- if (bucket) bucket.push(...subdirectories);
- else grouped.set(SUBDIRECTORY_SECTION, [...subdirectories]);
+ bucketFor(SUBDIRECTORY_SECTION).push(...subdirectories);
}
const blocks = [...grouped]
- .sort(([left], [right]) => compareSections(left, right))
- .map(([heading, links]) => {
- const body = links
- .slice()
- .sort(compareLinks)
- .map((link) => renderLink(link, directory))
- .join('\n');
- return `## ${heading}\n\n${body}`;
- });
+ .filter(([key]) => key !== titleKey)
+ .sort(([, left], [, right]) => compareSections(left.heading, right.heading))
+ .map(([, { heading, links }]) => `## ${heading}\n\n${renderBody(links, directory)}`);
const header = options.isRoot ? `---\nokf_version: "${GENERATED_OKF_VERSION}"\n---\n\n` : '';
- // An empty bundle keeps the title and gains no section: an empty `## Other`
- // would claim a grouping that describes nothing, while the title alone reads
- // as a bundle with no documents yet. Holding the title in both cases is what
- // makes "exactly one top-level heading" unconditional rather than a property
- // that lapses whenever the bundle happens to be empty.
- return `${header}${[`# ${INDEX_TITLE}`, ...blocks].join('\n\n')}\n`;
+ // The title renders alone when nothing shares its bucket and above a bullet
+ // list when documents do. Emitting it on both branches is what makes "exactly
+ // one top-level heading" unconditional rather than a property that lapses
+ // whenever the bundle happens to be empty. An empty bundle also gains no
+ // section, since a heading with no links would claim a grouping that
+ // describes nothing.
+ // DRIFT WARNING: bullets sitting directly under the level-one title depend on
+ // `okf/index-shape` opening a section on a heading at ANY level
+ // (`packages/core/src/markdown/lint/rules/index-shape.ts`). Tightening that
+ // rule so only `##` opens a section would make every index for a folder
+ // holding a title-colliding document report a finding against a file its
+ // author cannot edit. The two live in different packages and TypeScript
+ // cannot see the coupling.
+ const titleLinks = grouped.get(titleKey)?.links ?? [];
+ const title =
+ titleLinks.length === 0
+ ? `# ${INDEX_TITLE}`
+ : `# ${INDEX_TITLE}\n\n${renderBody(titleLinks, directory)}`;
+
+ return `${header}${[title, ...blocks].join('\n\n')}\n`;
}