Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
/**
* Markdown chunker — two-pass structural split + token-budget packer.
*
* Anchored in docs/plans/2026-05-16-brain-search-design.md §6.
*
* Token approximation: whitespace word count, except for scripts written
* without spaces between words (Han, Hiragana, Katakana, Hangul, Bopomofo,
* Thai, Lao, Khmer, CJK punctuation and fullwidth forms), where every
* character is one token - a paragraph of Chinese has no whitespace, and
* counting it as one "word" sent 9,000-character chunks to providers that
* reject them (issue #186). Long whitespace-free runs on the Latin side
* are the same trap with the sign flipped: a subword tokenizer spends
* roughly one token per two to four characters on a URL, a file path or a
* hash, so a run of sixteen or more such characters is charged one token
* per three characters instead of one (issue #272). Deterministic across
* Bun/Node, dependency-free, machine-independent — so the same vault
* hashes the same chunks on every Syncthing peer.
*
* Budget invariant: every chunk's `tokenCount`, overlap included, is at
* most `maxTokens`. A block too large for one chunk is split by line, then
* after sentence punctuation, then at a token boundary - and inside a
* long whitespace-free run those token boundaries fall every few
* characters, so one giant line is still distributed across chunks
* without dropping or duplicating source text. A cut that would land
* inside a `[[...]]` wikilink moves out of the link whenever any
* alternative exists, and severs the link only when the link alone
* exceeds the whole budget: the indexer extracts links per chunk, so a
* span cut in half would vanish from the link index even though its text
* is still indexed.
*/
/**
* Version of the chunking rules. Chunks are only recomputed when a file
* changes, so a change to how the SAME text is chunked must bump this: the
* indexer compares it with the value an index recorded and re-chunks every
* document once when they differ.
*
* 1 - whitespace word count, overlap outside the budget (through v1.57).
* 2 - per-character count for unspaced scripts; overlap inside the
* budget; oversize blocks split (issue #186).
* 3 - whitespace-free runs of sixteen or more characters charged about
* one token per three characters and split inside the run, so a
* long URL or hash can no longer pass the budget as one "word"
* (issue #272). Cut placement keeps `[[...]]` wikilinks whole
* whenever any alternative cut exists, because links are extracted
* per chunk.
*/
export const CHUNKER_VERSION = 3;
/**
* Token budget of one chunk when the caller names none. Exported so a
* renderer that sizes blocks to fit one chunk (the table note's row
* groups) states its budget against this one rather than a copy.
*/
export const DEFAULT_CHUNK_MAX_TOKENS = 800;
const DEFAULT_MIN_TOKENS = 100;
const DEFAULT_OVERLAP_TOKENS = 100;
export interface ChunkOptions {
readonly maxTokens?: number;
readonly minTokens?: number;
readonly overlapTokens?: number;
}
export interface MarkdownChunk {
readonly chunkIndex: number;
readonly content: string;
readonly startLine: number;
readonly endLine: number;
readonly tokenCount: number;
/**
* Breadcrumb of headings the chunk falls under, joined by " > "
* (e.g. "Top > Section A"). Computed at the chunk's end so a chunk
* spanning into a deeper subsection is anchored to the deepest
* heading it covers. Empty when no heading precedes the chunk.
* Indexed in a dedicated FTS column so a mid-document chunk keeps its
* topical anchor; never part of the display content.
*/
readonly headingPath: string;
}
export interface ChunkResult {
readonly title: string | null;
readonly chunks: ReadonlyArray<MarkdownChunk>;
readonly warnings: ReadonlyArray<string>;
}
interface Line {
readonly num: number;
readonly text: string;
}
type BlockKind = "frontmatter" | "code" | "heading" | "list" | "table" | "paragraph";
interface Block {
readonly kind: BlockKind;
readonly lines: ReadonlyArray<Line>;
readonly tokenCount: number;
}
/**
* A character of a script written without spaces between words, plus CJK
* punctuation (U+3000-303F) and halfwidth/fullwidth forms (U+FF00-FFEF).
* Sticky, so it tests the code point AT `lastIndex` without allocating.
* Cyrillic, Greek, Arabic and every other spaced script is deliberately
* absent: those keep the word count.
*/
const UNSPACED_RE =
/[\p{Script=Han}\p{Script=Hiragana}\p{Script=Katakana}\p{Script=Hangul}\p{Script=Bopomofo}\p{Script=Thai}\p{Script=Lao}\p{Script=Khmer} -〿-]/uy;
/** A combining mark (Thai vowel signs, the kana voicing marks, ...). */
const MARK_RE = /\p{M}/uy;
/** Every unspaced-script code point is at or above Thai (U+0E00). */
const FIRST_UNSPACED_CODE_POINT = 0x0e00;
/**
* A whitespace-free run shorter than this many code units keeps the plain
* word count: ordinary prose words, however long, stay one estimated
* token, and the space-separated corpus hashes exactly as before. The
* threshold sits above any real word with its punctuation attached, where
* a run stops looking like prose and starts looking like a URL, a path or
* a hash - the shapes that hide hundreds of real tokens behind a
* one-"word" estimate.
*/
const DENSE_RUN_MIN_CHARS = 16;
/**
* Code units charged per token inside a whitespace-free run of at least
* {@link DENSE_RUN_MIN_CHARS}. Subword tokenizers break such text into
* small fragments - about one token per two to four characters - so
* counting a 3,700-character URL as one "word" under-reported reality by
* three orders of magnitude and let a single chunk thousands of tokens
* past the model's input window through the budget (issue #272). Three is
* the conservative end of that range, chosen in the spirit of the
* unspaced scripts' one token per character: over-charging only splits a
* chunk earlier, while under-charging silently breaks the window.
*/
const DENSE_CHARS_PER_TOKEN = 3;
function matchesAt(re: RegExp, text: string, i: number): boolean {
re.lastIndex = i;
return re.test(text);
}
/**
* End (exclusive) of the whitespace-free run of code points below
* {@link FIRST_UNSPACED_CODE_POINT} that starts at `from`: the run stops
* at whitespace or at the first character of an unspaced script - exactly
* the places the word walk stops, so the measured run is the same span
* the old code counted as one "word". Cyrillic, Greek and the other
* scripts below Thai take part, as they took part in the word they
* extended.
*/
function wordRunEnd(text: string, from: number): number {
let end = from;
while (end < text.length) {
const c = text.charCodeAt(end);
if (c >= FIRST_UNSPACED_CODE_POINT) break;
if (c === 0x20 || c === 0x09 || c === 0x0a || c === 0x0d) break;
end++;
}
return end;
}
/**
* Walk `text`'s tokens: a run of non-whitespace is one token, except that
* each unspaced-script character is a token of its own (a combining mark
* after one stays part of it), and a whitespace-free run of at least
* {@link DENSE_RUN_MIN_CHARS} code units is charged one token per
* {@link DENSE_CHARS_PER_TOKEN} code units, with a token start every few
* characters so a cut can fall inside the run. Pushes each token's start
* offset onto `starts` when given; returns the count.
*
* Whitespace is exactly space, tab, LF and CR, as it always was: text
* with no unspaced-script character and no long unbroken run counts
* exactly as before.
*/
function walkTokens(text: string, starts: number[] | null): number {
let count = 0;
let inWord = false;
let afterUnspaced = false;
for (let i = 0; i < text.length;) {
const c = text.charCodeAt(i);
if (c < FIRST_UNSPACED_CODE_POINT) {
const isSpace = c === 0x20 || c === 0x09 || c === 0x0a || c === 0x0d;
if (isSpace) {
inWord = false;
afterUnspaced = false;
i++;
continue;
}
if (afterUnspaced && c >= 0x0300 && matchesAt(MARK_RE, text, i)) {
i++;
continue;
}
if (!inWord) {
const runEnd = wordRunEnd(text, i);
if (runEnd - i >= DENSE_RUN_MIN_CHARS) {
const runLen = runEnd - i;
count += Math.ceil(runLen / DENSE_CHARS_PER_TOKEN);
if (starts) {
for (let p = i; p < runEnd; p += DENSE_CHARS_PER_TOKEN) starts.push(p);
}
i = runEnd;
// The run stopped on whitespace or an unspaced-script
// character; both start a fresh token, so the word state resets.
inWord = false;
afterUnspaced = false;
continue;
}
count++;
starts?.push(i);
inWord = true;
}
afterUnspaced = false;
i++;
continue;
}
const cp = text.codePointAt(i)!;
const width = cp > 0xffff ? 2 : 1;
if (afterUnspaced && matchesAt(MARK_RE, text, i)) {
// Stays with the character it modifies.
} else if (matchesAt(UNSPACED_RE, text, i)) {
count++;
starts?.push(i);
inWord = false;
afterUnspaced = true;
} else {
if (!inWord) {
count++;
starts?.push(i);
inWord = true;
}
afterUnspaced = false;
}
i += width;
}
return count;
}
/**
* The chunker's token count of `text`, the same count every budget in this
* module is measured with. Exported so a renderer can size a block to fit
* one chunk without re-deriving the rules.
*/
export function countChunkTokens(text: string): number {
if (text.length === 0) return 0;
return walkTokens(text, null);
}
/** UTF-16 offset of every token start in `text`, in order. */
function tokenStarts(text: string): number[] {
const starts: number[] = [];
walkTokens(text, starts);
return starts;
}
/**
* Half-open offsets of every `[[...]]` wikilink span in `text`, in order.
* Wikilinks do not nest, so the first `]]` closes a span, and an
* unterminated `[[` is no span at all - the link extractor cannot resolve
* one either. The indexer extracts links per chunk, so a span severed at
* a chunk boundary disappears from the index even though its text is
* still indexed.
*/
function wikilinkSpans(text: string): Array<{ readonly start: number; readonly end: number }> {
const spans: Array<{ start: number; end: number }> = [];
let i = 0;
while (i < text.length) {
const open = text.indexOf("[[", i);
if (open === -1) break;
const close = text.indexOf("]]", open + 2);
if (close === -1) break;
spans.push({ start: open, end: close + 2 });
i = close + 2;
}
return spans;
}
/**
* The longest suffix of `text` holding at most `budget` tokens. A cut
* that would open the suffix inside a wikilink retreats past the link's
* closing brackets: the link stays whole in the chunk it came from and
* the next chunk never opens with a severed fragment of it. The suffix
* only shrinks, so the budget holds.
*/
function tailTokens(text: string, budget: number): string {
if (budget <= 0) return "";
const starts = tokenStarts(text);
if (starts.length <= budget) return text;
const from = starts[starts.length - budget]!;
for (const span of wikilinkSpans(text)) {
if (from > span.start && from < span.end) return text.slice(span.end);
}
return text.slice(from);
}
function tokensOfLines(lines: ReadonlyArray<Line>): number {
let total = 0;
for (const l of lines) total += countChunkTokens(l.text);
return total;
}
function makeBlock(kind: BlockKind, lines: Line[]): Block {
return { kind, lines, tokenCount: tokensOfLines(lines) };
}
function isBlank(s: string): boolean {
return s.trim() === "";
}
const FENCE_RE = /^(?:```|~~~)/;
const HEADING_RE = /^#{1,6}\s/;
const LIST_ITEM_RE = /^\s*(?:[-*+]\s|\d+\.\s)/;
const TABLE_SEP_RE = /^\s*\|?\s*:?-{3,}:?\s*(?:\|\s*:?-{3,}:?\s*)+\|?\s*$/;
/** Split the raw text into a 1-indexed line array. */
function splitLines(text: string): Line[] {
const out: Line[] = [];
let lineNum = 1;
let start = 0;
for (let i = 0; i < text.length; i++) {
const c = text.charCodeAt(i);
if (c === 0x0a /* \n */) {
const seg = text.slice(start, i);
const cleaned = seg.endsWith("\r") ? seg.slice(0, -1) : seg;
out.push({ num: lineNum++, text: cleaned });
start = i + 1;
}
}
if (start <= text.length) {
const seg = text.slice(start);
if (seg !== "" || out.length === 0) {
const cleaned = seg.endsWith("\r") ? seg.slice(0, -1) : seg;
out.push({ num: lineNum, text: cleaned });
}
}
return out;
}
function extractFrontmatter(lines: ReadonlyArray<Line>): {
block: Block | null;
remaining: ReadonlyArray<Line>;
warnings: string[];
} {
if (lines.length === 0 || lines[0]!.text.trim() !== "---") {
return { block: null, remaining: lines, warnings: [] };
}
for (let i = 1; i < lines.length; i++) {
if (lines[i]!.text.trim() === "---") {
const fmLines = lines.slice(0, i + 1);
return {
block: makeBlock("frontmatter", [...fmLines]),
remaining: lines.slice(i + 1),
warnings: [],
};
}
}
// Unterminated frontmatter: omit FM block, keep file otherwise indexable.
return {
block: null,
remaining: lines.slice(1),
warnings: ["malformed frontmatter (no closing '---')"],
};
}
function readCodeBlock(lines: ReadonlyArray<Line>, start: number): { end: number } {
// `lines[start]` is the opening fence.
const fenceText = lines[start]!.text.trim();
const fenceMark = fenceText.startsWith("```") ? "```" : "~~~";
for (let i = start + 1; i < lines.length; i++) {
if (lines[i]!.text.trim().startsWith(fenceMark)) return { end: i };
}
return { end: lines.length - 1 }; // unclosed: run to EOF
}
function readList(lines: ReadonlyArray<Line>, start: number): { end: number } {
let i = start + 1;
while (i < lines.length) {
const t = lines[i]!.text;
if (isBlank(t)) return { end: i - 1 };
if (HEADING_RE.test(t)) return { end: i - 1 };
if (FENCE_RE.test(t)) return { end: i - 1 };
// Continuation of list: list item or indented continuation.
if (LIST_ITEM_RE.test(t) || /^\s{2,}/.test(t)) {
i++;
continue;
}
return { end: i - 1 };
}
return { end: lines.length - 1 };
}
function isTableStart(lines: ReadonlyArray<Line>, start: number): boolean {
const a = lines[start];
const b = lines[start + 1];
if (!a || !b) return false;
if (!a.text.includes("|")) return false;
return TABLE_SEP_RE.test(b.text);
}
function readTable(lines: ReadonlyArray<Line>, start: number): { end: number } {
// Header (start), separator (start+1), then rows until blank or non-pipe line.
let i = start + 2;
while (i < lines.length) {
const t = lines[i]!.text;
if (isBlank(t)) return { end: i - 1 };
if (!t.includes("|")) return { end: i - 1 };
i++;
}
return { end: lines.length - 1 };
}
function readParagraph(lines: ReadonlyArray<Line>, start: number): { end: number } {
let i = start + 1;
while (i < lines.length) {
const t = lines[i]!.text;
if (isBlank(t)) return { end: i - 1 };
if (HEADING_RE.test(t)) return { end: i - 1 };
if (FENCE_RE.test(t)) return { end: i - 1 };
if (LIST_ITEM_RE.test(t)) return { end: i - 1 };
if (isTableStart(lines, i)) return { end: i - 1 };
i++;
}
return { end: lines.length - 1 };
}
function splitIntoBlocks(lines: ReadonlyArray<Line>): { blocks: Block[]; warnings: string[] } {
const blocks: Block[] = [];
const warnings: string[] = [];
const { block: fm, remaining, warnings: fmWarn } = extractFrontmatter(lines);
warnings.push(...fmWarn);
if (fm) blocks.push(fm);
let i = 0;
while (i < remaining.length) {
const text = remaining[i]!.text;
if (isBlank(text)) {
i++;
continue;
}
if (FENCE_RE.test(text)) {
const { end } = readCodeBlock(remaining, i);
blocks.push(makeBlock("code", remaining.slice(i, end + 1)));
i = end + 1;
continue;
}
if (HEADING_RE.test(text)) {
blocks.push(makeBlock("heading", [remaining[i]!]));
i++;
continue;
}
if (LIST_ITEM_RE.test(text)) {
const { end } = readList(remaining, i);
blocks.push(makeBlock("list", remaining.slice(i, end + 1)));
i = end + 1;
continue;
}
if (isTableStart(remaining, i)) {
const { end } = readTable(remaining, i);
blocks.push(makeBlock("table", remaining.slice(i, end + 1)));
i = end + 1;
continue;
}
const { end } = readParagraph(remaining, i);
blocks.push(makeBlock("paragraph", remaining.slice(i, end + 1)));
i = end + 1;
}
return { blocks, warnings };
}
interface DraftChunk {
blocks: Block[];
tokens: number;
}
function newDraft(): DraftChunk {
return { blocks: [], tokens: 0 };
}
/**
* The tail of `lines` holding at most `budget` tokens, in order. Whole
* lines are taken from the end while they fit; when not even the last line
* fits, its own tail is taken, cut at a token boundary - so a dense
* document still carries overlap, and the overlap never costs more than
* its budget.
*/
function tailLines(lines: ReadonlyArray<Line>, budget: number): Line[] {
if (budget <= 0) return [];
let acc = 0;
const tail: Line[] = [];
for (let i = lines.length - 1; i >= 0; i--) {
const line = lines[i]!;
const t = countChunkTokens(line.text);
if (acc + t > budget) {
if (tail.length === 0) {
const cut = tailTokens(line.text, budget);
if (cut !== "") tail.unshift({ num: line.num, text: cut });
}
break;
}
tail.unshift(line);
acc += t;
}
return tail;
}
function takeOverlap(prev: DraftChunk, overlapTokens: number): Line[] {
if (overlapTokens <= 0 || prev.blocks.length === 0) return [];
const allLines: Line[] = [];
for (const b of prev.blocks) for (const l of b.lines) allLines.push(l);
return tailLines(allLines, overlapTokens);
}
function emitChunk(
index: number,
overlap: ReadonlyArray<Line>,
draft: DraftChunk,
maxTokens: number,
): MarkdownChunk {
const bodyLines: Line[] = [];
for (const b of draft.blocks) for (const l of b.lines) bodyLines.push(l);
// The packer leaves room for the overlap; this only bites when the
// configured overlap is larger than a chunk can spare.
const fitted =
tokensOfLines(overlap) + draft.tokens > maxTokens
? tailLines(overlap, maxTokens - draft.tokens)
: overlap;
const allText = [...fitted.map((l) => l.text), ...bodyLines.map((l) => l.text)].join("\n");
const tokenCount = countChunkTokens(allText);
return Object.freeze({
chunkIndex: index,
content: allText,
startLine: bodyLines[0]?.num ?? fitted[0]?.num ?? 1,
endLine: bodyLines[bodyLines.length - 1]?.num ?? fitted[fitted.length - 1]?.num ?? 1,
tokenCount,
headingPath: "",
});
}
/**
* Sentence-final punctuation of the unspaced scripts. Each is a token of
* its own, so a cut right after one is a token boundary.
*/
const SENTENCE_ENDS: ReadonlySet<string> = new Set(["。", "!", "?", ";", "。", "."]);
/** Closing quotes and brackets that belong to the sentence before them. */
const SENTENCE_CLOSERS: ReadonlySet<string> = new Set([
"」",
"』",
")",
"】",
"〕",
"》",
"〉",
"”",
"’",
]);
/** `text` cut after each sentence end (and its closing quotes). Concatenates back to `text`. */
function sentences(text: string): string[] {
const out: string[] = [];
let start = 0;
let i = 0;
while (i < text.length) {
const ch = String.fromCodePoint(text.codePointAt(i)!);
i += ch.length;
if (!SENTENCE_ENDS.has(ch)) continue;
while (i < text.length) {
const next = String.fromCodePoint(text.codePointAt(i)!);
if (!SENTENCE_ENDS.has(next) && !SENTENCE_CLOSERS.has(next)) break;
i += next.length;
}
out.push(text.slice(start, i));
start = i;
}
if (start < text.length) out.push(text.slice(start));
return out;
}
/**
* `text` cut into pieces of about `budget` tokens (the last may be
* shorter), at token starts: never inside a word, a surrogate pair, or
* between a character and its combining mark. A boundary that would fall
* inside a `[[...]]` wikilink moves out of the link whenever any
* alternative exists - the whole link is absorbed into the piece when it
* still fits the budget, else the cut retreats to before the link - and
* only a link too large for the budget on its own is severed. Piece
* boundaries advance monotonically, so pieces still concatenate back to
* `text`.
*/
function hardCut(text: string, budget: number): string[] {
const starts = tokenStarts(text);
const spans = wikilinkSpans(text);
const out: string[] = [];
let from = 0;
let ki = 0; // first token start at or after `from`; advances monotonically
while (from < text.length) {
while (ki < starts.length && starts[ki]! < from) ki++;
let to = ki + budget < starts.length ? starts[ki + budget]! : text.length;
for (const span of spans) {
if (to > span.start && to < span.end) {
if (countChunkTokens(text.slice(from, span.end)) <= budget) to = span.end;
else if (span.start > from) to = span.start;
break; // spans are ordered and disjoint: at most one holds `to`
}
}
out.push(text.slice(from, to));
from = to;
}
return out;
}
/**
* One line longer than `budget`, cut into pieces within it: whole sentences
* packed together where they fit, a sentence longer than the budget cut at
* token boundaries. Concatenates back to the line.
*/
function splitLine(text: string, budget: number): string[] {
const parts: string[] = [];
let acc = "";
for (const s of sentences(text)) {
if (countChunkTokens(s) > budget) {
if (acc !== "") parts.push(acc);
const pieces = hardCut(s, budget);
parts.push(...pieces.slice(0, -1));
acc = pieces[pieces.length - 1] ?? "";
continue;
}
const merged = acc + s;
if (acc !== "" && countChunkTokens(merged) > budget) {
parts.push(acc);
acc = s;
} else {
acc = merged;
}
}
if (acc !== "") parts.push(acc);
return parts;
}
/**
* A block too large for one chunk, as a run of single-line blocks of the
* same kind, each within `budget`: its lines, and any line over the budget
* split by {@link splitLine}. Line numbers are kept, so a split line's
* pieces all point at the line they came from.
*/
function splitBlock(block: Block, budget: number): Block[] {
const units: Block[] = [];
for (const line of block.lines) {
const t = countChunkTokens(line.text);
if (t <= budget) {
units.push({ kind: block.kind, lines: [line], tokenCount: t });
continue;
}
for (const part of splitLine(line.text, budget)) {
units.push({
kind: block.kind,
lines: [{ num: line.num, text: part }],
tokenCount: countChunkTokens(part),
});
}
}
return units;
}
function packBlocks(
blocks: ReadonlyArray<Block>,
opts: { maxTokens: number; minTokens: number; overlapTokens: number },
): MarkdownChunk[] {
const out: MarkdownChunk[] = [];
let draft = newDraft();
let pendingOverlap: Line[] = [];
let pendingOverlapTokens = 0;
// Pieces of a split block leave room for a full overlap, so the overlap
// survives the split - up to half the chunk; a larger configured overlap
// is trimmed to fit instead (see emitChunk).
const unitBudget = Math.max(
1,
opts.maxTokens - Math.min(opts.overlapTokens, Math.floor(opts.maxTokens / 2)),
);
const setOverlap = (lines: Line[]) => {
pendingOverlap = lines;
pendingOverlapTokens = tokensOfLines(lines);
};
const flush = () => {
if (draft.blocks.length === 0) return;
out.push(emitChunk(out.length, pendingOverlap, draft, opts.maxTokens));
setOverlap(takeOverlap(draft, opts.overlapTokens));
draft = newDraft();
};
// The overlap is prepended to the chunk, so it spends the same budget.
const fits = (block: Block) =>
pendingOverlapTokens + draft.tokens + block.tokenCount <= opts.maxTokens;
const push = (block: Block) => {
draft.blocks.push(block);
draft.tokens += block.tokenCount;
};
for (const block of blocks) {
if (block.kind === "frontmatter") {
// Flush any in-progress (shouldn't happen — frontmatter is first), then
// emit the frontmatter as its own chunk(s) *without* overlap since
// nothing precedes it. Only an oversize frontmatter becomes several.
flush();
const units = block.tokenCount > opts.maxTokens ? splitBlock(block, opts.maxTokens) : [block];
let fm = newDraft();
for (const unit of units) {
if (fm.blocks.length > 0 && fm.tokens + unit.tokenCount > opts.maxTokens) {
out.push(emitChunk(out.length, [], fm, opts.maxTokens));
fm = newDraft();
}
fm.blocks.push(unit);
fm.tokens += unit.tokenCount;
}
out.push(emitChunk(out.length, [], fm, opts.maxTokens));
// The next chunk should NOT include frontmatter text as overlap.
setOverlap([]);
continue;
}
if (block.kind === "heading") {
const hasNonHeading = draft.blocks.some((b) => b.kind !== "heading");
if (hasNonHeading && draft.tokens >= opts.minTokens) {
flush();
}
}
if (draft.blocks.length > 0 && !fits(block)) {
flush();
}
if (fits(block)) {
push(block);
continue;
}
// Too large to share a chunk with its own overlap: split it, and pack
// the pieces like any other blocks.
for (const unit of splitBlock(block, unitBudget)) {
if (draft.blocks.length > 0 && !fits(unit)) flush();
push(unit);
}
}
flush();
return out;
}
function stripQuotes(s: string): string {
const t = s.trim();
if (
t.length >= 2 &&
((t.startsWith('"') && t.endsWith('"')) || (t.startsWith("'") && t.endsWith("'")))
) {
return t.slice(1, -1);
}
return t;
}
function resolveTitle(
frontmatterBlock: Block | null,
blocks: ReadonlyArray<Block>,
filenameBase: string | null,
): string | null {
if (frontmatterBlock) {
for (const l of frontmatterBlock.lines) {
const m = l.text.match(/^title:\s*(.*)$/);
if (m) {
const v = stripQuotes(m[1] ?? "");
if (v) return v;
}
}
}
for (const b of blocks) {
if (b.kind === "heading") {
const text = b.lines[0]!.text.replace(/^#{1,6}\s+/, "").trim();
if (text) return text;
}
if (b.kind !== "frontmatter") break;
}
if (filenameBase) return filenameBase.replace(/[-_]+/g, " ").trim() || filenameBase;
return null;
}
interface HeadingMark {
readonly line: number;
readonly level: number;
readonly text: string;
}
/** Collect heading marks (line, level, text) in document order. */
function collectHeadings(blocks: ReadonlyArray<Block>): HeadingMark[] {
const out: HeadingMark[] = [];
for (const b of blocks) {
if (b.kind !== "heading") continue;
const raw = b.lines[0]!.text;
const hashes = raw.match(/^#{1,6}/);
const level = hashes ? hashes[0].length : 1;
const text = raw.replace(/^#{1,6}\s+/, "").trim();
if (text) out.push({ line: b.lines[0]!.num, level, text });
}
return out;
}
/**
* Breadcrumb of headings in effect at `atLine` (the chunk's end line):
* replay heading marks at or before the line, maintaining a level-stack
* (a heading pops every entry at its level or deeper). Returns the stack
* texts joined " > ".
*/
function headingPathAt(headings: ReadonlyArray<HeadingMark>, atLine: number): string {
const stack: HeadingMark[] = [];
for (const h of headings) {
if (h.line > atLine) break;
while (stack.length > 0 && stack[stack.length - 1]!.level >= h.level) stack.pop();
stack.push(h);
}
return stack.map((h) => h.text).join(" > ");
}
/**
* Two-pass markdown chunker. Pure function: same input → same output.
*
* Returns `chunks: []` for empty or whitespace-only input; the indexer
* still records a `documents` row so the file is tracked.
*/
export function chunkMarkdown(
text: string,
filenameBase: string | null,
opts?: ChunkOptions,
): ChunkResult {
const maxTokens = opts?.maxTokens ?? DEFAULT_CHUNK_MAX_TOKENS;
const minTokens = opts?.minTokens ?? DEFAULT_MIN_TOKENS;
const overlapTokens = opts?.overlapTokens ?? DEFAULT_OVERLAP_TOKENS;
const allLines = splitLines(text);
if (allLines.every((l) => isBlank(l.text))) {
return Object.freeze({
title: filenameBase ? filenameBase.replace(/[-_]+/g, " ") : null,
chunks: Object.freeze([]),
warnings: Object.freeze([]),
});
}
const { blocks, warnings } = splitIntoBlocks(allLines);
const frontmatter = blocks.find((b) => b.kind === "frontmatter") ?? null;
const title = resolveTitle(frontmatter, blocks, filenameBase);
const packed = packBlocks(blocks, { maxTokens, minTokens, overlapTokens });
// Anchor each chunk to the heading breadcrumb active at its end (the
// deepest section it spans).
const headings = collectHeadings(blocks);
const chunks =
headings.length === 0
? packed
: packed.map((c) => Object.freeze({ ...c, headingPath: headingPathAt(headings, c.endLine) }));
return Object.freeze({
title,
chunks: Object.freeze(chunks),
warnings: Object.freeze(warnings),
});
}