Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript
28.8 KB
Raw Blame
/**
 * Markdown chunker — two-pass structural split + token-budget packer.
 *
 * Anchored in docs/plans/2026-05-16-brain-search-design.md §6.
 *
 * Token approximation: whitespace word count, except for scripts written
 * without spaces between words (Han, Hiragana, Katakana, Hangul, Bopomofo,
 * Thai, Lao, Khmer, CJK punctuation and fullwidth forms), where every
 * character is one token - a paragraph of Chinese has no whitespace, and
 * counting it as one "word" sent 9,000-character chunks to providers that
 * reject them (issue #186). Long whitespace-free runs on the Latin side
 * are the same trap with the sign flipped: a subword tokenizer spends
 * roughly one token per two to four characters on a URL, a file path or a
 * hash, so a run of sixteen or more such characters is charged one token
 * per three characters instead of one (issue #272). Deterministic across
 * Bun/Node, dependency-free, machine-independent — so the same vault
 * hashes the same chunks on every Syncthing peer.
 *
 * Budget invariant: every chunk's `tokenCount`, overlap included, is at
 * most `maxTokens`. A block too large for one chunk is split by line, then
 * after sentence punctuation, then at a token boundary - and inside a
 * long whitespace-free run those token boundaries fall every few
 * characters, so one giant line is still distributed across chunks
 * without dropping or duplicating source text. A cut that would land
 * inside a `[[...]]` wikilink moves out of the link whenever any
 * alternative exists, and severs the link only when the link alone
 * exceeds the whole budget: the indexer extracts links per chunk, so a
 * span cut in half would vanish from the link index even though its text
 * is still indexed.
 */

/**
 * Version of the chunking rules. Chunks are only recomputed when a file
 * changes, so a change to how the SAME text is chunked must bump this: the
 * indexer compares it with the value an index recorded and re-chunks every
 * document once when they differ.
 *
 *   1 - whitespace word count, overlap outside the budget (through v1.57).
 *   2 - per-character count for unspaced scripts; overlap inside the
 *       budget; oversize blocks split (issue #186).
 *   3 - whitespace-free runs of sixteen or more characters charged about
 *       one token per three characters and split inside the run, so a
 *       long URL or hash can no longer pass the budget as one "word"
 *       (issue #272). Cut placement keeps `[[...]]` wikilinks whole
 *       whenever any alternative cut exists, because links are extracted
 *       per chunk.
 */
export const CHUNKER_VERSION = 3;

/**
 * Token budget of one chunk when the caller names none. Exported so a
 * renderer that sizes blocks to fit one chunk (the table note's row
 * groups) states its budget against this one rather than a copy.
 */
export const DEFAULT_CHUNK_MAX_TOKENS = 800;
const DEFAULT_MIN_TOKENS = 100;
const DEFAULT_OVERLAP_TOKENS = 100;

export interface ChunkOptions {
  readonly maxTokens?: number;
  readonly minTokens?: number;
  readonly overlapTokens?: number;
}

export interface MarkdownChunk {
  readonly chunkIndex: number;
  readonly content: string;
  readonly startLine: number;
  readonly endLine: number;
  readonly tokenCount: number;
  /**
   * Breadcrumb of headings the chunk falls under, joined by " > "
   * (e.g. "Top > Section A"). Computed at the chunk's end so a chunk
   * spanning into a deeper subsection is anchored to the deepest
   * heading it covers. Empty when no heading precedes the chunk.
   * Indexed in a dedicated FTS column so a mid-document chunk keeps its
   * topical anchor; never part of the display content.
   */
  readonly headingPath: string;
}

export interface ChunkResult {
  readonly title: string | null;
  readonly chunks: ReadonlyArray<MarkdownChunk>;
  readonly warnings: ReadonlyArray<string>;
}

interface Line {
  readonly num: number;
  readonly text: string;
}

type BlockKind = "frontmatter" | "code" | "heading" | "list" | "table" | "paragraph";

interface Block {
  readonly kind: BlockKind;
  readonly lines: ReadonlyArray<Line>;
  readonly tokenCount: number;
}

/**
 * A character of a script written without spaces between words, plus CJK
 * punctuation (U+3000-303F) and halfwidth/fullwidth forms (U+FF00-FFEF).
 * Sticky, so it tests the code point AT `lastIndex` without allocating.
 * Cyrillic, Greek, Arabic and every other spaced script is deliberately
 * absent: those keep the word count.
 */
const UNSPACED_RE =
  /[\p{Script=Han}\p{Script=Hiragana}\p{Script=Katakana}\p{Script=Hangul}\p{Script=Bopomofo}\p{Script=Thai}\p{Script=Lao}\p{Script=Khmer} -〿＀-￯]/uy;

/** A combining mark (Thai vowel signs, the kana voicing marks, ...). */
const MARK_RE = /\p{M}/uy;

/** Every unspaced-script code point is at or above Thai (U+0E00). */
const FIRST_UNSPACED_CODE_POINT = 0x0e00;

/**
 * A whitespace-free run shorter than this many code units keeps the plain
 * word count: ordinary prose words, however long, stay one estimated
 * token, and the space-separated corpus hashes exactly as before. The
 * threshold sits above any real word with its punctuation attached, where
 * a run stops looking like prose and starts looking like a URL, a path or
 * a hash - the shapes that hide hundreds of real tokens behind a
 * one-"word" estimate.
 */
const DENSE_RUN_MIN_CHARS = 16;

/**
 * Code units charged per token inside a whitespace-free run of at least
 * {@link DENSE_RUN_MIN_CHARS}. Subword tokenizers break such text into
 * small fragments - about one token per two to four characters - so
 * counting a 3,700-character URL as one "word" under-reported reality by
 * three orders of magnitude and let a single chunk thousands of tokens
 * past the model's input window through the budget (issue #272). Three is
 * the conservative end of that range, chosen in the spirit of the
 * unspaced scripts' one token per character: over-charging only splits a
 * chunk earlier, while under-charging silently breaks the window.
 */
const DENSE_CHARS_PER_TOKEN = 3;

function matchesAt(re: RegExp, text: string, i: number): boolean {
  re.lastIndex = i;
  return re.test(text);
}

/**
 * End (exclusive) of the whitespace-free run of code points below
 * {@link FIRST_UNSPACED_CODE_POINT} that starts at `from`: the run stops
 * at whitespace or at the first character of an unspaced script - exactly
 * the places the word walk stops, so the measured run is the same span
 * the old code counted as one "word". Cyrillic, Greek and the other
 * scripts below Thai take part, as they took part in the word they
 * extended.
 */
function wordRunEnd(text: string, from: number): number {
  let end = from;
  while (end < text.length) {
    const c = text.charCodeAt(end);
    if (c >= FIRST_UNSPACED_CODE_POINT) break;
    if (c === 0x20 || c === 0x09 || c === 0x0a || c === 0x0d) break;
    end++;
  }
  return end;
}

/**
 * Walk `text`'s tokens: a run of non-whitespace is one token, except that
 * each unspaced-script character is a token of its own (a combining mark
 * after one stays part of it), and a whitespace-free run of at least
 * {@link DENSE_RUN_MIN_CHARS} code units is charged one token per
 * {@link DENSE_CHARS_PER_TOKEN} code units, with a token start every few
 * characters so a cut can fall inside the run. Pushes each token's start
 * offset onto `starts` when given; returns the count.
 *
 * Whitespace is exactly space, tab, LF and CR, as it always was: text
 * with no unspaced-script character and no long unbroken run counts
 * exactly as before.
 */
function walkTokens(text: string, starts: number[] | null): number {
  let count = 0;
  let inWord = false;
  let afterUnspaced = false;
  for (let i = 0; i < text.length;) {
    const c = text.charCodeAt(i);
    if (c < FIRST_UNSPACED_CODE_POINT) {
      const isSpace = c === 0x20 || c === 0x09 || c === 0x0a || c === 0x0d;
      if (isSpace) {
        inWord = false;
        afterUnspaced = false;
        i++;
        continue;
      }
      if (afterUnspaced && c >= 0x0300 && matchesAt(MARK_RE, text, i)) {
        i++;
        continue;
      }
      if (!inWord) {
        const runEnd = wordRunEnd(text, i);
        if (runEnd - i >= DENSE_RUN_MIN_CHARS) {
          const runLen = runEnd - i;
          count += Math.ceil(runLen / DENSE_CHARS_PER_TOKEN);
          if (starts) {
            for (let p = i; p < runEnd; p += DENSE_CHARS_PER_TOKEN) starts.push(p);
          }
          i = runEnd;
          // The run stopped on whitespace or an unspaced-script
          // character; both start a fresh token, so the word state resets.
          inWord = false;
          afterUnspaced = false;
          continue;
        }
        count++;
        starts?.push(i);
        inWord = true;
      }
      afterUnspaced = false;
      i++;
      continue;
    }
    const cp = text.codePointAt(i)!;
    const width = cp > 0xffff ? 2 : 1;
    if (afterUnspaced && matchesAt(MARK_RE, text, i)) {
      // Stays with the character it modifies.
    } else if (matchesAt(UNSPACED_RE, text, i)) {
      count++;
      starts?.push(i);
      inWord = false;
      afterUnspaced = true;
    } else {
      if (!inWord) {
        count++;
        starts?.push(i);
        inWord = true;
      }
      afterUnspaced = false;
    }
    i += width;
  }
  return count;
}

/**
 * The chunker's token count of `text`, the same count every budget in this
 * module is measured with. Exported so a renderer can size a block to fit
 * one chunk without re-deriving the rules.
 */
export function countChunkTokens(text: string): number {
  if (text.length === 0) return 0;
  return walkTokens(text, null);
}

/** UTF-16 offset of every token start in `text`, in order. */
function tokenStarts(text: string): number[] {
  const starts: number[] = [];
  walkTokens(text, starts);
  return starts;
}

/**
 * Half-open offsets of every `[[...]]` wikilink span in `text`, in order.
 * Wikilinks do not nest, so the first `]]` closes a span, and an
 * unterminated `[[` is no span at all - the link extractor cannot resolve
 * one either. The indexer extracts links per chunk, so a span severed at
 * a chunk boundary disappears from the index even though its text is
 * still indexed.
 */
function wikilinkSpans(text: string): Array<{ readonly start: number; readonly end: number }> {
  const spans: Array<{ start: number; end: number }> = [];
  let i = 0;
  while (i < text.length) {
    const open = text.indexOf("[[", i);
    if (open === -1) break;
    const close = text.indexOf("]]", open + 2);
    if (close === -1) break;
    spans.push({ start: open, end: close + 2 });
    i = close + 2;
  }
  return spans;
}

/**
 * The longest suffix of `text` holding at most `budget` tokens. A cut
 * that would open the suffix inside a wikilink retreats past the link's
 * closing brackets: the link stays whole in the chunk it came from and
 * the next chunk never opens with a severed fragment of it. The suffix
 * only shrinks, so the budget holds.
 */
function tailTokens(text: string, budget: number): string {
  if (budget <= 0) return "";
  const starts = tokenStarts(text);
  if (starts.length <= budget) return text;
  const from = starts[starts.length - budget]!;
  for (const span of wikilinkSpans(text)) {
    if (from > span.start && from < span.end) return text.slice(span.end);
  }
  return text.slice(from);
}

function tokensOfLines(lines: ReadonlyArray<Line>): number {
  let total = 0;
  for (const l of lines) total += countChunkTokens(l.text);
  return total;
}

function makeBlock(kind: BlockKind, lines: Line[]): Block {
  return { kind, lines, tokenCount: tokensOfLines(lines) };
}

function isBlank(s: string): boolean {
  return s.trim() === "";
}

const FENCE_RE = /^(?:```|~~~)/;
const HEADING_RE = /^#{1,6}\s/;
const LIST_ITEM_RE = /^\s*(?:[-*+]\s|\d+\.\s)/;
const TABLE_SEP_RE = /^\s*\|?\s*:?-{3,}:?\s*(?:\|\s*:?-{3,}:?\s*)+\|?\s*$/;

/** Split the raw text into a 1-indexed line array. */
function splitLines(text: string): Line[] {
  const out: Line[] = [];
  let lineNum = 1;
  let start = 0;
  for (let i = 0; i < text.length; i++) {
    const c = text.charCodeAt(i);
    if (c === 0x0a /* \n */) {
      const seg = text.slice(start, i);
      const cleaned = seg.endsWith("\r") ? seg.slice(0, -1) : seg;
      out.push({ num: lineNum++, text: cleaned });
      start = i + 1;
    }
  }
  if (start <= text.length) {
    const seg = text.slice(start);
    if (seg !== "" || out.length === 0) {
      const cleaned = seg.endsWith("\r") ? seg.slice(0, -1) : seg;
      out.push({ num: lineNum, text: cleaned });
    }
  }
  return out;
}

function extractFrontmatter(lines: ReadonlyArray<Line>): {
  block: Block | null;
  remaining: ReadonlyArray<Line>;
  warnings: string[];
} {
  if (lines.length === 0 || lines[0]!.text.trim() !== "---") {
    return { block: null, remaining: lines, warnings: [] };
  }
  for (let i = 1; i < lines.length; i++) {
    if (lines[i]!.text.trim() === "---") {
      const fmLines = lines.slice(0, i + 1);
      return {
        block: makeBlock("frontmatter", [...fmLines]),
        remaining: lines.slice(i + 1),
        warnings: [],
      };
    }
  }
  // Unterminated frontmatter: omit FM block, keep file otherwise indexable.
  return {
    block: null,
    remaining: lines.slice(1),
    warnings: ["malformed frontmatter (no closing '---')"],
  };
}

function readCodeBlock(lines: ReadonlyArray<Line>, start: number): { end: number } {
  // `lines[start]` is the opening fence.
  const fenceText = lines[start]!.text.trim();
  const fenceMark = fenceText.startsWith("```") ? "```" : "~~~";
  for (let i = start + 1; i < lines.length; i++) {
    if (lines[i]!.text.trim().startsWith(fenceMark)) return { end: i };
  }
  return { end: lines.length - 1 }; // unclosed: run to EOF
}

function readList(lines: ReadonlyArray<Line>, start: number): { end: number } {
  let i = start + 1;
  while (i < lines.length) {
    const t = lines[i]!.text;
    if (isBlank(t)) return { end: i - 1 };
    if (HEADING_RE.test(t)) return { end: i - 1 };
    if (FENCE_RE.test(t)) return { end: i - 1 };
    // Continuation of list: list item or indented continuation.
    if (LIST_ITEM_RE.test(t) || /^\s{2,}/.test(t)) {
      i++;
      continue;
    }
    return { end: i - 1 };
  }
  return { end: lines.length - 1 };
}

function isTableStart(lines: ReadonlyArray<Line>, start: number): boolean {
  const a = lines[start];
  const b = lines[start + 1];
  if (!a || !b) return false;
  if (!a.text.includes("|")) return false;
  return TABLE_SEP_RE.test(b.text);
}

function readTable(lines: ReadonlyArray<Line>, start: number): { end: number } {
  // Header (start), separator (start+1), then rows until blank or non-pipe line.
  let i = start + 2;
  while (i < lines.length) {
    const t = lines[i]!.text;
    if (isBlank(t)) return { end: i - 1 };
    if (!t.includes("|")) return { end: i - 1 };
    i++;
  }
  return { end: lines.length - 1 };
}

function readParagraph(lines: ReadonlyArray<Line>, start: number): { end: number } {
  let i = start + 1;
  while (i < lines.length) {
    const t = lines[i]!.text;
    if (isBlank(t)) return { end: i - 1 };
    if (HEADING_RE.test(t)) return { end: i - 1 };
    if (FENCE_RE.test(t)) return { end: i - 1 };
    if (LIST_ITEM_RE.test(t)) return { end: i - 1 };
    if (isTableStart(lines, i)) return { end: i - 1 };
    i++;
  }
  return { end: lines.length - 1 };
}

function splitIntoBlocks(lines: ReadonlyArray<Line>): { blocks: Block[]; warnings: string[] } {
  const blocks: Block[] = [];
  const warnings: string[] = [];
  const { block: fm, remaining, warnings: fmWarn } = extractFrontmatter(lines);
  warnings.push(...fmWarn);
  if (fm) blocks.push(fm);

  let i = 0;
  while (i < remaining.length) {
    const text = remaining[i]!.text;
    if (isBlank(text)) {
      i++;
      continue;
    }
    if (FENCE_RE.test(text)) {
      const { end } = readCodeBlock(remaining, i);
      blocks.push(makeBlock("code", remaining.slice(i, end + 1)));
      i = end + 1;
      continue;
    }
    if (HEADING_RE.test(text)) {
      blocks.push(makeBlock("heading", [remaining[i]!]));
      i++;
      continue;
    }
    if (LIST_ITEM_RE.test(text)) {
      const { end } = readList(remaining, i);
      blocks.push(makeBlock("list", remaining.slice(i, end + 1)));
      i = end + 1;
      continue;
    }
    if (isTableStart(remaining, i)) {
      const { end } = readTable(remaining, i);
      blocks.push(makeBlock("table", remaining.slice(i, end + 1)));
      i = end + 1;
      continue;
    }
    const { end } = readParagraph(remaining, i);
    blocks.push(makeBlock("paragraph", remaining.slice(i, end + 1)));
    i = end + 1;
  }

  return { blocks, warnings };
}

interface DraftChunk {
  blocks: Block[];
  tokens: number;
}

function newDraft(): DraftChunk {
  return { blocks: [], tokens: 0 };
}

/**
 * The tail of `lines` holding at most `budget` tokens, in order. Whole
 * lines are taken from the end while they fit; when not even the last line
 * fits, its own tail is taken, cut at a token boundary - so a dense
 * document still carries overlap, and the overlap never costs more than
 * its budget.
 */
function tailLines(lines: ReadonlyArray<Line>, budget: number): Line[] {
  if (budget <= 0) return [];
  let acc = 0;
  const tail: Line[] = [];
  for (let i = lines.length - 1; i >= 0; i--) {
    const line = lines[i]!;
    const t = countChunkTokens(line.text);
    if (acc + t > budget) {
      if (tail.length === 0) {
        const cut = tailTokens(line.text, budget);
        if (cut !== "") tail.unshift({ num: line.num, text: cut });
      }
      break;
    }
    tail.unshift(line);
    acc += t;
  }
  return tail;
}

function takeOverlap(prev: DraftChunk, overlapTokens: number): Line[] {
  if (overlapTokens <= 0 || prev.blocks.length === 0) return [];
  const allLines: Line[] = [];
  for (const b of prev.blocks) for (const l of b.lines) allLines.push(l);
  return tailLines(allLines, overlapTokens);
}

function emitChunk(
  index: number,
  overlap: ReadonlyArray<Line>,
  draft: DraftChunk,
  maxTokens: number,
): MarkdownChunk {
  const bodyLines: Line[] = [];
  for (const b of draft.blocks) for (const l of b.lines) bodyLines.push(l);
  // The packer leaves room for the overlap; this only bites when the
  // configured overlap is larger than a chunk can spare.
  const fitted =
    tokensOfLines(overlap) + draft.tokens > maxTokens
      ? tailLines(overlap, maxTokens - draft.tokens)
      : overlap;
  const allText = [...fitted.map((l) => l.text), ...bodyLines.map((l) => l.text)].join("\n");
  const tokenCount = countChunkTokens(allText);
  return Object.freeze({
    chunkIndex: index,
    content: allText,
    startLine: bodyLines[0]?.num ?? fitted[0]?.num ?? 1,
    endLine: bodyLines[bodyLines.length - 1]?.num ?? fitted[fitted.length - 1]?.num ?? 1,
    tokenCount,
    headingPath: "",
  });
}

/**
 * Sentence-final punctuation of the unspaced scripts. Each is a token of
 * its own, so a cut right after one is a token boundary.
 */
const SENTENCE_ENDS: ReadonlySet<string> = new Set(["。", "!", "?", ";", "。", "."]);

/** Closing quotes and brackets that belong to the sentence before them. */
const SENTENCE_CLOSERS: ReadonlySet<string> = new Set([
  "」",
  "』",
  ")",
  "】",
  "〕",
  "》",
  "〉",
  "”",
  "’",
]);

/** `text` cut after each sentence end (and its closing quotes). Concatenates back to `text`. */
function sentences(text: string): string[] {
  const out: string[] = [];
  let start = 0;
  let i = 0;
  while (i < text.length) {
    const ch = String.fromCodePoint(text.codePointAt(i)!);
    i += ch.length;
    if (!SENTENCE_ENDS.has(ch)) continue;
    while (i < text.length) {
      const next = String.fromCodePoint(text.codePointAt(i)!);
      if (!SENTENCE_ENDS.has(next) && !SENTENCE_CLOSERS.has(next)) break;
      i += next.length;
    }
    out.push(text.slice(start, i));
    start = i;
  }
  if (start < text.length) out.push(text.slice(start));
  return out;
}

/**
 * `text` cut into pieces of about `budget` tokens (the last may be
 * shorter), at token starts: never inside a word, a surrogate pair, or
 * between a character and its combining mark. A boundary that would fall
 * inside a `[[...]]` wikilink moves out of the link whenever any
 * alternative exists - the whole link is absorbed into the piece when it
 * still fits the budget, else the cut retreats to before the link - and
 * only a link too large for the budget on its own is severed. Piece
 * boundaries advance monotonically, so pieces still concatenate back to
 * `text`.
 */
function hardCut(text: string, budget: number): string[] {
  const starts = tokenStarts(text);
  const spans = wikilinkSpans(text);
  const out: string[] = [];
  let from = 0;
  let ki = 0; // first token start at or after `from`; advances monotonically
  while (from < text.length) {
    while (ki < starts.length && starts[ki]! < from) ki++;
    let to = ki + budget < starts.length ? starts[ki + budget]! : text.length;
    for (const span of spans) {
      if (to > span.start && to < span.end) {
        if (countChunkTokens(text.slice(from, span.end)) <= budget) to = span.end;
        else if (span.start > from) to = span.start;
        break; // spans are ordered and disjoint: at most one holds `to`
      }
    }
    out.push(text.slice(from, to));
    from = to;
  }
  return out;
}

/**
 * One line longer than `budget`, cut into pieces within it: whole sentences
 * packed together where they fit, a sentence longer than the budget cut at
 * token boundaries. Concatenates back to the line.
 */
function splitLine(text: string, budget: number): string[] {
  const parts: string[] = [];
  let acc = "";
  for (const s of sentences(text)) {
    if (countChunkTokens(s) > budget) {
      if (acc !== "") parts.push(acc);
      const pieces = hardCut(s, budget);
      parts.push(...pieces.slice(0, -1));
      acc = pieces[pieces.length - 1] ?? "";
      continue;
    }
    const merged = acc + s;
    if (acc !== "" && countChunkTokens(merged) > budget) {
      parts.push(acc);
      acc = s;
    } else {
      acc = merged;
    }
  }
  if (acc !== "") parts.push(acc);
  return parts;
}

/**
 * A block too large for one chunk, as a run of single-line blocks of the
 * same kind, each within `budget`: its lines, and any line over the budget
 * split by {@link splitLine}. Line numbers are kept, so a split line's
 * pieces all point at the line they came from.
 */
function splitBlock(block: Block, budget: number): Block[] {
  const units: Block[] = [];
  for (const line of block.lines) {
    const t = countChunkTokens(line.text);
    if (t <= budget) {
      units.push({ kind: block.kind, lines: [line], tokenCount: t });
      continue;
    }
    for (const part of splitLine(line.text, budget)) {
      units.push({
        kind: block.kind,
        lines: [{ num: line.num, text: part }],
        tokenCount: countChunkTokens(part),
      });
    }
  }
  return units;
}

function packBlocks(
  blocks: ReadonlyArray<Block>,
  opts: { maxTokens: number; minTokens: number; overlapTokens: number },
): MarkdownChunk[] {
  const out: MarkdownChunk[] = [];
  let draft = newDraft();
  let pendingOverlap: Line[] = [];
  let pendingOverlapTokens = 0;
  // Pieces of a split block leave room for a full overlap, so the overlap
  // survives the split - up to half the chunk; a larger configured overlap
  // is trimmed to fit instead (see emitChunk).
  const unitBudget = Math.max(
    1,
    opts.maxTokens - Math.min(opts.overlapTokens, Math.floor(opts.maxTokens / 2)),
  );

  const setOverlap = (lines: Line[]) => {
    pendingOverlap = lines;
    pendingOverlapTokens = tokensOfLines(lines);
  };

  const flush = () => {
    if (draft.blocks.length === 0) return;
    out.push(emitChunk(out.length, pendingOverlap, draft, opts.maxTokens));
    setOverlap(takeOverlap(draft, opts.overlapTokens));
    draft = newDraft();
  };

  // The overlap is prepended to the chunk, so it spends the same budget.
  const fits = (block: Block) =>
    pendingOverlapTokens + draft.tokens + block.tokenCount <= opts.maxTokens;

  const push = (block: Block) => {
    draft.blocks.push(block);
    draft.tokens += block.tokenCount;
  };

  for (const block of blocks) {
    if (block.kind === "frontmatter") {
      // Flush any in-progress (shouldn't happen — frontmatter is first), then
      // emit the frontmatter as its own chunk(s) *without* overlap since
      // nothing precedes it. Only an oversize frontmatter becomes several.
      flush();
      const units = block.tokenCount > opts.maxTokens ? splitBlock(block, opts.maxTokens) : [block];
      let fm = newDraft();
      for (const unit of units) {
        if (fm.blocks.length > 0 && fm.tokens + unit.tokenCount > opts.maxTokens) {
          out.push(emitChunk(out.length, [], fm, opts.maxTokens));
          fm = newDraft();
        }
        fm.blocks.push(unit);
        fm.tokens += unit.tokenCount;
      }
      out.push(emitChunk(out.length, [], fm, opts.maxTokens));
      // The next chunk should NOT include frontmatter text as overlap.
      setOverlap([]);
      continue;
    }

    if (block.kind === "heading") {
      const hasNonHeading = draft.blocks.some((b) => b.kind !== "heading");
      if (hasNonHeading && draft.tokens >= opts.minTokens) {
        flush();
      }
    }

    if (draft.blocks.length > 0 && !fits(block)) {
      flush();
    }
    if (fits(block)) {
      push(block);
      continue;
    }

    // Too large to share a chunk with its own overlap: split it, and pack
    // the pieces like any other blocks.
    for (const unit of splitBlock(block, unitBudget)) {
      if (draft.blocks.length > 0 && !fits(unit)) flush();
      push(unit);
    }
  }

  flush();
  return out;
}

function stripQuotes(s: string): string {
  const t = s.trim();
  if (
    t.length >= 2 &&
    ((t.startsWith('"') && t.endsWith('"')) || (t.startsWith("'") && t.endsWith("'")))
  ) {
    return t.slice(1, -1);
  }
  return t;
}

function resolveTitle(
  frontmatterBlock: Block | null,
  blocks: ReadonlyArray<Block>,
  filenameBase: string | null,
): string | null {
  if (frontmatterBlock) {
    for (const l of frontmatterBlock.lines) {
      const m = l.text.match(/^title:\s*(.*)$/);
      if (m) {
        const v = stripQuotes(m[1] ?? "");
        if (v) return v;
      }
    }
  }
  for (const b of blocks) {
    if (b.kind === "heading") {
      const text = b.lines[0]!.text.replace(/^#{1,6}\s+/, "").trim();
      if (text) return text;
    }
    if (b.kind !== "frontmatter") break;
  }
  if (filenameBase) return filenameBase.replace(/[-_]+/g, " ").trim() || filenameBase;
  return null;
}

interface HeadingMark {
  readonly line: number;
  readonly level: number;
  readonly text: string;
}

/** Collect heading marks (line, level, text) in document order. */
function collectHeadings(blocks: ReadonlyArray<Block>): HeadingMark[] {
  const out: HeadingMark[] = [];
  for (const b of blocks) {
    if (b.kind !== "heading") continue;
    const raw = b.lines[0]!.text;
    const hashes = raw.match(/^#{1,6}/);
    const level = hashes ? hashes[0].length : 1;
    const text = raw.replace(/^#{1,6}\s+/, "").trim();
    if (text) out.push({ line: b.lines[0]!.num, level, text });
  }
  return out;
}

/**
 * Breadcrumb of headings in effect at `atLine` (the chunk's end line):
 * replay heading marks at or before the line, maintaining a level-stack
 * (a heading pops every entry at its level or deeper). Returns the stack
 * texts joined " > ".
 */
function headingPathAt(headings: ReadonlyArray<HeadingMark>, atLine: number): string {
  const stack: HeadingMark[] = [];
  for (const h of headings) {
    if (h.line > atLine) break;
    while (stack.length > 0 && stack[stack.length - 1]!.level >= h.level) stack.pop();
    stack.push(h);
  }
  return stack.map((h) => h.text).join(" > ");
}

/**
 * Two-pass markdown chunker. Pure function: same input → same output.
 *
 * Returns `chunks: []` for empty or whitespace-only input; the indexer
 * still records a `documents` row so the file is tracked.
 */
export function chunkMarkdown(
  text: string,
  filenameBase: string | null,
  opts?: ChunkOptions,
): ChunkResult {
  const maxTokens = opts?.maxTokens ?? DEFAULT_CHUNK_MAX_TOKENS;
  const minTokens = opts?.minTokens ?? DEFAULT_MIN_TOKENS;
  const overlapTokens = opts?.overlapTokens ?? DEFAULT_OVERLAP_TOKENS;

  const allLines = splitLines(text);
  if (allLines.every((l) => isBlank(l.text))) {
    return Object.freeze({
      title: filenameBase ? filenameBase.replace(/[-_]+/g, " ") : null,
      chunks: Object.freeze([]),
      warnings: Object.freeze([]),
    });
  }

  const { blocks, warnings } = splitIntoBlocks(allLines);
  const frontmatter = blocks.find((b) => b.kind === "frontmatter") ?? null;
  const title = resolveTitle(frontmatter, blocks, filenameBase);
  const packed = packBlocks(blocks, { maxTokens, minTokens, overlapTokens });

  // Anchor each chunk to the heading breadcrumb active at its end (the
  // deepest section it spans).
  const headings = collectHeadings(blocks);
  const chunks =
    headings.length === 0
      ? packed
      : packed.map((c) => Object.freeze({ ...c, headingPath: headingPathAt(headings, c.endLine) }));

  return Object.freeze({
    title,
    chunks: Object.freeze(chunks),
    warnings: Object.freeze(warnings),
  });
}