Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
73.6 KB
Raw Blame
#!/usr/bin/env node
/**
 * Export gbrain conversation pages to an envelope-v0 file (a JSON serialization
 * of AI chat history; format spec: github.com/memvelope/memvelope). The
 * counterpart of scripts/envelope-to-gbrain.mjs - that script reads the format,
 * this one writes it.
 *
 * Usage:
 *   node scripts/gbrain-to-envelope.mjs <pagesDir> [out.mve.json]
 *
 *   A single-provider page set writes [out.mve.json] itself. A set spanning
 *   several providers writes one envelope per provider beside it
 *   (out.chatgpt.mve.json, out.claude.mve.json, ...), each named on stdout
 *   with its own counts; see the source_provider notes below.
 *
 * Zero dependencies. Deterministic. No network. It does NOT call gbrain - it
 * only reads Markdown files. Point it at a `gbrain export --dir` output tree, at
 * a brain repo, or at the directory envelope-to-gbrain.mjs wrote.
 *
 * Input selection:
 *   - Walks <pagesDir> recursively for *.md, in sorted path order. Dot
 *     directories are skipped.
 *   - Takes only pages whose frontmatter `type` is `conversation`. Everything
 *     else is skipped and counted on stderr.
 *
 * TWO PAGE SHAPES, TOLD APART BY THE `messages:` FRONTMATTER KEY.
 *
 * The importer writes per-message identity into frontmatter:
 *
 *   messages:
 *     - id: "m1"
 *       ts: "2025-11-02T14:22:51.000Z"
 *
 * and a body whose turn headers carry a speaker and a minute-resolution UTC
 * wall clock, nothing else:
 *
 *   **Me** (2025-11-02 14:22):
 *
 * A page CARRYING the `messages:` key is read that way: `messages[i]` is the
 * i-th turn of the body, so the id and the full RFC 3339 `ts` come from the
 * array, and the body header supplies only the speaker and the text. A page
 * WITHOUT the key is read the legacy way the pre-2026-08-02 importer wrote:
 * `**Me** (<ts> · <message id>):`, identity parsed out of the header
 * parenthetical. The legacy path is unchanged; everything it got right it
 * still gets right, and everything prose could do to it - break a header with
 * a newline id, miss a match over one trailing space, forge a boundary with a
 * header-shaped line - it can still do. Those defects are CLOSED only on the
 * recorded path, because identity moved somewhere message text cannot reach.
 *
 * HOW A RECORDED PAGE'S TURNS ARE FOUND. The positional join is only sound if
 * the body anchors exactly as many turns as the array records, so boundaries
 * are not taken on shape alone. For each recorded message the expected header
 * clock is recomputed from its `ts` - the same derivation the importer used
 * to write it (headerClock below is a code-identical copy from
 * envelope-to-gbrain.mjs, comments elided; keep them in lockstep) - and a
 * header-shaped line is a boundary ONLY when
 * its clock equals the clock expected for the next unfilled position. A
 * header-shaped line carrying any other clock is message text, reported on
 * stderr and kept in place. So forging a boundary from prose requires
 * predicting the next message's minute, not merely producing the shape - and
 * a page whose body still does not anchor one turn per recorded message is
 * SKIPPED, loudly, naming both numbers, rather than joined wrong.
 *
 * The recorded-path header is matched strictly - `**Me** (YYYY-MM-DD HH:MM):`
 * or `**Assistant** (...)`, two-digit fields, one space, no text after the
 * colon - with one tolerance: trailing spaces or tabs after the colon.
 * gbrain's own `imessage-slack` pattern (the one these headers are written
 * for) tolerates them too, and one trailing space added by an editor used to
 * absorb the whole turn into its neighbour in silence.
 *
 * What a recorded page REFUSES loudly (skipped, counted, named on stderr)
 * rather than guesses about:
 *   - a `messages:` value this script cannot read back as an array of
 *     `{id, ts}` items (anything but `[]`, or block items carrying both keys);
 *   - a recorded id that is not a string under YAML core-schema reading
 *     (`id: null`, an unquoted number or boolean, a flow collection).
 *     envelope-v0 requires a string message id, and inventing one is exactly
 *     the synthesis the spec forbids;
 *   - `messages: []` - envelope-v0 requires at least one message;
 *   - a body anchoring more or fewer turns than the array records.
 *
 * Fenced code blocks (``` or ~~~, up to three leading spaces) are not scanned
 * for turn headers, timeline sentinels or the H1 - but a fence reaches only to
 * the end of the turn it opened in. On the recorded path the scan runs in
 * stages so that balanced fences genuinely win: the first honors every fence
 * that closes, keeping even a quoted line that carries the next expected
 * clock as sample text - a pasted transcript quoting this very conversation
 * must not steal a boundary from outside its fence. Only when that stage
 * anchors fewer turns than the record holds is some fence plausibly
 * swallowing a real boundary, and then each fence is tried ALONE: the fence
 * to lose is the one whose demotion by itself anchors every recorded turn,
 * which is what keeps a balanced fence that merely quotes a boundary from
 * being demoted when a different fence caused the shortfall. Only if no
 * single demotion suffices does a greedy pass demote every fence still open
 * at a next-expected-clock line. Each demotion is warned, naming the line
 * the fence opened on; a fence still open at the end of the page is demoted
 * in every stage. All of it is the same trade: a fence must never be allowed
 * to swallow the turns after it, and a spurious stretch of ordinary text is
 * a far smaller failure than a turn deleted in silence. On
 * the legacy path the fence-versus-boundary signal is the blank / `---` /
 * blank separator the old importer wrote between turns, as before - and note
 * the demotion's reach, which an earlier revision of this header overstated:
 * the separator decides only WHETHER a spanning fence loses. Once it has
 * lost, every header-shaped line it had covered is read as an ordinary turn
 * header, separator above it or not, so a header-shaped line quoted inside
 * the demoted stretch does become a turn. (Measured: a fence spanning one
 * separator-preceded header with a separatorless header-shaped line above it
 * yields three messages, the middle one lifted from the quoted line, with the
 * demotion warned on stderr.) On the recorded path the same demotion is
 * narrower: an exposed line still becomes a boundary only by carrying the
 * next expected clock.
 *
 * The body is cut at a gbrain timeline sentinel, in all four forms gbrain's
 * own findTimelineSplitIndex accepts (src/core/markdown.ts, mirrored in
 * timelineSentinelAt below): `<!-- timeline -->`, `<!--timeline-->`, the
 * decorated rule under /^---\s+timeline\s+---$/i in any case and spacing, and
 * a bare `---` with content above it whose next non-blank line is a
 * `## Timeline` or `## History` heading. So a page's timeline never becomes
 * message text under any spelling gbrain itself would split on. Only a
 * sentinel with no accepted turn after it is treated as a boundary; one with
 * turns below it is message text and is reported rather than cut at - where
 * gbrain's parser cuts at the FIRST hit and truncates every turn below, the
 * one deliberate divergence, made because a quoted sentinel must not delete
 * the turns after it. A sentinel standing alone on its line INSIDE the final
 * message still cuts - from this side of the page it is indistinguishable
 * from gbrain's real delimiter, and gbrain's own parser reads that page the
 * same way - and the cut is what the timeline note on stderr is counting.
 * The bare-`---` form is safe against the legacy turn separator because of
 * its heading lookahead: a separator's next non-blank line is a turn header,
 * never the Timeline/History heading, and a final message ending in a plain
 * `---` with prose (or nothing) below keeps its bytes.
 *
 * Frontmatter is read line by line: top-level scalars only, in the plain,
 * single-quoted, double-quoted (escapes decoded, including `\U` beyond the
 * BMP), folded (`>`) and literal (`|`) block forms - plus, alone among
 * nested structures, the `messages:` block sequence described above. A UTF-8
 * BOM is stripped and CRLF normalized before any of it, as gray-matter does.
 * Trailing `# comments` are stripped the way js-yaml strips them - a reader
 * that kept them read `type: conversation # imported` as a different type
 * and `id: # placeholder` as an invented string id. Items accept the
 * importer's JSON-quoted scalars, the single-quoted / plain / block-scalar
 * forms gbrain's js-yaml rewrite produces (including `|-` literal ids
 * carrying newlines and whitespace-only content lines, verified against
 * js-yaml's own emissions), items at column 0 or indented, and `null`;
 * member keys beyond `id` and `ts` are ignored so a future third field does
 * not break the read; a quoted scalar that does not close on its own line is
 * refused (js-yaml folds long values into block scalars rather than wrapping
 * quotes, so that shape indicates a page this script does not understand).
 * One stated limit: this line reader is more PERMISSIVE than js-yaml - a
 * page whose frontmatter js-yaml rejects outright (an unknown escape, a
 * plain scalar opening with `@` or a backtick) may still export here under
 * this reader's simpler rules, where gbrain itself would call the page a
 * parse error.
 *
 * `title` falls back to the body's first H1 when frontmatter carries none,
 * the same precedence gbrain's own parser uses (src/core/markdown.ts,
 * inferTitleFromBody), and only then to `Untitled conversation` - a
 * whitespace-only title takes that fallback too, so a conversation titled
 * with pure whitespace comes back renamed. `title` and `source` are trimmed;
 * `memvelope_conversation_id` is VERBATIM, like the recorded message ids and
 * timestamps - the importer records ids verbatim precisely so that values
 * differing only by surrounding whitespace stay distinct, and trimming here
 * would collapse them back together on the way out.
 *
 * What survives the round trip envelope -> envelope-to-gbrain.mjs -> here,
 * measured on the recorded format (see STATUS for the corpus):
 *   - conversation id and title, including a null id
 *   - message id, role, timestamp and text, verbatim - including ids carrying
 *     newlines, YAML syntax, or a whole frontmatter block, and timestamps at
 *     full sub-second resolution with their original offsets
 *   - meta.source_provider, per envelope: every conversation travels in the
 *     envelope of its own page's `source:`
 *   - meta.conversation_count and meta.message_count, recomputed and equal
 *
 * What does not survive. Each of these was measured on a probe envelope or a
 * probe page built to carry it, not assumed:
 *   - `conversation.created_at` and `conversation.updated_at`. The page keeps
 *     only `date`, the first 10 characters of created_at, which is a day and
 *     not a date-time. Both fields are taken here from the first and last
 *     message timestamps instead. A conversation whose created_at was
 *     09:00:00Z with a first message at 10:00:00Z comes back as 10:00:00Z, and
 *     one whose first message has no timestamp comes back null.
 *   - `meta.source_export_date`. Never written to a page. Omitted here, which
 *     is what the spec asks for when the value is unknown.
 *   - Any other key. envelope-v0 allows additional properties at every level,
 *     the page carries none of them, and they are gone at meta, conversation
 *     and message level alike.
 *   - Conversation order. Output follows sorted file path, so a conversation
 *     listed first but dated last moves to the end.
 *   - Leading and trailing whitespace in message text, trimmed.
 *   - A message that is only whitespace. Dropped, with a warning, because
 *     envelope-v0 requires text of at least one character. On the recorded
 *     path its `{id, ts}` entry is dropped with it, so the join stays aligned.
 *     meta.message_count follows what was written, so it drops too.
 *   - Anything before the first turn header except the `# Title` heading,
 *     which is read as the title fallback described above. Reported on
 *     stderr when non-blank content is dropped this way.
 *   - Everything after a timeline sentinel that no accepted turn follows -
 *     a real gbrain timeline, or the tail of a final message that quoted
 *     the sentinel - in any of the four forms above, the bare-`---` form
 *     included: a final message that legitimately ends with `---` and a
 *     `## Timeline` heading is cut, because gbrain's own parser reads that
 *     page the same way. envelope-v0 has no field for a timeline. Each cut
 *     is warned per page, naming the body line, and tallied in the closing
 *     note.
 *   - A recorded `ts` that is not a strict RFC 3339 `date-time` - which is
 *     what the schema's `format: date-time` names. Emitted as null, with a
 *     warning, rather than as a value that fails validation. That includes a
 *     second of 60 anywhere but the instant a leap second is inserted; see
 *     asRfc3339DateTime below. A recorded `ts: null` stays null, silently -
 *     that is the agreed spelling of "the export had no timestamp", not a
 *     loss. On the legacy path the same rule judges the header parenthetical,
 *     and the literal `no timestamp` is the agreed null there.
 *   - A message whose text contains a line shaped like a turn header and
 *     carrying the very minute the NEXT message is recorded at. That line
 *     still steals the boundary, and the damage is worse than a tidy split:
 *     the forged line itself is consumed as the boundary (those bytes are
 *     deleted from the text); the stolen turn's ROLE comes from the forged
 *     line's speaker, not from any record, so an assistant message can come
 *     back as user or vice versa; the real header below is read as prose
 *     into the wrong turn's text; and a message left with NO text - one
 *     whose whole text was the forged line - is dropped outright with its
 *     `{id, ts}` entry, taking created_at/updated_at with it when it sat at
 *     an end. Every step is warned on stderr, and the ids and timestamps of
 *     the surviving messages stay aligned. This is the residue of defect D2,
 *     and the bar for forging is lower than "predict the future": two
 *     messages inside one minute make the next expected clock equal the
 *     current header's own clock, and a null-or-unusable `ts` makes it the
 *     page date at 00:00 - visible on the page itself.
 *   - CRLF line endings, normalized to LF on read.
 *
 * A page with no `memvelope_conversation_id` emits `id: null` rather than a
 * synthesized id. envelope-v0 permits null there and tells converters not to
 * invent one.
 *
 * `meta.source_provider` names ONE provider for a whole envelope, so a page
 * set spanning several is written as one envelope per provider - a mixed
 * directory cannot be described by one file without falsifying someone's
 * `source:`. (This script used to keep the first provider in sorted path
 * order for everyone; re-importing that envelope stamped the winner onto
 * every losing page's `source:` at exit 0, and envelope-v0 carries no
 * per-conversation provider field that could keep the truth.) The filename
 * token is the provider with everything outside [A-Za-z0-9._-] replaced by
 * `-`, deduplicated case-insensitively, so a hostile `source:` cannot steer
 * the write and case-only twins still get two files. A single-provider set is
 * unchanged: one file, at the path asked for.
 *
 * Pages carrying no `source:` key are their own group, never folded into a
 * named provider's envelope - absence of evidence is not membership, and the
 * fold would stamp the neighbour's provider onto them at re-import, the same
 * falsification through absence rather than collision. They ride under the
 * literal `unknown` - required because meta.source_provider cannot be
 * omitted, defined by no provider registry - warned on stderr naming the
 * token, in `out.unknown.mve.json` when named providers are present or at
 * the asked-for path when nothing in the set names one. Round trip,
 * measured: every conversation carrying a `memvelope_conversation_id`
 * re-imports into the very directory it came from and refreshes its page in
 * place, byte-identical - check 2 matches per-conversation ids - so no
 * page's `source:` changes hands. The residue sits on id-less pages: a
 * native gbrain page has no memvelope id, its conversation travels as
 * `id: null`, and re-importing the `unknown` envelope cannot find the
 * original page to refresh - it writes a NEW positional page
 * (`<date>-conv-N.md`) carrying `source: "unknown"` and leaves the original
 * untouched. A duplicate page, never a falsified one - the importer's
 * standing behavior for a null id, not new here.
 *
 * Memory: the whole page set is held in memory (no streaming), same posture as
 * the importer.
 *
 * Verify:
 *   node scripts/envelope-to-gbrain.mjs test/fixtures/memvelope/sample.mve.json /tmp/pages
 *   node scripts/gbrain-to-envelope.mjs /tmp/pages /tmp/out.mve.json
 *     -> expect "wrote 1 conversation(s), 4 message(s)"
 *   bun test test/gbrain-to-envelope.test.ts
 *
 * STATUS: measured 2026-08-02 against gbrain v0.42.72.1, on two paths - the
 * short one (envelope -> envelope-to-gbrain.mjs -> here) and the long one
 * through a throwaway HOME-redirected PGLite brain (envelope -> importer ->
 * `gbrain import` -> `gbrain export --dir` -> here), so the pages read are
 * the ones gbrain's own parseMarkdown / serializeMarkdown pair really
 * produces. The repo's sample fixture round-trips deep-equal on both paths
 * with zero stderr bytes. A 3-conversation / 17-message probe - ids carrying
 * an embedded newline, a whole `---`/`type:`/`---` block, a middle dot,
 * trailing space, emoji, and a folding-length 79-character value; timestamps
 * with microseconds, a +05:30 offset, and null; two messages sharing one
 * minute; a whole-line quoted sentinel mid-conversation; fences spanning
 * turns - comes back with 17/17 messages and every id, role, ts and text
 * verbatim on both paths. gbrain's serializer hands the array back as
 * literal (`|-`) block-scalar ids, a folded (`>-`) long id, single-quoted
 * timestamps and reordered top-level keys; all of it reads. The only fields
 * that moved are the documented ones: created_at / updated_at (replaced by
 * the first and last message timestamps) and one message's trailing newline
 * (trimmed). One long-path caveat sits UPSTREAM of this script: gbrain's own
 * parse/serialize pair re-spaces a whole-line HTML comment inside message
 * text (blank lines inserted around a sentinel written tight against its
 * neighbours) and swallows a sentinel on the last line of a body with no
 * real timeline - the probe's quoted sentinel was blank-padded, the one
 * spacing gbrain preserves, so those movements do not show above; they are
 * gbrain's, not this reader's.
 *
 * STATUS ADDENDUM, 2026-08-04, after the rebase onto the master that merged
 * the importer (#3788): the short path above re-verified against the merged
 * importer - the sample fixture round-trips deep-equal, zero stderr. The
 * per-provider fan-out measured on a mixed directory (chatgpt + claude
 * imports plus a native source-less page): three envelopes, every one
 * schema-conforming; re-importing all three into the same directory left
 * both id-carrying pages byte-identical and the native page untouched, plus
 * the one documented duplicate a null id mints. The sentinel closure is
 * canonical-path-invisible by construction: gbrain's serializer emits only
 * `<!-- timeline -->` (measured 2026-08-03 against a real brain), so the
 * decorated and bare-rule forms arrive only on hand-edited or legacy pages -
 * which is where they were exporting a timeline as message text.
 *
 * Conformance in CI is checked by test/gbrain-to-envelope.test.ts, which
 * validates against the published envelope-v0 JSON Schema vendored
 * byte-for-byte at test/fixtures/memvelope/envelope-v0.schema.json (sha256
 * 423813d563de394cde2798848e90fdadc85ba52458f5c18b1da897e6c8ae52b9, identical
 * across memvelope.com, raw.githubusercontent.com/memvelope/memvelope@main and
 * the memvelope package). The test walks those bytes with a draft-07 subset it
 * implements in full and refuses any keyword it does not, so a constraint added
 * upstream turns the suite red instead of going unchecked. No validator
 * dependency is added. Both round-trip outputs above additionally validate
 * under ajv + ajv-formats in full mode, with the validator first proven
 * non-trivial on a known-bad document.
 */

import { readFileSync, writeFileSync, readdirSync, statSync } from 'node:fs';
import { join } from 'node:path';

const [, , pagesDir, outPath = './envelope.mve.json'] = process.argv;
if (!pagesDir) {
  console.error('usage: node gbrain-to-envelope.mjs <pagesDir> [out.mve.json]');
  process.exit(1);
}

let dirStat;
try {
  dirStat = statSync(pagesDir);
} catch {
  console.error(`cannot read ${pagesDir}`);
  process.exit(1);
}
if (!dirStat.isDirectory()) {
  console.error(`${pagesDir} is not a directory`);
  process.exit(1);
}

// Sorted at every level, so the output is a function of the page set and not of
// the order the filesystem hands back. `gbrain export` writes nested by slug,
// the importer writes flat; both are covered by the same walk.
function markdownFiles(dir) {
  const found = [];
  for (const entry of readdirSync(dir, { withFileTypes: true }).sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0))) {
    if (entry.name.startsWith('.')) continue;
    const full = join(dir, entry.name);
    if (entry.isDirectory()) found.push(...markdownFiles(full));
    else if (entry.isFile() && entry.name.endsWith('.md')) found.push(full);
  }
  return found;
}

// Frontmatter is the block between the first `---` line and the next one. A
// page without that block is not a gbrain page and is skipped rather than
// guessed at.
function splitPage(content) {
  const lines = content.split('\n');
  if (lines[0] !== '---') return null;
  const close = lines.indexOf('---', 1);
  if (close === -1) return null;
  return { front: lines.slice(1, close), body: lines.slice(close + 1).join('\n') };
}

// YAML double-quoted scalars carry escapes JSON does not: `\U0001F680` for an
// astral code point, `\x41`, `\N`, `\_`. js-yaml reaches for this form whenever
// a string holds a tab or a non-printable, which includes any emoji - common in
// vendor conversation titles. Decoding it here is the difference between the
// real title and the literal text `"\U0001F680 ..."`, quotes and all.
const SIMPLE_ESCAPES = {
  '0': '\0', a: '\x07', b: '\b', t: '\t', '\t': '\t', n: '\n', v: '\v', f: '\f',
  r: '\r', e: '\x1b', ' ': ' ', '"': '"', '/': '/', '\\': '\\', N: '\x85',
  _: '\xa0', L: '\u2028', P: '\u2029',
};

function isCompleteDoubleQuoted(raw) {
  if (raw.length < 2 || !raw.startsWith('"') || !raw.endsWith('"')) return false;
  // The closing quote must not itself be escaped, and no unescaped quote may
  // appear before it - otherwise this is a fragment and not a whole scalar.
  let escaped = false;
  for (let i = 1; i < raw.length; i += 1) {
    if (escaped) {
      escaped = false;
      continue;
    }
    if (raw[i] === '\\') escaped = true;
    else if (raw[i] === '"') return i === raw.length - 1;
  }
  return false;
}

function decodeDoubleQuoted(raw) {
  const inner = raw.slice(1, -1);
  let out = '';
  for (let i = 0; i < inner.length; i += 1) {
    if (inner[i] !== '\\') {
      out += inner[i];
      continue;
    }
    const esc = inner[i + 1];
    if (esc === undefined) {
      out += '\\';
      break;
    }
    i += 1;
    if (esc in SIMPLE_ESCAPES) {
      out += SIMPLE_ESCAPES[esc];
      continue;
    }
    const width = esc === 'x' ? 2 : esc === 'u' ? 4 : esc === 'U' ? 8 : 0;
    if (width === 0) {
      // An escape YAML does not define. Keep both characters rather than
      // inventing a decoding.
      out += `\\${esc}`;
      continue;
    }
    const digits = inner.slice(i + 1, i + 1 + width);
    if (digits.length !== width || !/^[0-9A-Fa-f]+$/.test(digits)) {
      out += `\\${esc}`;
      continue;
    }
    const code = parseInt(digits, 16);
    if (code > 0x10ffff) {
      out += `\\${esc}${digits}`;
    } else {
      out += String.fromCodePoint(code);
    }
    i += width;
  }
  return out;
}

// A single-quoted scalar is complete when its closing quote is the last
// character and every interior quote is one half of an escaped `''` pair.
function isCompleteSingleQuoted(raw) {
  if (raw.length < 2 || !raw.startsWith("'") || !raw.endsWith("'")) return false;
  let i = 1;
  while (i < raw.length) {
    if (raw[i] !== "'") {
      i += 1;
      continue;
    }
    if (i === raw.length - 1) return true;
    if (raw[i + 1] === "'") {
      i += 2;
      continue;
    }
    return false;
  }
  return false;
}

// A block scalar's content, given its already-dedented lines. Folded (`>`)
// joins a paragraph's lines with a space and turns a blank line into a
// newline; a more-indented line keeps its breaks. Literal (`|`) keeps every
// break as written.
function joinBlockScalar(contentLines, style, chomping, trailingBlanks) {
  let value;
  if (style === '|') {
    value = contentLines.join('\n');
  } else {
    value = '';
    for (let i = 0; i < contentLines.length; i += 1) {
      const line = contentLines[i];
      if (i === 0) {
        value = line;
        continue;
      }
      const prev = contentLines[i - 1];
      if (line === '' || prev === '') {
        value += line === '' ? '\n' : line;
        continue;
      }
      value += (/^\s/.test(line) || /^\s/.test(prev) ? '\n' : ' ') + line;
    }
  }
  if (chomping === 'strip') return value;
  if (chomping === 'keep') {
    // The final content line contributes one break only if there IS content:
    // `|+` over a single empty line is the value "\n", not "\n\n".
    return value + '\n'.repeat(trailingBlanks + (contentLines.length > 0 ? 1 : 0));
  }
  return value === '' ? '' : `${value}\n`;
}

/** A block scalar header (`>-`, `|`, `>2-`, ...) or null. */
function blockScalarHeader(raw) {
  const block = /^([|>])([+-]?[0-9]?|[0-9][+-]?)$/.exec(raw);
  if (!block) return null;
  return {
    style: block[1],
    chomping: raw.includes('+') ? 'keep' : raw.includes('-') ? 'strip' : 'clip',
    explicit: /[0-9]/.exec(raw),
  };
}

/** Reads the indented block below `lines[headerIdx]` as a block scalar whose
 *  owner key sits at indentation `ownerLead`. Returns the value and the index
 *  of the first line after the block. */
function readBlockScalar(lines, headerIdx, header, ownerLead) {
  // A YAML indentation indicator is relative to the owner node; at the top
  // level (ownerLead 0) that is the absolute column, matching the previous
  // behavior of this parser exactly.
  let indent = header.explicit ? ownerLead + Number(header.explicit[0]) : -1;
  const collected = [];
  let j = headerIdx + 1;
  for (; j < lines.length; j += 1) {
    if (lines[j].trim() === '') {
      // A whitespace-only line's content beyond the block indent is real:
      // js-yaml writes the ` ` segment of "a\n \nb" as indent plus one space,
      // and collapsing it to an empty line silently corrupted the value. At
      // or below the indent (or before the indent is known) it is empty.
      collected.push(indent !== -1 && lines[j].length > indent ? lines[j].slice(indent) : '');
      continue;
    }
    const lead = lines[j].length - lines[j].trimStart().length;
    if (lead <= ownerLead) break;
    if (indent === -1) indent = lead;
    if (lead < indent) break;
    collected.push(lines[j].slice(indent));
  }
  let trailingBlanks = 0;
  while (collected.length > 0 && collected[collected.length - 1] === '') {
    collected.pop();
    trailingBlanks += 1;
  }
  return { value: joinBlockScalar(collected, header.style, header.chomping, trailingBlanks), next: j };
}

// Top-level scalars only. gbrain writes these through js-yaml, which quotes
// with single quotes, escapes with double quotes, and folds anything long into
// a block scalar; the importer writes them through JSON.stringify, which quotes
// with double quotes. All of those are read here, along with the unquoted form.
// A line that is indented belongs to a nested value or to a block scalar's
// content and is never read as a key. The one nested structure this script
// understands - the `messages:` sequence - is read by parseMessagesRecord,
// separately, over the same lines.
function parseFrontmatter(lines) {
  const out = {};
  for (let i = 0; i < lines.length; i += 1) {
    // The `s` flag lets `.` cross U+2028/U+2029. JSON.stringify does not
    // escape those two line terminators, so a conforming envelope value can
    // put one RAW inside a frontmatter line; without the flag the line
    // silently failed to parse and the id or title vanished with zero stderr.
    const m = /^([A-Za-z_][A-Za-z0-9_-]*): ?(.*)$/s.exec(lines[i]);
    if (!m) continue;
    const key = m[1];
    const raw = stripTrailingComment(m[2].trim());

    // `key: >-`, `key: |`, `key: >2-`: the value is the indented block below.
    const block = blockScalarHeader(raw);
    if (block) {
      const scalar = readBlockScalar(lines, i, block, 0);
      out[key] = scalar.value;
      i = scalar.next - 1;
      continue;
    }

    if (raw === '') continue;
    if (raw === 'null' || raw === '~') {
      out[key] = null;
      continue;
    }
    if (isCompleteDoubleQuoted(raw)) {
      out[key] = decodeDoubleQuoted(raw);
      continue;
    }
    if (raw.startsWith("'") && raw.endsWith("'") && raw.length >= 2) {
      out[key] = raw.slice(1, -1).replace(/''/g, "'");
      continue;
    }
    out[key] = raw;
  }
  return out;
}

// What a plain (unquoted) YAML scalar means to js-yaml 3.14's default schema
// - the reader and writer gbrain actually uses: null, boolean and number
// forms are not strings. The importer JSON-quotes every string it takes from
// an envelope and js-yaml re-quotes any string that resolves to one of these
// on the way back out, so an unquoted `id: 123` really is a number and
// refusing it is correct - reading it as the string "123" would fabricate an
// id the record does not hold. The int and float checks below are ports of
// js-yaml 3.14's own resolveYamlInteger / resolveYamlFloat (an approximating
// regex here once refused `089` - a STRING to js-yaml, which its dumper
// re-emits unquoted - and dropped the whole conversation after one brain
// cycle). js-yaml's octal is the 1.1 leading-zero form, `0o` is a string,
// trailing underscores make a string, and `yes`/`no`/`on`/`off` are strings.
const DEC_DIGIT = /[0-9]/;

function isJsYamlInteger(data) {
  const max = data.length;
  let index = 0;
  let hasDigits = false;
  if (max === 0) return false;
  let ch = data[index];
  if (ch === '-' || ch === '+') ch = data[(index += 1)];
  if (ch === '0') {
    if (index + 1 === max) return true;
    ch = data[(index += 1)];
    if (ch === 'b') {
      for (index += 1; index < max; index += 1) {
        ch = data[index];
        if (ch === '_') continue;
        if (ch !== '0' && ch !== '1') return false;
        hasDigits = true;
      }
      return hasDigits && ch !== '_';
    }
    if (ch === 'x') {
      for (index += 1; index < max; index += 1) {
        ch = data[index];
        if (ch === '_') continue;
        if (!/[0-9a-fA-F]/.test(ch)) return false;
        hasDigits = true;
      }
      return hasDigits && ch !== '_';
    }
    for (; index < max; index += 1) {
      ch = data[index];
      if (ch === '_') continue;
      if (!/[0-7]/.test(ch)) return false;
      hasDigits = true;
    }
    return hasDigits && ch !== '_';
  }
  if (ch === '_') return false;
  for (; index < max; index += 1) {
    ch = data[index];
    if (ch === '_') continue;
    if (ch === ':') break;
    if (!DEC_DIGIT.test(ch)) return false;
    hasDigits = true;
  }
  if (!hasDigits || ch === '_') return false;
  if (ch !== ':') return true;
  return /^(:[0-5]?[0-9])+$/.test(data.slice(index));
}

const JS_YAML_FLOAT = new RegExp(
  '^(?:[-+]?(?:0|[1-9][0-9_]*)(?:\\.[0-9_]*)?(?:[eE][-+]?[0-9]+)?' +
  '|\\.[0-9_]+(?:[eE][-+]?[0-9]+)?' +
  '|[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*' +
  '|[-+]?\\.(?:inf|Inf|INF)' +
  '|\\.(?:nan|NaN|NAN))$',
);

function isPlainNonString(data) {
  if (data === 'true' || data === 'True' || data === 'TRUE') return true;
  if (data === 'false' || data === 'False' || data === 'FALSE') return true;
  if (isJsYamlInteger(data)) return true;
  return JS_YAML_FLOAT.test(data) && data[data.length - 1] !== '_';
}

// A YAML comment starts at a `#` that opens the value or follows whitespace,
// outside any quoted run. js-yaml strips comments on read, so a reader that
// kept them was inventing values: `type: conversation # imported` must read
// as `conversation`, and `id: # placeholder` is a null id, not the string
// "# placeholder".
function stripTrailingComment(raw) {
  if (raw.startsWith('#')) return '';
  let quote = null;
  let escaped = false;
  let closedQuoted = false;
  for (let i = 0; i < raw.length; i += 1) {
    const ch = raw[i];
    if (quote === '"') {
      if (escaped) escaped = false;
      else if (ch === '\\') escaped = true;
      else if (ch === '"') {
        quote = null;
        closedQuoted = true;
      }
      continue;
    }
    if (quote === "'") {
      if (ch === "'") {
        if (raw[i + 1] === "'") i += 1;
        else {
          quote = null;
          closedQuoted = true;
        }
      }
      continue;
    }
    // A quote only opens a quoted scalar at the start of the value; later it
    // is ordinary content of a plain scalar.
    if (i === 0 && (ch === '"' || ch === "'")) {
      quote = ch;
      continue;
    }
    // In a plain scalar a comment needs whitespace before the '#'; after a
    // closed quoted scalar YAML ends the value at a '#' with or without one
    // ('"a"# c' reads as "a" to js-yaml).
    if (ch === '#' && (closedQuoted || raw[i - 1] === ' ' || raw[i - 1] === '\t')) {
      return raw.slice(0, i).trim();
    }
  }
  return raw;
}

/** Decodes one scalar value from a `messages:` item member line. Returns
 *  `{ value }` (string or null) or `{ error }` when the raw text is not a
 *  scalar this script can stand behind. */
function decodeMemberScalar(raw) {
  if (raw === '' || raw === 'null' || raw === '~' || raw === 'Null' || raw === 'NULL') return { value: null };
  if (raw.startsWith('"')) {
    if (isCompleteDoubleQuoted(raw)) return { value: decodeDoubleQuoted(raw) };
    return { error: 'a double-quoted scalar that does not close on its own line' };
  }
  if (raw.startsWith("'")) {
    if (isCompleteSingleQuoted(raw)) return { value: raw.slice(1, -1).replace(/''/g, "'") };
    return { error: 'a single-quoted scalar that does not close on its own line' };
  }
  if (raw.startsWith('[') || raw.startsWith('{') || raw.startsWith('&') || raw.startsWith('*') || raw.startsWith('!')) {
    return { error: `a value this script does not read (${JSON.stringify(raw)})` };
  }
  if (isPlainNonString(raw)) return { value: { nonString: raw } };
  return { value: raw };
}

/**
 * The `messages:` record, read out of the frontmatter lines: the per-message
 * identity the importer writes and gbrain's serializer rewrites (values and
 * order intact, quoting style not - so this parser accepts both quotings and
 * the block-scalar form js-yaml folds long values into).
 *
 * Returns `{ present: false }` when the page carries no `messages:` key - a
 * page written before this format existed - `{ present: true, items }` when
 * the record reads cleanly, and `{ present: true, error }` when the key is
 * there but this script cannot stand behind what it read. The error case must
 * never fall back to the legacy path: a page that declares a record it cannot
 * deliver is not a legacy page, it is an unreadable one.
 */
function parseMessagesRecord(lines) {
  let at = -1;
  for (let i = 0; i < lines.length; i += 1) {
    if (/^messages: ?/.test(lines[i]) || lines[i] === 'messages:') {
      if (at !== -1) return { present: true, error: 'the messages key appears twice' };
      at = i;
    }
  }
  if (at === -1) return { present: false };

  const inline = stripTrailingComment((/^messages: ?(.*)$/s.exec(lines[at]))[1].trim());
  if (inline === '[]') return { present: true, items: [] };
  if (inline !== '') return { present: true, error: `an inline value this script does not read (${JSON.stringify(inline)})` };

  const items = [];
  let itemLead = -1;
  let current = null;
  let i = at + 1;
  for (; i < lines.length; i += 1) {
    const line = lines[i];
    if (line.trim() === '') continue;
    const lead = line.length - line.trimStart().length;

    // Items may sit at column 0 (valid YAML, the noArrayIndent style) or
    // indented (what the importer and js-yaml's default write); the first
    // item fixes the indent for the rest.
    const item = /^(\s*)- (.*)$/s.exec(line);
    if (item && (itemLead === -1 || item[1].length === itemLead)) {
      if (itemLead === -1) itemLead = item[1].length;
      current = {};
      items.push(current);
      const first = readMember(item[2].trim(), itemLead + 2);
      if (first.error) return { present: true, error: first.error };
      continue;
    }
    // A new top-level key ends the sequence.
    if (lead === 0) break;
    if (current === null) return { present: true, error: `a line before the first item (${JSON.stringify(line)})` };
    if (lead <= itemLead) return { present: true, error: `an indentation this script does not read (${JSON.stringify(line)})` };
    const member = readMember(line.trim(), lead);
    if (member.error) return { present: true, error: member.error };
  }

  // One member line, already trimmed, belonging to `current`. `memberLead` is
  // the column its key starts at, needed when its value is a block scalar.
  function readMember(text, memberLead) {
    const m = /^([A-Za-z_][A-Za-z0-9_-]*): ?(.*)$/s.exec(text);
    if (!m) return { error: `an item line that is not a key: value pair (${JSON.stringify(text)})` };
    const key = m[1];
    const raw = stripTrailingComment(m[2].trim());
    if (key !== 'id' && key !== 'ts') {
      // A future third field. Skip its value, block scalar and all.
      const block = blockScalarHeader(raw);
      if (block) i = readBlockScalar(lines, i, block, memberLead).next - 1;
      return {};
    }
    if (key in current) return { error: `the ${key} key appears twice in one item` };
    const block = blockScalarHeader(raw);
    if (block) {
      const scalar = readBlockScalar(lines, i, block, memberLead);
      current[key] = scalar.value;
      i = scalar.next - 1;
      return {};
    }
    const decoded = decodeMemberScalar(raw);
    if (decoded.error) return { error: `${key}: ${decoded.error}` };
    current[key] = decoded.value;
    return {};
  }

  for (const [n, item] of items.entries()) {
    if (!('id' in item) || !('ts' in item)) {
      return { present: true, error: `item ${n + 1} does not carry both id and ts` };
    }
  }
  return { present: true, items };
}

// ---------------------------------------------------------------------------
// The header-clock derivation, a code-identical copy from
// envelope-to-gbrain.mjs (that file's longer comments elided) so the expected
// clock recomputed here is the clock the importer wrote. If one of these
// functions changes, change both files - the round-trip tests fail loudly (a
// clock mismatch skips the page) if they drift.
// ---------------------------------------------------------------------------

/** The date a turn header is allowed to carry: exactly `YYYY-MM-DD`. */
const HEADER_DATE = /^\d{4}-\d{2}-\d{2}$/;

/** What `deriveDateContext()` in gbrain's conversation parser falls back to when
 *  a page carries no date at all. */
const EPOCH_DATE = '1970-01-01';

/** The RFC 3339 shapes the importer will read a wall clock out of.
 *  Groups: 1=Y 2=M 3=D 4=hh 5=mm, then an optional offset 6=sign 7=hh 8=mm. */
const TS_SHAPE =
  /^(\d{4})-(\d{2})-(\d{2})[Tt ](\d{2}):(\d{2})(?::\d{2}(?:\.\d+)?)?(?:[Zz]|([+-])(\d{2}):?(\d{2}))?$/;

/**
 * The `YYYY-MM-DD HH:MM` a turn header carries, or null when the message's `ts`
 * cannot supply one. 24-hour, and UTC - see envelope-to-gbrain.mjs for the
 * full reasoning; this copy exists so the expected boundary clock is derived
 * from the recorded `ts` by the exact function that wrote it.
 */
function headerClock(ts) {
  if (typeof ts !== 'string') return null;
  const m = TS_SHAPE.exec(ts.trim());
  if (m === null) return null;
  const [, year, month, day, hour, minute, sign, offsetHour, offsetMinute] = m;
  const [y, mo, d, h, mi] = [year, month, day, hour, minute].map(Number);
  if (h > 23 || mi > 59) return null;
  const utc = new Date(0);
  utc.setUTCFullYear(y, mo - 1, d);
  utc.setUTCHours(h, mi, 0, 0);
  if (utc.getUTCFullYear() !== y || utc.getUTCMonth() !== mo - 1 || utc.getUTCDate() !== d) {
    return null;
  }
  if (sign === undefined) return `${year}-${month}-${day} ${hour}:${minute}`;
  const [oh, om] = [offsetHour, offsetMinute].map(Number);
  if (oh > 23 || om > 59) return null;
  utc.setUTCMinutes(utc.getUTCMinutes() - (oh * 60 + om) * (sign === '-' ? -1 : 1));
  const pad = (n, width = 2) => String(n).padStart(width, '0');
  return `${pad(utc.getUTCFullYear(), 4)}-${pad(utc.getUTCMonth() + 1)}-${pad(utc.getUTCDate())} ${pad(utc.getUTCHours())}:${pad(utc.getUTCMinutes())}`;
}

// ---------------------------------------------------------------------------
// Body reading - shared pieces.
// ---------------------------------------------------------------------------

// The legacy turn header, matched one line at a time so a fenced block can be
// excluded. The timestamp group is lazy so the first middle dot separates it
// from the message id, which leaves an id free to contain one.
const TURN_HEADER = /^\*\*(Me|Assistant)\*\* \((.*?) · (.*)\):$/;
// The recorded-format turn header: speaker and minute-resolution clock, no
// identity. Strict except for trailing whitespace, which gbrain's own
// `imessage-slack` pattern also tolerates - and which used to absorb a turn.
const RECORDED_HEADER = /^\*\*(Me|Assistant)\*\* \((\d{4}-\d{2}-\d{2} \d{2}:\d{2})\):[ \t]*$/;
// A whole line, after trimming - never a substring. A message that quotes the
// sentinel mid-sentence is prose, and cutting there destroys every later turn.
// gbrain's findTimelineSplitIndex (src/core/markdown.ts) accepts four forms;
// the set holds the exact spellings, DECORATED_SENTINEL the case/space-tolerant
// `--- timeline ---` family, and timelineSentinelAt adds the fourth - a bare
// `---` whose next non-blank line is a `## Timeline`/`## History` heading.
const TIMELINE_SENTINELS = new Set(['<!-- timeline -->', '<!--timeline-->', '--- timeline ---']);
const DECORATED_SENTINEL = /^---\s+timeline\s+---$/i;
const TIMELINE_HEADING = /^##\s+(timeline|history)\b/i;

// Decision-identical to gbrain's own reading (findTimelineSplitIndex; keep in
// lockstep like headerClock): a bare `---` is a sentinel only when something
// stands above it and the next non-blank line is the Timeline/History heading.
// The two conditions are checked heading-first here because a legacy body puts
// a `---` separator between every pair of turns, and the lookahead ends at the
// very next non-blank line where the leading-content scan walks the whole
// prefix. The caller keeps its fence mask: a fenced line is sample text, never
// a candidate - the one deliberate divergence from gbrain's fence-blind parser,
// and it predates this predicate.
function timelineSentinelAt(lines, i) {
  const trimmed = lines[i].trim();
  if (TIMELINE_SENTINELS.has(trimmed) || DECORATED_SENTINEL.test(trimmed)) return true;
  if (trimmed !== '---') return false;
  for (let j = i + 1; j < lines.length; j += 1) {
    const next = lines[j].trim();
    if (next.length === 0) continue;
    if (!TIMELINE_HEADING.test(next)) return false;
    // gbrain skips a rule with nothing above it, so a body OPENING on
    // `---` + `## Timeline` is content there and stays content here.
    return lines.slice(0, i).join('\n').trim().length > 0;
  }
  return false;
}
const NO_TIMESTAMP = 'no timestamp';
const FALLBACK_PROVIDER = 'unknown';
const DAYS_IN_MONTH = [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31];

// Strict RFC 3339 `date-time`, the profile the schema's `format: date-time`
// names. Shape first, then real calendar and clock bounds, so `2026-02-30` and
// `T25:00:00` are rejected rather than passed through. Returns the value when it
// conforms, else null.
//
// Second 60 is a leap second, and RFC 3339 5.6 permits it only at the instant a
// leap second is inserted - midnight UTC. The local clock may read anything, as
// long as it names that instant: the RFC's own example set has
// `1990-12-31T15:59:60-08:00`. So the check normalizes the offset away and asks
// whether the UTC time-of-day is 23:59. `2026-02-01T09:00:60Z` and
// `2026-12-31T23:59:60+01:00` both fail it, and both are rejected by ajv +
// ajv-formats 3.0.1 in strict/full mode - which is what a consumer validating
// the published schema runs. The whole 11-value boundary set was measured
// against ajv 8.20.0 on 2026-08-02 and re-measured against 8.18.0 the same
// day; both agree with this function value for value.
function asRfc3339DateTime(value) {
  if (typeof value !== 'string') return null;
  const m = /^(\d{4})-(\d{2})-(\d{2})[Tt](\d{2}):(\d{2}):(\d{2})(\.\d+)?([Zz]|[+-]\d{2}:\d{2})$/.exec(value);
  if (!m) return null;
  const year = Number(m[1]);
  const month = Number(m[2]);
  const day = Number(m[3]);
  if (month < 1 || month > 12) return null;
  const leap = (year % 4 === 0 && year % 100 !== 0) || year % 400 === 0;
  if (day < 1 || day > (month === 2 && leap ? 29 : DAYS_IN_MONTH[month - 1])) return null;
  const hour = Number(m[4]);
  const minute = Number(m[5]);
  const second = Number(m[6]);
  if (hour > 23 || minute > 59 || second > 60) return null;
  const offset = m[8];
  let offsetMinutes = 0;
  if (offset !== 'Z' && offset !== 'z') {
    const offsetHour = Number(offset.slice(1, 3));
    const offsetMinute = Number(offset.slice(4, 6));
    if (offsetHour > 23 || offsetMinute > 59) return null;
    offsetMinutes = (offset[0] === '-' ? -1 : 1) * (offsetHour * 60 + offsetMinute);
  }
  if (second === 60) {
    const utcMinuteOfDay = (((hour * 60 + minute - offsetMinutes) % 1440) + 1440) % 1440;
    if (utcMinuteOfDay !== 23 * 60 + 59) return null;
  }
  return value;
}

const FENCE_LINE = /^ {0,3}(`{3,}|~{3,})(.*)$/s;

// The separator the LEGACY importer wrote between two turns: a blank line, a
// `---` rule, a blank line. On the legacy path it is the one structural signal
// on the page that separates a real turn boundary from a transcript pasted
// into a code block, and it decides the only case fence state cannot decide on
// its own - a header-shaped line reached while a fence is open.
function precededByTurnSeparator(lines, i) {
  return i >= 3
    && lines[i - 1].trim() === ''
    && lines[i - 2].trim() === '---'
    && lines[i - 3].trim() === '';
}

// One left-to-right pass producing the fence mask and the turn headers
// together, so that a fence can never hide a header belonging to a LATER turn.
// `disabled` holds the fence openers already proven not to close inside their
// own turn; those read as ordinary text. Returns `{ reopen }` when it meets such
// an opener, and the caller disables it and runs the pass again.
function scanPass(lines, disabled) {
  const fenced = new Array(lines.length).fill(false);
  const headers = [];
  const quotedHeaders = [];
  let open = null;
  for (let i = 0; i < lines.length; i += 1) {
    const fence = FENCE_LINE.exec(lines[i]);
    if (open === null) {
      const header = TURN_HEADER.exec(lines[i]);
      if (header) {
        headers.push({ line: i, match: header });
        continue;
      }
      // A backtick fence's info string may not itself contain a backtick.
      if (fence && !disabled.has(i) && !(fence[1][0] === '`' && fence[2].includes('`'))) {
        open = { char: fence[1][0], len: fence[1].length, at: i };
        fenced[i] = true;
      }
      continue;
    }
    if (TURN_HEADER.test(lines[i])) {
      // A fence's reach ends with the turn it opened in. A header carrying the
      // separator above it belongs to the next turn, so the fence is the thing
      // that is wrong and it loses: masking on from here would delete that
      // turn's id, role and timestamp outright and swallow its text into the
      // turn above, which is the loss no count in the output contradicts.
      if (precededByTurnSeparator(lines, i)) return { reopen: { at: open.at, spanningAt: i } };
      quotedHeaders.push(i);
    }
    fenced[i] = true;
    if (fence && fence[1][0] === open.char && fence[1].length >= open.len && fence[2].trim() === '') open = null;
  }
  // A fence left open at the end of the body would mask everything after it.
  if (open !== null) return { reopen: { at: open.at, spanningAt: -1 } };
  return { reopen: null, fenced, headers, quotedHeaders };
}

// The fence mask and the turn headers. A fence that cannot close inside its own
// turn is demoted to ordinary text and the pass is rerun without it, which is
// the same trade the unclosed case already made: a spurious extra message is a
// far smaller failure than a silently truncated conversation. Each rerun
// disables one more opener, so the loop runs at most once per fence line.
function scanFencesAndHeaders(lines, warn) {
  const disabled = new Set();
  for (;;) {
    const pass = scanPass(lines, disabled);
    if (pass.reopen === null) return pass;
    disabled.add(pass.reopen.at);
    warn(pass.reopen.spanningAt === -1
      ? `an unclosed code fence opens at body line ${pass.reopen.at + 1}; read as ordinary text so the turns after it are not lost.`
      : `the code fence opened at body line ${pass.reopen.at + 1} is still open at the turn header on body line ${pass.reopen.spanningAt + 1}; a fence does not span a turn, so it was read as ordinary text and that turn was kept.`);
  }
}

// The recorded-path twin of scanPass. Boundaries are accepted by position:
// a header-shaped line is boundary k only when its clock is the one expected
// for position k. When `demoteOnClock` is set, that same rule is the
// fence-demotion signal - a fence still open at a line carrying the next
// expected clock loses; when it is not set, such a line stays sample text.
// A fence still open at the end of the body is demoted either way (it would
// otherwise mask the timeline and title scans with no turn to bound it).
function scanRecordedPass(lines, expected, disabled, demoteOnClock) {
  const fenced = new Array(lines.length).fill(false);
  const boundaries = [];
  const proseHeaders = [];
  const quotedHeaders = [];
  const openers = [];
  let open = null;
  for (let i = 0; i < lines.length; i += 1) {
    const fence = FENCE_LINE.exec(lines[i]);
    if (open === null) {
      const header = RECORDED_HEADER.exec(lines[i]);
      if (header) {
        if (boundaries.length < expected.length && header[2] === expected[boundaries.length]) {
          boundaries.push({ line: i, speaker: header[1] });
        } else {
          proseHeaders.push(i);
        }
        continue;
      }
      if (fence && !disabled.has(i) && !(fence[1][0] === '`' && fence[2].includes('`'))) {
        open = { char: fence[1][0], len: fence[1].length, at: i, entry: { at: i, closedAt: -1 } };
        openers.push(open.entry);
        fenced[i] = true;
      }
      continue;
    }
    const header = RECORDED_HEADER.exec(lines[i]);
    if (demoteOnClock && header && boundaries.length < expected.length && header[2] === expected[boundaries.length]) {
      return { reopen: { at: open.at, spanningAt: i } };
    }
    if (header) quotedHeaders.push(i);
    fenced[i] = true;
    if (fence && fence[1][0] === open.char && fence[1].length >= open.len && fence[2].trim() === '') {
      open.entry.closedAt = i;
      open = null;
    }
  }
  if (open !== null) return { reopen: { at: open.at, spanningAt: -1 } };
  return { reopen: null, fenced, boundaries, proseHeaders, quotedHeaders, openers };
}

// Balanced fences genuinely win, in two stages. Stage 1 honors every fence
// that closes: even a quoted line carrying the next expected clock stays
// sample text inside one - a pasted transcript quoting this very
// conversation must not steal a boundary from outside its fence. (Fences
// still open at the end of the body are demoted in every stage.) Only when
// stage 1 anchors fewer turns than the record holds is some fence plausibly
// swallowing a real boundary - and then each fence is tried ALONE: the fence
// to lose is the one whose demotion by itself anchors every recorded turn.
// Trying them one at a time is what keeps a balanced fence that merely
// QUOTES a boundary from being demoted when a different fence caused the
// shortfall: demoting the quoting fence cannot recover the turn the other
// fence swallowed, so its trial fails and the right fence is found. Only if
// no single demotion suffices (two swallowing fences, or a genuinely
// mismatched body) does the greedy clock-armed demotion run as a last
// resort. Warnings are buffered per attempt and only the attempt whose
// result is used speaks.
function scanRecordedFences(lines, expected, warn) {
  const runToCompletion = (seed, demoteOnClock) => {
    const disabled = new Set(seed);
    const buffered = [];
    for (;;) {
      const pass = scanRecordedPass(lines, expected, disabled, demoteOnClock);
      if (pass.reopen === null) return { pass, buffered };
      disabled.add(pass.reopen.at);
      buffered.push(pass.reopen.spanningAt === -1
        ? `an unclosed code fence opens at body line ${pass.reopen.at + 1}; read as ordinary text so the turns after it are not lost.`
        : `the code fence opened at body line ${pass.reopen.at + 1} is still open at the turn header on body line ${pass.reopen.spanningAt + 1}; a fence does not span a turn, so it was read as ordinary text and that turn was kept.`);
    }
  };
  const first = runToCompletion([], false);
  if (first.pass.boundaries.length === expected.length) {
    for (const message of first.buffered) warn(message);
    return first.pass;
  }
  for (const opener of first.pass.openers) {
    // The opener and its closer are disabled TOGETHER: dropping only the
    // opener re-pairs every later fence line (the old closer becomes a new
    // opener), which let a wrong trial anchor the right COUNT of turns on
    // the wrong lines. Removing the pair leaves every other fence exactly
    // where the first stage saw it, so a wrong trial genuinely fails.
    const seed = opener.closedAt === -1 ? [opener.at] : [opener.at, opener.closedAt];
    const trial = runToCompletion(seed, false);
    if (trial.pass.boundaries.length === expected.length) {
      warn(`the code fence opened at body line ${opener.at + 1} hides the header of a recorded turn; a fence does not span a turn, so it was read as ordinary text and that turn was kept.`);
      for (const message of trial.buffered) warn(message);
      return trial.pass;
    }
  }
  const second = runToCompletion([], true);
  for (const message of second.buffered) warn(message);
  return second.pass;
}

// gbrain's own title precedence, minus the filename fallback this script must
// not use: frontmatter `title:` first, then the body's first H1.
function titleFromBody(lines, fenced) {
  for (let i = 0; i < lines.length; i += 1) {
    if (fenced[i]) continue;
    const m = /^#(?!#)\s+(.+?)\s*$/s.exec(lines[i]);
    if (m) return m[1].replace(/\s+#+\s*$/, '').trim();
  }
  return '';
}

// The timeline cut and the slice-into-messages step, shared by both paths.
// `anchors` carries each accepted boundary's line; `build` turns a boundary and
// its raw text into a message or null (null = dropped, already warned).
function cutAndSlice(lines, anchors, fenced, warn, stripLegacySeparator, build) {
  // Body content ahead of the first turn header (except the H1, which is the
  // title fallback) is not exported - a documented loss, but it was the one
  // loss class with no stderr trace at all, so it is reported here.
  if (anchors.length > 0) {
    let h1Seen = false;
    const preamble = [];
    for (let i = 0; i < anchors[0].line; i += 1) {
      if (fenced[i] || lines[i].trim() === '') continue;
      if (!h1Seen && /^#(?!#)\s+/.test(lines[i])) {
        h1Seen = true;
        continue;
      }
      preamble.push(i);
    }
    if (preamble.length > 0) {
      warn(`${preamble.length} line(s) of body content before the first turn header were not exported (first at body line ${preamble[0] + 1}).`);
    }
  }

  const sentinels = [];
  for (let i = 0; i < lines.length; i += 1) {
    if (fenced[i]) continue;
    if (timelineSentinelAt(lines, i)) sentinels.push(i);
  }

  // Cut only at a sentinel that no accepted turn follows. gbrain writes the
  // timeline after the whole body, so a sentinel with turns below it is a
  // message quoting the marker - the case that used to destroy every later
  // turn in silence.
  const lastAnchor = anchors.length > 0 ? anchors[anchors.length - 1].line : -1;
  let cut = lines.length;
  let hasTimeline = false;
  for (const at of sentinels) {
    if (at > lastAnchor) {
      cut = at;
      hasTimeline = true;
      // Named per page, not only tallied in the aggregate note: the tail cut
      // away here may be a real gbrain timeline or the end of a message that
      // quoted the sentinel - either way the operator can find the page.
      warn(`the body was cut at the timeline sentinel on body line ${at + 1}; nothing after it was exported.`);
      break;
    }
    warn(`the line \`${lines[at].trim()}\` at body line ${at + 1} has speaker turns after it, so it is not a timeline boundary; the body was not cut there.`);
  }

  const messages = [];
  for (const [i, anchor] of anchors.entries()) {
    const hasNext = i + 1 < anchors.length;
    const end = hasNext ? anchors[i + 1].line : cut;
    let raw = lines.slice(anchor.line + 1, end).join('\n');
    if (hasNext && stripLegacySeparator) {
      // Between two turns the legacy importer wrote exactly one `---`
      // separator line. Strip that one occurrence, never a `---` the message
      // itself ended with after the last turn.
      raw = `${raw}\n`.replace(/\n\n---\n\n$/, '');
    }
    const message = build(anchor, i, raw.trim());
    if (message !== null) messages.push(message);
  }
  return { messages, hasTimeline };
}

function readPageBody(body, warn) {
  const lines = body.split('\n');
  const { fenced, headers, quotedHeaders } = scanFencesAndHeaders(lines, warn);

  // Reported for the same reason a quoted timeline sentinel is: the script made
  // a call about what a line means, and the operator gets to see it.
  for (const at of quotedHeaders) {
    warn(`the line \`${lines[at].trim()}\` at body line ${at + 1} sits inside a fenced code block, so it was read as sample text and not as a turn.`);
  }

  let droppedEmpty = 0;
  let nulledTimestamps = 0;
  const { messages, hasTimeline } = cutAndSlice(lines, headers, fenced, warn, true, (header, _i, text) => {
    const id = header.match[3];
    if (text === '') {
      droppedEmpty += 1;
      warn(`message ${JSON.stringify(id)} has no text; dropped (envelope-v0 requires at least one character).`);
      return null;
    }
    // The legacy importer wrote the literal `no timestamp` when the envelope
    // had none. Read it back as the null the schema asks for, not as prose.
    let ts = null;
    if (header.match[2] !== NO_TIMESTAMP) {
      ts = asRfc3339DateTime(header.match[2]);
      if (ts === null) {
        nulledTimestamps += 1;
        warn(`message ${JSON.stringify(id)} carries ${JSON.stringify(header.match[2])} where a turn header's timestamp goes; it is not an RFC 3339 date-time, so ts was written as null.`);
      }
    }
    return { id, role: header.match[1] === 'Me' ? 'user' : 'assistant', ts, text };
  });

  return { messages, title: titleFromBody(lines, fenced), hasTimeline, droppedEmpty, nulledTimestamps };
}

/**
 * The recorded path: identity from the frontmatter record, speaker and text
 * from the body. Returns `{ mismatch, anchored }` when the body does not
 * anchor exactly one turn per recorded message - the caller skips the page
 * loudly; a positional join over the wrong count assigns real ids to the
 * wrong text, which is worse than refusing.
 */
function readRecordedBody(body, record, pageDate, warn) {
  const lines = body.split('\n');
  // The clock the importer wrote for each recorded message: derived from its
  // `ts` where one is usable, else the page-date fallback the importer used.
  const expected = record.map((r) => {
    const clock = headerClock(r.ts);
    return clock === null ? `${pageDate} 00:00` : clock;
  });
  const { fenced, boundaries, proseHeaders, quotedHeaders } = scanRecordedFences(lines, expected, warn);

  for (const at of quotedHeaders) {
    warn(`the line \`${lines[at].trim()}\` at body line ${at + 1} sits inside a fenced code block, so it was read as sample text and not as a turn.`);
  }
  for (const at of proseHeaders) {
    warn(`the line \`${lines[at].trim()}\` at body line ${at + 1} is shaped like a turn header but does not carry the clock the messages record expects next, so it was read as prose.`);
  }

  if (boundaries.length !== record.length) {
    return { mismatch: true, anchored: boundaries.length };
  }

  let droppedEmpty = 0;
  let nulledTimestamps = 0;
  const { messages, hasTimeline } = cutAndSlice(lines, boundaries, fenced, warn, false, (anchor, i, text) => {
    const { id, ts } = record[i];
    if (text === '') {
      // The `{id, ts}` entry is dropped with its turn, so the join between the
      // remaining turns and their entries stays aligned. Worded as what this
      // side can see: the SOURCE message may well have had text - a message
      // whose whole text was consumed as a forged boundary, or cut away at a
      // sentinel, lands here too, and "has no text" would be false about it.
      droppedEmpty += 1;
      warn(`no text remained for message ${JSON.stringify(id)} between its turn header and the next boundary; dropped (envelope-v0 requires at least one character). If the source message had text, it was consumed as a boundary line or cut at a sentinel.`);
      return null;
    }
    let outTs = null;
    if (ts !== null) {
      outTs = asRfc3339DateTime(ts);
      if (outTs === null) {
        nulledTimestamps += 1;
        warn(`message ${JSON.stringify(id)} records ${JSON.stringify(ts)} as its timestamp; it is not an RFC 3339 date-time, so ts was written as null.`);
      }
    }
    return { id, role: anchor.speaker === 'Me' ? 'user' : 'assistant', ts: outTs, text };
  });

  return { mismatch: false, messages, title: titleFromBody(lines, fenced), hasTimeline, droppedEmpty, nulledTimestamps };
}

const files = markdownFiles(pagesDir);
// provider (the page's trimmed `source:`, or null when it carries none) ->
// { conversations, messageCount }. Insertion order is first appearance in
// sorted path order - what the old "first provider" collapse meant by first;
// the difference is that nobody wins anymore.
const groups = new Map();
const seenIds = new Set();
let skippedNotConversation = 0;
let skippedNoFrontmatter = 0;
let skippedNoMessages = 0;
let sawSourceKey = false;
let skippedUnreadableRecord = 0;
let skippedJoinMismatch = 0;
let droppedEmptyMessages = 0;
let droppedTimelines = 0;
let nulledTimestamps = 0;

for (const file of files) {
  // Normalize to LF up front. Every match below is line-anchored, so a page
  // saved with CRLF would otherwise fail to parse as a whole and be reported as
  // frontmatter-less. The cost is stated in the header: CRLF inside a message
  // comes back as LF.
  // A UTF-8 BOM is stripped like CRLF is normalized: gray-matter strips it
  // too, so a BOM page is a first-class conversation to gbrain, and refusing
  // to see its frontmatter made the whole conversation vanish into the
  // "without frontmatter" count.
  const page = splitPage(readFileSync(file, 'utf8').replace(/^/, '').replace(/\r\n/g, '\n'));
  if (!page) {
    skippedNoFrontmatter += 1;
    continue;
  }
  const front = parseFrontmatter(page.front);
  if (front.type !== 'conversation') {
    skippedNotConversation += 1;
    continue;
  }
  // Noted before any skip below, so the no-provider warning can tell "no page
  // carries a source: key" apart from "the pages that carry one were skipped".
  if (typeof front.source === 'string' && front.source.trim() !== '') sawSourceKey = true;
  const warn = (message) => console.warn(`warning: ${file} - ${message}`);

  let read;
  const record = parseMessagesRecord(page.front);
  if (record.present) {
    if (record.error) {
      skippedUnreadableRecord += 1;
      warn(`the messages record could not be read (${record.error}); skipped rather than joined wrong.`);
      continue;
    }
    if (record.items.length === 0) {
      skippedNoMessages += 1;
      warn('the messages record is empty; skipped (envelope-v0 requires at least one message per conversation).');
      continue;
    }
    const badId = record.items.findIndex((item) => typeof item.id !== 'string');
    if (badId !== -1) {
      skippedUnreadableRecord += 1;
      const shown = record.items[badId].id === null ? 'null' : JSON.stringify(record.items[badId].id.nonString);
      warn(`messages[${badId}] records ${shown} where its id goes; envelope-v0 requires a string message id and inventing one is the synthesis the spec forbids, so the page was skipped.`);
      continue;
    }
    // A non-string `ts` (an unquoted number, say) is what the importer writes
    // for a non-conforming envelope; it is not a clock the header could have
    // been derived from, so it takes the fallback-clock path and comes back
    // null like every other unusable timestamp.
    const items = record.items.map((item) => ({
      id: item.id,
      ts: typeof item.ts === 'string' || item.ts === null ? item.ts : item.ts.nonString,
    }));
    const pageDate = typeof front.date === 'string' && HEADER_DATE.test(front.date) ? front.date : EPOCH_DATE;
    read = readRecordedBody(page.body, items, pageDate, warn);
    if (read.mismatch) {
      skippedJoinMismatch += 1;
      warn(`the frontmatter records ${record.items.length} message(s) but the body anchors ${read.anchored} turn(s); a positional join over unequal counts would assign identity to the wrong text, so the page was skipped.`);
      continue;
    }
  } else {
    read = readPageBody(page.body, warn);
  }

  droppedEmptyMessages += read.droppedEmpty;
  nulledTimestamps += read.nulledTimestamps;
  if (read.hasTimeline) droppedTimelines += 1;
  if (read.messages.length === 0) {
    skippedNoMessages += 1;
    warn('no speaker turns found; skipped (envelope-v0 requires at least one message per conversation).');
    continue;
  }
  const provider = typeof front.source === 'string' && front.source.trim() !== '' ? front.source.trim() : null;
  // Never synthesize. An absent id is null, which the schema permits and the
  // spec requires of converters.
  // VERBATIM, not trimmed. The importer records the id verbatim precisely so
  // that two ids differing only by surrounding whitespace stay distinct -
  // trimming here re-collapsed them on the way out, the exact ambiguity that
  // once let one import destroy another. (Blank-or-absent still maps to null:
  // the importer never writes the key for an id that trims to nothing.)
  const id = typeof front.memvelope_conversation_id === 'string' && front.memvelope_conversation_id.trim() !== ''
    ? front.memvelope_conversation_id
    : null;
  // envelope-v0 says ids should be unique within an envelope and that consumers
  // must tolerate duplicates. Emit both conversations and say so, rather than
  // dropping one to keep the field clean.
  if (id !== null && seenIds.has(id)) {
    warn(`conversation id ${JSON.stringify(id)} already used by another page; both are emitted.`);
  }
  if (id !== null) seenIds.add(id);
  // Frontmatter first, then the body's H1 - gbrain's own precedence - so a
  // title that js-yaml folded into a block scalar has a second way home.
  const title = (typeof front.title === 'string' && front.title.trim() !== '' ? front.title.trim() : read.title)
    || 'Untitled conversation';
  let group = groups.get(provider);
  if (group === undefined) {
    group = { conversations: [], messageCount: 0 };
    groups.set(provider, group);
  }
  group.conversations.push({
    id,
    title,
    // The page's `date` is a day, not a date-time, so it cannot fill these.
    // First and last message timestamps are the only date-times on the page.
    created_at: read.messages[0].ts,
    updated_at: read.messages[read.messages.length - 1].ts,
    messages: read.messages,
  });
  group.messageCount += read.messages.length;
}

// One envelope per provider. meta.source_provider names ONE provider for a
// whole envelope, so a mixed directory cannot be described by one file without
// falsifying someone's `source:` - the old collapse-to-first did exactly that,
// and re-importing the collapsed envelope stamped the winner onto every losing
// page at exit 0, with nothing on disk retaining the truth. Pages with no
// `source:` at all are their own group under the placeholder: absence of
// evidence is not membership in whichever provider the directory also holds,
// and folding them in would be the same falsification arriving through
// absence rather than collision.
const named = [...groups.keys()].filter((p) => p !== null);

function envelopeFor(provider, group) {
  return {
    memvelope: 'envelope-v0',
    meta: {
      source_provider: provider ?? FALLBACK_PROVIDER,
      conversation_count: group.conversations.length,
      message_count: group.messageCount,
    },
    conversations: group.conversations,
  };
}

// `out.mve.json` -> `out.<token>.mve.json`, keeping whichever json suffix the
// operator chose. The token is the provider with everything outside
// [A-Za-z0-9._-] replaced by `-`, so a hostile `source:` cannot steer the
// write out of the output directory, then deduplicated case-insensitively so
// providers differing only by case still get two files on the
// case-insensitive filesystems macOS ships - and the same two names
// everywhere else.
function providerOutPath(basePath, token) {
  const mve = /^(.*)\.mve\.json$/.exec(basePath);
  if (mve) return `${mve[1]}.${token}.mve.json`;
  const json = /^(.*)\.json$/.exec(basePath);
  if (json) return `${json[1]}.${token}.json`;
  return `${basePath}.${token}`;
}

if (groups.size <= 1) {
  // A single-provider page set (or an empty one): one file, the same name and
  // the same words as before the fan-out existed.
  const provider = named[0] ?? null;
  const group = groups.get(provider) ?? { conversations: [], messageCount: 0 };
  // envelope-v0 requires meta.source_provider, so it cannot be omitted the way
  // meta.source_export_date is. Minting a token no registry defines is a
  // guess, and a guess gets reported like every other lossy edge here.
  if (provider === null) {
    // Two different situations, two true statements: pages with no source: key
    // at all, versus sourced pages that were all skipped before they could
    // contribute one - telling the second audience to "set source:" would be
    // advice about a key they already set.
    console.warn(sawSourceKey
      ? `warning: every page carrying a \`source:\` key was skipped before it could contribute one, and envelope-v0 requires meta.source_provider; wrote the placeholder ${JSON.stringify(FALLBACK_PROVIDER)}, which is not a registered provider token. Fix the skipped pages to name the real provider.`
      : `warning: no page carries a \`source:\` key, and envelope-v0 requires meta.source_provider; wrote the placeholder ${JSON.stringify(FALLBACK_PROVIDER)}, which is not a registered provider token. Set \`source:\` on the pages to name the real provider.`);
  }
  writeFileSync(outPath, JSON.stringify(envelopeFor(provider, group), null, 2) + '\n');
  console.log(`wrote ${group.conversations.length} conversation(s), ${group.messageCount} message(s) to ${outPath}`);
} else {
  if (named.length > 1) {
    console.warn(`note: pages name ${named.length} source providers (${named.join(', ')}); wrote one envelope per provider.`);
  }
  const usedTokens = new Set();
  const ordered = groups.has(null) ? [...named, null] : named;
  for (const provider of ordered) {
    const group = groups.get(provider);
    const base = (provider ?? FALLBACK_PROVIDER).replace(/[^A-Za-z0-9._-]/g, '-');
    let token = base;
    for (let n = 2; usedTokens.has(token.toLowerCase()); n += 1) token = `${base}-${n}`;
    usedTokens.add(token.toLowerCase());
    const file = providerOutPath(outPath, token);
    if (provider === null) {
      console.warn(`warning: ${group.conversations.length} page(s) carry no \`source:\` key; their conversation(s) were written to ${file} with the placeholder ${JSON.stringify(FALLBACK_PROVIDER)}, which is not a registered provider token. Set \`source:\` on the pages to name the real provider.`);
    }
    writeFileSync(file, JSON.stringify(envelopeFor(provider, group), null, 2) + '\n');
    console.log(`wrote ${group.conversations.length} conversation(s), ${group.messageCount} message(s) to ${file} (provider ${JSON.stringify(provider ?? FALLBACK_PROVIDER)})`);
  }
}
// Every page that did not become a conversation is accounted for on stderr, so
// a mistargeted directory reads as a diagnosis rather than an empty file.
if (skippedNoFrontmatter || skippedNotConversation || skippedNoMessages || skippedUnreadableRecord || skippedJoinMismatch || droppedEmptyMessages) {
  console.warn(`scanned ${files.length} markdown file(s): ${skippedNoFrontmatter} without frontmatter, ${skippedNotConversation} not type conversation, ${skippedNoMessages} without speaker turns, ${skippedUnreadableRecord} with a messages record that could not be read, ${skippedJoinMismatch} whose body does not match their messages record, ${droppedEmptyMessages} empty message(s) dropped.`);
}
// Two losses that leave the output conforming and would otherwise be invisible.
// Worded as what happened - a cut at a sentinel - not as an assertion that a
// timeline was really there: a message that ends by quoting the sentinel on
// its own line is cut identically, and a note claiming it "carried a timeline"
// would be false about that page.
if (droppedTimelines) {
  console.warn(`note: ${droppedTimelines} page(s) were cut at a timeline sentinel; envelope-v0 has no field for a timeline, so nothing after the sentinel was exported.`);
}
if (nulledTimestamps) {
  console.warn(`note: ${nulledTimestamps} turn timestamp(s) were not RFC 3339 date-times and were written as null.`);
}