Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
Conversation archive reads OpenClaw in far fewer process starts, and archive status reads no session (#9482)
Garry Tanaf51e19ea261
diff --git a/changelog.d/capy-conversation-archive-reads.md b/changelog.d/capy-conversation-archive-reads.mdnew file mode 100644index 0000000000..5ac9ebc881--- /dev/null+++ b/changelog.d/capy-conversation-archive-reads.md@@ -0,0 +1,8 @@+**The conversation archive reads OpenClaw's Gateway in far fewer process starts, and `gbrain transcripts archive status` no longer reads any session.**++### Changed+- The OpenClaw Gateway reader asks `chat.history` for pages of up to 1,000 messages within a 4 MiB budget, and reads up to 4 sessions at once. Each `openclaw gateway call` is its own process, so on a test install with 350 sessions a read went from 162 process starts and 164 s to 52 starts and 16 s. A Gateway older than 2026.10 that refuses the byte budget is asked again without it. Pages of one session stay in order, conversations keep their `sessions.list` order, and a failed session still fails the whole read before any day is written.+- `gbrain transcripts archive status` and `gbrain doctor --only transcript_archive_freshness` report what the last archive run read for each reader (conversations, newest session day, time and any error) instead of running the readers. They start no `openclaw` process and parse no session file. A reader no run has read yet says so. Each real run records this in `transcripts.archive.last_read.<source>`; a dry run records nothing.++### Fixed+- Doctor warns when no archive run has read a reader successfully within `transcripts.archive.stale_days`, for example when maintenance passes keep stopping at their deadline before that reader, and suggests `gbrain transcripts archive run`.diff --git a/docs/architecture/key-files/commands-2-continued.md b/docs/architecture/key-files/commands-2-continued.mdindex 2da6482109..bcc5a66fdc 100644--- a/docs/architecture/key-files/commands-2-continued.md+++ b/docs/architecture/key-files/commands-2-continued.md@@ -44,5 +44,5 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/commands/doctor/checks/purge-parks.ts` — `purge_parked_pages` (#6741 C): names parked purges with their last error (only this version's parks, `isParked`); fix `gbrain pages purge-deleted --retry-parked --yes` (preview `--dry-run --json`), consent destructive, ask_user. Also `derived_origin_parked` (GST-2689, `checkDerivedOriginParked`): names each origin `persistence/derived-origin-hold.ts` parked with the page that blocks it and the reason (`details.origins`); fix the read-only `gbrain repair visibility --source <id> --slug <blocking page>` preview. An origin whose owner purge was held (`held_request_id`) also gets `finish_argv` (`gbrain delete <slug> --purge --source-id <id>`, a new request), named in the message, which finishes it at once.- `src/commands/doctor/checks/unlinked-facts.ts` — `unlinked_facts` (#5836), registered before `search_mode` and mirrored in `doctorReportRemote`: per the doctor's source scope (`--source`, else every source; the caller's grant remotely), the unlinked share of the newest 100,000 active facts per source (`idx_facts_since` LATERAL walk), the 7-day unlinked share of new facts, `no_subject` judgments and the 7-day `linked` tiers from `fact_relink_attempts`, fence-owned unlinked rows (row_num or source_markdown_slug set, conversation rows excluded), the conversation extractor's unlinked derived rows (`cli:extract-conversation-facts*`, #6764: count, origin `conversation`, and `details.conversation` naming the source with the most and its pages, most rows first, up to `CONVERSATION_FIX_SLUGS` = 50), origins and 7-day write-time inferred links by tier (`entity inferred from <tier>` context, bare or after ` — `). Warns when over 25% of at least 20 new facts are unlinked, facts judged `no_subject` and conversation rows memoized `conversation_unresolved` excluded (`new_7d.conversation_unresolved`); the fix is `gbrain facts relink --dry-run`; when conversation rows are unlinked the envelope `fix` (`details.conversation.fix`) is, for rows the free conversation relink has not tried, `gbrain facts relink --conversation --source <id> --slugs <csv>` (preview `--dry-run`, consent none, actor agent, host operator remotely; test `test/conversation-relink-6764.test.ts`) and, once only rows it memoized `conversation_unresolved` remain (a memo that a page created later re-arms unless the reason is memoized, `core/facts/conversation-memo.ts`), a bounded re-extraction of those pages not yet re-extracted on their current content (`gbrain extract-conversation-facts --source-id <id> --slugs <csv> --force --max-cost-usd <N>`, input `N` the approved USD, preview `--dry-run`, consent paid + egress, ask_user; test `test/conversation-facts-unlinked-6764.test.ts`), both verified by `gbrain doctor --only unlinked_facts --json`, because plain relink never examines derived rows; the remote report asks the brain host operator to run it. When only rows on pages already re-extracted on their current content remain, `details.conversation.fix` is `manual`: no envelope `fix`, and the message names linking by hand with `--entity` (test `test/conversation-memo-6764.test.ts`). Without `fact_relink_attempts` the relink figures are null and nothing is excluded. Non-user facts are skipped (#7270). Tests: `test/doctor-unlinked-facts.test.ts`, `test/relink-attribution-7270.test.ts`.-- `src/commands/doctor/checks/transcript-archive.ts` — `transcript_archive_freshness` (GST-2768, brain category): off is ok and reads nothing; on, it reads `archiveStatus` and `transcripts.archive.last_maintenance` and warns on a harness more than `transcripts.archive.stale_days` (2) behind its newest session day (fix `gbrain transcripts archive run`, or `backfill --from <day>` past one run window), on no maintenance pass in that long (fix `gbrain autopilot --install`, `persistent_install` consent), on held days (with codes), reader failures and pass errors; no readable sessions is info. Pinned by `test/doctor-transcript-archive-freshness.test.ts`.+- `src/commands/doctor/checks/transcript-archive.ts` — `transcript_archive_freshness` (GST-2768, brain category): off is ok and reads nothing; on, reads `archiveStatus` (no reader run) and `transcripts.archive.last_maintenance` and warns on a harness `transcripts.archive.stale_days` (2) behind the newest session day seen or a reader unread that long (fix `gbrain transcripts archive run`, or `backfill --from <day>` past one run window), on no maintenance pass that long (fix `gbrain autopilot --install`, `persistent_install` consent), on held days, reader and pass errors; no readable sessions is info. Pinned by `test/doctor-transcript-archive-freshness.test.ts`.- `src/commands/doctor/checks/unattended-maintenance.ts` — `unattended_maintenance` (GBRA-438): `readUpkeepStatus` as a check; off and autopilot-owned are ok, a brain under 48h with no run is info, a run older than 48h, a step failing 3 runs in a row or an item serve parked (`details.parked`) warns with `gbrain maintain --unattended` (`consent: []` unless an embedding key and a non-zero cap make it paid); on an overdue brain `details.alternative_fix` renders `gbrain maintain --schedule` as the second fix.diff --git a/docs/guides/conversation-archive.md b/docs/guides/conversation-archive.mdindex 56fb53bb20..0c6cd3a96e 100644--- a/docs/guides/conversation-archive.md+++ b/docs/guides/conversation-archive.md@@ -11,7 +11,7 @@ gbrain transcripts archive on # turn it on (a shared brgbrain transcripts archive run --dry-run # preview the last two local daysgbrain transcripts archive run # write themgbrain transcripts archive backfill --from 2026-09-01 # older history, newest days first-gbrain transcripts archive status # readers, timezone, archived and held days+gbrain transcripts archive status # readers as of the last run, timezone, archived and held daysgbrain doctor --only transcript_archive_freshness # is it keeping up?```@@ -71,14 +71,22 @@ false` turns both off.`gbrain doctor --only transcript_archive_freshness` (brain category) warns when:- a harness's newest archived day is more than `transcripts.archive.stale_days` (default 2) behind the newest session- day its reader sees. The fix is `gbrain transcripts archive run`, or+ day the last run read. The fix is `gbrain transcripts archive run`, or`gbrain transcripts archive backfill --from <first missing day>` when the gap is wider than one run;+- no archive run has read a reader successfully in that long, for example when maintenance passes keep stopping at+ their deadline before that reader. The fix is `gbrain transcripts archive run`;- the archive is on but no maintenance pass has run it in that long. The fix is `gbrain autopilot --install`(a persistent service, so it asks first);- days are held (each is listed with its code, such as `archive_owner_unset` or `archive_redaction_failed`), a reader- failed, or the last pass recorded an error.+ failed on the last run, or the last pass recorded an error.-Off is ok with no detail, and a harness with no readable sessions is informational.+Off is ok with no detail, and a harness with no readable sessions, or one no run has read yet, is informational.++Neither this check nor `gbrain transcripts archive status` reads a session or calls the OpenClaw Gateway, so both+answer in about a second however many sessions there are. They read the archive's ledger and config, and what each+run (`run`, `backfill` or the maintenance pass, never a `--dry-run`) recorded about every reader: how many+conversations it read, the newest session day any run has seen, and the reader's error if the last run failed. Status+shows those as of that run, with its time; a reader no run has read yet says so.Every command takes `--json` and `--source <id>` (the archive source, `conversations` by default).@@ -180,6 +188,12 @@ Current OpenClaw keeps live sessions in a database owned by its Gateway. gbrain`archive_gateway_auth` and `archive_gateway_failed`; a failed read leaves the archive ledger unchanged, so the nextrun retries. Checked against OpenClaw 2026.10.9 and 2026.11.3, which list sessions and return history the same way.+Each `openclaw gateway call` is a new process, and starting one takes far longer than the Gateway takes to answer,+so gbrain asks for the largest history page the Gateway gives (1,000 messages within 4 MiB; OpenClaw before 2026.11+caps a page at 512 KiB, and one before 2026.10 refuses the byte budget, so gbrain asks again without it) and reads up+to 4 sessions at once. The messages of each session, and the order of the conversations, are the same as one read at+a time; if any session cannot be read, the whole Gateway read fails once the reads already running have ended.+Session files OpenClaw keeps on disk are read too, even when the Gateway is down: `<id>.jsonl.reset.<stamp>`, thezstd-compressed `<id>.jsonl.deleted.<stamp>.<hex>.zst` and the legacy compact backup `<id>.jsonl.bak.<stamp>`.Several files of one session merge into one conversation (`gbrain transcripts ingest` reads only live files). Migration copies (`*.migrated`, `*.migrated.<n>`,diff --git a/src/commands/doctor/checks/transcript-archive.ts b/src/commands/doctor/checks/transcript-archive.tsindex 298434bc42..d1e150a8a4 100644--- a/src/commands/doctor/checks/transcript-archive.ts+++ b/src/commands/doctor/checks/transcript-archive.ts@@ -1,16 +1,18 @@/*** transcript_archive_freshness doctor check (GST-2768 PR 3): is the conversation archive keeping up?*- * Off (switched off, or a shared brain that is only offered it) is one quiet ok row and reads no session. On, it- * reads archiveStatus (the readers this machine has and the archive ledger) and the maintenance record- * (transcripts.archive.last_maintenance), and warns when:+ * Off (switched off, or a shared brain that is only offered it) is one quiet ok row. On, it reads archiveStatus (the+ * readers this machine has, the archive ledger and what the last run read; no reader runs, so no session is read and+ * no Gateway is called) and the maintenance record (transcripts.archive.last_maintenance), and warns when:* - a harness's newest archived day is more than transcripts.archive.stale_days (default 2) behind the newest- * session day its reader sees: fix gbrain transcripts archive run, or backfill --from <first missing day> when+ * session day the last run read: fix gbrain transcripts archive run, or backfill --from <first missing day> when* that day is older than one run's window;+ * - no run has read a reader successfully in stale_days (a maintenance pass that keeps stopping at its deadline+ * before that reader): fix gbrain transcripts archive run;* - no maintenance pass has run the archive in stale_days (or ever, on a brain older than that): fix* gbrain autopilot --install (persistent, so ask the user first);* - days are held (listed with their codes), a reader failed, or the last pass recorded an error.- * A harness with no readable sessions is informational.+ * A harness with no readable sessions, or one no run has read yet, is informational.*/import type { Check } from '../../doctor.ts';import type { BrainEngine } from '../../../core/engine.ts';@@ -54,6 +56,10 @@ export async function checkTranscriptArchiveFreshness(let fix: Action | undefined;const behind: Array<{ harness: string; newest_session_day: string; newest_archived_day: string | null; days_behind: number }> = [];for (const h of status.harnesses) {+ if (h.read_at && nowMs - Date.parse(h.read_at) > staleDays * DAY_MS) {+ problems.push(h.harness + ': no archive run has read ' + h.reader + ' successfully since ' + h.read_at);+ fix ??= agentFix(['gbrain', 'transcripts', 'archive', 'run'], 'Archives the last ' + (runWindow + 1) + ' local days from every readable harness now.', name);+ }if (h.info) continue;if (!h.newest_session_day) continue;const gap = h.newest_archived_day ? Math.round((dayMs(h.newest_session_day) - dayMs(h.newest_archived_day)) / DAY_MS) : Infinity;@@ -94,7 +100,11 @@ export async function checkTranscriptArchiveFreshness(if (problems.length) return { name, status: 'warn', message: 'Conversation archive: ' + problems.join('; ') + '.', details, ...(fix ? { fix } : {}) };const readable = status.harnesses.filter(h => !h.info && h.newest_session_day);const idle = [...new Set(status.harnesses.filter(h => h.info).map(h => h.harness))];- const idleNote = idle.length ? 'no readable sessions for ' + idle.join(', ') + ' (gbrain transcripts archive status names each reader)' : '';+ const unread = [...new Set(status.harnesses.filter(h => !h.info && !h.read_at && !h.newest_session_day).map(h => h.harness))];+ const idleNote = [+ ...(idle.length ? ['no readable sessions for ' + idle.join(', ') + ' (gbrain transcripts archive status names each reader)'] : []),+ ...(unread.length ? ['not read by an archive run yet: ' + unread.join(', ')] : []),+ ].join('; ');if (!readable.length) return infoCheck(name, 'Conversation archive is on; ' + (idleNote || 'no harness has readable sessions yet') + '.', 'not_applicable', undefined, details);return { name, status: 'ok', message: 'Conversation archive is up to date (' + readable.map(h => h.harness + ' ' + h.newest_archived_day).join(', ') + ')' + (idleNote ? '; ' + idleNote : '') + '.', details };} catch (error) {diff --git a/src/commands/transcripts-archive.ts b/src/commands/transcripts-archive.tsindex 3f9d870251..f2d6215e4a 100644--- a/src/commands/transcripts-archive.ts+++ b/src/commands/transcripts-archive.ts@@ -3,7 +3,7 @@** run archive the last few local days from every readable harness* backfill --from D [--to D] archive a date range, newest first, capped per run; embedding held for approval- * status readers, pinned timezone, archived and held days+ * status readers as of the last run, pinned timezone, archived and held days (reads no session)* adopt-legacy --dir <folder> list the days an old archive folder covers (--apply records them so backfill skips them)* on | off turn the archive on or off for this brain*@@ -58,10 +58,13 @@ export async function runTranscriptsArchive(engine: BrainEngine, args: string[],const status = await archive.archiveStatus(engine, { sourceId });out(status, () => {console.log('conversation archive: ' + (status.enabled ? 'on' : 'off') + ' (source: ' + status.source_id + ', timezone: ' + (status.timezone ?? status.configured_timezone + ', pinned at the first run') + ')');- for (const h of status.harnesses) console.log(' ' + h.reader.padEnd(18) + ' ' + (h.info ?? (h.conversations + ' conversation(s), newest session day ' + (h.newest_session_day ?? '-') + ', newest archived day ' + (h.newest_archived_day ?? '-'))));+ for (const h of status.harnesses) {+ const read = h.read_at ? h.conversations + ' conversation(s) read by the last ' + (h.read_mode ?? 'run') + ' (' + h.read_at + '), newest session day ' + (h.newest_session_day ?? '-') : 'not read by an archive run yet';+ console.log(' ' + h.reader.padEnd(18) + ' ' + (h.info ?? read + ', newest archived day ' + (h.newest_archived_day ?? '-')));+ }console.log(' archived days: ' + status.archived_days + (status.legacy_days ? ', covered by a legacy archive: ' + status.legacy_days : ''));for (const h of status.held) console.log(' held: ' + h.harness + ' ' + h.day + ' (' + h.code + ')');- for (const e of status.errors) console.log(' error: ' + e.reader + ' ' + e.code + ': ' + e.message);+ for (const e of status.errors) console.log(' error on the last run' + (e.at ? ' (' + e.at + ')' : '') + ': ' + e.reader + ' ' + e.code + ': ' + e.message);if (status.rescope) console.log(' not kept to the owner yet (' + status.rescope.code + '): ' + status.rescope.message);});if (status.errors.length) setCliExitVerdict(1);diff --git a/src/core/transcripts/archive.ts b/src/core/transcripts/archive.tsindex 115d6361f5..3d43657894 100644--- a/src/core/transcripts/archive.ts+++ b/src/core/transcripts/archive.ts@@ -85,6 +85,8 @@ const AUDIENCE_PREFIX = 'transcripts.archive.audience.';const RESCOPE_CAPPED_PREFIX = 'transcripts.archive.rescope_capped.';/** Days whose index page still has to be written (a failed or interrupted index write), per source. */const INDEX_PENDING_PREFIX = 'transcripts.archive.index_pending.';+/** What the last run that read each reader found, per source: status and doctor report it without reading again. */+const LAST_READ_PREFIX = 'transcripts.archive.last_read.';const INGESTED_VIA = 'cli:transcripts-archive';const DAY_RE = /^\d{4}-\d{2}-\d{2}$/;@@ -170,7 +172,27 @@ export interface ArchiveDayOutcome {message?: string;}-export interface ArchiveReaderError { harness: string; reader: string; code: string; message: string; fix?: Action }+export interface ArchiveReaderError { harness: string; reader: string; code: string; message: string; fix?: Action; /** When the run that met it read (status only). */ at?: string }++/** One reader as the last run that read it found it: the newest message any run saw, and the last run's error until a later read succeeds. */+interface ArchiveReadRecord {+ harness: string;+ at?: string;+ mode?: 'run' | 'backfill';+ conversations?: number;+ newest?: string;+ error?: { at: string; code: string; message: string; fix?: Action };+}++async function readArchiveReads(engine: BrainEngine, sourceId: string): Promise<Record<string, ArchiveReadRecord>> {+ const raw = await engine.getConfig(`${LAST_READ_PREFIX}${sourceId}`);+ try {+ const doc = raw ? JSON.parse(raw) as unknown : null;+ return doc && typeof doc === 'object' && !Array.isArray(doc) ? doc as Record<string, ArchiveReadRecord> : {};+ } catch {+ return {};+ }+}export interface ArchiveRunResult {mode: 'run' | 'backfill';@@ -515,6 +537,49 @@ interface DayWrite {pending?: string[];}+/** Reads every reader once for a run, and unless it is a dry run records what each one found (transcripts.archive.last_read.<source>) for status and doctor. */+async function readReaders(engine: BrainEngine, readers: ArchiveReader[], scanOpts: ArchiveScanOpts,+ ctx: { sourceId: string; mode: 'run' | 'backfill'; dryRun: boolean; at: string; result: ArchiveRunResult; pastDeadline: () => boolean }): Promise<ArchiveScanEntry[]> {+ const entries: ArchiveScanEntry[] = [];+ const { result, pastDeadline } = ctx;+ const reads = new Map<string, { harness: string; conversations: number; newest?: string; error?: ArchiveReaderError }>();+ for (const reader of readers) {+ if (entries.length && pastDeadline()) break;+ if (reader.unavailable) { result.info.push(reader.unavailable); continue; }+ try {+ let found = 0;+ let newest: string | undefined;+ for await (const entry of scanReader(reader, scanOpts)) {+ entries.push(entry);+ found++;+ if (entry.newest && (!newest || entry.newest > newest)) newest = entry.newest;+ }+ reads.set(reader.name, { harness: reader.harness, conversations: found, ...(newest ? { newest } : {}) });+ if (found === 0) result.info.push('no readable sessions for ' + reader.harness + '; use gbrain connectors or an export with gbrain transcripts ingest');+ } catch (err) {+ if (classifyItemFailure(err).scope === 'run') throw err;+ const e = err as { code?: string; message?: string; fix?: Action };+ const error = { harness: reader.harness, reader: reader.name, code: e.code ?? 'archive_reader_failed', message: e.message ?? String(err), ...(e.fix ? { fix: e.fix } : {}) };+ result.errors.push(error);+ reads.set(reader.name, { harness: reader.harness, conversations: 0, error });+ }+ }+ if (ctx.dryRun || !reads.size) return entries;+ const { at } = ctx;+ const record = await readArchiveReads(engine, ctx.sourceId);+ for (const [name, read] of reads) {+ const prior = record[name];+ if (read.error) {+ record[name] = { ...prior, harness: read.harness, error: { at, code: read.error.code, message: read.error.message, ...(read.error.fix ? { fix: read.error.fix } : {}) } };+ continue;+ }+ const newest = [prior?.newest, read.newest].filter((v): v is string => !!v).sort().pop();+ record[name] = { harness: read.harness, at, mode: ctx.mode, conversations: read.conversations, ...(newest ? { newest } : {}) };+ }+ await engine.setConfig(LAST_READ_PREFIX + ctx.sourceId, JSON.stringify(record));+ return entries;+}+export async function runTranscriptArchive(engine: BrainEngine, opts: ArchiveRunOpts): Promise<ArchiveRunResult> {const dryRun = opts.dryRun === true;const enabled = await archiveEnabled(engine);@@ -591,20 +656,7 @@ async function archiveSource(engine: BrainEngine, opts: ArchiveRunOpts, ctx: { econst minMtimeMs = opts.mode === 'backfill' ? Date.parse(from + 'T00:00:00.000Z') - 2 * 86_400_000 : sinceMs;const dayOf = (ts: string) => localDay(ts, tz.timezone);const pastDeadline = () => opts.deadlineAtMs !== undefined && Date.now() >= opts.deadlineAtMs && (result.stopped_at_deadline = true);- const entries: ArchiveScanEntry[] = [];- for (const reader of readers) {- if (entries.length && pastDeadline()) break;- if (reader.unavailable) { result.info.push(reader.unavailable); continue; }- try {- let found = 0;- for await (const entry of scanReader(reader, { sinceMs, minMtimeMs, dayOf })) { entries.push(entry); found++; }- if (found === 0) result.info.push('no readable sessions for ' + reader.harness + '; use gbrain connectors or an export with gbrain transcripts ingest');- } catch (err) {- if (classifyItemFailure(err).scope === 'run') throw err;- const e = err as { code?: string; message?: string; fix?: Action };- result.errors.push({ harness: reader.harness, reader: reader.name, code: e.code ?? 'archive_reader_failed', message: e.message ?? String(err), ...(e.fix ? { fix: e.fix } : {}) });- }- }+ const entries = await readReaders(engine, readers, { sinceMs, minMtimeMs, dayOf }, { sourceId, mode: opts.mode, dryRun, at: now.toISOString(), result, pastDeadline });// A Gateway conversation and the session files of the same session id are one conversation: a reset in place leaves// the earlier messages only in the .jsonl.reset.* file, so the file messages merge in, each message once.const filesBySession = new Map<string, ArchiveScanEntry[]>();@@ -967,15 +1019,25 @@ export interface ArchiveStatus {source_id: string;timezone: string | null;configured_timezone: string;- harnesses: Array<{ harness: string; reader: string; conversations: number; newest_session_day: string | null; newest_archived_day: string | null; info?: string }>;+ /**+ * One row per reader on this machine. conversations and newest_session_day are as of the last run that read it+ * (read_at, read_mode; null when no run has read it yet): status never reads a reader itself. In run mode a run+ * reads only recently active conversations, so conversations counts those; newest_session_day is the newest any run saw.+ */+ harnesses: Array<{ harness: string; reader: string; conversations: number; newest_session_day: string | null; newest_archived_day: string | null; read_at: string | null; read_mode?: 'run' | 'backfill'; info?: string }>;archived_days: number;held: Array<{ harness: string; day: string; code: string }>;legacy_days: number;+ /** Readers whose last run failed, with when (at); a later successful read clears one. */errors: ArchiveReaderError[];/** The brain became shared but the pages archived while personal could not be kept to the owner yet. */rescope?: ArchiveRescopeSkip;}+/**+ * The archive's own records: config, ledger and what the last run read. It runs no reader (no Gateway call, no session+ * file parsed); building the reader list only checks which harness folders exist and whether openclaw is on PATH.+ */export async function archiveStatus(engine: BrainEngine, opts: { sourceId?: string; readers?: ArchiveReader[] } = {}): Promise<ArchiveStatus> {const { sourceId } = await effectiveSource(engine, opts.sourceId);const pinned = (await engine.getConfig(`${PIN_PREFIX}${sourceId}`))?.trim() || null;@@ -987,25 +1049,18 @@ export async function archiveStatus(engine: BrainEngine, opts: { sourceId?: striheld: entries.filter(e => e.held).map(e => ({ harness: e.harness, day: e.day, code: e.held!.code })),legacy_days: (await legacyCovered(engine, sourceId)).size, errors: [],};+ const reads = await readArchiveReads(engine, sourceId);+ const zone = pinned ?? configured;for (const reader of opts.readers ?? await defaultArchiveReaders()) {const archived = entries.filter(e => e.harness === reader.harness && e.hash).map(e => e.day).sort();- const row = { harness: reader.harness, reader: reader.name, conversations: 0, newest_session_day: null as string | null, newest_archived_day: archived[archived.length - 1] ?? null };+ const row = { harness: reader.harness, reader: reader.name, conversations: 0, newest_session_day: null as string | null, newest_archived_day: archived[archived.length - 1] ?? null, read_at: null as string | null };if (reader.unavailable) { out.harnesses.push({ ...row, info: reader.unavailable }); continue; }- try {- const zone = pinned ?? configured;- let conversations = 0;- let newest: string | undefined;- for await (const entry of scanReader(reader, { dayOf: ts => localDay(ts, zone) })) {- conversations++;- if (entry.newest && (!newest || entry.newest > newest)) newest = entry.newest;- }- out.harnesses.push({ ...row, conversations, newest_session_day: newest ? localDay(newest, zone) : null,- ...(conversations ? {} : { info: 'no readable sessions for ' + reader.harness + '; use gbrain connectors or an export with gbrain transcripts ingest' }) });- } catch (err) {- const e = err as { code?: string; message?: string; fix?: Action };- out.errors.push({ harness: reader.harness, reader: reader.name, code: e.code ?? 'archive_reader_failed', message: e.message ?? String(err), ...(e.fix ? { fix: e.fix } : {}) });- out.harnesses.push(row);- }+ const read = reads[reader.name];+ if (read?.error) out.errors.push({ harness: reader.harness, reader: reader.name, code: read.error.code, message: read.error.message, ...(read.error.fix ? { fix: read.error.fix } : {}), at: read.error.at });+ const newest = read?.newest ? localDay(read.newest, zone) : null;+ if (!read?.at) { out.harnesses.push({ ...row, newest_session_day: newest }); continue; }+ out.harnesses.push({ ...row, conversations: read.conversations ?? 0, newest_session_day: newest, read_at: read.at, ...(read.mode ? { read_mode: read.mode } : {}),+ ...(newest ? {} : { info: 'no readable sessions for ' + reader.harness + '; use gbrain connectors or an export with gbrain transcripts ingest' }) });}const rescope = await archiveRescopeSkipped(engine, opts.sourceId);if (rescope) out.rescope = rescope;diff --git a/src/core/transcripts/openclaw-gateway.ts b/src/core/transcripts/openclaw-gateway.tsindex d85dbfaa1b..b2d7a08e77 100644--- a/src/core/transcripts/openclaw-gateway.ts+++ b/src/core/transcripts/openclaw-gateway.ts@@ -9,10 +9,16 @@* - sessions.list {limit, offset}: rows with key, sessionId, agentId, chatType (direct | group | room), createdAt,* updatedAt (epoch ms); paged by hasMore / nextOffset. Archived sessions are not listed, and the call takes no* parameter that lists them.- * - chat.history {sessionKey, limit, offset}: messages {role, content (string or [{type: text, text}]),+ * - chat.history {sessionKey, limit, maxBytes, offset}: messages {role, content (string or [{type: text, text}]),* timestamp (epoch ms), __openclaw {id, seq}}; pages walk backward from the newest (hasMore / nextOffset), each* page oldest first. It still reads an archived session by key. A reset in place keeps the sessionId and drops- * the earlier messages from this read.+ * the earlier messages from this read. A page holds at most limit messages (up to 1000) within maxBytes (default+ * 512 KiB; since 2026.11 up to a 6 MiB ceiling, before that never above 512 KiB); a Gateway from before 2026.10+ * refuses maxBytes as an unexpected property.+ *+ * Every call is one openclaw process, and a process start costs far more than the Gateway's answer, so the reader+ * asks for the largest page the Gateway gives and reads up to HISTORY_CONCURRENCY sessions side by side. Pages of one+ * session stay in order, and the conversations come back in sessions.list order whichever read finished first.* - Errors come back as a JSON envelope {ok: false, error: {type, code?, message, details?}}: a closed port is* gateway_transport_error, a missing credential gateway_credentials_required, a wrong one gateway_request_error* with details.code AUTH_*.@@ -22,7 +28,8 @@* group and chatType direct). Anything else is unknown, which the archive resolves owner-only.** Failures throw archive_gateway_unreachable, archive_gateway_auth or archive_gateway_failed with the Gateway's- * message redacted; the reader yields nothing before every read finished, so a failed run changes no ledger row.+ * message redacted; the reader yields nothing before every read finished, and after a failure it starts no new read+ * and throws once the reads already running have ended, so a failed run changes no ledger row.*/import { execFileBounded } from '../bounded-child-exec.ts';@@ -50,7 +57,9 @@ export interface GatewayConversation {}const LIST_PAGE = 100;-const HISTORY_PAGE = 200;+const HISTORY_PAGE = 1000;+const HISTORY_MAX_BYTES = 4 * 1024 * 1024;+const HISTORY_CONCURRENCY = 4;const MAX_PAGES = 10_000;/** Redact and bound text that came from the Gateway before it reaches a message, a log or a receipt. */@@ -167,11 +176,27 @@ function messageTimestamp(value: unknown): string {const asRecord = (v: unknown): Record<string, unknown> => (v && typeof v === 'object' ? v as Record<string, unknown> : {});-async function readHistory(transport: GatewayTransport, sessionKey: string): Promise<{ sessionId?: string; messages: TranscriptMessage[] }> {+/** Whether this Gateway takes the maxBytes page budget; a Gateway from before 2026.10 refuses it as an unexpected property. */+interface HistoryBudget { maxBytes: boolean }++async function historyPage(transport: GatewayTransport, budget: HistoryBudget, sessionKey: string, offset: number): Promise<Record<string, unknown>> {+ const params = { sessionKey, limit: HISTORY_PAGE, ...(offset ? { offset } : {}) };+ if (!budget.maxBytes) return asRecord(await transport.call('chat.history', params));+ try {+ return asRecord(await transport.call('chat.history', { ...params, maxBytes: HISTORY_MAX_BYTES }));+ } catch (err) {+ const e = err as { code?: string; message?: string };+ if (e.code !== 'archive_gateway_failed' || !/unexpected property '?maxBytes/.test(String(e.message))) throw err;+ budget.maxBytes = false;+ return asRecord(await transport.call('chat.history', params));+ }+}++async function readHistory(transport: GatewayTransport, budget: HistoryBudget, sessionKey: string): Promise<{ sessionId?: string; messages: TranscriptMessage[] }> {const pages: Array<Record<string, unknown>> = [];let offset = 0;for (let i = 0; i < MAX_PAGES; i++) {- const page = asRecord(await transport.call('chat.history', { sessionKey, limit: HISTORY_PAGE, ...(offset ? { offset } : {}) }));+ const page = await historyPage(transport, budget, sessionKey, offset);pages.push(page);if (page.hasMore !== true || typeof page.nextOffset !== 'number' || page.nextOffset <= offset) break;offset = page.nextOffset;@@ -211,13 +236,31 @@ export function openclawGatewayReader(opts: { transport?: GatewayTransport } = {if (page.hasMore !== true || typeof page.nextOffset !== 'number' || page.nextOffset <= offset) break;offset = page.nextOffset;}- const out: GatewayConversation[] = [];- for (const row of rows) {- const key = typeof row.key === 'string' ? row.key : '';- if (!key) continue;+ const wanted = rows.filter(row => {+ if (typeof row.key !== 'string' || !row.key) return false;const updated = typeof row.updatedAt === 'number' ? row.updatedAt : null;- if (listOpts.sinceMs !== undefined && updated !== null && updated < listOpts.sinceMs) continue;- const history = await readHistory(transport, key);+ return listOpts.sinceMs === undefined || updated === null || updated >= listOpts.sinceMs;+ });+ const budget: HistoryBudget = { maxBytes: true };+ const histories: Array<Awaited<ReturnType<typeof readHistory>>> = new Array(wanted.length);+ let next = 0;+ let failure: { error: unknown } | null = null;+ const worker = async () => {+ while (!failure && next < wanted.length) {+ const i = next++;+ try {+ histories[i] = await readHistory(transport, budget, String(wanted[i]!.key));+ } catch (error) {+ failure ??= { error };+ }+ }+ };+ await Promise.all(Array.from({ length: Math.min(HISTORY_CONCURRENCY, wanted.length) }, worker));+ if (failure) throw (failure as { error: unknown }).error;+ const out: GatewayConversation[] = [];+ for (const [i, row] of wanted.entries()) {+ const key = String(row.key);+ const history = histories[i]!;if (history.messages.length === 0) continue;const sessionId = typeof row.sessionId === 'string' ? row.sessionId : history.sessionId ?? key;out.push({diff --git a/test/doctor-transcript-archive-freshness.test.ts b/test/doctor-transcript-archive-freshness.test.tsindex 85848112d0..0f088aea2f 100644--- a/test/doctor-transcript-archive-freshness.test.ts+++ b/test/doctor-transcript-archive-freshness.test.ts@@ -1,7 +1,7 @@/*** GST-2768 PR 3: the transcript_archive_freshness doctor check. Off is ok with no noise; with the archive on it warns- * when the newest archived day is more than transcripts.archive.stale_days behind the newest session day a reader- * sees (fix: archive run, or backfill --from when the gap is wider than one run), when no maintenance pass has run+ * when the newest archived day is more than transcripts.archive.stale_days behind the newest session day the last run+ * read (fix: archive run, or backfill --from when the gap is wider than one run); the check itself reads no reader, when no maintenance pass has run* the archive in that long (fix: gbrain autopilot --install), and lists held days with their codes; a harness with* no readable sessions is informational. PGLite, synthetic sessions only.*/@@ -40,6 +40,16 @@ async function recordPass(atMs: number): Promise<void> {await engine.setConfig(ARCHIVE_MAINTENANCE_KEY, JSON.stringify({ at: new Date(atMs).toISOString(), runner: 'serve', status: 'ok', written: 0, held: 0, errors: [] }));}+/** A run at NOW that read a conversation on `day` but could not read it again to write it, so the day stays unarchived. */+async function runSawUnarchived(day: string): Promise<void> {+ const ts = day + 'T08:00:00.000Z';+ const reader: ArchiveReader = {+ harness: 'codex', name: 'codex',+ async *scan({ dayOf }) { yield { harness: 'codex', conversationKey: 'codex-new', sessionId: 'codex-new', days: [dayOf(ts)], newest: ts, load: async () => null }; },+ };+ await runTranscriptArchive(engine, { mode: 'run', now: NOW, readers: [reader] });+}+async function archiveDay(day: string): Promise<void> {await runTranscriptArchive(engine, { mode: 'run', now: new Date(day + 'T12:00:00Z'), readers: [staticReader('codex', [said('codex-old-' + day, day + 'T08:00:00.000Z')])] });}@@ -62,6 +72,7 @@ describe('transcript_archive_freshness', () => {test('a 3-day gap warns with the run fix', async () => {await archiveDay('2026-11-07');+ await runSawUnarchived('2026-11-10');await recordPass(NOW.getTime() - 60_000);const readers = [staticReader('codex', [said('codex-old-2026-11-07', '2026-11-07T08:00:00.000Z'), said('codex-new', '2026-11-10T08:00:00.000Z')])];const check = await checkTranscriptArchiveFreshness(engine, { readers, nowMs: NOW.getTime() });@@ -74,6 +85,7 @@ describe('transcript_archive_freshness', () => {test('a gap wider than one run names the backfill with --from', async () => {await archiveDay('2026-11-01');+ await runSawUnarchived('2026-11-09');await recordPass(NOW.getTime() - 60_000);const readers = [staticReader('codex', [said('codex-old-2026-11-01', '2026-11-01T08:00:00.000Z'), said('codex-new', '2026-11-09T08:00:00.000Z')])];const check = await checkTranscriptArchiveFreshness(engine, { readers, nowMs: NOW.getTime() });@@ -89,6 +101,16 @@ describe('transcript_archive_freshness', () => {expect(check.status).toBe('ok');}, 60_000);+ test('a reader no run has read in stale_days warns even when its archive matches the last read', async () => {+ await archiveDay('2026-11-01');+ await recordPass(NOW.getTime() - 60_000);+ const readers = [staticReader('codex', [said('codex-old-2026-11-01', '2026-11-01T08:00:00.000Z')])];+ const check = await checkTranscriptArchiveFreshness(engine, { readers, nowMs: NOW.getTime() });+ expect(check.status).toBe('warn');+ expect(check.message).toContain('codex: no archive run has read codex successfully since 2026-11-01T12:00:00.000Z');+ expect(check.fix?.argv).toEqual(['gbrain', 'transcripts', 'archive', 'run']);+ }, 60_000);+test('a harness with no readable sessions is informational', async () => {await recordPass(NOW.getTime() - 60_000);const check = await checkTranscriptArchiveFreshness(engine, { readers: [staticReader('grok', []), { harness: 'hermes', name: 'hermes', unavailable: 'no Hermes sessions on this machine' }], nowMs: NOW.getTime() });diff --git a/test/helpers/config-reader-registry.ts b/test/helpers/config-reader-registry.tsindex 6b317a06f4..f07f763afb 100644--- a/test/helpers/config-reader-registry.ts+++ b/test/helpers/config-reader-registry.ts@@ -30,6 +30,7 @@ export const INTERNAL_CONFIG_STATE: readonly InternalConfigState[] = [{ key: 'transcripts.archive.rescope_capped.', prefix: true, reason: 'the number of archive folders when the owner-only re-scope passed the per-source rule limit and kept the whole conversations/ folder to the owner instead (transcripts/archive.ts); status and doctor report it' },{ key: 'transcripts.archive.last_maintenance', reason: 'the conversation archive\'s maintenance-pass record (transcripts/archive-maintenance.ts); doctor transcript_archive_freshness reads it, so a hand edit would misreport the archive' },{ key: 'transcripts.archive.backfill_state', reason: 'whether the maintenance pass has finished backfilling older conversation days (transcripts/archive-maintenance.ts); a hand edit would stop or restart the backfill' },+ { key: 'transcripts.archive.last_read.', prefix: true, reason: 'what the last conversation archive run read from each reader per source (transcripts/archive.ts): status and doctor report it as of that run instead of reading every session again' },{ key: 'transcripts.archive.index_pending.', prefix: true, reason: 'days whose conversation archive day index still has to be written (transcripts/archive.ts); the next run retries them' },{ key: 'maintenance.last_run', reason: 'the free-upkeep run record (steps, cursor, parked items, next due time) that maintenance/run.ts writes after each gbrain maintain --unattended run; maintain --status and doctor read it, so a hand edit would misreport upkeep' },{ key: 'maintenance.embed_spend', reason: 'the per-day dollar total the capped upkeep embed step has spent (maintenance/run.ts); the daily cap reads it, so a hand edit would let upkeep spend past maintenance.embed_daily_usd' },diff --git a/test/helpers/openclaw-gateway-stub.ts b/test/helpers/openclaw-gateway-stub.tsnew file mode 100644index 0000000000..130c4ddc7d--- /dev/null+++ b/test/helpers/openclaw-gateway-stub.ts@@ -0,0 +1,113 @@+/**+ * A stub `openclaw` executable for the Gateway reader suites: it waits a configurable start-up delay (standing in for+ * the CLI's own process start), answers `gateway call sessions.list | chat.history` from a synthetic session store the+ * way OpenClaw 2026.11.3 pages them (chat.history: limit up to 1000, default 200; a byte budget of maxBytes, default+ * 512 KiB, hard ceiling 6 MiB; pages walk backward from the newest, each page oldest first), and appends one line per+ * invocation (method, params, start and end time) to a log. Every value is synthetic.+ */+import { chmodSync, readFileSync, writeFileSync } from 'node:fs';+import { join } from 'node:path';++export interface StubSession { key: string; sessionId: string; messages: number; updatedAt: number; chatType?: string }+export interface StubCall { method: string; params: Record<string, unknown>; start: number; end: number; pid: number }++export interface GatewayStubOpts {+ dir: string;+ sessions: StubSession[];+ /** Milliseconds the stub waits before it answers, per invocation. */+ delayMs: number;+ /** Extra Gateway time per chat.history page, drawn uniformly from [min, max] ms. */+ answerMs?: [number, number];+ /** Bytes each message counts against the page budget. */+ messageBytes?: number;+ /** A Gateway from before 2026.10, whose closed chat.history schema refuses maxBytes. */+ rejectMaxBytes?: boolean;+ /** chat.history on this session key answers with an auth error envelope. */+ failKey?: string;+ /** Name of the executable file (default openclaw). */+ name?: string;+}++export function gatewayStub(opts: GatewayStubOpts): { bin: string; calls: () => StubCall[] } {+ const store = join(opts.dir, (opts.name ?? 'openclaw') + '-store.json');+ const log = join(opts.dir, (opts.name ?? 'openclaw') + '-calls.jsonl');+ writeFileSync(store, JSON.stringify({ ...opts, dir: undefined }));+ writeFileSync(log, '');+ const bin = join(opts.dir, opts.name ?? 'openclaw');+ writeFileSync(bin, `#!${process.execPath}+const fs = require('node:fs');+const start = Date.now();+const cfg = JSON.parse(fs.readFileSync(${JSON.stringify(store)}, 'utf8'));+const argv = process.argv.slice(2);+const method = argv[2];+const params = JSON.parse(argv[argv.indexOf('--params') + 1] || '{}');+const sleep = ms => new Promise(r => setTimeout(r, ms));+const done = (doc, code) => {+ fs.appendFileSync(${JSON.stringify(log)}, JSON.stringify({ method, params, start, end: Date.now(), pid: process.pid }) + '\\n');+ process.stdout.write(JSON.stringify(doc) + '\\n', () => process.exit(code));+};+(async () => {+ await sleep(cfg.delayMs);+ if (method === 'sessions.list') {+ const limit = params.limit ?? 50;+ const offset = params.offset ?? 0;+ const rows = cfg.sessions.slice(offset, offset + limit).map(s => ({ key: s.key, sessionId: s.sessionId, chatType: s.chatType ?? 'direct', updatedAt: s.updatedAt }));+ const next = offset + rows.length;+ return done({ sessions: rows, hasMore: next < cfg.sessions.length, ...(next < cfg.sessions.length ? { nextOffset: next } : {}) }, 0);+ }+ if (method !== 'chat.history') return done({ ok: false, error: { type: 'gateway_request_error', code: 'INVALID_REQUEST', message: 'unknown method ' + method } }, 1);+ if (cfg.rejectMaxBytes && params.maxBytes !== undefined) {+ return done({ ok: false, error: { type: 'gateway_request_error', code: 'INVALID_REQUEST', message: "invalid chat.history params: at root: unexpected property 'maxBytes'" } }, 1);+ }+ if (cfg.failKey && params.sessionKey === cfg.failKey) {+ return done({ ok: false, error: { type: 'gateway_request_error', code: 'UNAUTHORIZED', message: 'unauthorized', details: { code: 'AUTH_TOKEN_MISMATCH' } } }, 1);+ }+ const [lo, hi] = cfg.answerMs ?? [0, 0];+ if (hi > 0) await sleep(lo + Math.random() * (hi - lo));+ const s = cfg.sessions.find(x => x.key === params.sessionKey);+ const total = s ? s.messages : 0;+ const budget = Math.min(params.maxBytes ?? 512 * 1024, 6 * 1024 * 1024);+ const size = Math.max(1, Math.min(params.limit ?? 200, 1000, Math.floor(budget / (cfg.messageBytes ?? 2150))));+ const offset = params.offset ?? 0;+ const endIdx = Math.max(0, total - offset);+ const startIdx = Math.max(0, endIdx - size);+ const messages = [];+ for (let i = startIdx; i < endIdx; i++) {+ messages.push({ role: i % 2 ? 'assistant' : 'user', content: 'synthetic message ' + i + ' in ' + s.key, timestamp: s.updatedAt - (total - i) * 1000, __openclaw: { id: s.sessionId + ':' + i, seq: i + 1 } });+ }+ const next = offset + messages.length;+ done({ sessionKey: params.sessionKey, sessionId: s ? s.sessionId : undefined, messages, hasMore: startIdx > 0, ...(startIdx > 0 ? { nextOffset: next } : {}) }, 0);+})();+`);+ chmodSync(bin, 0o755);+ return {+ bin,+ calls: () => readFileSync(log, 'utf8').trim().split('\n').filter(Boolean).map(l => JSON.parse(l) as StubCall),+ };+}++/** The most stub invocations of one method that ran at the same time. */+export function maxOverlap(calls: StubCall[], method: string): number {+ const edges = calls.filter(c => c.method === method).flatMap(c => [[c.start, 1], [c.end, -1]] as Array<[number, number]>)+ .sort((a, b) => a[0] - b[0] || a[1] - b[1]);+ let now = 0;+ let max = 0;+ for (const [, d] of edges) { now += d; max = Math.max(max, now); }+ return max;+}++/** The large-install shape the reader is tuned for: 350 sessions, 40 active recently, 158 history pages at 200 a page. */+export function largeInstall(nowMs: number): StubSession[] {+ const sessions: StubSession[] = [];+ for (let i = 0; i < 350; i++) {+ const recent = i % 8 === 0 && sessions.filter(s => s.updatedAt >= nowMs - 86_400_000).length < 40;+ const n = sessions.filter(s => s.updatedAt >= nowMs - 86_400_000).length;+ sessions.push({+ key: 'agent:main:example-session-' + i,+ sessionId: '00000000-0000-4000-8000-' + String(i).padStart(12, '0'),+ messages: recent ? (n < 2 ? 4300 : 500) : 20,+ updatedAt: recent ? nowMs - 60_000 * (i + 1) : nowMs - 30 * 86_400_000,+ });+ }+ return sessions;+}diff --git a/test/helpers/transcript-archive.ts b/test/helpers/transcript-archive.tsindex 90fbac8c22..4abd49d894 100644--- a/test/helpers/transcript-archive.ts+++ b/test/helpers/transcript-archive.ts@@ -37,6 +37,7 @@ export async function archivedPages(engine: BrainEngine, sourceId = 'conversatio}));}+/** Every archive config row except transcripts.archive.last_read.*, which records what each run read (and when), not what it archived. */export async function ledgerRows(engine: BrainEngine): Promise<Array<{ key: string; value: string }>> {- return engine.executeRaw<{ key: string; value: string }>("SELECT key, value FROM config WHERE key LIKE 'transcripts.archive.%' ORDER BY key");+ return engine.executeRaw<{ key: string; value: string }>("SELECT key, value FROM config WHERE key LIKE 'transcripts.archive.%' AND key NOT LIKE 'transcripts.archive.last\\_read.%' ORDER BY key");}diff --git a/test/transcripts-archive-status-reads.test.ts b/test/transcripts-archive-status-reads.test.tsnew file mode 100644index 0000000000..4df1d110eb--- /dev/null+++ b/test/transcripts-archive-status-reads.test.ts@@ -0,0 +1,102 @@+/**+ * gbrain transcripts archive status (and the doctor check built on it) reads the archive's own records only: the+ * ledger, the config and what the last run recorded about each reader. It never runs a reader, so a slow Gateway or a+ * large session folder cannot slow it down. PGLite, synthetic sessions only.+ */+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';+import { mkdtempSync, rmSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { resetPgliteState } from './helpers/reset-pglite.ts';+import { ARCHIVE_TIMEZONE_KEY, archiveStatus, runTranscriptArchive, setArchiveEnabled, type ArchiveReader } from '../src/core/transcripts/archive.ts';+import { cliGatewayTransport, openclawGatewayReader } from '../src/core/transcripts/openclaw-gateway.ts';+import { conversation } from './helpers/transcript-archive.ts';+import { gatewayStub } from './helpers/openclaw-gateway-stub.ts';++const dir = mkdtempSync(join(tmpdir(), 'archive-status-'));+const NOW = new Date('2026-11-10T09:00:00Z');+let engine: PGLiteEngine;++beforeAll(async () => {+ engine = new PGLiteEngine();+ await engine.connect({});+ await engine.initSchema();+});++afterAll(async () => {+ await engine.disconnect();+ rmSync(dir, { recursive: true, force: true });+});++beforeEach(async () => {+ await resetPgliteState(engine);+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');+ await engine.setConfig('brain.audience', 'personal');+ await setArchiveEnabled(engine, true);+});++function counting(harness: 'codex' | 'grok', fail?: Error): ArchiveReader & { reads: number } {+ const reader = {+ harness, name: harness, reads: 0,+ async list() {+ reader.reads++;+ if (fail) throw fail;+ return [conversation(harness, harness + '-a', [['user', '2026-11-09T08:00:00.000Z', 'first'], ['assistant', '2026-11-10T08:00:00.000Z', 'second']])];+ },+ };+ return reader;+}++describe('archive status reads no reader', () => {+ test('it reports what the last run read, as of that run, without reading again', async () => {+ const codex = counting('codex');+ await runTranscriptArchive(engine, { mode: 'run', readers: [codex], now: NOW });+ expect(codex.reads).toBe(1);+ const status = await archiveStatus(engine, { readers: [codex] });+ expect(codex.reads).toBe(1);+ expect(status.harnesses).toEqual([expect.objectContaining({+ harness: 'codex', reader: 'codex', conversations: 1, newest_session_day: '2026-11-10', newest_archived_day: '2026-11-10', read_mode: 'run',+ })]);+ expect(Date.parse(status.harnesses[0]!.read_at!)).toBeGreaterThan(0);+ expect(status.errors).toEqual([]);+ }, 60_000);++ test('a reader no run has read yet is listed as not read, and still not read', async () => {+ const grok = counting('grok');+ const status = await archiveStatus(engine, { readers: [grok, { harness: 'hermes', name: 'hermes', unavailable: 'no Hermes sessions on this machine' }] });+ expect(grok.reads).toBe(0);+ expect(status.harnesses.map(h => [h.reader, h.read_at, h.conversations, h.newest_session_day, h.info ?? null])).toEqual([+ ['grok', null, 0, null, null],+ ['hermes', null, 0, null, 'no Hermes sessions on this machine'],+ ]);+ });++ test('a reader that failed on the last run is reported with its code and when it failed, keeping the newest day it read before', async () => {+ await runTranscriptArchive(engine, { mode: 'run', readers: [counting('codex')], now: NOW });+ const failing = counting('codex', Object.assign(new Error('synthetic gateway refusal'), { code: 'archive_gateway_auth' }));+ await runTranscriptArchive(engine, { mode: 'run', readers: [failing], now: NOW });+ const status = await archiveStatus(engine, { readers: [failing] });+ expect(failing.reads).toBe(1);+ expect(status.errors).toEqual([expect.objectContaining({ reader: 'codex', code: 'archive_gateway_auth', message: 'synthetic gateway refusal' })]);+ expect(Date.parse(status.errors[0]!.at!)).toBeGreaterThan(0);+ expect(status.harnesses[0]).toMatchObject({ newest_session_day: '2026-11-10' });+ }, 60_000);++ test('a dry run records nothing about its reads', async () => {+ await runTranscriptArchive(engine, { mode: 'run', readers: [counting('codex')], now: NOW, dryRun: true });+ const status = await archiveStatus(engine, { readers: [counting('codex')] });+ expect(status.harnesses[0]).toMatchObject({ read_at: null, conversations: 0 });+ }, 60_000);++ test('status with the OpenClaw Gateway reader starts no openclaw process', async () => {+ const stub = gatewayStub({ dir, sessions: [{ key: 'agent:main:example-status', sessionId: 'status-1', messages: 4, updatedAt: NOW.getTime() }], delayMs: 50 });+ const gateway = openclawGatewayReader({ transport: cliGatewayTransport({ bin: stub.bin, timeoutMs: 10_000 }) });+ await runTranscriptArchive(engine, { mode: 'run', readers: [gateway], now: NOW });+ const started = stub.calls().length;+ expect(started).toBeGreaterThan(0);+ const status = await archiveStatus(engine, { readers: [gateway] });+ expect(stub.calls().length).toBe(started);+ expect(status.harnesses[0]).toMatchObject({ harness: 'openclaw', reader: 'openclaw-gateway', conversations: 1, newest_session_day: '2026-11-10' });+ }, 60_000);+});diff --git a/test/transcripts-openclaw-gateway-argv.test.ts b/test/transcripts-openclaw-gateway-argv.test.tsindex 371997c771..31a493b7ef 100644--- a/test/transcripts-openclaw-gateway-argv.test.ts+++ b/test/transcripts-openclaw-gateway-argv.test.ts@@ -53,7 +53,7 @@ describe('openclaw gateway call child process', () => {expect(argv.slice(6)).toEqual(['--timeout', '5000']);}const historyKeys = calls.filter(c => c.argv[2] === 'chat.history').map(c => (JSON.parse(c.argv[5]!) as { sessionKey: string }).sessionKey);- expect(historyKeys).toEqual(HOSTILE);+ expect([...historyKeys].sort()).toEqual([...HOSTILE].sort());expect(() => readFileSync(join(dir, 'pwned'))).toThrow();}, 30_000);diff --git a/test/transcripts-openclaw-gateway-pages.test.ts b/test/transcripts-openclaw-gateway-pages.test.tsnew file mode 100644index 0000000000..44af295be9--- /dev/null+++ b/test/transcripts-openclaw-gateway-pages.test.ts@@ -0,0 +1,66 @@+/**+ * The OpenClaw Gateway reader's process count and concurrency, against a stub `openclaw` that waits a start-up delay+ * per invocation and pages chat.history the way OpenClaw 2026.11.3 does. The large-install shape is 350 sessions, 40+ * of them active, whose history is 158 pages at the old 200-message page. Every value is synthetic.+ */+import { afterAll, describe, expect, test } from 'bun:test';+import { mkdtempSync, rmSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import { cliGatewayTransport, openclawGatewayReader } from '../src/core/transcripts/openclaw-gateway.ts';+import { gatewayStub, largeInstall, maxOverlap, type StubSession } from './helpers/openclaw-gateway-stub.ts';++const dir = mkdtempSync(join(tmpdir(), 'openclaw-pages-'));+afterAll(() => rmSync(dir, { recursive: true, force: true }));++const NOW = Date.parse('2026-11-10T12:00:00Z');+const SINCE = NOW - 4 * 86_400_000;+const token = ['tok', 'synthetic', 'pages'].join('-');+const reader = (bin: string) => openclawGatewayReader({ transport: cliGatewayTransport({ bin, url: 'ws://127.0.0.1:18789', token, timeoutMs: 20_000 }) });+const expected = (s: StubSession) => Array.from({ length: s.messages }, (_, i) => 'synthetic message ' + i + ' in ' + s.key);++describe('openclaw gateway reader: pages and processes', () => {+ test('the large-install shape reads in 52 process starts instead of 162, at most 4 side by side, every message in order', async () => {+ const sessions = largeInstall(NOW);+ const stub = gatewayStub({ dir, sessions, delayMs: 100, answerMs: [0, 80] });+ const convs = await reader(stub.bin).list({ sinceMs: SINCE });+ const calls = stub.calls();+ expect(calls.filter(c => c.method === 'sessions.list')).toHaveLength(4);+ expect(calls.filter(c => c.method === 'chat.history')).toHaveLength(48);+ const overlap = maxOverlap(calls, 'chat.history');+ expect(overlap).toBeGreaterThan(1);+ expect(overlap).toBeLessThanOrEqual(4);+ for (const c of calls) expect(JSON.stringify(c.params)).not.toContain(token);+ const recent = sessions.filter(s => s.updatedAt >= SINCE);+ expect(convs.map(c => c.scopeKey)).toEqual(recent.map(s => s.key));+ for (const [i, conv] of convs.entries()) {+ expect(conv.conversationKey).toBe(recent[i]!.sessionId);+ expect(conv.session.messages.map(m => m.text)).toEqual(expected(recent[i]!));+ const stamps = conv.session.messages.map(m => m.timestamp);+ expect(stamps).toEqual([...stamps].sort());+ }+ }, 120_000);++ test('a Gateway from before 2026.10 that refuses maxBytes is read in full without it after the first refusals', async () => {+ const sessions: StubSession[] = Array.from({ length: 6 }, (_, i) => ({ key: 'agent:main:example-old-' + i, sessionId: 'old-' + i, messages: 500, updatedAt: NOW - 1000 * i }));+ const stub = gatewayStub({ dir, sessions, delayMs: 20, rejectMaxBytes: true, name: 'openclaw-old' });+ const convs = await reader(stub.bin).list({});+ expect(convs.map(c => c.session.messages.map(m => m.text))).toEqual(sessions.map(expected));+ const history = stub.calls().filter(c => c.method === 'chat.history');+ const refused = history.filter(c => c.params.maxBytes !== undefined);+ expect(refused.length).toBeGreaterThan(0);+ expect(refused.length).toBeLessThanOrEqual(4);+ expect(history.length - refused.length).toBe(6 * 3);+ }, 60_000);++ test('one failed session read fails the whole read, and no read is still running when it does', async () => {+ const sessions: StubSession[] = Array.from({ length: 12 }, (_, i) => ({ key: 'agent:main:example-fail-' + i, sessionId: 'fail-' + i, messages: 30, updatedAt: NOW - 1000 * i }));+ const stub = gatewayStub({ dir, sessions, delayMs: 150, failKey: 'agent:main:example-fail-2', name: 'openclaw-fail' });+ const err = await reader(stub.bin).list({}).then(() => null, (e: { code?: string }) => e);+ expect(err?.code).toBe('archive_gateway_auth');+ const settled = stub.calls().length;+ await new Promise(r => setTimeout(r, 600));+ expect(stub.calls().length).toBe(settled);+ expect(stub.calls().filter(c => c.method === 'chat.history').length).toBeLessThan(12);+ }, 60_000);+});