Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Conversation archive reads OpenClaw in far fewer process starts, and archive status reads no session (#9482)

Garry Tanaf51e19ea261
diff --git a/changelog.d/capy-conversation-archive-reads.md b/changelog.d/capy-conversation-archive-reads.md
new file mode 100644
index 0000000000..5ac9ebc881
--- /dev/null
+++ b/changelog.d/capy-conversation-archive-reads.md
@@ -0,0 +1,8 @@
+**The conversation archive reads OpenClaw's Gateway in far fewer process starts, and `gbrain transcripts archive status` no longer reads any session.**
+
+### Changed
+- The OpenClaw Gateway reader asks `chat.history` for pages of up to 1,000 messages within a 4 MiB budget, and reads up to 4 sessions at once. Each `openclaw gateway call` is its own process, so on a test install with 350 sessions a read went from 162 process starts and 164 s to 52 starts and 16 s. A Gateway older than 2026.10 that refuses the byte budget is asked again without it. Pages of one session stay in order, conversations keep their `sessions.list` order, and a failed session still fails the whole read before any day is written.
+- `gbrain transcripts archive status` and `gbrain doctor --only transcript_archive_freshness` report what the last archive run read for each reader (conversations, newest session day, time and any error) instead of running the readers. They start no `openclaw` process and parse no session file. A reader no run has read yet says so. Each real run records this in `transcripts.archive.last_read.<source>`; a dry run records nothing.
+
+### Fixed
+- Doctor warns when no archive run has read a reader successfully within `transcripts.archive.stale_days`, for example when maintenance passes keep stopping at their deadline before that reader, and suggests `gbrain transcripts archive run`.
diff --git a/docs/architecture/key-files/commands-2-continued.md b/docs/architecture/key-files/commands-2-continued.md
index 2da6482109..bcc5a66fdc 100644
--- a/docs/architecture/key-files/commands-2-continued.md
+++ b/docs/architecture/key-files/commands-2-continued.md
@@ -44,5 +44,5 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/commands/doctor/checks/purge-parks.ts` — `purge_parked_pages` (#6741 C): names parked purges with their last error (only this version's parks, `isParked`); fix `gbrain pages purge-deleted --retry-parked --yes` (preview `--dry-run --json`), consent destructive, ask_user. Also `derived_origin_parked` (GST-2689, `checkDerivedOriginParked`): names each origin `persistence/derived-origin-hold.ts` parked with the page that blocks it and the reason (`details.origins`); fix the read-only `gbrain repair visibility --source <id> --slug <blocking page>` preview. An origin whose owner purge was held (`held_request_id`) also gets `finish_argv` (`gbrain delete <slug> --purge --source-id <id>`, a new request), named in the message, which finishes it at once.
- `src/commands/doctor/checks/unlinked-facts.ts` — `unlinked_facts` (#5836), registered before `search_mode` and mirrored in `doctorReportRemote`: per the doctor's source scope (`--source`, else every source; the caller's grant remotely), the unlinked share of the newest 100,000 active facts per source (`idx_facts_since` LATERAL walk), the 7-day unlinked share of new facts, `no_subject` judgments and the 7-day `linked` tiers from `fact_relink_attempts`, fence-owned unlinked rows (row_num or source_markdown_slug set, conversation rows excluded), the conversation extractor's unlinked derived rows (`cli:extract-conversation-facts*`, #6764: count, origin `conversation`, and `details.conversation` naming the source with the most and its pages, most rows first, up to `CONVERSATION_FIX_SLUGS` = 50), origins and 7-day write-time inferred links by tier (`entity inferred from <tier>` context, bare or after ` — `). Warns when over 25% of at least 20 new facts are unlinked, facts judged `no_subject` and conversation rows memoized `conversation_unresolved` excluded (`new_7d.conversation_unresolved`); the fix is `gbrain facts relink --dry-run`; when conversation rows are unlinked the envelope `fix` (`details.conversation.fix`) is, for rows the free conversation relink has not tried, `gbrain facts relink --conversation --source <id> --slugs <csv>` (preview `--dry-run`, consent none, actor agent, host operator remotely; test `test/conversation-relink-6764.test.ts`) and, once only rows it memoized `conversation_unresolved` remain (a memo that a page created later re-arms unless the reason is memoized, `core/facts/conversation-memo.ts`), a bounded re-extraction of those pages not yet re-extracted on their current content (`gbrain extract-conversation-facts --source-id <id> --slugs <csv> --force --max-cost-usd <N>`, input `N` the approved USD, preview `--dry-run`, consent paid + egress, ask_user; test `test/conversation-facts-unlinked-6764.test.ts`), both verified by `gbrain doctor --only unlinked_facts --json`, because plain relink never examines derived rows; the remote report asks the brain host operator to run it. When only rows on pages already re-extracted on their current content remain, `details.conversation.fix` is `manual`: no envelope `fix`, and the message names linking by hand with `--entity` (test `test/conversation-memo-6764.test.ts`). Without `fact_relink_attempts` the relink figures are null and nothing is excluded. Non-user facts are skipped (#7270). Tests: `test/doctor-unlinked-facts.test.ts`, `test/relink-attribution-7270.test.ts`.
-- `src/commands/doctor/checks/transcript-archive.ts` — `transcript_archive_freshness` (GST-2768, brain category): off is ok and reads nothing; on, it reads `archiveStatus` and `transcripts.archive.last_maintenance` and warns on a harness more than `transcripts.archive.stale_days` (2) behind its newest session day (fix `gbrain transcripts archive run`, or `backfill --from <day>` past one run window), on no maintenance pass in that long (fix `gbrain autopilot --install`, `persistent_install` consent), on held days (with codes), reader failures and pass errors; no readable sessions is info. Pinned by `test/doctor-transcript-archive-freshness.test.ts`.
+- `src/commands/doctor/checks/transcript-archive.ts` — `transcript_archive_freshness` (GST-2768, brain category): off is ok and reads nothing; on, reads `archiveStatus` (no reader run) and `transcripts.archive.last_maintenance` and warns on a harness `transcripts.archive.stale_days` (2) behind the newest session day seen or a reader unread that long (fix `gbrain transcripts archive run`, or `backfill --from <day>` past one run window), on no maintenance pass that long (fix `gbrain autopilot --install`, `persistent_install` consent), on held days, reader and pass errors; no readable sessions is info. Pinned by `test/doctor-transcript-archive-freshness.test.ts`.
- `src/commands/doctor/checks/unattended-maintenance.ts` — `unattended_maintenance` (GBRA-438): `readUpkeepStatus` as a check; off and autopilot-owned are ok, a brain under 48h with no run is info, a run older than 48h, a step failing 3 runs in a row or an item serve parked (`details.parked`) warns with `gbrain maintain --unattended` (`consent: []` unless an embedding key and a non-zero cap make it paid); on an overdue brain `details.alternative_fix` renders `gbrain maintain --schedule` as the second fix.
diff --git a/docs/guides/conversation-archive.md b/docs/guides/conversation-archive.md
index 56fb53bb20..0c6cd3a96e 100644
--- a/docs/guides/conversation-archive.md
+++ b/docs/guides/conversation-archive.md
@@ -11,7 +11,7 @@ gbrain transcripts archive on # turn it on (a shared br
gbrain transcripts archive run --dry-run # preview the last two local days
gbrain transcripts archive run # write them
gbrain transcripts archive backfill --from 2026-09-01 # older history, newest days first
-gbrain transcripts archive status # readers, timezone, archived and held days
+gbrain transcripts archive status # readers as of the last run, timezone, archived and held days
gbrain doctor --only transcript_archive_freshness # is it keeping up?
```
@@ -71,14 +71,22 @@ false` turns both off.
`gbrain doctor --only transcript_archive_freshness` (brain category) warns when:
- a harness's newest archived day is more than `transcripts.archive.stale_days` (default 2) behind the newest session
- day its reader sees. The fix is `gbrain transcripts archive run`, or
+ day the last run read. The fix is `gbrain transcripts archive run`, or
`gbrain transcripts archive backfill --from <first missing day>` when the gap is wider than one run;
+- no archive run has read a reader successfully in that long, for example when maintenance passes keep stopping at
+ their deadline before that reader. The fix is `gbrain transcripts archive run`;
- the archive is on but no maintenance pass has run it in that long. The fix is `gbrain autopilot --install`
(a persistent service, so it asks first);
- days are held (each is listed with its code, such as `archive_owner_unset` or `archive_redaction_failed`), a reader
- failed, or the last pass recorded an error.
+ failed on the last run, or the last pass recorded an error.
-Off is ok with no detail, and a harness with no readable sessions is informational.
+Off is ok with no detail, and a harness with no readable sessions, or one no run has read yet, is informational.
+
+Neither this check nor `gbrain transcripts archive status` reads a session or calls the OpenClaw Gateway, so both
+answer in about a second however many sessions there are. They read the archive's ledger and config, and what each
+run (`run`, `backfill` or the maintenance pass, never a `--dry-run`) recorded about every reader: how many
+conversations it read, the newest session day any run has seen, and the reader's error if the last run failed. Status
+shows those as of that run, with its time; a reader no run has read yet says so.
Every command takes `--json` and `--source <id>` (the archive source, `conversations` by default).
@@ -180,6 +188,12 @@ Current OpenClaw keeps live sessions in a database owned by its Gateway. gbrain
`archive_gateway_auth` and `archive_gateway_failed`; a failed read leaves the archive ledger unchanged, so the next
run retries. Checked against OpenClaw 2026.10.9 and 2026.11.3, which list sessions and return history the same way.
+Each `openclaw gateway call` is a new process, and starting one takes far longer than the Gateway takes to answer,
+so gbrain asks for the largest history page the Gateway gives (1,000 messages within 4 MiB; OpenClaw before 2026.11
+caps a page at 512 KiB, and one before 2026.10 refuses the byte budget, so gbrain asks again without it) and reads up
+to 4 sessions at once. The messages of each session, and the order of the conversations, are the same as one read at
+a time; if any session cannot be read, the whole Gateway read fails once the reads already running have ended.
+
Session files OpenClaw keeps on disk are read too, even when the Gateway is down: `<id>.jsonl.reset.<stamp>`, the
zstd-compressed `<id>.jsonl.deleted.<stamp>.<hex>.zst` and the legacy compact backup `<id>.jsonl.bak.<stamp>`.
Several files of one session merge into one conversation (`gbrain transcripts ingest` reads only live files). Migration copies (`*.migrated`, `*.migrated.<n>`,
diff --git a/src/commands/doctor/checks/transcript-archive.ts b/src/commands/doctor/checks/transcript-archive.ts
index 298434bc42..d1e150a8a4 100644
--- a/src/commands/doctor/checks/transcript-archive.ts
+++ b/src/commands/doctor/checks/transcript-archive.ts
@@ -1,16 +1,18 @@
/**
* transcript_archive_freshness doctor check (GST-2768 PR 3): is the conversation archive keeping up?
*
- * Off (switched off, or a shared brain that is only offered it) is one quiet ok row and reads no session. On, it
- * reads archiveStatus (the readers this machine has and the archive ledger) and the maintenance record
- * (transcripts.archive.last_maintenance), and warns when:
+ * Off (switched off, or a shared brain that is only offered it) is one quiet ok row. On, it reads archiveStatus (the
+ * readers this machine has, the archive ledger and what the last run read; no reader runs, so no session is read and
+ * no Gateway is called) and the maintenance record (transcripts.archive.last_maintenance), and warns when:
* - a harness's newest archived day is more than transcripts.archive.stale_days (default 2) behind the newest
- * session day its reader sees: fix gbrain transcripts archive run, or backfill --from <first missing day> when
+ * session day the last run read: fix gbrain transcripts archive run, or backfill --from <first missing day> when
* that day is older than one run's window;
+ * - no run has read a reader successfully in stale_days (a maintenance pass that keeps stopping at its deadline
+ * before that reader): fix gbrain transcripts archive run;
* - no maintenance pass has run the archive in stale_days (or ever, on a brain older than that): fix
* gbrain autopilot --install (persistent, so ask the user first);
* - days are held (listed with their codes), a reader failed, or the last pass recorded an error.
- * A harness with no readable sessions is informational.
+ * A harness with no readable sessions, or one no run has read yet, is informational.
*/
import type { Check } from '../../doctor.ts';
import type { BrainEngine } from '../../../core/engine.ts';
@@ -54,6 +56,10 @@ export async function checkTranscriptArchiveFreshness(
let fix: Action | undefined;
const behind: Array<{ harness: string; newest_session_day: string; newest_archived_day: string | null; days_behind: number }> = [];
for (const h of status.harnesses) {
+ if (h.read_at && nowMs - Date.parse(h.read_at) > staleDays * DAY_MS) {
+ problems.push(h.harness + ': no archive run has read ' + h.reader + ' successfully since ' + h.read_at);
+ fix ??= agentFix(['gbrain', 'transcripts', 'archive', 'run'], 'Archives the last ' + (runWindow + 1) + ' local days from every readable harness now.', name);
+ }
if (h.info) continue;
if (!h.newest_session_day) continue;
const gap = h.newest_archived_day ? Math.round((dayMs(h.newest_session_day) - dayMs(h.newest_archived_day)) / DAY_MS) : Infinity;
@@ -94,7 +100,11 @@ export async function checkTranscriptArchiveFreshness(
if (problems.length) return { name, status: 'warn', message: 'Conversation archive: ' + problems.join('; ') + '.', details, ...(fix ? { fix } : {}) };
const readable = status.harnesses.filter(h => !h.info && h.newest_session_day);
const idle = [...new Set(status.harnesses.filter(h => h.info).map(h => h.harness))];
- const idleNote = idle.length ? 'no readable sessions for ' + idle.join(', ') + ' (gbrain transcripts archive status names each reader)' : '';
+ const unread = [...new Set(status.harnesses.filter(h => !h.info && !h.read_at && !h.newest_session_day).map(h => h.harness))];
+ const idleNote = [
+ ...(idle.length ? ['no readable sessions for ' + idle.join(', ') + ' (gbrain transcripts archive status names each reader)'] : []),
+ ...(unread.length ? ['not read by an archive run yet: ' + unread.join(', ')] : []),
+ ].join('; ');
if (!readable.length) return infoCheck(name, 'Conversation archive is on; ' + (idleNote || 'no harness has readable sessions yet') + '.', 'not_applicable', undefined, details);
return { name, status: 'ok', message: 'Conversation archive is up to date (' + readable.map(h => h.harness + ' ' + h.newest_archived_day).join(', ') + ')' + (idleNote ? '; ' + idleNote : '') + '.', details };
} catch (error) {
diff --git a/src/commands/transcripts-archive.ts b/src/commands/transcripts-archive.ts
index 3f9d870251..f2d6215e4a 100644
--- a/src/commands/transcripts-archive.ts
+++ b/src/commands/transcripts-archive.ts
@@ -3,7 +3,7 @@
*
* run archive the last few local days from every readable harness
* backfill --from D [--to D] archive a date range, newest first, capped per run; embedding held for approval
- * status readers, pinned timezone, archived and held days
+ * status readers as of the last run, pinned timezone, archived and held days (reads no session)
* adopt-legacy --dir <folder> list the days an old archive folder covers (--apply records them so backfill skips them)
* on | off turn the archive on or off for this brain
*
@@ -58,10 +58,13 @@ export async function runTranscriptsArchive(engine: BrainEngine, args: string[],
const status = await archive.archiveStatus(engine, { sourceId });
out(status, () => {
console.log('conversation archive: ' + (status.enabled ? 'on' : 'off') + ' (source: ' + status.source_id + ', timezone: ' + (status.timezone ?? status.configured_timezone + ', pinned at the first run') + ')');
- for (const h of status.harnesses) console.log(' ' + h.reader.padEnd(18) + ' ' + (h.info ?? (h.conversations + ' conversation(s), newest session day ' + (h.newest_session_day ?? '-') + ', newest archived day ' + (h.newest_archived_day ?? '-'))));
+ for (const h of status.harnesses) {
+ const read = h.read_at ? h.conversations + ' conversation(s) read by the last ' + (h.read_mode ?? 'run') + ' (' + h.read_at + '), newest session day ' + (h.newest_session_day ?? '-') : 'not read by an archive run yet';
+ console.log(' ' + h.reader.padEnd(18) + ' ' + (h.info ?? read + ', newest archived day ' + (h.newest_archived_day ?? '-')));
+ }
console.log(' archived days: ' + status.archived_days + (status.legacy_days ? ', covered by a legacy archive: ' + status.legacy_days : ''));
for (const h of status.held) console.log(' held: ' + h.harness + ' ' + h.day + ' (' + h.code + ')');
- for (const e of status.errors) console.log(' error: ' + e.reader + ' ' + e.code + ': ' + e.message);
+ for (const e of status.errors) console.log(' error on the last run' + (e.at ? ' (' + e.at + ')' : '') + ': ' + e.reader + ' ' + e.code + ': ' + e.message);
if (status.rescope) console.log(' not kept to the owner yet (' + status.rescope.code + '): ' + status.rescope.message);
});
if (status.errors.length) setCliExitVerdict(1);
diff --git a/src/core/transcripts/archive.ts b/src/core/transcripts/archive.ts
index 115d6361f5..3d43657894 100644
--- a/src/core/transcripts/archive.ts
+++ b/src/core/transcripts/archive.ts
@@ -85,6 +85,8 @@ const AUDIENCE_PREFIX = 'transcripts.archive.audience.';
const RESCOPE_CAPPED_PREFIX = 'transcripts.archive.rescope_capped.';
/** Days whose index page still has to be written (a failed or interrupted index write), per source. */
const INDEX_PENDING_PREFIX = 'transcripts.archive.index_pending.';
+/** What the last run that read each reader found, per source: status and doctor report it without reading again. */
+const LAST_READ_PREFIX = 'transcripts.archive.last_read.';
const INGESTED_VIA = 'cli:transcripts-archive';
const DAY_RE = /^\d{4}-\d{2}-\d{2}$/;
@@ -170,7 +172,27 @@ export interface ArchiveDayOutcome {
message?: string;
}
-export interface ArchiveReaderError { harness: string; reader: string; code: string; message: string; fix?: Action }
+export interface ArchiveReaderError { harness: string; reader: string; code: string; message: string; fix?: Action; /** When the run that met it read (status only). */ at?: string }
+
+/** One reader as the last run that read it found it: the newest message any run saw, and the last run's error until a later read succeeds. */
+interface ArchiveReadRecord {
+ harness: string;
+ at?: string;
+ mode?: 'run' | 'backfill';
+ conversations?: number;
+ newest?: string;
+ error?: { at: string; code: string; message: string; fix?: Action };
+}
+
+async function readArchiveReads(engine: BrainEngine, sourceId: string): Promise<Record<string, ArchiveReadRecord>> {
+ const raw = await engine.getConfig(`${LAST_READ_PREFIX}${sourceId}`);
+ try {
+ const doc = raw ? JSON.parse(raw) as unknown : null;
+ return doc && typeof doc === 'object' && !Array.isArray(doc) ? doc as Record<string, ArchiveReadRecord> : {};
+ } catch {
+ return {};
+ }
+}
export interface ArchiveRunResult {
mode: 'run' | 'backfill';
@@ -515,6 +537,49 @@ interface DayWrite {
pending?: string[];
}
+/** Reads every reader once for a run, and unless it is a dry run records what each one found (transcripts.archive.last_read.<source>) for status and doctor. */
+async function readReaders(engine: BrainEngine, readers: ArchiveReader[], scanOpts: ArchiveScanOpts,
+ ctx: { sourceId: string; mode: 'run' | 'backfill'; dryRun: boolean; at: string; result: ArchiveRunResult; pastDeadline: () => boolean }): Promise<ArchiveScanEntry[]> {
+ const entries: ArchiveScanEntry[] = [];
+ const { result, pastDeadline } = ctx;
+ const reads = new Map<string, { harness: string; conversations: number; newest?: string; error?: ArchiveReaderError }>();
+ for (const reader of readers) {
+ if (entries.length && pastDeadline()) break;
+ if (reader.unavailable) { result.info.push(reader.unavailable); continue; }
+ try {
+ let found = 0;
+ let newest: string | undefined;
+ for await (const entry of scanReader(reader, scanOpts)) {
+ entries.push(entry);
+ found++;
+ if (entry.newest && (!newest || entry.newest > newest)) newest = entry.newest;
+ }
+ reads.set(reader.name, { harness: reader.harness, conversations: found, ...(newest ? { newest } : {}) });
+ if (found === 0) result.info.push('no readable sessions for ' + reader.harness + '; use gbrain connectors or an export with gbrain transcripts ingest');
+ } catch (err) {
+ if (classifyItemFailure(err).scope === 'run') throw err;
+ const e = err as { code?: string; message?: string; fix?: Action };
+ const error = { harness: reader.harness, reader: reader.name, code: e.code ?? 'archive_reader_failed', message: e.message ?? String(err), ...(e.fix ? { fix: e.fix } : {}) };
+ result.errors.push(error);
+ reads.set(reader.name, { harness: reader.harness, conversations: 0, error });
+ }
+ }
+ if (ctx.dryRun || !reads.size) return entries;
+ const { at } = ctx;
+ const record = await readArchiveReads(engine, ctx.sourceId);
+ for (const [name, read] of reads) {
+ const prior = record[name];
+ if (read.error) {
+ record[name] = { ...prior, harness: read.harness, error: { at, code: read.error.code, message: read.error.message, ...(read.error.fix ? { fix: read.error.fix } : {}) } };
+ continue;
+ }
+ const newest = [prior?.newest, read.newest].filter((v): v is string => !!v).sort().pop();
+ record[name] = { harness: read.harness, at, mode: ctx.mode, conversations: read.conversations, ...(newest ? { newest } : {}) };
+ }
+ await engine.setConfig(LAST_READ_PREFIX + ctx.sourceId, JSON.stringify(record));
+ return entries;
+}
+
export async function runTranscriptArchive(engine: BrainEngine, opts: ArchiveRunOpts): Promise<ArchiveRunResult> {
const dryRun = opts.dryRun === true;
const enabled = await archiveEnabled(engine);
@@ -591,20 +656,7 @@ async function archiveSource(engine: BrainEngine, opts: ArchiveRunOpts, ctx: { e
const minMtimeMs = opts.mode === 'backfill' ? Date.parse(from + 'T00:00:00.000Z') - 2 * 86_400_000 : sinceMs;
const dayOf = (ts: string) => localDay(ts, tz.timezone);
const pastDeadline = () => opts.deadlineAtMs !== undefined && Date.now() >= opts.deadlineAtMs && (result.stopped_at_deadline = true);
- const entries: ArchiveScanEntry[] = [];
- for (const reader of readers) {
- if (entries.length && pastDeadline()) break;
- if (reader.unavailable) { result.info.push(reader.unavailable); continue; }
- try {
- let found = 0;
- for await (const entry of scanReader(reader, { sinceMs, minMtimeMs, dayOf })) { entries.push(entry); found++; }
- if (found === 0) result.info.push('no readable sessions for ' + reader.harness + '; use gbrain connectors or an export with gbrain transcripts ingest');
- } catch (err) {
- if (classifyItemFailure(err).scope === 'run') throw err;
- const e = err as { code?: string; message?: string; fix?: Action };
- result.errors.push({ harness: reader.harness, reader: reader.name, code: e.code ?? 'archive_reader_failed', message: e.message ?? String(err), ...(e.fix ? { fix: e.fix } : {}) });
- }
- }
+ const entries = await readReaders(engine, readers, { sinceMs, minMtimeMs, dayOf }, { sourceId, mode: opts.mode, dryRun, at: now.toISOString(), result, pastDeadline });
// A Gateway conversation and the session files of the same session id are one conversation: a reset in place leaves
// the earlier messages only in the .jsonl.reset.* file, so the file messages merge in, each message once.
const filesBySession = new Map<string, ArchiveScanEntry[]>();
@@ -967,15 +1019,25 @@ export interface ArchiveStatus {
source_id: string;
timezone: string | null;
configured_timezone: string;
- harnesses: Array<{ harness: string; reader: string; conversations: number; newest_session_day: string | null; newest_archived_day: string | null; info?: string }>;
+ /**
+ * One row per reader on this machine. conversations and newest_session_day are as of the last run that read it
+ * (read_at, read_mode; null when no run has read it yet): status never reads a reader itself. In run mode a run
+ * reads only recently active conversations, so conversations counts those; newest_session_day is the newest any run saw.
+ */
+ harnesses: Array<{ harness: string; reader: string; conversations: number; newest_session_day: string | null; newest_archived_day: string | null; read_at: string | null; read_mode?: 'run' | 'backfill'; info?: string }>;
archived_days: number;
held: Array<{ harness: string; day: string; code: string }>;
legacy_days: number;
+ /** Readers whose last run failed, with when (at); a later successful read clears one. */
errors: ArchiveReaderError[];
/** The brain became shared but the pages archived while personal could not be kept to the owner yet. */
rescope?: ArchiveRescopeSkip;
}
+/**
+ * The archive's own records: config, ledger and what the last run read. It runs no reader (no Gateway call, no session
+ * file parsed); building the reader list only checks which harness folders exist and whether openclaw is on PATH.
+ */
export async function archiveStatus(engine: BrainEngine, opts: { sourceId?: string; readers?: ArchiveReader[] } = {}): Promise<ArchiveStatus> {
const { sourceId } = await effectiveSource(engine, opts.sourceId);
const pinned = (await engine.getConfig(`${PIN_PREFIX}${sourceId}`))?.trim() || null;
@@ -987,25 +1049,18 @@ export async function archiveStatus(engine: BrainEngine, opts: { sourceId?: stri
held: entries.filter(e => e.held).map(e => ({ harness: e.harness, day: e.day, code: e.held!.code })),
legacy_days: (await legacyCovered(engine, sourceId)).size, errors: [],
};
+ const reads = await readArchiveReads(engine, sourceId);
+ const zone = pinned ?? configured;
for (const reader of opts.readers ?? await defaultArchiveReaders()) {
const archived = entries.filter(e => e.harness === reader.harness && e.hash).map(e => e.day).sort();
- const row = { harness: reader.harness, reader: reader.name, conversations: 0, newest_session_day: null as string | null, newest_archived_day: archived[archived.length - 1] ?? null };
+ const row = { harness: reader.harness, reader: reader.name, conversations: 0, newest_session_day: null as string | null, newest_archived_day: archived[archived.length - 1] ?? null, read_at: null as string | null };
if (reader.unavailable) { out.harnesses.push({ ...row, info: reader.unavailable }); continue; }
- try {
- const zone = pinned ?? configured;
- let conversations = 0;
- let newest: string | undefined;
- for await (const entry of scanReader(reader, { dayOf: ts => localDay(ts, zone) })) {
- conversations++;
- if (entry.newest && (!newest || entry.newest > newest)) newest = entry.newest;
- }
- out.harnesses.push({ ...row, conversations, newest_session_day: newest ? localDay(newest, zone) : null,
- ...(conversations ? {} : { info: 'no readable sessions for ' + reader.harness + '; use gbrain connectors or an export with gbrain transcripts ingest' }) });
- } catch (err) {
- const e = err as { code?: string; message?: string; fix?: Action };
- out.errors.push({ harness: reader.harness, reader: reader.name, code: e.code ?? 'archive_reader_failed', message: e.message ?? String(err), ...(e.fix ? { fix: e.fix } : {}) });
- out.harnesses.push(row);
- }
+ const read = reads[reader.name];
+ if (read?.error) out.errors.push({ harness: reader.harness, reader: reader.name, code: read.error.code, message: read.error.message, ...(read.error.fix ? { fix: read.error.fix } : {}), at: read.error.at });
+ const newest = read?.newest ? localDay(read.newest, zone) : null;
+ if (!read?.at) { out.harnesses.push({ ...row, newest_session_day: newest }); continue; }
+ out.harnesses.push({ ...row, conversations: read.conversations ?? 0, newest_session_day: newest, read_at: read.at, ...(read.mode ? { read_mode: read.mode } : {}),
+ ...(newest ? {} : { info: 'no readable sessions for ' + reader.harness + '; use gbrain connectors or an export with gbrain transcripts ingest' }) });
}
const rescope = await archiveRescopeSkipped(engine, opts.sourceId);
if (rescope) out.rescope = rescope;
diff --git a/src/core/transcripts/openclaw-gateway.ts b/src/core/transcripts/openclaw-gateway.ts
index d85dbfaa1b..b2d7a08e77 100644
--- a/src/core/transcripts/openclaw-gateway.ts
+++ b/src/core/transcripts/openclaw-gateway.ts
@@ -9,10 +9,16 @@
* - sessions.list {limit, offset}: rows with key, sessionId, agentId, chatType (direct | group | room), createdAt,
* updatedAt (epoch ms); paged by hasMore / nextOffset. Archived sessions are not listed, and the call takes no
* parameter that lists them.
- * - chat.history {sessionKey, limit, offset}: messages {role, content (string or [{type: text, text}]),
+ * - chat.history {sessionKey, limit, maxBytes, offset}: messages {role, content (string or [{type: text, text}]),
* timestamp (epoch ms), __openclaw {id, seq}}; pages walk backward from the newest (hasMore / nextOffset), each
* page oldest first. It still reads an archived session by key. A reset in place keeps the sessionId and drops
- * the earlier messages from this read.
+ * the earlier messages from this read. A page holds at most limit messages (up to 1000) within maxBytes (default
+ * 512 KiB; since 2026.11 up to a 6 MiB ceiling, before that never above 512 KiB); a Gateway from before 2026.10
+ * refuses maxBytes as an unexpected property.
+ *
+ * Every call is one openclaw process, and a process start costs far more than the Gateway's answer, so the reader
+ * asks for the largest page the Gateway gives and reads up to HISTORY_CONCURRENCY sessions side by side. Pages of one
+ * session stay in order, and the conversations come back in sessions.list order whichever read finished first.
* - Errors come back as a JSON envelope {ok: false, error: {type, code?, message, details?}}: a closed port is
* gateway_transport_error, a missing credential gateway_credentials_required, a wrong one gateway_request_error
* with details.code AUTH_*.
@@ -22,7 +28,8 @@
* group and chatType direct). Anything else is unknown, which the archive resolves owner-only.
*
* Failures throw archive_gateway_unreachable, archive_gateway_auth or archive_gateway_failed with the Gateway's
- * message redacted; the reader yields nothing before every read finished, so a failed run changes no ledger row.
+ * message redacted; the reader yields nothing before every read finished, and after a failure it starts no new read
+ * and throws once the reads already running have ended, so a failed run changes no ledger row.
*/
import { execFileBounded } from '../bounded-child-exec.ts';
@@ -50,7 +57,9 @@ export interface GatewayConversation {
}
const LIST_PAGE = 100;
-const HISTORY_PAGE = 200;
+const HISTORY_PAGE = 1000;
+const HISTORY_MAX_BYTES = 4 * 1024 * 1024;
+const HISTORY_CONCURRENCY = 4;
const MAX_PAGES = 10_000;
/** Redact and bound text that came from the Gateway before it reaches a message, a log or a receipt. */
@@ -167,11 +176,27 @@ function messageTimestamp(value: unknown): string {
const asRecord = (v: unknown): Record<string, unknown> => (v && typeof v === 'object' ? v as Record<string, unknown> : {});
-async function readHistory(transport: GatewayTransport, sessionKey: string): Promise<{ sessionId?: string; messages: TranscriptMessage[] }> {
+/** Whether this Gateway takes the maxBytes page budget; a Gateway from before 2026.10 refuses it as an unexpected property. */
+interface HistoryBudget { maxBytes: boolean }
+
+async function historyPage(transport: GatewayTransport, budget: HistoryBudget, sessionKey: string, offset: number): Promise<Record<string, unknown>> {
+ const params = { sessionKey, limit: HISTORY_PAGE, ...(offset ? { offset } : {}) };
+ if (!budget.maxBytes) return asRecord(await transport.call('chat.history', params));
+ try {
+ return asRecord(await transport.call('chat.history', { ...params, maxBytes: HISTORY_MAX_BYTES }));
+ } catch (err) {
+ const e = err as { code?: string; message?: string };
+ if (e.code !== 'archive_gateway_failed' || !/unexpected property '?maxBytes/.test(String(e.message))) throw err;
+ budget.maxBytes = false;
+ return asRecord(await transport.call('chat.history', params));
+ }
+}
+
+async function readHistory(transport: GatewayTransport, budget: HistoryBudget, sessionKey: string): Promise<{ sessionId?: string; messages: TranscriptMessage[] }> {
const pages: Array<Record<string, unknown>> = [];
let offset = 0;
for (let i = 0; i < MAX_PAGES; i++) {
- const page = asRecord(await transport.call('chat.history', { sessionKey, limit: HISTORY_PAGE, ...(offset ? { offset } : {}) }));
+ const page = await historyPage(transport, budget, sessionKey, offset);
pages.push(page);
if (page.hasMore !== true || typeof page.nextOffset !== 'number' || page.nextOffset <= offset) break;
offset = page.nextOffset;
@@ -211,13 +236,31 @@ export function openclawGatewayReader(opts: { transport?: GatewayTransport } = {
if (page.hasMore !== true || typeof page.nextOffset !== 'number' || page.nextOffset <= offset) break;
offset = page.nextOffset;
}
- const out: GatewayConversation[] = [];
- for (const row of rows) {
- const key = typeof row.key === 'string' ? row.key : '';
- if (!key) continue;
+ const wanted = rows.filter(row => {
+ if (typeof row.key !== 'string' || !row.key) return false;
const updated = typeof row.updatedAt === 'number' ? row.updatedAt : null;
- if (listOpts.sinceMs !== undefined && updated !== null && updated < listOpts.sinceMs) continue;
- const history = await readHistory(transport, key);
+ return listOpts.sinceMs === undefined || updated === null || updated >= listOpts.sinceMs;
+ });
+ const budget: HistoryBudget = { maxBytes: true };
+ const histories: Array<Awaited<ReturnType<typeof readHistory>>> = new Array(wanted.length);
+ let next = 0;
+ let failure: { error: unknown } | null = null;
+ const worker = async () => {
+ while (!failure && next < wanted.length) {
+ const i = next++;
+ try {
+ histories[i] = await readHistory(transport, budget, String(wanted[i]!.key));
+ } catch (error) {
+ failure ??= { error };
+ }
+ }
+ };
+ await Promise.all(Array.from({ length: Math.min(HISTORY_CONCURRENCY, wanted.length) }, worker));
+ if (failure) throw (failure as { error: unknown }).error;
+ const out: GatewayConversation[] = [];
+ for (const [i, row] of wanted.entries()) {
+ const key = String(row.key);
+ const history = histories[i]!;
if (history.messages.length === 0) continue;
const sessionId = typeof row.sessionId === 'string' ? row.sessionId : history.sessionId ?? key;
out.push({
diff --git a/test/doctor-transcript-archive-freshness.test.ts b/test/doctor-transcript-archive-freshness.test.ts
index 85848112d0..0f088aea2f 100644
--- a/test/doctor-transcript-archive-freshness.test.ts
+++ b/test/doctor-transcript-archive-freshness.test.ts
@@ -1,7 +1,7 @@
/**
* GST-2768 PR 3: the transcript_archive_freshness doctor check. Off is ok with no noise; with the archive on it warns
- * when the newest archived day is more than transcripts.archive.stale_days behind the newest session day a reader
- * sees (fix: archive run, or backfill --from when the gap is wider than one run), when no maintenance pass has run
+ * when the newest archived day is more than transcripts.archive.stale_days behind the newest session day the last run
+ * read (fix: archive run, or backfill --from when the gap is wider than one run); the check itself reads no reader, when no maintenance pass has run
* the archive in that long (fix: gbrain autopilot --install), and lists held days with their codes; a harness with
* no readable sessions is informational. PGLite, synthetic sessions only.
*/
@@ -40,6 +40,16 @@ async function recordPass(atMs: number): Promise<void> {
await engine.setConfig(ARCHIVE_MAINTENANCE_KEY, JSON.stringify({ at: new Date(atMs).toISOString(), runner: 'serve', status: 'ok', written: 0, held: 0, errors: [] }));
}
+/** A run at NOW that read a conversation on `day` but could not read it again to write it, so the day stays unarchived. */
+async function runSawUnarchived(day: string): Promise<void> {
+ const ts = day + 'T08:00:00.000Z';
+ const reader: ArchiveReader = {
+ harness: 'codex', name: 'codex',
+ async *scan({ dayOf }) { yield { harness: 'codex', conversationKey: 'codex-new', sessionId: 'codex-new', days: [dayOf(ts)], newest: ts, load: async () => null }; },
+ };
+ await runTranscriptArchive(engine, { mode: 'run', now: NOW, readers: [reader] });
+}
+
async function archiveDay(day: string): Promise<void> {
await runTranscriptArchive(engine, { mode: 'run', now: new Date(day + 'T12:00:00Z'), readers: [staticReader('codex', [said('codex-old-' + day, day + 'T08:00:00.000Z')])] });
}
@@ -62,6 +72,7 @@ describe('transcript_archive_freshness', () => {
test('a 3-day gap warns with the run fix', async () => {
await archiveDay('2026-11-07');
+ await runSawUnarchived('2026-11-10');
await recordPass(NOW.getTime() - 60_000);
const readers = [staticReader('codex', [said('codex-old-2026-11-07', '2026-11-07T08:00:00.000Z'), said('codex-new', '2026-11-10T08:00:00.000Z')])];
const check = await checkTranscriptArchiveFreshness(engine, { readers, nowMs: NOW.getTime() });
@@ -74,6 +85,7 @@ describe('transcript_archive_freshness', () => {
test('a gap wider than one run names the backfill with --from', async () => {
await archiveDay('2026-11-01');
+ await runSawUnarchived('2026-11-09');
await recordPass(NOW.getTime() - 60_000);
const readers = [staticReader('codex', [said('codex-old-2026-11-01', '2026-11-01T08:00:00.000Z'), said('codex-new', '2026-11-09T08:00:00.000Z')])];
const check = await checkTranscriptArchiveFreshness(engine, { readers, nowMs: NOW.getTime() });
@@ -89,6 +101,16 @@ describe('transcript_archive_freshness', () => {
expect(check.status).toBe('ok');
}, 60_000);
+ test('a reader no run has read in stale_days warns even when its archive matches the last read', async () => {
+ await archiveDay('2026-11-01');
+ await recordPass(NOW.getTime() - 60_000);
+ const readers = [staticReader('codex', [said('codex-old-2026-11-01', '2026-11-01T08:00:00.000Z')])];
+ const check = await checkTranscriptArchiveFreshness(engine, { readers, nowMs: NOW.getTime() });
+ expect(check.status).toBe('warn');
+ expect(check.message).toContain('codex: no archive run has read codex successfully since 2026-11-01T12:00:00.000Z');
+ expect(check.fix?.argv).toEqual(['gbrain', 'transcripts', 'archive', 'run']);
+ }, 60_000);
+
test('a harness with no readable sessions is informational', async () => {
await recordPass(NOW.getTime() - 60_000);
const check = await checkTranscriptArchiveFreshness(engine, { readers: [staticReader('grok', []), { harness: 'hermes', name: 'hermes', unavailable: 'no Hermes sessions on this machine' }], nowMs: NOW.getTime() });
diff --git a/test/helpers/config-reader-registry.ts b/test/helpers/config-reader-registry.ts
index 6b317a06f4..f07f763afb 100644
--- a/test/helpers/config-reader-registry.ts
+++ b/test/helpers/config-reader-registry.ts
@@ -30,6 +30,7 @@ export const INTERNAL_CONFIG_STATE: readonly InternalConfigState[] = [
{ key: 'transcripts.archive.rescope_capped.', prefix: true, reason: 'the number of archive folders when the owner-only re-scope passed the per-source rule limit and kept the whole conversations/ folder to the owner instead (transcripts/archive.ts); status and doctor report it' },
{ key: 'transcripts.archive.last_maintenance', reason: 'the conversation archive\'s maintenance-pass record (transcripts/archive-maintenance.ts); doctor transcript_archive_freshness reads it, so a hand edit would misreport the archive' },
{ key: 'transcripts.archive.backfill_state', reason: 'whether the maintenance pass has finished backfilling older conversation days (transcripts/archive-maintenance.ts); a hand edit would stop or restart the backfill' },
+ { key: 'transcripts.archive.last_read.', prefix: true, reason: 'what the last conversation archive run read from each reader per source (transcripts/archive.ts): status and doctor report it as of that run instead of reading every session again' },
{ key: 'transcripts.archive.index_pending.', prefix: true, reason: 'days whose conversation archive day index still has to be written (transcripts/archive.ts); the next run retries them' },
{ key: 'maintenance.last_run', reason: 'the free-upkeep run record (steps, cursor, parked items, next due time) that maintenance/run.ts writes after each gbrain maintain --unattended run; maintain --status and doctor read it, so a hand edit would misreport upkeep' },
{ key: 'maintenance.embed_spend', reason: 'the per-day dollar total the capped upkeep embed step has spent (maintenance/run.ts); the daily cap reads it, so a hand edit would let upkeep spend past maintenance.embed_daily_usd' },
diff --git a/test/helpers/openclaw-gateway-stub.ts b/test/helpers/openclaw-gateway-stub.ts
new file mode 100644
index 0000000000..130c4ddc7d
--- /dev/null
+++ b/test/helpers/openclaw-gateway-stub.ts
@@ -0,0 +1,113 @@
+/**
+ * A stub `openclaw` executable for the Gateway reader suites: it waits a configurable start-up delay (standing in for
+ * the CLI's own process start), answers `gateway call sessions.list | chat.history` from a synthetic session store the
+ * way OpenClaw 2026.11.3 pages them (chat.history: limit up to 1000, default 200; a byte budget of maxBytes, default
+ * 512 KiB, hard ceiling 6 MiB; pages walk backward from the newest, each page oldest first), and appends one line per
+ * invocation (method, params, start and end time) to a log. Every value is synthetic.
+ */
+import { chmodSync, readFileSync, writeFileSync } from 'node:fs';
+import { join } from 'node:path';
+
+export interface StubSession { key: string; sessionId: string; messages: number; updatedAt: number; chatType?: string }
+export interface StubCall { method: string; params: Record<string, unknown>; start: number; end: number; pid: number }
+
+export interface GatewayStubOpts {
+ dir: string;
+ sessions: StubSession[];
+ /** Milliseconds the stub waits before it answers, per invocation. */
+ delayMs: number;
+ /** Extra Gateway time per chat.history page, drawn uniformly from [min, max] ms. */
+ answerMs?: [number, number];
+ /** Bytes each message counts against the page budget. */
+ messageBytes?: number;
+ /** A Gateway from before 2026.10, whose closed chat.history schema refuses maxBytes. */
+ rejectMaxBytes?: boolean;
+ /** chat.history on this session key answers with an auth error envelope. */
+ failKey?: string;
+ /** Name of the executable file (default openclaw). */
+ name?: string;
+}
+
+export function gatewayStub(opts: GatewayStubOpts): { bin: string; calls: () => StubCall[] } {
+ const store = join(opts.dir, (opts.name ?? 'openclaw') + '-store.json');
+ const log = join(opts.dir, (opts.name ?? 'openclaw') + '-calls.jsonl');
+ writeFileSync(store, JSON.stringify({ ...opts, dir: undefined }));
+ writeFileSync(log, '');
+ const bin = join(opts.dir, opts.name ?? 'openclaw');
+ writeFileSync(bin, `#!${process.execPath}
+const fs = require('node:fs');
+const start = Date.now();
+const cfg = JSON.parse(fs.readFileSync(${JSON.stringify(store)}, 'utf8'));
+const argv = process.argv.slice(2);
+const method = argv[2];
+const params = JSON.parse(argv[argv.indexOf('--params') + 1] || '{}');
+const sleep = ms => new Promise(r => setTimeout(r, ms));
+const done = (doc, code) => {
+ fs.appendFileSync(${JSON.stringify(log)}, JSON.stringify({ method, params, start, end: Date.now(), pid: process.pid }) + '\\n');
+ process.stdout.write(JSON.stringify(doc) + '\\n', () => process.exit(code));
+};
+(async () => {
+ await sleep(cfg.delayMs);
+ if (method === 'sessions.list') {
+ const limit = params.limit ?? 50;
+ const offset = params.offset ?? 0;
+ const rows = cfg.sessions.slice(offset, offset + limit).map(s => ({ key: s.key, sessionId: s.sessionId, chatType: s.chatType ?? 'direct', updatedAt: s.updatedAt }));
+ const next = offset + rows.length;
+ return done({ sessions: rows, hasMore: next < cfg.sessions.length, ...(next < cfg.sessions.length ? { nextOffset: next } : {}) }, 0);
+ }
+ if (method !== 'chat.history') return done({ ok: false, error: { type: 'gateway_request_error', code: 'INVALID_REQUEST', message: 'unknown method ' + method } }, 1);
+ if (cfg.rejectMaxBytes && params.maxBytes !== undefined) {
+ return done({ ok: false, error: { type: 'gateway_request_error', code: 'INVALID_REQUEST', message: "invalid chat.history params: at root: unexpected property 'maxBytes'" } }, 1);
+ }
+ if (cfg.failKey && params.sessionKey === cfg.failKey) {
+ return done({ ok: false, error: { type: 'gateway_request_error', code: 'UNAUTHORIZED', message: 'unauthorized', details: { code: 'AUTH_TOKEN_MISMATCH' } } }, 1);
+ }
+ const [lo, hi] = cfg.answerMs ?? [0, 0];
+ if (hi > 0) await sleep(lo + Math.random() * (hi - lo));
+ const s = cfg.sessions.find(x => x.key === params.sessionKey);
+ const total = s ? s.messages : 0;
+ const budget = Math.min(params.maxBytes ?? 512 * 1024, 6 * 1024 * 1024);
+ const size = Math.max(1, Math.min(params.limit ?? 200, 1000, Math.floor(budget / (cfg.messageBytes ?? 2150))));
+ const offset = params.offset ?? 0;
+ const endIdx = Math.max(0, total - offset);
+ const startIdx = Math.max(0, endIdx - size);
+ const messages = [];
+ for (let i = startIdx; i < endIdx; i++) {
+ messages.push({ role: i % 2 ? 'assistant' : 'user', content: 'synthetic message ' + i + ' in ' + s.key, timestamp: s.updatedAt - (total - i) * 1000, __openclaw: { id: s.sessionId + ':' + i, seq: i + 1 } });
+ }
+ const next = offset + messages.length;
+ done({ sessionKey: params.sessionKey, sessionId: s ? s.sessionId : undefined, messages, hasMore: startIdx > 0, ...(startIdx > 0 ? { nextOffset: next } : {}) }, 0);
+})();
+`);
+ chmodSync(bin, 0o755);
+ return {
+ bin,
+ calls: () => readFileSync(log, 'utf8').trim().split('\n').filter(Boolean).map(l => JSON.parse(l) as StubCall),
+ };
+}
+
+/** The most stub invocations of one method that ran at the same time. */
+export function maxOverlap(calls: StubCall[], method: string): number {
+ const edges = calls.filter(c => c.method === method).flatMap(c => [[c.start, 1], [c.end, -1]] as Array<[number, number]>)
+ .sort((a, b) => a[0] - b[0] || a[1] - b[1]);
+ let now = 0;
+ let max = 0;
+ for (const [, d] of edges) { now += d; max = Math.max(max, now); }
+ return max;
+}
+
+/** The large-install shape the reader is tuned for: 350 sessions, 40 active recently, 158 history pages at 200 a page. */
+export function largeInstall(nowMs: number): StubSession[] {
+ const sessions: StubSession[] = [];
+ for (let i = 0; i < 350; i++) {
+ const recent = i % 8 === 0 && sessions.filter(s => s.updatedAt >= nowMs - 86_400_000).length < 40;
+ const n = sessions.filter(s => s.updatedAt >= nowMs - 86_400_000).length;
+ sessions.push({
+ key: 'agent:main:example-session-' + i,
+ sessionId: '00000000-0000-4000-8000-' + String(i).padStart(12, '0'),
+ messages: recent ? (n < 2 ? 4300 : 500) : 20,
+ updatedAt: recent ? nowMs - 60_000 * (i + 1) : nowMs - 30 * 86_400_000,
+ });
+ }
+ return sessions;
+}
diff --git a/test/helpers/transcript-archive.ts b/test/helpers/transcript-archive.ts
index 90fbac8c22..4abd49d894 100644
--- a/test/helpers/transcript-archive.ts
+++ b/test/helpers/transcript-archive.ts
@@ -37,6 +37,7 @@ export async function archivedPages(engine: BrainEngine, sourceId = 'conversatio
}));
}
+/** Every archive config row except transcripts.archive.last_read.*, which records what each run read (and when), not what it archived. */
export async function ledgerRows(engine: BrainEngine): Promise<Array<{ key: string; value: string }>> {
- return engine.executeRaw<{ key: string; value: string }>("SELECT key, value FROM config WHERE key LIKE 'transcripts.archive.%' ORDER BY key");
+ return engine.executeRaw<{ key: string; value: string }>("SELECT key, value FROM config WHERE key LIKE 'transcripts.archive.%' AND key NOT LIKE 'transcripts.archive.last\\_read.%' ORDER BY key");
}
diff --git a/test/transcripts-archive-status-reads.test.ts b/test/transcripts-archive-status-reads.test.ts
new file mode 100644
index 0000000000..4df1d110eb
--- /dev/null
+++ b/test/transcripts-archive-status-reads.test.ts
@@ -0,0 +1,102 @@
+/**
+ * gbrain transcripts archive status (and the doctor check built on it) reads the archive's own records only: the
+ * ledger, the config and what the last run recorded about each reader. It never runs a reader, so a slow Gateway or a
+ * large session folder cannot slow it down. PGLite, synthetic sessions only.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { ARCHIVE_TIMEZONE_KEY, archiveStatus, runTranscriptArchive, setArchiveEnabled, type ArchiveReader } from '../src/core/transcripts/archive.ts';
+import { cliGatewayTransport, openclawGatewayReader } from '../src/core/transcripts/openclaw-gateway.ts';
+import { conversation } from './helpers/transcript-archive.ts';
+import { gatewayStub } from './helpers/openclaw-gateway-stub.ts';
+
+const dir = mkdtempSync(join(tmpdir(), 'archive-status-'));
+const NOW = new Date('2026-11-10T09:00:00Z');
+let engine: PGLiteEngine;
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+});
+
+afterAll(async () => {
+ await engine.disconnect();
+ rmSync(dir, { recursive: true, force: true });
+});
+
+beforeEach(async () => {
+ await resetPgliteState(engine);
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await engine.setConfig('brain.audience', 'personal');
+ await setArchiveEnabled(engine, true);
+});
+
+function counting(harness: 'codex' | 'grok', fail?: Error): ArchiveReader & { reads: number } {
+ const reader = {
+ harness, name: harness, reads: 0,
+ async list() {
+ reader.reads++;
+ if (fail) throw fail;
+ return [conversation(harness, harness + '-a', [['user', '2026-11-09T08:00:00.000Z', 'first'], ['assistant', '2026-11-10T08:00:00.000Z', 'second']])];
+ },
+ };
+ return reader;
+}
+
+describe('archive status reads no reader', () => {
+ test('it reports what the last run read, as of that run, without reading again', async () => {
+ const codex = counting('codex');
+ await runTranscriptArchive(engine, { mode: 'run', readers: [codex], now: NOW });
+ expect(codex.reads).toBe(1);
+ const status = await archiveStatus(engine, { readers: [codex] });
+ expect(codex.reads).toBe(1);
+ expect(status.harnesses).toEqual([expect.objectContaining({
+ harness: 'codex', reader: 'codex', conversations: 1, newest_session_day: '2026-11-10', newest_archived_day: '2026-11-10', read_mode: 'run',
+ })]);
+ expect(Date.parse(status.harnesses[0]!.read_at!)).toBeGreaterThan(0);
+ expect(status.errors).toEqual([]);
+ }, 60_000);
+
+ test('a reader no run has read yet is listed as not read, and still not read', async () => {
+ const grok = counting('grok');
+ const status = await archiveStatus(engine, { readers: [grok, { harness: 'hermes', name: 'hermes', unavailable: 'no Hermes sessions on this machine' }] });
+ expect(grok.reads).toBe(0);
+ expect(status.harnesses.map(h => [h.reader, h.read_at, h.conversations, h.newest_session_day, h.info ?? null])).toEqual([
+ ['grok', null, 0, null, null],
+ ['hermes', null, 0, null, 'no Hermes sessions on this machine'],
+ ]);
+ });
+
+ test('a reader that failed on the last run is reported with its code and when it failed, keeping the newest day it read before', async () => {
+ await runTranscriptArchive(engine, { mode: 'run', readers: [counting('codex')], now: NOW });
+ const failing = counting('codex', Object.assign(new Error('synthetic gateway refusal'), { code: 'archive_gateway_auth' }));
+ await runTranscriptArchive(engine, { mode: 'run', readers: [failing], now: NOW });
+ const status = await archiveStatus(engine, { readers: [failing] });
+ expect(failing.reads).toBe(1);
+ expect(status.errors).toEqual([expect.objectContaining({ reader: 'codex', code: 'archive_gateway_auth', message: 'synthetic gateway refusal' })]);
+ expect(Date.parse(status.errors[0]!.at!)).toBeGreaterThan(0);
+ expect(status.harnesses[0]).toMatchObject({ newest_session_day: '2026-11-10' });
+ }, 60_000);
+
+ test('a dry run records nothing about its reads', async () => {
+ await runTranscriptArchive(engine, { mode: 'run', readers: [counting('codex')], now: NOW, dryRun: true });
+ const status = await archiveStatus(engine, { readers: [counting('codex')] });
+ expect(status.harnesses[0]).toMatchObject({ read_at: null, conversations: 0 });
+ }, 60_000);
+
+ test('status with the OpenClaw Gateway reader starts no openclaw process', async () => {
+ const stub = gatewayStub({ dir, sessions: [{ key: 'agent:main:example-status', sessionId: 'status-1', messages: 4, updatedAt: NOW.getTime() }], delayMs: 50 });
+ const gateway = openclawGatewayReader({ transport: cliGatewayTransport({ bin: stub.bin, timeoutMs: 10_000 }) });
+ await runTranscriptArchive(engine, { mode: 'run', readers: [gateway], now: NOW });
+ const started = stub.calls().length;
+ expect(started).toBeGreaterThan(0);
+ const status = await archiveStatus(engine, { readers: [gateway] });
+ expect(stub.calls().length).toBe(started);
+ expect(status.harnesses[0]).toMatchObject({ harness: 'openclaw', reader: 'openclaw-gateway', conversations: 1, newest_session_day: '2026-11-10' });
+ }, 60_000);
+});
diff --git a/test/transcripts-openclaw-gateway-argv.test.ts b/test/transcripts-openclaw-gateway-argv.test.ts
index 371997c771..31a493b7ef 100644
--- a/test/transcripts-openclaw-gateway-argv.test.ts
+++ b/test/transcripts-openclaw-gateway-argv.test.ts
@@ -53,7 +53,7 @@ describe('openclaw gateway call child process', () => {
expect(argv.slice(6)).toEqual(['--timeout', '5000']);
}
const historyKeys = calls.filter(c => c.argv[2] === 'chat.history').map(c => (JSON.parse(c.argv[5]!) as { sessionKey: string }).sessionKey);
- expect(historyKeys).toEqual(HOSTILE);
+ expect([...historyKeys].sort()).toEqual([...HOSTILE].sort());
expect(() => readFileSync(join(dir, 'pwned'))).toThrow();
}, 30_000);
diff --git a/test/transcripts-openclaw-gateway-pages.test.ts b/test/transcripts-openclaw-gateway-pages.test.ts
new file mode 100644
index 0000000000..44af295be9
--- /dev/null
+++ b/test/transcripts-openclaw-gateway-pages.test.ts
@@ -0,0 +1,66 @@
+/**
+ * The OpenClaw Gateway reader's process count and concurrency, against a stub `openclaw` that waits a start-up delay
+ * per invocation and pages chat.history the way OpenClaw 2026.11.3 does. The large-install shape is 350 sessions, 40
+ * of them active, whose history is 158 pages at the old 200-message page. Every value is synthetic.
+ */
+import { afterAll, describe, expect, test } from 'bun:test';
+import { mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { cliGatewayTransport, openclawGatewayReader } from '../src/core/transcripts/openclaw-gateway.ts';
+import { gatewayStub, largeInstall, maxOverlap, type StubSession } from './helpers/openclaw-gateway-stub.ts';
+
+const dir = mkdtempSync(join(tmpdir(), 'openclaw-pages-'));
+afterAll(() => rmSync(dir, { recursive: true, force: true }));
+
+const NOW = Date.parse('2026-11-10T12:00:00Z');
+const SINCE = NOW - 4 * 86_400_000;
+const token = ['tok', 'synthetic', 'pages'].join('-');
+const reader = (bin: string) => openclawGatewayReader({ transport: cliGatewayTransport({ bin, url: 'ws://127.0.0.1:18789', token, timeoutMs: 20_000 }) });
+const expected = (s: StubSession) => Array.from({ length: s.messages }, (_, i) => 'synthetic message ' + i + ' in ' + s.key);
+
+describe('openclaw gateway reader: pages and processes', () => {
+ test('the large-install shape reads in 52 process starts instead of 162, at most 4 side by side, every message in order', async () => {
+ const sessions = largeInstall(NOW);
+ const stub = gatewayStub({ dir, sessions, delayMs: 100, answerMs: [0, 80] });
+ const convs = await reader(stub.bin).list({ sinceMs: SINCE });
+ const calls = stub.calls();
+ expect(calls.filter(c => c.method === 'sessions.list')).toHaveLength(4);
+ expect(calls.filter(c => c.method === 'chat.history')).toHaveLength(48);
+ const overlap = maxOverlap(calls, 'chat.history');
+ expect(overlap).toBeGreaterThan(1);
+ expect(overlap).toBeLessThanOrEqual(4);
+ for (const c of calls) expect(JSON.stringify(c.params)).not.toContain(token);
+ const recent = sessions.filter(s => s.updatedAt >= SINCE);
+ expect(convs.map(c => c.scopeKey)).toEqual(recent.map(s => s.key));
+ for (const [i, conv] of convs.entries()) {
+ expect(conv.conversationKey).toBe(recent[i]!.sessionId);
+ expect(conv.session.messages.map(m => m.text)).toEqual(expected(recent[i]!));
+ const stamps = conv.session.messages.map(m => m.timestamp);
+ expect(stamps).toEqual([...stamps].sort());
+ }
+ }, 120_000);
+
+ test('a Gateway from before 2026.10 that refuses maxBytes is read in full without it after the first refusals', async () => {
+ const sessions: StubSession[] = Array.from({ length: 6 }, (_, i) => ({ key: 'agent:main:example-old-' + i, sessionId: 'old-' + i, messages: 500, updatedAt: NOW - 1000 * i }));
+ const stub = gatewayStub({ dir, sessions, delayMs: 20, rejectMaxBytes: true, name: 'openclaw-old' });
+ const convs = await reader(stub.bin).list({});
+ expect(convs.map(c => c.session.messages.map(m => m.text))).toEqual(sessions.map(expected));
+ const history = stub.calls().filter(c => c.method === 'chat.history');
+ const refused = history.filter(c => c.params.maxBytes !== undefined);
+ expect(refused.length).toBeGreaterThan(0);
+ expect(refused.length).toBeLessThanOrEqual(4);
+ expect(history.length - refused.length).toBe(6 * 3);
+ }, 60_000);
+
+ test('one failed session read fails the whole read, and no read is still running when it does', async () => {
+ const sessions: StubSession[] = Array.from({ length: 12 }, (_, i) => ({ key: 'agent:main:example-fail-' + i, sessionId: 'fail-' + i, messages: 30, updatedAt: NOW - 1000 * i }));
+ const stub = gatewayStub({ dir, sessions, delayMs: 150, failKey: 'agent:main:example-fail-2', name: 'openclaw-fail' });
+ const err = await reader(stub.bin).list({}).then(() => null, (e: { code?: string }) => e);
+ expect(err?.code).toBe('archive_gateway_auth');
+ const settled = stub.calls().length;
+ await new Promise(r => setTimeout(r, 600));
+ expect(stub.calls().length).toBe(settled);
+ expect(stub.calls().filter(c => c.method === 'chat.history').length).toBeLessThan(12);
+ }, 60_000);
+});