Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Add a built-in conversation archive, off by default (GST-2768) (#9432)

Garry Tanecc7a62cfd55
diff --git a/.gitleaksignore b/.gitleaksignore
index d27ccac04f..d2a7efb7d4 100644
--- a/.gitleaksignore
+++ b/.gitleaksignore
@@ -67,3 +67,10 @@ d7a54effbc2e4ab26496bebb0d680ec5b82aaeed:test/storage-manifest.test.ts:generic-a
f60d3d827947be5836e7a1ab530596494f3f4e25:test/storage-manifest.test.ts:generic-api-key:141
f60d3d827947be5836e7a1ab530596494f3f4e25:test/storage-manifest.test.ts:generic-api-key:142
f60d3d827947be5836e7a1ab530596494f3f4e25:test/storage-manifest.test.ts:generic-api-key:143
+# Conversation archive (GST-2768): synthetic Gateway tokens (tok-synthetic-000N) in test/transcripts-openclaw-gateway-argv.test.ts
+# and a synthetic redaction canary constant in test/transcripts-archive-redaction.test.ts read as generic-api-key.
+# History-only fingerprints of test fixtures, not secrets.
+abef0b6d4b3c475ab8a6ab991744562f1f579a4f:test/transcripts-openclaw-gateway-argv.test.ts:generic-api-key:105
+abef0b6d4b3c475ab8a6ab991744562f1f579a4f:test/transcripts-openclaw-gateway-argv.test.ts:generic-api-key:124
+5cdad604afa992a5af330606719f41299d05c29f:test/transcripts-openclaw-gateway-argv.test.ts:generic-api-key:34
+8d727757c32db0479ebc33e48c3ac6cb8a7f023c:test/transcripts-archive-redaction.test.ts:generic-api-key:31
diff --git a/changelog.d/transcript-archive-core.md b/changelog.d/transcript-archive-core.md
new file mode 100644
index 0000000000..11e8474e21
--- /dev/null
+++ b/changelog.d/transcript-archive-core.md
@@ -0,0 +1,14 @@
+**New, off by default: `gbrain transcripts archive` writes one redacted page per conversation per local day from your agent harnesses, plus a daily index (GST-2768).**
+
+### Added
+- `gbrain transcripts archive run | backfill --from <day> | status | adopt-legacy --dir <folder> | on | off`. Off until `gbrain transcripts archive on`; nothing is scheduled yet. Pages go to `conversations/<harness>/<day>/<label>-<hash8>` and `conversations/days/<day>` in the `conversations` source, redacted before any write; reruns change nothing; the first run pins the timezone so days never re-slice. Backfilled pages are keyword-searchable at once and their embedding waits for `gbrain embed --held`. On a shared brain every page names its readers (the owner, plus an existing `chat-<hash8>` group for a verified group chat); with no owner named, days are held with `archive_owner_unset`. Works on repo-less brains (database is the record), personal repo brains (files committed with `gbrain git commit-push`, then synced) and managed brains (the write coordinator).
+- OpenClaw: live sessions are read through the Gateway (`openclaw gateway call`, with OpenClaw's own config or `OPENCLAW_GATEWAY_URL` / `OPENCLAW_GATEWAY_TOKEN`, or `GBRAIN_OPENCLAW_GATEWAY_URL` / `GBRAIN_OPENCLAW_GATEWAY_TOKEN` to override them), checked against OpenClaw 2026.10.9 and 2026.11.3. The archive also reads OpenClaw's retained session files (`.jsonl.reset.<stamp>`, zstd `.jsonl.deleted.<stamp>.<hex>.zst` and `.jsonl.bak.<stamp>`) and merges them with the Gateway's copy of the same session, so a backfill works with the Gateway down and messages from before a reset are kept; migration and Doctor copies are skipped, and `gbrain transcripts ingest` still reads live files only. A day page is rewritten only when it keeps every message it already held, and backfill re-redacts a finished day when the redaction rules changed since it was written. The `openclaw` child gets only PATH, HOME, locale and the resolved Gateway URL and token, and a working-folder `.env` can no longer set any `OPENCLAW_*` or gbrain Gateway variable. Backfill reads large session folders in a bounded way and resumes where the last run stopped.
+- New error codes: `archive_gateway_unreachable`, `archive_gateway_auth`, `archive_gateway_failed`, `archive_owner_unset`, `archive_redaction_failed`, `archive_run_busy` (one run per source at a time), `archive_audience_unproven` (the brain's audience could not be checked, so nothing was written).
+- When a brain becomes shared, the next archive run first keeps every earlier archive page to the owner, then records the new audience. A failed day index write is retried on the next run.
+
+- Docs: `docs/guides/conversation-archive.md`.
+- Tests: `test/transcripts-archive-*.test.ts` (PGLite, including a real git remote and child CLI commit-push), `test/transcripts-openclaw-gateway.test.ts`, `test/transcripts-openclaw-archived-files.test.ts`, `test/transcripts-openclaw-files-only.test.ts`.
+
+### Known limits
+- Redaction works on what each day's own text marks as a secret. A value that a later day labels as a credential stays visible in the earlier day's page.
+- Pages written while the brain was personal are closed on the next archive run after it becomes shared, so they stay readable to other members for up to one maintenance interval.
diff --git a/docs/architecture/canonical-writers.tsv b/docs/architecture/canonical-writers.tsv
index 9dd4c65c83..c4e90f02d4 100644
--- a/docs/architecture/canonical-writers.tsv
+++ b/docs/architecture/canonical-writers.tsv
@@ -146,6 +146,7 @@ src/core/takes-write.ts 8 filesystem_guard Managed operation routes through jour
src/core/think/index.ts 1 engine_guard Canonical table SQL trigger requires coordinator capability; legacy calls fail before canonical DB publication.
src/core/timeline-dedup-repair.ts 3 engine_guard Canonical table SQL trigger requires coordinator capability; legacy calls fail before canonical DB publication.
src/core/timeline-write-through.ts 2 filesystem_guard Managed semantic operation journals page changes; legacy atomic writes check root.
+src/core/transcripts/archive.ts 2 engine_guard GST-2768: on writer-claimed brains the archive submits day parts and stale-part deletions as put_page/delete_page write requests through the coordinator (submitTranscriptPart, deleteStaleTranscriptPart). The direct importFromContent and deletePages calls run only in repo-less database mode and refuse on managed brains before any write (assertUnmanagedCanonicalWriter / canonical table trigger).
src/core/transcripts/ingest.ts 2 engine_guard On writer-claimed brains the CLI ingest submits rendered parts as put_page write requests (submitPageMutation with its trusted-local context). The remaining direct importFromContent (unmanaged brains) and stale-part deletePage calls refuse on managed brains before any write (assertUnmanagedCanonicalWriter / canonical table trigger).
src/core/url-source-origin.ts 1 projection Source settings (the sources.config merge in recordSourceOrigin, for the trusted local sources record-origin command, under a FOR UPDATE row lock in one transaction, inside the reclone's topology locks on a managed brain); canonical fields remain guarded by SQL triggers.
src/core/workspace-push.ts 0 filesystem_guard The resolved git root is checked before staging, committing or pulling.
diff --git a/docs/architecture/key-files/commands-6.md b/docs/architecture/key-files/commands-6.md
index a5326d4b09..9206728885 100644
--- a/docs/architecture/key-files/commands-6.md
+++ b/docs/architecture/key-files/commands-6.md
@@ -10,7 +10,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/commands/takes-mutation.ts` — one parser and operation adapter for local, resident PGLite, and HTTP take mutations. Allocates a request UUID before delivery; retries retain that UUID and caller intent. `--source` remains claim provenance, `--source-id` selects the logical source, and `--dir` becomes a trusted local canonical-root assertion. Default owner/resolver choices remain unset until admission freezes them. The CLI and thin-client router share this parser for all four mutations before connecting; `runTakesRebuild` (W9F item 8) is `gbrain takes rebuild`'s parser and renderer, running `takes_rebuild` through a resident owner or the local engine (thin clients refused); queued/error responses never print completed-mutation text. Remote failures use the shared typed reporter directly, retaining UUIDs, unknown delivery status and original receipt fields in JSON. `test/persistence-cli-delegation.test.ts` drives the real CLI through a held PGLite owner's socket, checks all four parameter mappings and pending receipts, and rejects malformed numeric/precondition input.
-- `src/commands/transcripts.ts` — the transcripts command family, all local-only (`ctx.remote=false` by construction). `recent`: raw `.txt` corpus reads via `listRecentTranscripts` (same library as the gated `get_recent_transcripts` op). `ingest <path-or-glob>`: the cross-harness session importer — resolves ONE source id (6-tier chain), threads activePack once, streams progress (phase `transcripts.ingest`), and calls `runTranscriptsIngest` with the CLI's `makeContext` context, so on a writer-claimed brain each rendered part is submitted as a `put_page` write request naming the page's current revision (a part whose publication would change nothing is screened request-free first and takes no admission; unmanaged brains and dry runs keep the direct importer; `recover codex --apply` passes the same context; pinned by `test/transcripts-managed-ingest.test.ts`); embedding is OFF by default (embed backfill is the catch-up lane; the embed flag opts in); the max-bytes flag (validated size string, e.g. `4gb`) overrides the per-format file/store byte caps for oversized stores while omission preserves each adapter's native default; the since-last watermark is an op-checkpoint (op `transcripts-ingest`, fingerprint = source + pathspec + format + adapter version + any explicit byte cap via `ingestCheckpointFingerprintInput` — a checkpoint written under one cap is never silently reused under another, so a capped run's skipped tail can't read as scanned; the pathspec binds the user-stated paths resolved + sorted, and the no-arg/all lane substitutes hostname + sorted harness roots so DB-backed checkpoints shared across a brain's machines never let one machine inherit another's watermark and skip sessions it never scanned) advanced ONLY after a clean, untruncated, non-dry scan (a Codex sub-agent rollout, `session_meta` payload.source.subagent, has no user turn by design: the adapter reports `subagentRollout`, the scan counts it in `subagentRollouts` and prints `N sub-agent rollouts (no user turns by design)`, and it keeps the scan clean; an ordinary assistant-only rollout is still `userTurnsMissing` drift); no-arg = confined discovery table, the all flag imports the discovered set. `status`: found-vs-imported gap table (disk scan vs ONE conversation-pages frontmatter query — `executeRaw`, both engines — in `src/core/transcripts/discover.ts`) — the correctness surface that catches late-arriving sessions no watermark can. The facts flag hands EVERY touched slug (including hash-skipped) to `src/core/transcripts/ingest-facts.ts`; a page that fails extraction counts in `facts.pagesFailed`, stays unfinished, and makes the command exit 1; a pass that stopped at its budget reports `facts.budgetExhausted` and `facts.budgetReason`, and pages skipped as too large or held by another run count in `facts.pagesSkippedTooLarge` and `facts.pagesLockSkipped` (#8722). Registered in CLI_ONLY + CLI_ONLY_SELF_HELP + SELF_HELP_WITHOUT_ENGINE (engine-free help). `expandTilde(raw)` (bare `~`, leading `~/` or `~\\` → home dir; `~user` forms and a mid-string `~` stay literal) is the one tilde expansion `expandPaths` applies to every path spec. `recover codex` (`src/commands/transcripts-recover.ts`, logic `src/core/transcripts/recover.ts`; D15 for #5163): finds codex conversation pages in one source with no `**User**` anchor, re-parses the retained rollouts (discovery root + archived store, or given paths) and classifies each session recoverable / still userless / unrecoverable (rollout gone); preview by default, `--apply` re-imports only the recoverable rollouts through `runTranscriptsIngest` (content-hash dedup, no facts, no watermark write). Pinned by `test/transcripts-recover-codex.test.ts`. `audit-secrets [--source-id S] [--json]` (`src/commands/transcripts-audit.ts`, logic `src/core/transcripts/secret-audit.ts`): the read-only installed-base audit that streams imported conversation pages (`type = 'conversation'` with `transcript_import` frontmatter) in id-keyset batches of 25, scans title and body with the transcript-lane detectors, lists slug, source, hit counts per pattern and up to 20 line numbers (never values or previews), and caches the summary without slugs in config row `transcripts.secret_audit` for the doctor check `transcript_secret_exposure` (`src/commands/doctor/checks/transcript-secrets.ts`, brain category), which counts imported conversation pages and reads that summary (never scans): no audit, an older `TRANSCRIPT_SECRET_AUDIT_VERSION` or a source-scoped audit warns with the audit as fix; hits warn with the audit's age; a clean full audit is ok. Pinned by `test/transcripts-secret-audit.test.ts`.
+- `src/commands/transcripts.ts` — the transcripts command family, all local-only (`ctx.remote=false` by construction). `archive run|backfill|status|adopt-legacy|on|off` dispatches to `src/commands/transcripts-archive.ts` (core `src/core/transcripts/archive.ts`, see core-services-3; no MCP operation). `recent`: raw `.txt` corpus reads via `listRecentTranscripts` (same library as the gated `get_recent_transcripts` op). `ingest <path-or-glob>`: the cross-harness session importer — resolves ONE source id (6-tier chain), threads activePack once, streams progress (phase `transcripts.ingest`), and calls `runTranscriptsIngest` with the CLI's `makeContext` context, so on a writer-claimed brain each rendered part is submitted as a `put_page` write request naming the page's current revision (a part whose publication would change nothing is screened request-free first and takes no admission; unmanaged brains and dry runs keep the direct importer; `recover codex --apply` passes the same context; pinned by `test/transcripts-managed-ingest.test.ts`); embedding is OFF by default (embed backfill is the catch-up lane; the embed flag opts in); the max-bytes flag (validated size string, e.g. `4gb`) overrides the per-format file/store byte caps for oversized stores while omission preserves each adapter's native default; the since-last watermark is an op-checkpoint (op `transcripts-ingest`, fingerprint = source + pathspec + format + adapter version + any explicit byte cap via `ingestCheckpointFingerprintInput` — a checkpoint written under one cap is never silently reused under another, so a capped run's skipped tail can't read as scanned; the pathspec binds the user-stated paths resolved + sorted, and the no-arg/all lane substitutes hostname + sorted harness roots so DB-backed checkpoints shared across a brain's machines never let one machine inherit another's watermark and skip sessions it never scanned) advanced ONLY after a clean, untruncated, non-dry scan (a Codex sub-agent rollout, `session_meta` payload.source.subagent, has no user turn by design: the adapter reports `subagentRollout`, the scan counts it in `subagentRollouts` and prints `N sub-agent rollouts (no user turns by design)`, and it keeps the scan clean; an ordinary assistant-only rollout is still `userTurnsMissing` drift); no-arg = confined discovery table, the all flag imports the discovered set. `status`: found-vs-imported gap table (disk scan vs ONE conversation-pages frontmatter query — `executeRaw`, both engines — in `src/core/transcripts/discover.ts`) — the correctness surface that catches late-arriving sessions no watermark can. The facts flag hands EVERY touched slug (including hash-skipped) to `src/core/transcripts/ingest-facts.ts`; a page that fails extraction counts in `facts.pagesFailed`, stays unfinished, and makes the command exit 1; a pass that stopped at its budget reports `facts.budgetExhausted` and `facts.budgetReason`, and pages skipped as too large or held by another run count in `facts.pagesSkippedTooLarge` and `facts.pagesLockSkipped` (#8722). Registered in CLI_ONLY + CLI_ONLY_SELF_HELP + SELF_HELP_WITHOUT_ENGINE (engine-free help). `expandTilde(raw)` (bare `~`, leading `~/` or `~\\` → home dir; `~user` forms and a mid-string `~` stay literal) is the one tilde expansion `expandPaths` applies to every path spec. `recover codex` (`src/commands/transcripts-recover.ts`, logic `src/core/transcripts/recover.ts`; D15 for #5163): finds codex conversation pages in one source with no `**User**` anchor, re-parses the retained rollouts (discovery root + archived store, or given paths) and classifies each session recoverable / still userless / unrecoverable (rollout gone); preview by default, `--apply` re-imports only the recoverable rollouts through `runTranscriptsIngest` (content-hash dedup, no facts, no watermark write). Pinned by `test/transcripts-recover-codex.test.ts`. `audit-secrets [--source-id S] [--json]` (`src/commands/transcripts-audit.ts`, logic `src/core/transcripts/secret-audit.ts`): the read-only installed-base audit that streams imported conversation pages (`type = 'conversation'` with `transcript_import` frontmatter) in id-keyset batches of 25, scans title and body with the transcript-lane detectors, lists slug, source, hit counts per pattern and up to 20 line numbers (never values or previews), and caches the summary without slugs in config row `transcripts.secret_audit` for the doctor check `transcript_secret_exposure` (`src/commands/doctor/checks/transcript-secrets.ts`, brain category), which counts imported conversation pages and reads that summary (never scans): no audit, an older `TRANSCRIPT_SECRET_AUDIT_VERSION` or a source-scoped audit warns with the audit as fix; hits warn with the audit's age; a clean full audit is ok. Pinned by `test/transcripts-secret-audit.test.ts`.
- `src/commands/upgrade.ts` — self-update CLI. The package has no postinstall, so the install step runs no migrations; post-upgrade owns them (`GBRAIN_UPGRADE_OWNS_MIGRATIONS=1` is still set for the install step). An install step that fails or times out after `gbrain --version` already reports a newer version counts as installed. The run ends with a `Binary:` line (installed version, or upgrade failed) and a `Migrations:` line: complete; running, with the lock holder's host and pid when post-upgrade exits 75; or failed, with the recovery command (`--force-retry <v>` for wedged versions). `runPostUpgrade()` enumerates migrations from the TS registry (`src/commands/migrations/index.ts`) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally. After schema apply it prints the one-shot auto_chronicle default-on `[AGENT]` notice (`core/chronicle/upgrade-notice.ts`). Install methods are bun, bun-link and binary (a compiled binary is found by Bun's `/$bunfs/` entrypoint, so a release asset keeps its published name); anything else is `unknown` and prints manual steps. It never runs `clawhub update gbrain`, which would install whatever skill anyone publishes under that name (#8464; `test/upgrade.serial.test.ts`). Before the swap of a bun-link or package install it refuses (exit 78, nothing changed) a target whose Bun floor is above the host's Bun or unreadable (`--no-bun-floor-check` overrides, manual only; `src/core/bun-floor.ts`); after the swap it runs `gbrain --help` as a smoke that passes the startup Bun check and, on failure, exits 1 with per-method recovery (#5855). Everything the post-swap tail needs is imported statically, so the old process never loads new-release modules into its cached graph (#6604; the supervisor step constant lives in the import-free leaf `src/commands/movement-step.ts`; guard: `test/upgrade-post-swap-imports.test.ts`). A tag-pinned reinstall hint (`bunReinstallCommand`, `reinstallHintLines`) probes `git ls-remote --exit-code --tags … refs/tags/v<x>` before `bun remove -g gbrain`, prints a restore line, and on Windows prints one step per line; `planPinnedReinstall` probes the tag itself (published / not published / unknown) and falls back to the newest published `v<x>` tag (`latestPublishedVersion`) when the requested one is missing (#6662, D24; `test/upgrade-release-tag-hint.test.ts`). A verified version equal to the running one after the install step is no swap (#6828): it prints `No newer version was installed`, skips `last_upgrade`, `JUST_UPGRADED`, the cache reset and the smoke, and on a tag pin with no newer published tag the hint moves the install off the pin with `to_version: latest`, never a cached unpublished version (#6718). With `GBRAIN_SWAP_RECEIPT` set it writes a `SwapReceipt` (`installed` / `not_installed` / `installed_broken`) for the autopilot channel (`test/autopilot-unpublished-target-6828.test.ts`). Tests: `upgrade-migration-outcome.test.ts`, `upgrade-no-autopilot.serial.test.ts`, `upgrade-bun-link-arc.serial.test.ts`. `post-upgrade --json` (D2, `json: 'document'`): human lines go to stderr and stdout is one report `{status, apply_migrations:{exit_code}, schema, warnings[]}`; it calls `applyMigrations()` and still ends on any status apply-migrations returns (a failure is one envelope leading with the report). GBRA-175: `--channel stable|alpha` (`latest` is an alias of `alpha`; resolution order in the Release channels entry below): `resolveStableRelease` maps the commit `refs/tags/stable` peels to onto its `v<version>` tag through `git ls-remote`, and `installStableRelease` installs exactly that release (bun: remove then add `#v<version>`, restoring the previous release on failure; bun-link: fast-forward only; binary: `fetchReleaseByTag`), never downgrading. Before the swap `preflightHostRestart` refuses (`restart_refused`, exit 3) when a running process could not be restarted; after post-upgrade the new build runs `gbrain upgrade --restart-only` (`--no-restart` skips it). Tests: `upgrade-stable-channel.test.ts`. A global install pinned to a tag or commit (`bunGlobalExactTagPin(root, true)`) is replaced with the newest published release the same way instead of `bun update`, which keeps the pin. `parseChannelArg` and `resolveStableRelease` live in `src/core/stable-channel.ts` (re-exported).
- Release channels: `src/core/self-upgrade.ts` resolves the channel (`resolveReleaseChannel`: `--channel`, `GBRAIN_RELEASE_CHANNEL`, `self_upgrade.channel`, default `stable`; `latest` normalizes to `alpha`; an invalid value at the first present source is an error) and keeps the update cache (`update-check.<channel>.json`, tagged with the evidence policy; an older answer never overwrites a newer one) and the time-based snooze (`update-snoozed.<channel>.json`) per channel; `offeredVersion` never offers a downgrade except to move off a `known_bad` version. `src/core/stable-channel.ts` reads and strictly parses the evidence record (`stable.json` on `stable-record`; `requiredEvidenceKeys`: `soak_run`, plus `matrix_run` from `MATRIX_REQUIRED_FROM`), maps failures to `stable_missing` / `stable_invalid` / `stable_unreadable` with sanitized causes, and `resolveStableRelease` checks the `stable` and `v<version>` tags against the record commit. `src/commands/upgrade-channel.ts` plans an upgrade's target (`channelTarget`, `runUpgradeDryRun` for `gbrain upgrade --dry-run --json`) and refuses with the typed code. `src/commands/check-update.ts` `refreshUpdateCache` never reads a bare VERSION for stable and never falls back to alpha. `src/core/release-channel-notice.ts` prints the one-time `[gbrain notice release_channel]` line on stderr (marker file under `notices/`; not under `--json`, `serve` or `mcp`; only when no channel is set). `src/core/install-ref.ts` is the one ref every install surface names, linted by `scripts/check-install-ref.ts`. Tests: `test/release-channel-w1.serial.test.ts`, `test/upgrade-stable-channel.test.ts`, `test/install-ref.test.ts`. Docs: `docs/guides/upgrades-auto-update.md#stable-channel`.
diff --git a/docs/architecture/key-files/core-services-3.md b/docs/architecture/key-files/core-services-3.md
index ad8c75fc14..fa72208eb5 100644
--- a/docs/architecture/key-files/core-services-3.md
+++ b/docs/architecture/key-files/core-services-3.md
@@ -15,6 +15,8 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/transcripts/` (directory) — the transcript-adapter seam. `types.ts`: the `TranscriptAdapter` contract — `parse(path): AsyncGenerator<ParsedSession, FileDiagnostics>` (one FILE may hold many sessions; the generator RETURN value carries bytes/skipped-lines/zero-session diagnostics so an empty file explains itself), format-specific byte caps (JSONL 50MB hard cap — the 10MB default in `claude-code-jsonl.ts` belongs to the hook-lane tail reader, not imports; monolithic export JSON rejects-not-truncates at 200MB), and the ONE `buildTranscriptSlug` helper (per-provider dirs matching the conversation-archive layout; id suffixes are sha256 prefixes of the session id — hash12 in slugs, hash16 for the dedup identity — never a cleaned prefix of the source id, which would let same-prefix session ids collide). `detect.ts`: adapter registry + head-sample detection (filesystem admission first for every path, explicit format included: symlinks lstat-rejected, an unreadable path carries its `errno`, which ingest copies to the file outcome with a per-cause next step; then the explicit format wins without sniffing) + injectable `harnessRoots` (the confined discovery surface). Adapters, each with a DATED `SPEC_TARGET` + scrubbed fixture + `bytes>0 && sessions==0` drift alarm: `claude-code.ts` (thin wrapper over the shipped `claude-code-jsonl.ts`, which also exposes the full-file `parseClaudeSessionFile` with real per-message timestamps — hook-lane `parseTranscript` output is pinned byte-identical; owns `isClaudeCodeSubagentFile` (a literal `subagents` path segment at ANY depth + `agent-*.jsonl` basename — both `<session>/subagents/agent-*.jsonl` and the workflow-nested `<session>/subagents/workflows/<wf>/agent-*.jsonl` are all-sidechain, zero-turn files carrying the PARENT session id; a project slug containing "subagents" is one segment and never matches) and `isClaudeCodeWorkflowArtifactFile` (everything under `<session>/subagents/workflows/<wf>/` — the agent logs plus the `journal.jsonl` run log that matches no adapter format; a UUID-named session directly under `subagents/` stays discoverable); `discover.ts` and the CLI's `expandPaths` skip both, so they never read as gap-table backlog, drift, or `unknown format` errors that hold the since-last watermark frozen; the adapter sets `expectedEmpty` when zero accepted turn-shaped records and zero skipped lines remain after root metadata/sidechain and explicit non-human user-text filtering. Metadata-only, non-human-text-only, and all-subagent files are understood empties and do not freeze the watermark; accepted turn records that yield no text and unparseable lines still signal drift. Pinned by `test/claude-code-jsonl-origin.test.ts`), `codex.ts` (turn selection is STRUCTURAL: user turns from `event_msg` user_message (through 0.152) or `event_msg` item_completed with item.type UserMessage, text blocks only (0.153+, #5163; a turn recorded in both shapes is kept once, other item types incl. AgentMessage skipped), assistant from `response_item` output_text; a file with assistant turns and zero user turns sets `FileDiagnostics.userTurnsMissing` (codex and claude-code; E-N4), which ingest counts as drift and the session-end hook heartbeats as `degraded`/`no_user_turns` on whole-file reads (pinned by `test/codex-rollout-0154.serial.test.ts`); role user/developer response_items are injected preambles and never leak; identity is `payload.id` — the per-thread id codex embeds in the rollout filename — with `payload.session_id` (the ROOT session shared by every forked/subagent thread) only as the legacy fallback, and the FIRST `session_meta` header wins because a child rollout carries its inherited parent header later in the file; the hook lane's `parseCodexHookTranscript` follows the same two rules), `openclaw.ts` (session header + message lines; `.checkpoint.*.jsonl` snapshots rejected), `hermes.ts` (SNAPSHOT-THEN-READ: a read-only connection writes one coherent `VACUUM INTO` copy of state.db, WAL frames included, to a temp dir that is removed after the read; copying the db and its sidecars separately tore under a concurrent checkpoint, pinned by `test/hermes-snapshot-7067.serial.test.ts`; schema from the installed hermes-agent source; `json:`-prefixed and bare block arrays unwrap to text, image blocks to `[image]`; #7396: the answered responses of a `clarify` tool result (`role=tool`, `tool_name=clarify`; `status: answered` under `outcome: submitted` or `timed_out`, whose locked answers Hermes keeps, or the earlier batch shape without `timed_out`) import as user turns at the result's timestamp, each after its question as an assistant turn, with the question and tool-call id as flat `clarify_<n>_*` raw keys, while offered choices, unanswered responses, cancelled/undelivered forms, the legacy single-question shape and other tool rows never do; pinned by `test/hermes-clarify-transcripts-7396.test.ts`; PROVISIONAL), `grok.ts` (Grok Build session dirs — structural turn selection, sidecar-aware discovery; see its own entry), `chatgpt-export.ts` (the `mapping`-tree current_node walk; branches dropped by design, orphaned parents terminate quietly, latest-leaf fallback; extracted conversations.json only) + `claude-export.ts` (flat chat_messages); both export adapters load through the shared `export-json.ts` (monolithic JSON over the cap REJECTS with a split hint; a zip or wrong-shape file gets the unzip-first hint) and give id-less conversations content-derived fallback ids so two id-less exports cannot dedup-skip or overwrite each other. `render.ts`: session → part pages — imessage-slack line format with the regex IMPORTED from `conversation-parser/builtins.ts` (round-trip pinned), REAL timestamps (missing ones carry forward; zero-timestamp sessions refused — provenance is never fabricated), anchor-shaped BODY lines backslash-escaped (hostile message content cannot forge speakers), quoted facts/takes fence-marker tokens backslash-escaped at the same per-message seam (`gbrain\:facts:begin` — a session that read another page cannot plant a live or unbalanced fence on the transcript page; every fence consumer is a substring matcher, so the escape lands inside the token), fail-closed redaction (secret-scan + `harvest-private-patterns.txt` user patterns, slack-channel default excluded because it eats issue refs; imperatives COUNTED into hash-covered `transcript_import` frontmatter, never content_flag), message-boundary splitting at `PART_TARGET_BYTES` = 45,000 bytes (0.9 × the 50,000-byte content-sanity warn line, so no part takes the oversize-warn branch; far under the embed-skip threshold) with 2-message overlap, part 1 keeps the base slug, `frontmatter.id` unique per part. `ingest.ts`: the engine-facing core — a session whose part-1 `frontmatter.id` already has a live page imports at that page's slug (a renamed or re-dated conversation updates in place instead of being skipped as a cross-slug duplicate), SESSION atomicity (a failed session records its `classifyItemFailure` code and the run continues; a part the coordinator accepted but has not published leaves the session pending under a request id derived from source, slug, revision and content, so a rerun waits on that write and a write that ended without committing is retried under the next attempt id; only run-scoped failures and the `item_failures_systemic` breaker abort the run, GBRA-130 A-2, pinned by `test/pathological/a2-transcripts-ingest.test.ts`), stale-part reconciliation (deletes `part > of` leftovers), session metadata banked in the base page's raw_data as the REDACTED copy and healed by content-compare even on hash-skipped re-runs (a run that died before `putRawData`, or a private pattern added after first import, repairs on the next pass), `cleanScan`/`maxSessionTs` for the watermark. `ingest-facts.ts`: ONE `runExtractConversationFactsCore` invocation (batch `slugs` selector) inside ONE `withBudgetTracker`, `isFactsExtractionEnabled` pre-checked. Pinned by `test/transcript-adapters.test.ts`, `test/transcript-render.test.ts`, `test/transcripts-retitle.test.ts`, `test/transcripts-ingest-pglite.test.ts`, `test/e2e/transcripts-writeback-fidelity.test.ts` (raw files through the adapters into the gold-extractor facts path).
+- `src/core/transcripts/archive.ts` + `archive-day.ts` + `archive-readers.ts` + `openclaw-gateway.ts` (GST-2768) — the day-sliced conversation archive behind `gbrain transcripts archive`, off until `transcripts.archive.enabled` is `'true'` (`archive on`; a dry run previews while off). `runTranscriptArchive` reads every `ArchiveReader` (file readers over `discoverTranscriptFiles`, which includes OpenClaw's `.jsonl.reset.<stamp>` and zstd `.jsonl.deleted.<stamp>.<hex>.zst` archives; the OpenClaw Gateway reader over `openclaw gateway call sessions.list`/`chat.history` when `openclaw` is on PATH or `GBRAIN_OPENCLAW_GATEWAY_URL` is set), redacts the whole session once with `redactSession` (a throw holds every day of it, `archive_redaction_failed`), slices messages by local day in the pinned timezone (`transcripts.archive.pinned_timezone.<source>`, pinned on first run), and renders `conversations/<harness>/<day>/<label>-<hash8>` part pages plus `conversations/days/<day>` indexes (`archive-day.ts`, reusing `renderSessionParts`). One ledger row per (source, harness, conversation hash16, day) in config makes reruns no-ops; a day whose source now has fewer messages than the ledger recorded is `kept` (OpenClaw's in-place reset). Shared brains (non-personal or unknown audience, or 2+ enabled members) stamp `readable_by` with the owner (no owner: every day held, `archive_owner_unset`) plus `group:chat-<hash8>` only for a verified group chat whose group exists; a shared brain whose archive source has a checkout writes to database-only `<source>-owner`. Write paths: coordinator on managed brains (`submitTranscriptPart`), files + `gbrain git commit-push -- <paths>` (`commitPushChild` with paths) + `performSync` on a personal repo source, `importFromContent` otherwise (repo-less is first class). `backfill` needs `--from`, caps distinct days per run newest first, and holds embedding with `page_embedding_budget` reason `archive_backfill` (applies at any page size: `budgetRowAppliesFilter`, `smallPageHold` in `page-embedding-budget.ts`); `adoptLegacyArchive` records the (harness or `*`, day) pairs an old archive folder covers so backfill skips them. Pinned by `test/transcripts-archive-*.test.ts`, `test/transcripts-openclaw-gateway.test.ts` (recorded OpenClaw 2026.11.3 fixtures under `test/fixtures/openclaw-gateway/`) and `test/transcripts-openclaw-archived-files.test.ts`. Guide: `docs/guides/conversation-archive.md`.
+
- `src/core/verbs.ts` — MEMORY_VERBS v1: the four frozen protocol verbs (`remember`, `entity`, `synthesize`, `forget`) as first-class Operations, plus `MEMORY_VERBS_VERSION` (single source of truth, =1), `VERB_NAMES`, the hand-authored `RESPONSE_SCHEMAS` registry (Operation carries input params only; response shapes live here and conformance validates LIVE responses against them), and `ERROR_SCHEMA`. The extended `recall` op is defined in ops/facts.ts and registered through operations.ts. Its response registry advertises optional `budget_packing` policy/applied/reason and per-arm integer accounting without changing required default fields or protocol version; `test/protocol-document.test.ts` pins discovery and malformed metadata rejection. The `context_pack` and `delta` fact items advertise optional `fact_id` and `provenance` (#6146). The recall registry advertises optional `facts_order`, `facts_degraded` and per-fact `relevance` (question ranking). RUNTIME LEAF invariant: operations.ts spreads `verbOperations` into its array at module-eval time, so this file must never statically import operations.ts VALUES (type-only imports fine; handlers use dynamic import) — violating it causes a TDZ crash on whichever module evaluates second. Every verb error carries a populated `suggestion` + `protocol_version` (via `verbError` in operations.ts). The `forget` verb deliberately has NO cliHints (CLI_ONLY `forget` dispatches first and would shadow it). Frozen contract: docs/protocol/MEMORY_VERBS_v1.md; pinned by test/memory-verbs-conformance.test.ts. The starter list leaves out remember's `expected_revision`/`force`/`valid_from`/`infer_entity`/`visibility` and, in `ops/facts.ts`, recall's `budget_policy`/`include_expired`/`supersessions`/`limit`/`grep`/`include_pending`/`return_unit`/`return_window`, context_pack's `since`/`session_id`/`include_private` and delta's `since_slug`/`entities`/`include_private` (`fullSurfaceOnly`); the verbs surface lists every param.
- `src/core/verbs/conformance.ts` + `src/core/verbs/conformance-fixtures.ts` — the conformance runner core (transport-agnostic: minimal `ConformanceClient` = list_tools + call_tool) and the embedded fixture set. Deterministic by construction: shape/enum/behavior/round-trip checks only, never ranking quality. Validation is NON-STRICT on extra fields (additive-forever means unknown fields are always legal). Entity-card cases seed via put_page when the target exposes it and skip honestly on verbs-only targets; synthesize is cost-gated behind --synthesize. `validateAgainstSchema` is a minimal JSON-Schema-subset validator (type unions, required, properties, enum, const, items). Fixtures mirror to `test/fixtures/memory-verbs/cases.json` (BrainBench seeds; drift-guarded by the conformance test). The negative self-test (test/memory-verbs-conformance.test.ts) proves the runner FAILS a lying server.
diff --git a/docs/architecture/key-files/security.md b/docs/architecture/key-files/security.md
index 6553e008d4..3a3e9d4675 100644
--- a/docs/architecture/key-files/security.md
+++ b/docs/architecture/key-files/security.md
@@ -5,7 +5,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/destructive-guard.ts` — three-layer protection against accidental data loss. `assessDestructiveImpact(engine, sourceId)` counts pages/chunks/embeddings/files/facts for a source (a fact-only source — a revoked agent's workspace, since facts are the primary agent write lane — is data at stake, not empty; pre-facts brains degrade to 0), plus `oauthClientCount` — OAuth clients whose `source_id` references it. `checkDestructiveConfirmation(impact, opts)` is the fail-closed gate (`--confirm-destructive` required when data is present; `--yes` alone is rejected). FK-RESTRICT lifecycle: `clientsReferencingSource(engine, sourceId)` lists ALL physical OAuth-client rows referencing a source via `oauth_clients.source_id` — the FK is PHYSICAL (ON DELETE RESTRICT ignores `deleted_at`), so soft-deleted (revoked-but-retained) rows BLOCK a hard delete too and come back tagged `deleted`; pre-migration brains without `deleted_at` fall back to untagged referents (42703-retry idiom) and brains without the table have none by construction. `formatClientReferentsBlock` renders the shared refusal `sources remove`/`sources purge` print (naming each client — `[revoked, retained]` for soft-deleted rows — + the revoke command) so the raw Postgres FK violation never reaches the operator. `softDeleteSource` / `restoreSource` / `listArchivedSources` / `purgeExpiredSources` drive the source-level archive lifecycle via `sources.archived BOOLEAN`, `archived_at TIMESTAMPTZ`, `archive_expires_at TIMESTAMPTZ`; `purgeExpiredSources` SKIPS client-referenced sources via a physical `NOT EXISTS` (soft-deleted clients count) so recurring maintenance keeps sweeping the rest instead of aborting; an archive with a missing or epoch `archived_at`/`archive_expires_at` is never treated as expired and is reported in `blocked`. On an unmanaged brain any other failure of one source's delete is classified (ItemRun site `retention.expired_sources`): an item-scoped one (a statement timeout on a large cascade, a lock timeout) is reported in `blocked` with its `code` and a rendered `fix` (`tell_user_to_run gbrain sources purge <id> --confirm-destructive`) and the sweep continues; a run-scoped one, or a sweep where most sources fail, re-raises (`test/pathological/a50-unmanaged-expired-sources.test.ts`). Page-level analog: `BrainEngine.softDeletePage` / `restorePage` / `purgeDeletedPages` plus `pages.deleted_at TIMESTAMPTZ` and a partial purge index. The MCP `delete_page` op rewires to `softDeletePage`; ops `restore_page` (`scope: write`) and `purge_deleted_pages` (`scope: admin`, `localOnly: true`) round out the surface. `get_page`/`delete_page`/`restore_page` (`src/core/ops/pages.ts`) accept an explicit per-call `source_id` that is honored or rejected loudly, never silently dropped: `get_page` resolves it through the caller's read grant (`'__all__'` spans every source for trusted local callers, the granted sources for remote callers), while `delete_page`/`restore_page` reject `'__all__'`, target exactly one source, and — for every caller other than trusted local CLI (`ctx.remote === false`) — accept only the caller's write authority (`ctx.auth.sourceId`, falling back to `ctx.sourceId` for legacy tokens and unauthenticated transports; the federated read grant `allowedSources` confers no delete/restore access), echoing the targeted `source_id` in the response. Pinned by `test/pages-source-scoping-4329.test.ts`. Search visibility (`buildVisibilityClause` in `src/core/search/sql-ranking.ts`) hides soft-deleted pages and archived sources from `searchKeyword` / `searchKeywordChunks` / `searchVector` in both engines. The autopilot cycle's `purge` phase calls `purgeExpiredSources` + `engine.purgeDeletedPages(72)` so the 72h TTL is real.
-- `src/core/env-trust.ts` — Trust boundary for environment variables Bun auto-loaded from the cwd `.env` family. `CWD_DOTENV_FILES` (the eight names Bun auto-loads across NODE_ENV values), `splitDotenvLines` (Bun's line grammar: `\n`, `\r\n` AND a bare `\r` terminate a line, leading BOM stripped — shared with gbrain-env-file.ts), `dotenvValuesForKey(key, dir)` (value set across all assignments; the #427 `DATABASE_URL` value-match guard in config.ts, which re-exports both), `cwdDotenvAssignsKey(key, dir | assignments)` (KEY-PRESENCE predicate; an empty assignment counts) and `parseCwdDotenv(dir)` → `DotenvAssignment[]` (one read the two preflight consumers share). The assignment RHS is matched as `([^\r\n]*)`, never `.*$`. Protection is one predicate, `isCwdDotenvProtectedKey(key)`: `CWD_DOTENV_PROTECTED_KEYS` (code-loading / exec-target / root- and endpoint-redirect incl. `GBRAIN_DATABASE_URL` (key-presence; the #427 config.ts VALUE guard is separate and covers only bare `DATABASE_URL`), `GBRAIN_DIRECT_DATABASE_URL`, `GBRAIN_REMOTE_MCP_URL`, `GBRAIN_REMOTE_ISSUER_URL`, `GBRAIN_OAUTH_RELAY_URL` / posture-widening `GBRAIN_*`, plus the re-run marker `GBRAIN_CWD_ENV_QUARANTINED`; one why-comment per key, plus the deliberate exclusions and deferrals) PLUS `CWD_DOTENV_PROTECTED_PREFIXES` (`LD_`, `DYLD_`, `GIT_`, `BUN_`, `NPM_CONFIG_`, `npm_config_`, `BASH_FUNC_`) and `CWD_DOTENV_PROTECTED_TOOLCHAIN_KEYS` (NODE_OPTIONS/NODE_PATH/CA/TLS, SSH_ASKPASS*, XDG_CONFIG/DATA/CACHE_HOME, GNUPGHOME, SSL_CERT_FILE/DIR, CURL_CA_BUNDLE, REQUESTS_CA_BUNDLE, OPENSSL_CONF/ENGINES/MODULES, TMPDIR/TMP/TEMP, HOME/USERPROFILE, the shell-startup / interpreter-home group BASH_ENV, SHELLOPTS, PS4, BASHOPTS, PROMPT_COMMAND, ZDOTDIR, PYTHONHOME, PERLLIB, GCONV_PATH (sh/ksh `ENV` deliberately excluded: interactive-only and a routine project key), EDITOR/VISUAL/PAGER, PYTHON/PERL/RUBY preload, proxy vars both spellings, CLAUDE_CONFIG_DIR, ANTHROPIC_*/OPENAI/OPENROUTER/LITELLM/OLLAMA/LMSTUDIO/LLAMA_SERVER(_RERANKER)_BASE_URL) — the families a cwd .env could plant for the programs gbrain spawns; a denylist that grows as families are identified. `quarantineCwdDotenv(env = process.env, dir = process.cwd(), { warn, assignments })` iterates the keys the cwd .env files ASSIGN, deletes each protected one present in `env`, prints ONE stderr line (`formatQuarantineWarning`, composed from the exported `CWD_DOTENV_REMEDIATION` that guardrails.ts reuses) naming them sorted, returns them sorted. INVARIANTS: key presence, never value match; the parser accepts every line shape Bun's loader does; in-process only — children spawned without an explicit `env` see Bun's startup snapshot, which is why `cli-preflight.ts` re-runs gbrain from a neutral cwd whenever the list is non-empty; fs/path only, no config import. Pinned by `test/env-trust.test.ts`; the GBRAIN list is guarded against rot by the structural `test/env-trust-protected-keys.test.ts` (every `process.env.GBRAIN_*` / `Bun.env.GBRAIN_*` / `env.GBRAIN_*` read in src/ whose name matches `_BIN|_CLI|_MODULE|_PATH|_HOME|_URL` suffix or `GBRAIN_ALLOW_` prefix must be on the list or carry a `cwd-dotenv-ok: <why>` annotation).
+- `src/core/env-trust.ts` — Trust boundary for environment variables Bun auto-loaded from the cwd `.env` family. `CWD_DOTENV_FILES` (the eight names Bun auto-loads across NODE_ENV values), `splitDotenvLines` (Bun's line grammar: `\n`, `\r\n` AND a bare `\r` terminate a line, leading BOM stripped — shared with gbrain-env-file.ts), `dotenvValuesForKey(key, dir)` (value set across all assignments; the #427 `DATABASE_URL` value-match guard in config.ts, which re-exports both), `cwdDotenvAssignsKey(key, dir | assignments)` (KEY-PRESENCE predicate; an empty assignment counts) and `parseCwdDotenv(dir)` → `DotenvAssignment[]` (one read the two preflight consumers share). The assignment RHS is matched as `([^\r\n]*)`, never `.*$`. Protection is one predicate, `isCwdDotenvProtectedKey(key)`: `CWD_DOTENV_PROTECTED_KEYS` (code-loading / exec-target / root- and endpoint-redirect incl. `GBRAIN_DATABASE_URL` (key-presence; the #427 config.ts VALUE guard is separate and covers only bare `DATABASE_URL`), `GBRAIN_DIRECT_DATABASE_URL`, `GBRAIN_REMOTE_MCP_URL`, `GBRAIN_REMOTE_ISSUER_URL`, `GBRAIN_OAUTH_RELAY_URL` / posture-widening `GBRAIN_*`, plus the re-run marker `GBRAIN_CWD_ENV_QUARANTINED`; one why-comment per key, plus the deliberate exclusions and deferrals) PLUS `CWD_DOTENV_PROTECTED_PREFIXES` (`LD_`, `DYLD_`, `GIT_`, `BUN_`, `NPM_CONFIG_`, `npm_config_`, `BASH_FUNC_`, `OPENCLAW_`) and `CWD_DOTENV_PROTECTED_TOOLCHAIN_KEYS` (NODE_OPTIONS/NODE_PATH/CA/TLS, SSH_ASKPASS*, XDG_CONFIG/DATA/CACHE_HOME, GNUPGHOME, SSL_CERT_FILE/DIR, CURL_CA_BUNDLE, REQUESTS_CA_BUNDLE, OPENSSL_CONF/ENGINES/MODULES, TMPDIR/TMP/TEMP, HOME/USERPROFILE, the shell-startup / interpreter-home group BASH_ENV, SHELLOPTS, PS4, BASHOPTS, PROMPT_COMMAND, ZDOTDIR, PYTHONHOME, PERLLIB, GCONV_PATH (sh/ksh `ENV` deliberately excluded: interactive-only and a routine project key), EDITOR/VISUAL/PAGER, PYTHON/PERL/RUBY preload, proxy vars both spellings, CLAUDE_CONFIG_DIR, ANTHROPIC_*/OPENAI/OPENROUTER/LITELLM/OLLAMA/LMSTUDIO/LLAMA_SERVER(_RERANKER)_BASE_URL) — the families a cwd .env could plant for the programs gbrain spawns; a denylist that grows as families are identified. `quarantineCwdDotenv(env = process.env, dir = process.cwd(), { warn, assignments })` iterates the keys the cwd .env files ASSIGN, deletes each protected one present in `env`, prints ONE stderr line (`formatQuarantineWarning`, composed from the exported `CWD_DOTENV_REMEDIATION` that guardrails.ts reuses) naming them sorted, returns them sorted. INVARIANTS: key presence, never value match; the parser accepts every line shape Bun's loader does; in-process only — children spawned without an explicit `env` see Bun's startup snapshot, which is why `cli-preflight.ts` re-runs gbrain from a neutral cwd whenever the list is non-empty; fs/path only, no config import. Pinned by `test/env-trust.test.ts`; the GBRAIN list is guarded against rot by the structural `test/env-trust-protected-keys.test.ts` (every `process.env.GBRAIN_*` / `Bun.env.GBRAIN_*` / `env.GBRAIN_*` read in src/ whose name matches `_BIN|_CLI|_MODULE|_PATH|_HOME|_URL` suffix or `GBRAIN_ALLOW_` prefix must be on the list or carry a `cwd-dotenv-ok: <why>` annotation).
- `src/core/grants/` — canonical OAuth client grant model shared by CLI, admin HTTP, provisioning, and worker admission. `model.ts` validates active sources, independent direct/delegated fences, explicit reviewed tools, namespaces, concurrency, spend, and TTL; `host`/`current` brain aliases normalize to the serving brain (`null`), while other IDs need independently verified host identity. `profiles.ts` snapshots current eligible operation names for six explicit profiles; new defaults are renewable 1 hour (static 30 days), concurrency 1, unlimited delegated spend, and an operator-set `full` surface for every profile except `delegating-agent` (`starter`); only an explicit profile application changes an existing client's surface. `catalogProvenance()` is the operation catalog a snapshot is written against: `service.ts` adds it as `catalogProvenance` to the audit document of every write that sets a new snapshot (registration included), and `readClientCatalogProvenance` returns it only while that audit row's snapshot is still the client's snapshot (null for grants written without it; never synthesized). `allowed_operations = NULL` preserves legacy scope-based behavior; a new profile missing its snapshot fails closed. `service.ts` locks the client, checks `grant_revision`, and commits the grant, secret-free audit, and original ceilings for pending/active legacy jobs atomically; stale edits fail, repeated repair is a no-op, and repair preserves populated limits. Issued scopes intersect live scopes on access and refresh; source/path/operation/delegation changes are live, newly added scopes require new issuance, and TTL changes do not rewrite existing expiry. `migration.ts` splits legacy fences and removes only invalid `agent` grants with repair reasons; it loads registry validation lazily only after locking existing delegated clients, so empty engine imports and cold migrations do not register unrelated background sinks. This justified engine-live import is line-marked; constraints still install when no clients exist. F3 one grant shape: `model.ts` also holds `PrincipalGrant` (`GrantSources` = `default` | `scalar` | `federated` | `none`), `grantFromTokenRow` (unified rows read the `access_tokens` grant columns and any axis whose `permissions` mirror disagrees evaluates deny-all and is listed in `drift`; an unconverted `source_grant IS NULL` row reads as the conversion `migrateLegacyTokens` would write, and a malformed `permissions` value denies every axis), `GRANT_MIRROR_WINDOW_ENDS` (the date until which the `permissions` mirror is written and drift is enforced; doctor reports it), `authSourcesFromGrant` (the AuthInfo source fields every token reader uses: both HTTP transports, publication `authority.ts`, job admission, shared-skill membership and policy), `grantFromClient`, `validatePrincipalGrant` (tokens and clients may hold `none`); `ClientGrant.sourcesNone` (`oauth_clients.source_grant = 'none'`: NULL `source_id`, empty `federated_read`, verified as `NO_SOURCES`) and `ClientGrant.takesHolders` (`oauth_clients.takes_holders`, NULL = `['world']`; read by `verifyAccessToken` and `authorizeTakeHolder`), both added by migration v203 (fragment `oauth-client-axes-schema.ts`) and set through `auth rescope --client --sources none|--takes-holders` and `permissionsMirror`. `access-token-schema.ts` is the column fragment (migration `f3_access_token_grants`); v202 (`legacy_token_grant_conversion`) converts every legacy grant in bulk. Tests: `test/legacy-token-grant-conversion.test.ts`. `legacy-token.ts` is the token grant editor behind `gbrain auth rescope --token` (aliases `rescope-token`, `permissions set-takes-holders`): one `FOR UPDATE` transaction that writes the columns, bumps `grant_revision` (`--if-version` compare-and-set) and rewrites `permissions` as a mirror with every other key preserved; `none` = explicit empty list; refresh previews and widens only by named operations; a drifted token refuses edits until `--adopt-permissions` or `--adopt-columns`; `migrateLegacyTokens` (`--migrate-legacy` and schema migration v202) writes the columns for every unmigrated token without changing a grant and skips malformed JSONB; `resolveTokenGrant` is the HTTP auth read (both transports): a legacy-shape row converts on that read with one `source_grant IS NULL`-guarded UPDATE (concurrent first reads convert once; a failed write such as a read-only role authorizes with the identical in-memory conversion); `resolveRescopeTarget` refuses a bare name that matches a token and a client. `cli.ts` splits `auth rescope` targets and maps the unified client flags onto `auth rescope-client`'s patch. An explicit empty `permissions.source_id` decodes (`parseLegacyTokenScope` in `src/core/legacy-token-scope.ts`) to the `NO_SOURCES` sentinel, which `sourceScopeOpts`, the dispatch fence (`enforceBoundClientOpAllowList`, `fence=no_source_grant`), `submissionAuthority` and `POST /ingest` refuse and publication reauthorization rejects; it never regains the `default` floor. `schema.ts` supplies migration147/bootstrap columns and invariants for both engines. SQL JSON passes through `text::jsonb` for driver parity. Tests: `test/client-grants.test.ts`, OAuth compatibility suites, and `test/e2e/client-grants.test.ts` (Postgres CAS, job ceilings, and actual HTTP preview/edit). GBRA-156 private-pages axis: `PrivatePagesGrant` (`none` | `granted_sources`) lives in `access_tokens.private_pages` and `oauth_clients.private_pages` (migration v280, never in the legacy permissions mirror); `privatePagesSources` counts only explicitly named sources (scalar write source, federated write plus read sources; never the `default` floor), and `privatePagesAuth` builds `AuthInfo.privatePages` for HTTP and OAuth verification; MCP dispatch runs a remote read under `withPrivateReadableSources` (`src/core/search/private-visibility.ts`, an AsyncLocalStorage), whose `*ForCaller` predicate twins and `orPrivateReadableSource` OR those sources in at read sites only, never at stored-visibility decisions.
diff --git a/docs/guides/conversation-archive.md b/docs/guides/conversation-archive.md
new file mode 100644
index 0000000000..8f5b67f3bf
--- /dev/null
+++ b/docs/guides/conversation-archive.md
@@ -0,0 +1,106 @@
+# Conversation archive
+
+The conversation archive writes one redacted page per conversation per local day, read from your agent harnesses
+(Claude Code, Codex, OpenClaw, Hermes, Grok), plus one index page per day. It is **off** until you turn it on, and
+it only runs when you run it: nothing is scheduled yet.
+
+```bash
+gbrain transcripts archive on # turn it on for this brain
+gbrain transcripts archive run --dry-run # preview the last two local days
+gbrain transcripts archive run # write them
+gbrain transcripts archive backfill --from 2026-09-01 # older history, newest days first
+gbrain transcripts archive status # readers, timezone, archived and held days
+gbrain transcripts archive off # stop; pages already written stay
+```
+
+Every command takes `--json` and `--source <id>` (the archive source, `conversations` by default).
+
+## What it writes
+
+- `conversations/<harness>/<YYYY-MM-DD>/<label>-<hash8>`: the messages of one conversation that fall on that local
+ day, in the same speaker format as `gbrain transcripts ingest`. A long day splits into `-p2`, `-p3` parts.
+- `conversations/days/<YYYY-MM-DD>`: an index of every conversation archived for that day.
+
+Frontmatter carries `transcript_archive` (harness, day, timezone, message count, first and last timestamp, whether
+the day is complete) beside the usual `transcript_import` block.
+
+Running the archive again on the same input changes nothing. A day still in progress is rewritten in place as it
+grows, and closes once its last message is older than `transcripts.archive.quiet_minutes` (default 180).
+
+## Days and timezone
+
+Days are local days in `transcripts.archive.timezone` (an IANA name such as `America/Los_Angeles`; the machine's
+zone when unset). The first run pins the zone for the archive source, so a later timezone change never re-slices
+days already written. `run` covers today and the previous `transcripts.archive.run_window_days` days (default 2).
+
+## Backfill
+
+`backfill` needs `--from` (and takes `--to`, default today). One run writes at most
+`transcripts.archive.backfill_days_per_run` days, newest first; `status` and the run output say how many wait for
+the next run, and the next run skips the days already finished. A run reads each session file once to learn its
+days, then reads again only the conversations on the days it writes, and writes them in batches, so a folder of
+thousands of sessions stays bounded in memory. Backfilled pages are searchable by keyword at once, and their
+embedding is held until you approve its cost with `gbrain embed --held`.
+
+## Old archive folders
+
+If an earlier tool already archived conversations into a folder, `adopt-legacy` lists the days it covers so
+backfill skips them instead of writing a second copy:
+
+```bash
+gbrain transcripts archive adopt-legacy --dir ~/brain/conversations # preview, writes nothing
+gbrain transcripts archive adopt-legacy --dir ~/brain/conversations --apply # record the covered days
+```
+
+The old pages are never changed or deleted.
+
+## Privacy
+
+Every page is redacted before anything is written (the same secret scan and your pattern file as
+`gbrain transcripts ingest`). If redaction fails for a conversation, its days are held with
+`archive_redaction_failed` and retried on the next run; a day is never written unredacted.
+
+On a personal brain, archive pages carry no `readable_by`. On a shared brain (a non-personal audience, or two or
+more enabled members), every page names its readers:
+
+- the owner member always reads it. With no owner named, every day is held with `archive_owner_unset` until you
+ run `gbrain access owner set <member>`;
+- a group chat whose chat type the harness verified also lists `group:chat-<hash8>`, where the hash is the first
+ 8 hex characters of sha256 of `<harness>:<conversation key>`, but only when that group already exists. Create
+ the group (`gbrain access group add chat-<hash8>`) and add members to it to share that chat's pages with them.
+
+On a shared brain whose archive source is a git checkout, pages go to a database-only `<source>-owner` source so
+private conversations never land in the shared repository.
+
+## OpenClaw
+
+Current OpenClaw keeps live sessions in a database owned by its Gateway. gbrain reads them with
+`openclaw gateway call` (`sessions.list`, then `chat.history` per session), so the `openclaw` CLI must be on
+`PATH`. Point it at a Gateway with `GBRAIN_OPENCLAW_GATEWAY_URL` and pass its token with
+`GBRAIN_OPENCLAW_GATEWAY_TOKEN`; gbrain hands both to `openclaw` through its environment (`OPENCLAW_GATEWAY_URL`,
+`OPENCLAW_GATEWAY_TOKEN`), never on the command line. Errors map to `archive_gateway_unreachable`,
+`archive_gateway_auth` and `archive_gateway_failed`; a failed read leaves the archive ledger unchanged, so the next
+run retries. Checked against OpenClaw 2026.10.9 and 2026.11.3, which list sessions and return history the same way.
+
+Session files OpenClaw keeps on disk are read too, even when the Gateway is down: `<id>.jsonl.reset.<stamp>`, the
+zstd-compressed `<id>.jsonl.deleted.<stamp>.<hex>.zst` and the legacy compact backup `<id>.jsonl.bak.<stamp>`.
+Several files of one session merge into one conversation (`gbrain transcripts ingest` reads only live files). Migration copies (`*.migrated`, `*.migrated.<n>`,
+`<id>.jsonl.pre-doctor-*-repair-<stamp>.bak`) and Doctor's recovery copies (`<id>.jsonl.bak-<pid>-<stamp>`) are
+skipped: their sessions live in the Gateway's database, which the next run with the Gateway up reads.
+
+Three OpenClaw limits to know:
+
+- `sessions.list` does not list archived sessions, so a session archived before gbrain first read it is only
+ found through its file on disk.
+- On 2026.10.9 a session created by an operator has no `chatType`, so the archive treats it as unverified and
+ keeps it to the owner.
+- `sessions.reset` keeps the same session id and the Gateway then returns only the messages after the reset. The
+ archive merges the `.jsonl.reset.<stamp>` file of the same session with the Gateway's history, each message once,
+ so the earlier messages stay. When neither holds a message the day page already has, the page is kept as it is
+ (day status `kept`), so a reset never shrinks or replaces the archive.
+
+## Repo-less brains
+
+A brain with no git repository archives straight into the database; the archive source is created database-only
+when it does not exist. On a personal brain whose archive source is a git checkout, the archive writes the files,
+commits only those paths with `gbrain git commit-push`, and syncs the checkout.
diff --git a/docs/guides/data-ingestion.md b/docs/guides/data-ingestion.md
index 6649137f81..b40c1abdf6 100644
--- a/docs/guides/data-ingestion.md
+++ b/docs/guides/data-ingestion.md
@@ -81,6 +81,8 @@ gbrain transcripts ingest --max-bytes 4gb <store> # oversized store; om
gbrain transcripts status # found vs imported, per harness
```
+For one redacted page per conversation per local day plus a daily index (off until `gbrain transcripts archive on`), see the [conversation archive](conversation-archive.md).
+
**Say to your agent:** *"Import my conversations from my chatgpt export at ~/Downloads/conversations.json"* — *"Archive my session transcripts"* — and later, *"When did I first discuss agent memory?"* (the archive answers origin questions with dated quotes).
Codex sessions imported before gbrain read codex 0.153+ rollouts lost their
diff --git a/docs/guides/error-codes.md b/docs/guides/error-codes.md
index a17b4592c3..e13e4afd03 100644
--- a/docs/guides/error-codes.md
+++ b/docs/guides/error-codes.md
@@ -342,6 +342,76 @@ More: [docs/guides/google-connect.md#troubleshooting](../../docs/guides/google-c
|---|---|---|---|---|---|---|
| The change needs the owner's explicit approval before it can proceed. | The step needs the user's decision before it runs. | Stop and ask the user; re-run only with the authorization the message names. | user | `repeat the read that failed` | 1 | no |
+### archive_audience_unproven
+
+<a id="archive_audience_unproven"></a>
+
+| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable |
+|---|---|---|---|---|---|---|
+| The conversation archive could not check whether the brain is shared (its client-usage read failed), so it wrote nothing this run. | A brain with no audience declared counts as shared when several clients use it; without that read a page written as personal could be open to other members, so the run publishes nothing. | Rerun gbrain transcripts archive run; if it repeats, declare the audience with gbrain config set brain.audience personal\|shared. | agent | `repeat the read that failed` | 1 | yes |
+
+More: [docs/guides/conversation-archive.md#privacy](../../docs/guides/conversation-archive.md#privacy)
+
+### archive_gateway_auth
+
+<a id="archive_gateway_auth"></a>
+
+| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable |
+|---|---|---|---|---|---|---|
+| The OpenClaw Gateway refused the credential the conversation archive used, so no OpenClaw conversation was read this run; the archive ledger is unchanged. | The Gateway token or device pairing belongs to the user; gbrain never mints or edits one. | Ask the user to give gbrain the Gateway token (GBRAIN_OPENCLAW_GATEWAY_TOKEN, or gateway.auth in the OpenClaw config on this machine), then rerun gbrain transcripts archive run. Run: gbrain transcripts archive status --json | user | `gbrain doctor --json` | 1 | no |
+
+More: [docs/guides/conversation-archive.md#openclaw](../../docs/guides/conversation-archive.md#openclaw)
+
+### archive_gateway_failed
+
+<a id="archive_gateway_failed"></a>
+
+| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable |
+|---|---|---|---|---|---|---|
+| The OpenClaw Gateway answered the conversation archive with an error, so no OpenClaw conversation was read this run; the archive ledger is unchanged. | A temporary condition (a lock, a pending write, a rate limit); the same request can succeed later. | Rerun gbrain transcripts archive run; if it repeats, report the redacted message from gbrain transcripts archive status --json. Run: gbrain transcripts archive status --json | agent | `repeat the read that failed` | 1 | yes |
+
+More: [docs/guides/conversation-archive.md#openclaw](../../docs/guides/conversation-archive.md#openclaw)
+
+### archive_gateway_unreachable
+
+<a id="archive_gateway_unreachable"></a>
+
+| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable |
+|---|---|---|---|---|---|---|
+| The conversation archive could not reach the OpenClaw Gateway, so no OpenClaw conversation was read this run; the archive ledger is unchanged, so the next run retries safely. | Current OpenClaw keeps live sessions in SQLite owned by its Gateway, and gbrain reads them only through the Gateway's documented RPCs (openclaw gateway call), never the file. | Start the Gateway (openclaw gateway) or point gbrain at it with GBRAIN_OPENCLAW_GATEWAY_URL, then rerun gbrain transcripts archive run. Run: gbrain transcripts archive status --json | agent | `repeat the read that failed` | 1 | yes |
+
+More: [docs/guides/conversation-archive.md#openclaw](../../docs/guides/conversation-archive.md#openclaw)
+
+### archive_owner_unset
+
+<a id="archive_owner_unset"></a>
+
+| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable |
+|---|---|---|---|---|---|---|
+| On a shared brain the conversation archive held every day and wrote nothing, because no owner member is named and an archive page without readable_by would be open to the whole source. | Only the operator of the brain host can change what blocks this. | Ask the user which enabled member owns the brain, run gbrain access owner set <member>, then rerun the archive; held days are written then. Run: gbrain access owner | user | `gbrain doctor --json` | 1 | no |
+
+More: [docs/guides/conversation-archive.md#privacy](../../docs/guides/conversation-archive.md#privacy)
+
+### archive_redaction_failed
+
+<a id="archive_redaction_failed"></a>
+
+| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable |
+|---|---|---|---|---|---|---|
+| Redaction failed for a conversation, so the conversation archive held its days instead of writing them; the next run retries them. | Archive pages are written only after redaction succeeded over the whole session; a day is never written unredacted and never dropped. | Rerun gbrain transcripts archive run; if the day stays held, check the user redaction pattern file named in the held message. Run: gbrain transcripts archive status --json | agent | `repeat the read that failed` | 1 | yes |
+
+More: [docs/guides/conversation-archive.md#privacy](../../docs/guides/conversation-archive.md#privacy)
+
+### archive_run_busy
+
+<a id="archive_run_busy"></a>
+
+| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable |
+|---|---|---|---|---|---|---|
+| Another conversation archive run holds this source, so this run read and wrote nothing. | A run decides each day from the archive ledger it read when it started, so two runs at once could publish older history over newer; runs on one source take turns on a database lease. | Wait for the other run to finish, then rerun gbrain transcripts archive run. Run: gbrain status --section locks --json | agent | `repeat the read that failed` | 1 | yes |
+
+More: [docs/guides/conversation-archive.md#openclaw](../../docs/guides/conversation-archive.md#openclaw)
+
### atom_drain_incomplete
<a id="atom_drain_incomplete"></a>
diff --git a/plugin-variants/gbrain-daily/skills/conversation-archive/SKILL.md b/plugin-variants/gbrain-daily/skills/conversation-archive/SKILL.md
index 0a35a7a272..e646443546 100644
--- a/plugin-variants/gbrain-daily/skills/conversation-archive/SKILL.md
+++ b/plugin-variants/gbrain-daily/skills/conversation-archive/SKILL.md
@@ -76,6 +76,7 @@ gbrain transcripts ingest ~/Downloads/conversations.json # unzip first
gbrain transcripts ingest # discover harness logs
gbrain transcripts ingest --max-bytes 4gb <store> # oversized store (omit = per-format caps)
gbrain transcripts status # found vs imported gaps
+gbrain transcripts archive run # one page per conversation per local day (off until `archive on`)
```
`--max-bytes` note: the cap is part of the `--since last` checkpoint
diff --git a/plugin/skills/conversation-archive/SKILL.md b/plugin/skills/conversation-archive/SKILL.md
index 0a35a7a272..e646443546 100644
--- a/plugin/skills/conversation-archive/SKILL.md
+++ b/plugin/skills/conversation-archive/SKILL.md
@@ -76,6 +76,7 @@ gbrain transcripts ingest ~/Downloads/conversations.json # unzip first
gbrain transcripts ingest # discover harness logs
gbrain transcripts ingest --max-bytes 4gb <store> # oversized store (omit = per-format caps)
gbrain transcripts status # found vs imported gaps
+gbrain transcripts archive run # one page per conversation per local day (off until `archive on`)
```
`--max-bytes` note: the cap is part of the `--since last` checkpoint
diff --git a/scripts/core-world-reads-baseline.tsv b/scripts/core-world-reads-baseline.tsv
index afd4a347a4..f81b6181f7 100644
--- a/scripts/core-world-reads-baseline.tsv
+++ b/scripts/core-world-reads-baseline.tsv
@@ -1170,6 +1170,9 @@ src/core/transcription.ts fs_import 2 W7: filesystem boundary moves to the local
src/core/transcription.ts process_env 5 W6: settings plane out of the core
src/core/transcription.ts spawn_import 2 W7: process-wide effects move to src/cli
src/core/transcripts.ts fs_import 1 W7: filesystem boundary moves to the local app
+src/core/transcripts/archive-readers.ts fs_import 1 W7: filesystem boundary moves to the local app
+src/core/transcripts/archive-readers.ts process_env 1 W6: settings plane out of the core (GBRAIN_OPENCLAW_GATEWAY_URL picks the Gateway reader)
+src/core/transcripts/archive.ts fs_import 1 W7: filesystem boundary moves to the local app (repo-source archive files, legacy folder scan)
src/core/transcripts/claude-code-jsonl.ts fs_import 1 W7: filesystem boundary moves to the local app
src/core/transcripts/claude-code.ts fs_import 1 W7: filesystem boundary moves to the local app
src/core/transcripts/codex-hook-lane.ts fs_import 1 W7: filesystem boundary moves to the local app
@@ -1181,6 +1184,7 @@ src/core/transcripts/export-json.ts fs_import 1 W7: filesystem boundary moves to
src/core/transcripts/grok.ts fs_import 1 W7: filesystem boundary moves to the local app
src/core/transcripts/hermes.ts fs_import 1 W7: filesystem boundary moves to the local app
src/core/transcripts/openclaw.ts fs_import 1 W7: filesystem boundary moves to the local app
+src/core/transcripts/openclaw-gateway.ts process_env 2 W6: settings plane out of the core (Gateway URL and token env, passed to the openclaw child)
src/core/transcripts/recover.ts fs_import 1 W7: filesystem boundary moves to the local app
src/core/trust/marker-repair.ts fs_import 1 W7: filesystem boundary moves to the local app
src/core/trust/page-handlers.ts fs_import 1 W7: filesystem boundary moves to the local app (#7960 restores the page file a failed owner repair wrote)
diff --git a/skills/conversation-archive/SKILL.md b/skills/conversation-archive/SKILL.md
index 2f8bc4e886..babc27d6b5 100644
--- a/skills/conversation-archive/SKILL.md
+++ b/skills/conversation-archive/SKILL.md
@@ -75,6 +75,7 @@ gbrain transcripts ingest ~/Downloads/conversations.json # unzip first
gbrain transcripts ingest # discover harness logs
gbrain transcripts ingest --max-bytes 4gb <store> # oversized store (omit = per-format caps)
gbrain transcripts status # found vs imported gaps
+gbrain transcripts archive run # one page per conversation per local day (off until `archive on`)
```
`--max-bytes` note: the cap is part of the `--since last` checkpoint
diff --git a/skills/skills.lock.history.json b/skills/skills.lock.history.json
index 3b330d701a..cede221e90 100644
--- a/skills/skills.lock.history.json
+++ b/skills/skills.lock.history.json
@@ -383,6 +383,7 @@
"895b63ecc46ab7bd173ba9bb8d0cb740f355fbfaf429584b74842823edf68a42",
"90bbc9f5b3e8dd915e3bcda6e0a49fdfbaa3d1ba16e61040a21e3d5b665dd7c9",
"a59d3dc7076871b8605b5be0cd48fb0b7ae5c0d97bc65447233d2addc2680ac4",
+ "cf7be090bff1f77dde978d78f2ec27f737596566a9da90638d38fb9b44784bdc",
"d156d71460b91d8b4f878b26fb5f2cf129ccbe2083b2b2dcdd8ed1d2b86162dc",
"de2b44da329c4601da8c77536a627b094e52a93c8f45032335571b29b311d736",
"df603c367ef604b9b0a326b04d1c91a67dca87418431080f21f1122ed94b9960",
diff --git a/skills/skills.lock.json b/skills/skills.lock.json
index c452c67a32..c04a11cf0f 100644
--- a/skills/skills.lock.json
+++ b/skills/skills.lock.json
@@ -63,7 +63,7 @@
"conventions/subagent-routing.md": "8b8830b815a9a8581a12b489f966c0b0a39eb9b5f66e905a691a03653eef348d",
"conventions/test-before-bulk.md": "6b2c52cda9e2cd5f04c15152b3d92aeb7187ab193a15082be0f8a3991a6a5725",
"conventions/untrusted-content.md": "e3c8feae0510039d13991a139c6a0d857647a6db50206c2940d829e069649d35",
- "conversation-archive/SKILL.md": "d156d71460b91d8b4f878b26fb5f2cf129ccbe2083b2b2dcdd8ed1d2b86162dc",
+ "conversation-archive/SKILL.md": "cf7be090bff1f77dde978d78f2ec27f737596566a9da90638d38fb9b44784bdc",
"conversation-archive/routing-eval.jsonl": "ae087a84b1fd5b108b7cdab8d035a09b3ccecd8aad53ba5f71e463059108cfca",
"correction-pipeline/SKILL.md": "04b4aee06d7fcdd75db314da432ed7c08b5fbc20f117f18062c64aaf795bc564",
"correction-pipeline/routing-eval.jsonl": "7f8d96606a8d7bed3d79fdcee6904764c8abb9fa0b506adb414b5c4805b69d0b",
@@ -226,7 +226,7 @@
"skillpack-check/SKILL.md": "0d2d2d201ed72e19b47d7e3743b5bbdfb5c2655d1f34d29447c601fca3908f9e",
"skillpack-harvest/SKILL.md": "0f86314b58d25b5511809f567c04aedc3fba6fe35ed642ac52d8216ae365b6eb",
"skillpack-harvest/routing-eval.jsonl": "cb4783288e95af3132b32ecb40a54cffc095f57b36240f96a25c2d2adf5e68c6",
- "skills.lock.history.json": "b94d36832095a449c2ff32d676b7a1d9f620e53729f45d7f8f003f504ae4e91f",
+ "skills.lock.history.json": "28f2522cc52fbc4f1b1828b22fe31304790067b1748c04672f2dfa1a3f8edc4f",
"smoke-test/SKILL.md": "f5099e1d2ab9dc6827096249e68414f7ae357c29b39fb069a54bb7fb74f414a5",
"soul-audit/SKILL.md": "7a48e180849f7502d332878c443397f90a3756f45caa94f76bcfcc7c997469c8",
"storage-manifest/SKILL.md": "1debbefba61007a17cda1525496875d1edfde90642adf3ec031f25a71724807e",
diff --git a/src/commands/transcripts-archive.ts b/src/commands/transcripts-archive.ts
new file mode 100644
index 0000000000..c07b0d6333
--- /dev/null
+++ b/src/commands/transcripts-archive.ts
@@ -0,0 +1,111 @@
+/**
+ * gbrain transcripts archive — the day-sliced conversation archive (GST-2768). Off unless the owner turns it on.
+ *
+ * run archive the last few local days from every readable harness
+ * backfill --from D [--to D] archive a date range, newest first, capped per run; embedding held for approval
+ * status readers, pinned timezone, archived and held days
+ * adopt-legacy --dir <folder> list the days an old archive folder covers (--apply records them so backfill skips them)
+ * on | off turn the archive on or off for this brain
+ *
+ * Core: src/core/transcripts/archive.ts. Trusted local CLI only; no MCP operation.
+ */
+
+import type { BrainEngine } from '../core/engine.ts';
+import type { CliDispatchContext } from '../cli/command-table.ts';
+import { setCliExitVerdict } from '../core/cli-force-exit.ts';
+
+const HELP = `Usage:
+ gbrain transcripts archive run [--dry-run] [--source S] [--json]
+ gbrain transcripts archive backfill --from YYYY-MM-DD [--to YYYY-MM-DD] [--dry-run] [--source S] [--json]
+ gbrain transcripts archive status [--source S] [--json]
+ gbrain transcripts archive adopt-legacy --dir <folder> [--dry-run | --apply] [--source S] [--json]
+ gbrain transcripts archive on | off [--json]
+
+Writes one redacted page per conversation per local day (conversations/<harness>/<day>/...) plus a day index
+(conversations/days/<day>) into the archive source (default "conversations"). Off unless turned on with
+"gbrain transcripts archive on". Backfilled pages are keyword-only until "gbrain embed --held" approves their cost.
+Guide: docs/guides/conversation-archive.md`;
+
+function flag(args: string[], name: string): string | undefined {
+ const i = args.indexOf(name);
+ return i >= 0 ? args[i + 1] : undefined;
+}
+
+export async function runTranscriptsArchive(engine: BrainEngine, args: string[], dispatch: Pick<CliDispatchContext, 'makeContext'> = {}): Promise<void> {
+ const sub = args[0];
+ const rest = args.slice(1);
+ const json = rest.includes('--json');
+ const sourceId = flag(rest, '--source');
+ const out = (doc: unknown, human: () => void) => { if (json) console.log(JSON.stringify(doc, null, 2)); else human(); };
+ const archive = await import('../core/transcripts/archive.ts');
+
+ if (!sub || sub === '--help' || sub === '-h' || rest.includes('--help') || rest.includes('-h')) {
+ console.log(HELP);
+ if (sub && sub !== '--help' && sub !== '-h' && !rest.includes('--help') && !rest.includes('-h')) setCliExitVerdict(2);
+ return;
+ }
+
+ if (sub === 'on' || sub === 'off') {
+ await archive.setArchiveEnabled(engine, sub === 'on');
+ out({ enabled: sub === 'on' }, () => console.log(sub === 'on'
+ ? 'conversation archive: on. Run "gbrain transcripts archive run" now, or "gbrain transcripts archive backfill --from <day>" for history.'
+ : 'conversation archive: off. Pages already written stay; nothing new is archived.'));
+ return;
+ }
+
+ if (sub === 'status') {
+ const status = await archive.archiveStatus(engine, { sourceId });
+ out(status, () => {
+ console.log('conversation archive: ' + (status.enabled ? 'on' : 'off') + ' (source: ' + status.source_id + ', timezone: ' + (status.timezone ?? status.configured_timezone + ', pinned at the first run') + ')');
+ for (const h of status.harnesses) console.log(' ' + h.reader.padEnd(18) + ' ' + (h.info ?? (h.conversations + ' conversation(s), newest session day ' + (h.newest_session_day ?? '-') + ', newest archived day ' + (h.newest_archived_day ?? '-'))));
+ console.log(' archived days: ' + status.archived_days + (status.legacy_days ? ', covered by a legacy archive: ' + status.legacy_days : ''));
+ for (const h of status.held) console.log(' held: ' + h.harness + ' ' + h.day + ' (' + h.code + ')');
+ for (const e of status.errors) console.log(' error: ' + e.reader + ' ' + e.code + ': ' + e.message);
+ });
+ if (status.errors.length) setCliExitVerdict(1);
+ return;
+ }
+
+ if (sub === 'adopt-legacy') {
+ const dir = flag(rest, '--dir');
+ if (!dir) {
+ const { exitCliError, usageError } = await import('../cli/cli-error.ts');
+ exitCliError(usageError('adopt-legacy needs --dir <folder>.', 'Run gbrain transcripts archive adopt-legacy --dir <old archive folder> --dry-run.'), 'transcripts', { json });
+ return;
+ }
+ const result = await archive.adoptLegacyArchive(engine, { dir, apply: rest.includes('--apply'), sourceId });
+ out(result, () => {
+ console.log((result.applied ? 'adopted' : 'would adopt (dry run; pass --apply to record)') + ' ' + result.covered.length + ' day(s) from ' + result.dir);
+ for (const [h, n] of Object.entries(result.by_harness)) console.log(' ' + (h === '*' ? 'every harness' : h).padEnd(14) + ' ' + n + ' day(s)');
+ console.log('The old files are left as they are. Remove the old scheduled script yourself, then run "gbrain transcripts archive backfill --from <day>" for the gap.');
+ });
+ return;
+ }
+
+ if (sub !== 'run' && sub !== 'backfill') {
+ console.log(HELP);
+ setCliExitVerdict(2);
+ return;
+ }
+ const dryRun = rest.includes('--dry-run');
+ const makeContext = dispatch.makeContext;
+ const result = await archive.runTranscriptArchive(engine, {
+ mode: sub, dryRun, sourceId,
+ ...(makeContext ? { makeContext: (source: string) => makeContext(engine, { source }) } : {}),
+ ...(sub === 'backfill' ? { from: flag(rest, '--from'), to: flag(rest, '--to') } : {}),
+ });
+ out(result, () => {
+ if (result.message) console.log(result.message);
+ if (result.fix?.argv) console.log(' next: ' + result.fix.argv.join(' '));
+ if (!result.enabled && !result.dry_run) return;
+ console.log('conversation archive ' + sub + (result.dry_run ? ' (dry run)' : '') + ': ' + result.from + ' to ' + result.to + ' in ' + result.timezone + ', source ' + result.source_id + ' (' + result.write_mode + ')');
+ console.log(' days written ' + result.written + ', unchanged ' + result.unchanged + ', held ' + result.held + ', planned ' + result.planned + ', covered by legacy ' + result.skipped_legacy);
+ const redacted = result.days.filter(d => (d.redactions ?? 0) > 0);
+ if (redacted.length) console.log(' redacted values on ' + redacted.length + ' page(s): ' + redacted.map(d => d.slug + ' (' + d.redactions + ')').join(', '));
+ for (const d of result.days.filter(d => d.status === 'held' || d.status === 'error')) console.log(' ' + d.status + ': ' + d.harness + ' ' + d.day + ' (' + d.code + ')');
+ for (const line of result.info) console.log(' ' + line);
+ for (const e of result.errors) console.log(' error: ' + e.reader + ' ' + e.code + ': ' + e.message);
+ if (sub === 'backfill' && result.written) console.log(' backfilled pages are keyword-only; preview their embedding cost with: gbrain embed --held --dry-run');
+ });
+ if (result.errors.length || result.days.some(d => d.status === 'error') || (result.held && result.fix)) setCliExitVerdict(1);
+}
diff --git a/src/commands/transcripts.ts b/src/commands/transcripts.ts
index c09df09b36..c8d3ff323f 100644
--- a/src/commands/transcripts.ts
+++ b/src/commands/transcripts.ts
@@ -197,6 +197,7 @@ const HELP = `Usage:
gbrain transcripts status # found vs imported gap table
gbrain transcripts recover codex # restore user turns lost to #5163 (preview; --apply)
gbrain transcripts audit-secrets # list imported pages that still carry a credential (read-only)
+ gbrain transcripts archive run|backfill|status|adopt-legacy|on|off # day-sliced conversation archive (off unless turned on)
gbrain transcripts recent [options]
ingest — import dead session logs and chat exports as conversation pages
@@ -627,6 +628,11 @@ export async function runTranscripts(engine: BrainEngine, args: string[], dispat
await runTranscriptsAuditSecrets(engine, args.slice(1));
return;
}
+ if (sub === 'archive') {
+ const { runTranscriptsArchive } = await import('./transcripts-archive.ts');
+ await runTranscriptsArchive(engine, args.slice(1), dispatch);
+ return;
+ }
if (sub === 'recover') {
const { runTranscriptsRecover } = await import('./transcripts-recover.ts');
await runTranscriptsRecover(engine, args.slice(1), dispatch);
@@ -635,7 +641,7 @@ export async function runTranscripts(engine: BrainEngine, args: string[], dispat
if (sub !== 'recent') {
if (sub !== '--help' && sub !== '-h' && args.includes('--json')) {
const { exitCliError, usageError } = await import('../cli/cli-error.ts');
- exitCliError(usageError(sub && !sub.startsWith('-') ? `Unknown transcripts subcommand: ${sub}` : 'gbrain transcripts needs a subcommand: ingest, status, recover, audit-secrets or recent.',
+ exitCliError(usageError(sub && !sub.startsWith('-') ? `Unknown transcripts subcommand: ${sub}` : 'gbrain transcripts needs a subcommand: ingest, status, recover, audit-secrets, archive or recent.',
'Run `gbrain transcripts recent --json` to list recent transcripts, or `gbrain transcripts --help`.'), 'transcripts', { json: true });
}
console.log(HELP);
diff --git a/src/core/cli-flag-registry.generated.ts b/src/core/cli-flag-registry.generated.ts
index 7d2470d467..bc2730d360 100644
--- a/src/core/cli-flag-registry.generated.ts
+++ b/src/core/cli-flag-registry.generated.ts
@@ -139,7 +139,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {
'sync': ['--abbrev-ref', '--absolute-git-dir', '--accept-pending', '--access-policy', '--account', '--acknowledge-managed', '--admin-intent', '--adopt-columns', '--adopt-permissions', '--after', '--alert', '--all', '--allow-empty', '--allow-outside-source-root', '--apply', '--apply-rewrites', '--as', '--assign-corpus', '--attest-read', '--audit', '--auto', '--background', '--batch', '--batch-check', '--batch-size', '--brain', '--branch', '--break-lock', '--budget-ms', '--by-mention', '--cached', '--catch-up', '--check', '--check-only', '--class', '--clear', '--client', '--close', '--code', '--concurrency', '--config', '--confirm-destructive', '--confirm-quiesced', '--confirm-relocated-root', '--conflicts', '--contacts-dir', '--content-audit', '--continue', '--conversation', '--count', '--cursor', '--cwd', '--db-url', '--delete-brain', '--depth', '--detach', '--detail', '--diff-filter', '--dim', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--empty', '--enable', '--end-of-options', '--entity', '--exclude', '--exclude-standard', '--expect', '--expect-local', '--expect-upstream', '--expected-epoch', '--expected-revision', '--expected-state', '--explain', '--facts', '--fast', '--ff-only', '--filter', '--filters', '--first-parent', '--fix', '--follow', '--force', '--force-break-lock', '--force-rechunk', '--force-retry', '--force-schema', '--format', '--fresh', '--from', '--from-meetings', '--full', '--full-history', '--get', '--get-regexp', '--git-dir', '--git-path', '--grammar', '--group-by', '--hard-deadline', '--harness', '--held', '--help', '--hold-state', '--home', '--http', '--id', '--ignore-missing', '--images', '--include-ambiguous', '--include-flagged', '--include-frontmatter', '--include-gitignored', '--include-hidden', '--include-null-signature', '--include-repairs', '--index-audit', '--infer-dates', '--input', '--install', '--instructions-only', '--interval', '--is-ancestor', '--is-bare-repository', '--is-inside-work-tree', '--is-shallow-repository', '--json', '--keep-checkout', '--kind', '--lanes', '--lang', '--left-right', '--legacy-lock', '--limit', '--link-source', '--link-type', '--list', '--local', '--locks', '--log-noop', '--manifest', '--markdown', '--max-age', '--max-cost', '--max-cost-usd', '--max-jobs', '--max-rss', '--max-sources', '--max-usd', '--may-act-for', '--migrate-legacy', '--migrate-only', '--missing-path', '--multimodal', '--name', '--name-only', '--name-status', '--ner', '--no-accept-pending', '--no-auto-embed', '--no-autopilot-install', '--no-bulk', '--no-capture', '--no-delegate', '--no-embed', '--no-extensions', '--no-extract', '--no-gpg-sign', '--no-hard-deadline', '--no-instructions', '--no-lanes', '--no-llm', '--no-pull', '--no-renames', '--no-repair', '--no-schema-pack', '--no-verify', '--no-walk', '--once', '--only', '--operations', '--others', '--output', '--oversized', '--overwrite', '--pace', '--pace-max-concurrency', '--pack', '--page', '--parallel', '--params', '--path', '--path-format', '--pending', '--pglite', '--phase', '--pid-file', '--plan', '--porcelain', '--prefer', '--prefer-postgres', '--prefix', '--preview', '--priority', '--privacy-url', '--probe', '--probe-pglite', '--progress-interval', '--progress-json', '--project', '--providers', '--prune-orphans', '--pull', '--purge', '--push', '--push-only', '--query', '--question', '--queue', '--quiet', '--raw', '--raw-only', '--reader-mode', '--readers', '--reads-unscoped', '--reauth', '--rebase', '--rebuild', '--redact-paths', '--refresh', '--regenerate', '--relative', '--remediate', '--remediation-plan', '--remote', '--remove', '--remove-core', '--replace', '--repo', '--request-id', '--reset', '--reset-checkpoint', '--reset-default', '--restart-only', '--restore-only', '--resume', '--retry-failed', '--retry-held', '--retry-parked', '--reverse', '--room', '--scan-secrets', '--scope', '--scopes', '--self-transfer', '--serial', '--serves', '--session', '--short', '--show-current', '--show-toplevel', '--side', '--since', '--skill-loading', '--skills-dir', '--skills-index-only', '--skip', '--skip-failed', '--slot', '--slug', '--slugs', '--source', '--source-id', '--sources', '--src-subpath', '--staged', '--stale', '--stat', '--state', '--status', '--stdin', '--strategy', '--supabase', '--surface', '--symbol-kind', '--symbolic-full-name', '--takes-holders', '--target', '--target-score', '--terms-url', '--tier', '--timeout', '--timings', '--to', '--token', '--token-env', '--top-k', '--type', '--unindexed', '--unshallow', '--untracked-files', '--url', '--url-env', '--verbose', '--verify', '--version', '--wait', '--wakeshift', '--watch', '--where', '--window', '--workers', '--working-tree', '--workspace', '--yes'],
'takes': ['--about', '--accept', '--applied', '--apply', '--batch-size', '--before', '--brain', '--bucket-size', '--by', '--claim', '--date', '--dim', '--dir', '--domain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--evidence', '--expected-revision', '--expired', '--failed', '--force', '--from-pages', '--group-by', '--help', '--holder', '--include-covered', '--json', '--kind', '--limit', '--max-pages', '--min-confidence', '--min-n', '--no-embed', '--no-embedding', '--offset', '--outcome', '--path', '--quality', '--reject', '--request-id', '--retry-failed', '--row', '--semantic', '--since', '--slug', '--sort', '--source', '--source-id', '--stale', '--state', '--status', '--supabase', '--to', '--unapplied', '--unit', '--until', '--value', '--verdict', '--weight', '--who', '--yes'],
'think': ['--anchor', '--brain', '--calibration-holder', '--help', '--images', '--json', '--mcp-only', '--model', '--only', '--reference-date', '--request-id', '--rounds', '--save', '--since', '--source', '--source-id', '--stale', '--take', '--until', '--with-calibration'],
- 'transcripts': ['--accept-pending', '--all', '--all-discovery', '--apply', '--as', '--background', '--brain', '--days', '--dry-run', '--embed', '--explain', '--facts', '--follow', '--format', '--full', '--help', '--include-self', '--json', '--limit', '--max-bytes', '--max-cost-usd', '--no-accept-pending', '--only', '--progress-interval', '--progress-json', '--purge', '--quiet', '--room', '--since', '--source', '--source-id', '--timeout', '--timings', '--wait'],
+ 'transcripts': ['--accept-pending', '--all', '--all-discovery', '--apply', '--as', '--background', '--brain', '--days', '--dir', '--dry-run', '--embed', '--explain', '--facts', '--follow', '--format', '--from', '--full', '--held', '--help', '--include-self', '--json', '--limit', '--max-bytes', '--max-cost-usd', '--no-accept-pending', '--only', '--progress-interval', '--progress-json', '--purge', '--quiet', '--room', '--since', '--source', '--source-id', '--timeout', '--timings', '--to', '--wait'],
'trust': ['--accept-all', '--accept-pending', '--all', '--as', '--background', '--batch-size', '--brain', '--break-lock', '--db-url', '--dry-run', '--expect', '--explain', '--follow', '--from', '--help', '--http', '--json', '--kind', '--label-lowerings', '--legacy-lock', '--no-accept-pending', '--only', '--path', '--probe', '--progress-interval', '--progress-json', '--quiet', '--reason', '--reason-family', '--relower', '--remove', '--repair-unknown-markers', '--resume', '--room', '--since', '--slug', '--source', '--source-id', '--timeout', '--timings', '--undo', '--uri-prefix', '--version', '--wait', '--yes'],
'upgrade': ['--accept-reviewed-inventory', '--acknowledge-no-backup', '--admin-intent', '--all', '--all-applied', '--apply', '--apply-clean-hunks', '--backend', '--backup-confirmed', '--brain', '--catch-up', '--changed', '--channel', '--check', '--check-downgrade', '--check-only', '--client', '--concurrency', '--confirm-quiesced', '--confirm-relocated-root', '--content-root', '--cost-estimate', '--dry-run', '--embedding-dimensions', '--embedding-model', '--exit-code', '--expand', '--expect', '--expected-epoch', '--expected-state', '--explain', '--export-db-only', '--export-source', '--ff-only', '--force', '--force-all', '--force-orchestrator', '--force-retry', '--force-schema', '--format', '--held', '--help', '--host-dir', '--id', '--images', '--inject-bootstrap', '--inline', '--install', '--instructions-only', '--interval', '--is-ancestor', '--json', '--limit', '--list', '--manifest', '--markdown', '--max-rss', '--max-usd', '--may-act-for', '--member', '--migrate-only', '--migration', '--mode', '--multimodal', '--name', '--name-only', '--no-autopilot-install', '--no-bun-floor-check', '--no-embed', '--no-expand', '--no-inject', '--no-restart', '--no-worker', '--non-interactive', '--now', '--once', '--only', '--oversized', '--path', '--pause', '--phase', '--prefix', '--quiet', '--readers', '--reads-unscoped', '--reason', '--refs', '--remediation-plan', '--repo', '--require-db', '--restart-only', '--resume', '--retry-global', '--scope', '--segmented', '--self-transfer', '--skills-dir', '--skip-verify', '--slug', '--slugs-file', '--source', '--stale', '--statement-timeout', '--status', '--stop-mcp', '--swap-only', '--synopsis', '--tags', '--target', '--token', '--uninstall', '--user', '--vectors', '--verify', '--version', '--workers', '--yes'],
'waiting': ['--all', '--as-of', '--brain', '--dry-run', '--fail-fast', '--help', '--json', '--note', '--source', '--stale-ok', '--status', '--tools-json', '--top', '--type'],
diff --git a/src/core/config-runtime-keys.ts b/src/core/config-runtime-keys.ts
index ad69a50d12..597ebaea23 100644
--- a/src/core/config-runtime-keys.ts
+++ b/src/core/config-runtime-keys.ts
@@ -34,5 +34,7 @@ export const RUNTIME_READER_CONFIG_KEYS: readonly string[] = [
'access.requester_scope_enforce', // access/requester-scope.ts (GST-363): false turns the unscoped-read refusal on a brain that serves several people into a notice (hosted, until connections are bound); default on
'mcp.stdio_may_act_for', // access/audience.ts (GBRA-231): whom a local stdio server may act for (_meta.on_behalf_of); unset, stdio refuses every audience
'proposals.ttl_hours', 'proposals.max_pending_per_actor', 'proposals.max_pending_per_credential', // persistence/memory-proposals.ts: held QM proposal TTL (168 h) and pending caps (10 per person, 50 per credential)
+ 'transcripts.archive.enabled', 'transcripts.archive.source', 'transcripts.archive.timezone', // transcripts/archive.ts (GST-2768): the conversation archive switch (gbrain transcripts archive on|off), its source (default conversations) and the timezone its days are cut in (pinned per source on the first run)
+ 'transcripts.archive.backfill_days_per_run', 'transcripts.archive.run_window_days', 'transcripts.archive.quiet_minutes', // transcripts/archive.ts: days one backfill run writes, days a run re-reads, and minutes of quiet before an open day embeds
'proposals.retention_days', 'proposals.max_payload_chars', 'proposals.max_reason_chars', // persistence/memory-proposals.ts: closed-row retention (30 d), payload (1,000) and deny-reason (500) character caps
];
diff --git a/src/core/env-trust.ts b/src/core/env-trust.ts
index 1cfa7121d2..baf68f706c 100644
--- a/src/core/env-trust.ts
+++ b/src/core/env-trust.ts
@@ -187,6 +187,8 @@ export const CWD_DOTENV_PROTECTED_KEYS: readonly string[] = [
'GBRAIN_REMOTE_MCP_URL', // `init --remote` default: where the thin client sends its MCP traffic and bearer token
'GBRAIN_REMOTE_ISSUER_URL', // `init --remote` default: the OAuth issuer the client trusts and hands its credential to
'GBRAIN_OAUTH_RELAY_URL', // the OAuth relay that hands back Google tokens (creds/relay-client.ts): a planted relay is token theft
+ 'GBRAIN_OPENCLAW_GATEWAY_URL', // GST-2768: the Gateway the archive's openclaw child calls with GBRAIN_OPENCLAW_GATEWAY_TOKEN; a planted URL would receive the token and serve forged history
+ 'GBRAIN_OPENCLAW_GATEWAY_TOKEN', // GST-2768: the credential that child presents; a planted one pairs with a planted Gateway to archive forged history
'GBRAIN_HOOK_CWD', // #8424: the session directory the hook launcher records; a planted value moves the yield checks and the turn-rule project match
// --- re-run marker ------------------------------------------------------
'GBRAIN_CWD_ENV_QUARANTINED', // cli-preflight.ts's sanitized re-run marker; only honoured when the startup cwd IS the .env-free dir it names, so a planted one is inert — listed so it is also dropped and named in the warning
@@ -219,6 +221,7 @@ export const CWD_DOTENV_PROTECTED_PREFIXES: readonly string[] = [
'NPM_CONFIG_', // npm registry / script-shell / node-options redirection for npm-based children
'npm_config_', // the lowercase spelling npm actually reads
'BASH_FUNC_', // exported-function injection (`BASH_FUNC_name%%=() {…}`): bash imports the body as a function into every child shell
+ 'OPENCLAW_', // GST-2768: the openclaw child's Gateway URL, token, config and state roots; a planted value redirects the Gateway token or serves forged history
];
/**
diff --git a/src/core/error-registry.ts b/src/core/error-registry.ts
index ea90a691fe..08eaa7a4a1 100644
--- a/src/core/error-registry.ts
+++ b/src/core/error-registry.ts
@@ -66,6 +66,13 @@ export const CODES = {
access_command_failed: { class: 'caller', summary: "Google connect credential error: access command failed.", docs: 'docs/guides/google-connect.md#troubleshooting' },
access_identity_unavailable: { class: 'retryable', actor: 'provider', summary: "The member a credential (access token, OAuth client or stdio binding) is bound to could not be looked up, so the request was refused and nothing was read or written.", why: "Reading the binding failed with a database error, or with a lock or statement timeout or a dropped connection while the request was submitted. Reading the credential as unbound instead would open every unlabelled page to a labelled-only member, so the request stops, and the same request succeeds once the database answers.", docs: 'docs/guides/write-refusals.md#access_identity_unavailable', suggestion: 'Retry the same request in a few seconds (a write with the same request_id); if it keeps failing, ask the brain operator to run gbrain doctor --json on the brain host.', fix: { argv: ['gbrain', 'whoami', '--json'], consent: [], actor: 'provider', requires_exclusive: false, why: 'Read-only: after a few seconds whoami reads the binding again and shows the member this connection reads as; then repeat the original request.', verify: { argv: ['gbrain', 'whoami', '--json'] } } },
access_member_disabled: { class: 'host_only', actor: 'user', reasons: ['disabled', 'removed', 'missing', 'malformed'], summary: "The credential (access token, OAuth client or stdio binding) is bound to a member who is disabled, removed or missing, so it was refused at verification and nothing was read.", why: 'A credential bound to a member reads as that member. While the member cannot read, the credential is refused rather than silently treated as unbound.', docs: 'docs/guides/write-refusals.md#access_member_disabled', suggestion: 'Ask the brain operator to enable the member (gbrain access member enable <id>) or rebind the credential (gbrain auth rescope --token <name>|--client <id> --member <id>|none).' },
+ archive_audience_unproven: { class: 'retryable', retryable: true, summary: "The conversation archive could not check whether the brain is shared (its client-usage read failed), so it wrote nothing this run.", why: 'A brain with no audience declared counts as shared when several clients use it; without that read a page written as personal could be open to other members, so the run publishes nothing.', docs: 'docs/guides/conversation-archive.md#privacy', suggestion: 'Rerun gbrain transcripts archive run; if it repeats, declare the audience with gbrain config set brain.audience personal|shared.' },
+ archive_gateway_unreachable: { class: 'retryable', retryable: true, summary: "The conversation archive could not reach the OpenClaw Gateway, so no OpenClaw conversation was read this run; the archive ledger is unchanged, so the next run retries safely.", why: "Current OpenClaw keeps live sessions in SQLite owned by its Gateway, and gbrain reads them only through the Gateway's documented RPCs (openclaw gateway call), never the file.", docs: 'docs/guides/conversation-archive.md#openclaw', suggestion: 'Start the Gateway (openclaw gateway) or point gbrain at it with GBRAIN_OPENCLAW_GATEWAY_URL, then rerun gbrain transcripts archive run.', fix: { argv: ['gbrain', 'transcripts', 'archive', 'status', '--json'], consent: [], actor: 'agent', why: 'Shows each harness the archive reads, the last error per reader and the held days, read-only.', requires_exclusive: false } },
+ archive_gateway_auth: { class: 'host_only', actor: 'user', summary: "The OpenClaw Gateway refused the credential the conversation archive used, so no OpenClaw conversation was read this run; the archive ledger is unchanged.", why: 'The Gateway token or device pairing belongs to the user; gbrain never mints or edits one.', docs: 'docs/guides/conversation-archive.md#openclaw', suggestion: 'Ask the user to give gbrain the Gateway token (GBRAIN_OPENCLAW_GATEWAY_TOKEN, or gateway.auth in the OpenClaw config on this machine), then rerun gbrain transcripts archive run.', fix: { argv: ['gbrain', 'transcripts', 'archive', 'status', '--json'], consent: [], actor: 'agent', why: 'Shows the reader error and the held days, read-only.', requires_exclusive: false } },
+ archive_gateway_failed: { class: 'retryable', retryable: true, summary: "The OpenClaw Gateway answered the conversation archive with an error, so no OpenClaw conversation was read this run; the archive ledger is unchanged.", docs: 'docs/guides/conversation-archive.md#openclaw', suggestion: 'Rerun gbrain transcripts archive run; if it repeats, report the redacted message from gbrain transcripts archive status --json.', fix: { argv: ['gbrain', 'transcripts', 'archive', 'status', '--json'], consent: [], actor: 'agent', why: 'Shows the reader error, read-only.', requires_exclusive: false } },
+ archive_owner_unset: { class: 'host_only', actor: 'user', summary: "On a shared brain the conversation archive held every day and wrote nothing, because no owner member is named and an archive page without readable_by would be open to the whole source.", docs: 'docs/guides/conversation-archive.md#privacy', suggestion: 'Ask the user which enabled member owns the brain, run gbrain access owner set <member>, then rerun the archive; held days are written then.', fix: { argv: ['gbrain', 'access', 'owner'], consent: [], actor: 'agent', why: 'Shows whether an owner member is named, read-only.', requires_exclusive: false } },
+ archive_redaction_failed: { class: 'retryable', retryable: true, summary: "Redaction failed for a conversation, so the conversation archive held its days instead of writing them; the next run retries them.", why: 'Archive pages are written only after redaction succeeded over the whole session; a day is never written unredacted and never dropped.', docs: 'docs/guides/conversation-archive.md#privacy', suggestion: 'Rerun gbrain transcripts archive run; if the day stays held, check the user redaction pattern file named in the held message.', fix: { argv: ['gbrain', 'transcripts', 'archive', 'status', '--json'], consent: [], actor: 'agent', why: 'Lists held days with their codes, read-only.', requires_exclusive: false } },
+ archive_run_busy: { class: 'retryable', retryable: true, summary: "Another conversation archive run holds this source, so this run read and wrote nothing.", why: 'A run decides each day from the archive ledger it read when it started, so two runs at once could publish older history over newer; runs on one source take turns on a database lease.', docs: 'docs/guides/conversation-archive.md#openclaw', suggestion: 'Wait for the other run to finish, then rerun gbrain transcripts archive run.', fix: { argv: ['gbrain', 'status', '--section', 'locks', '--json'], consent: [], actor: 'agent', why: 'Shows which process holds the archive lease and when it expires, read-only.', requires_exclusive: false } },
autopilot_no_repo: { class: 'caller', summary: "gbrain autopilot --install found no absolute repo directory (sync.repo_path and the default source's local_path are unset or relative), so it installed nothing.", why: 'The autopilot daemon syncs a git checkout. A repo-less brain needs no daemon: free upkeep (gbrain maintain --unattended) keeps it maintained.', suggestion: 'Pass --repo <absolute-directory> for a brain kept in git, or check free upkeep with gbrain maintain --status.' },
autopilot_pglite_daemon: { class: 'caller', summary: 'gbrain autopilot --install refused on a PGLite brain: a daemon would hold the single-writer lock all day and block every other gbrain process.', why: 'PGLite has one writer. gbrain serve owns the lock and runs the resident sweep; free upkeep runs through gbrain maintain --unattended.', suggestion: 'Run gbrain serve. Check upkeep with gbrain maintain --status. For a dedicated-brain daemon, re-run gbrain autopilot --install --force.' },
cycle_already_running: { class: 'retryable', summary: 'Another maintenance cycle holds the brain cycle lock, so this run skipped instead of waiting.', why: 'Cycles serialize on the gbrain-cycle lease; the holder releases it when it finishes (or its TTL lapses after a crash).', suggestion: 'Wait for the other cycle; unattended upkeep retries on its next turn (gbrain maintain --status shows the last skip).' },
diff --git a/src/core/git/reconcile.ts b/src/core/git/reconcile.ts
index 8bc683b6b4..bfb916443d 100644
--- a/src/core/git/reconcile.ts
+++ b/src/core/git/reconcile.ts
@@ -90,12 +90,12 @@ const MOVED_REMOTE_ATTEMPTS = 4;
/** The bound on one `gbrain git commit-push` child (a fetch, an in-memory merge and a push, repeated while the remote keeps moving). */
export const COMMIT_PUSH_CHILD_TIMEOUT_MS = 600_000;
-/** Run `gbrain git commit-push <args> --path <root> --json` in a bounded child; its JSON document, or its refusal rethrown. */
-export async function commitPushChild(root: string, args: string[], timeoutMs = COMMIT_PUSH_CHILD_TIMEOUT_MS): Promise<Record<string, unknown>> {
+/** Run `gbrain git commit-push <args> --path <root> --json [-- <paths>]` in a bounded child; its JSON document, or its refusal rethrown. */
+export async function commitPushChild(root: string, args: string[], timeoutMs = COMMIT_PUSH_CHILD_TIMEOUT_MS, paths: readonly string[] = []): Promise<Record<string, unknown>> {
const cli = resolveChildCliInvocation(process.env, process.execPath, process.argv[1], () => null);
if (!cli) throw opError('git_push_unavailable', `This process could not locate its own CLI executable to reconcile ${root}; nothing changed.`,
`Set ${CHILD_ENV.childCliOverride} to the path of the CLI executable and retry.`);
- const child = await execFileBounded(cli.cmd, [...cli.argsPrefix, 'git', 'commit-push', ...args, '--path', root, '--json'],
+ const child = await execFileBounded(cli.cmd, [...cli.argsPrefix, 'git', 'commit-push', ...args, '--path', root, '--json', ...(paths.length ? ['--', ...paths] : [])],
{ timeout: timeoutMs, termGraceMs: 10_000, maxBuffer: 1024 * 1024, env: process.env, killTree: true });
let doc: Record<string, unknown> | null = null;
try { doc = JSON.parse(String(child.stdout).trim()); } catch { /* reported below */ }
diff --git a/src/core/page-embedding-budget.ts b/src/core/page-embedding-budget.ts
index 0e52c7e5c1..e98980d399 100644
--- a/src/core/page-embedding-budget.ts
+++ b/src/core/page-embedding-budget.ts
@@ -116,6 +116,37 @@ async function segmentedSampleWindows(engine: BrainEngine, pageId: number): Prom
return [await forward(0), await forward(Math.max(0, middle)), tail.slice(-JUNK_SAMPLE_CHARS)];
}
+/**
+ * GST-2768: holds that apply to a page of any size. A small page is otherwise unmetered, so the conversation archive's
+ * backfill (archive_backfill: waits for the owner's approval of the previewed cost) and its open day (archive_open_day:
+ * waits until the day closes or goes quiet) name the page here and the embed paths skip it until then.
+ */
+export const ANY_SIZE_HOLD_REASONS: readonly string[] = ['archive_backfill', 'archive_open_day'];
+
+async function smallPageHold(engine: BrainEngine, pageId: number, approved: boolean): Promise<PageEmbedDecision> {
+ const [row] = await engine.executeRaw<{ reason: string }>(
+ "SELECT reason FROM page_embedding_budget WHERE page_id = $1 AND state = 'held' AND reason = ANY($2::text[])", [pageId, [...ANY_SIZE_HOLD_REASONS]]);
+ if (!row) return { kind: 'unmetered' };
+ if (!approved) return { kind: 'held', reason: row.reason };
+ await engine.executeRaw("UPDATE page_embedding_budget SET state = 'open', reason = NULL, updated_at = now() WHERE page_id = $1", [pageId]);
+ return { kind: 'unmetered' };
+}
+
+/**
+ * An archive page names its hold in frontmatter (transcript_archive.embedding_hold) from its first write. The first
+ * embedding plan for it, before any budget row exists, records that hold, so a page published ahead of its hold row
+ * is never embedded without the owner's approval. An approved page keeps its row (open), so the marker holds once.
+ */
+async function frontmatterHold(engine: BrainEngine, pageId: number): Promise<string | null> {
+ const [row] = await engine.executeRaw<{ marker: string | null; has_row: boolean }>(
+ `SELECT p.frontmatter->'transcript_archive'->>'embedding_hold' AS marker,
+ EXISTS (SELECT 1 FROM page_embedding_budget b WHERE b.page_id = p.id) AS has_row
+ FROM pages p WHERE p.id = $1`, [pageId]);
+ if (!row?.marker || row.has_row || !ANY_SIZE_HOLD_REASONS.includes(row.marker)) return null;
+ await engine.executeRaw("INSERT INTO page_embedding_budget (page_id, state, reason) VALUES ($1, 'held', $2) ON CONFLICT (page_id) DO NOTHING", [pageId, row.marker]);
+ return row.marker;
+}
+
/** Hold a large page's embedding until the user approves it (`reason` names why). Idempotent; never touches open spend. */
export async function holdPageEmbedding(engine: Pick<BrainEngine, 'executeRaw'>, pageId: number, reason: string): Promise<void> {
await engine.executeRaw(`INSERT INTO page_embedding_budget (page_id, state, reason) VALUES ($1, 'held', $2)
@@ -131,8 +162,13 @@ export async function holdPageEmbedding(engine: Pick<BrainEngine, 'executeRaw'>,
export async function planPageEmbedding(engine: BrainEngine, page: BudgetPage,
stale: ReadonlyArray<{ chunk_index: number; chunk_text: string }>,
opts: { signature?: string | null; approvedCapUsd?: number; maxMarkupRatio?: number } = {}): Promise<PageEmbedDecision> {
- if (!isLargePageBytes(pageBodyBytes(page)) || stale.length === 0) return { kind: 'unmetered' };
+ if (stale.length === 0) return { kind: 'unmetered' };
const approved = opts.approvedCapUsd !== undefined;
+ if (!approved) {
+ const marker = await frontmatterHold(engine, page.id);
+ if (marker) return { kind: 'held', reason: marker };
+ }
+ if (!isLargePageBytes(pageBodyBytes(page))) return smallPageHold(engine, page.id, approved);
const configured = await resolvePageEmbedCapUsd(engine);
const { rate, priced } = ledgerPricePerMTok(opts.signature);
return engine.transaction(async tx => {
diff --git a/src/core/page-size-policy.ts b/src/core/page-size-policy.ts
index bcb9c6a203..d00b56caab 100644
--- a/src/core/page-size-policy.ts
+++ b/src/core/page-size-policy.ts
@@ -48,12 +48,13 @@ export function largePageBodyFilter(alias: string): string {
* SQL over a `page_embedding_budget` row aliased `budget` joined to its
* `pages` row aliased `page`: the row is held or capped and still applies.
* The budget meters large pages only, so a row left on a page edited down to
- * the inline threshold or under no longer applies. The one rule the embedding
+ * the inline threshold or under no longer applies, except a conversation-archive
+ * hold (archive_backfill, archive_open_day; GST-2768), which applies at any size. The one rule the embedding
* queue, the held list (`gbrain embed --held`) and the held count (doctor
* `partial_embedding`) share.
*/
export function budgetRowAppliesFilter(budget: string, page: string): string {
- return `${budget}.state <> 'open' AND ${largePageBodyFilter(page)}`;
+ return `${budget}.state <> 'open' AND (${largePageBodyFilter(page)} OR ${budget}.reason IN ('archive_backfill', 'archive_open_day'))`;
}
/** SQL over a `pages` row aliased `alias`: the page has a held or capped budget row that still applies (budgetRowAppliesFilter). */
diff --git a/src/core/transcripts/archive-day.ts b/src/core/transcripts/archive-day.ts
new file mode 100644
index 0000000000..f72754b0ac
--- /dev/null
+++ b/src/core/transcripts/archive-day.ts
@@ -0,0 +1,186 @@
+/**
+ * archive-day.ts — the conversation archive's day layout (GST-2768).
+ *
+ * One page per (harness, conversation, local day). The day comes from the source's pinned timezone, while the
+ * message anchors inside the body stay UTC exactly as the per-session renderer writes them. Day pages live in their
+ * own slug and id namespace, so they never meet a per-session page:
+ *
+ * slug conversations/<harness>/<YYYY-MM-DD>/<label>-<id8> (parts 2..N append -pN)
+ * id <harness>-<hash16>-d<YYYYMMDD>-pN (per-session pages use <harness>-<hash16>-pN)
+ *
+ * The label never carries a raw chat id or name: dm, group-<hash8>, room-<hash8>, cron-<job> from verified channel
+ * metadata, else the harness name. The body, escaping and part split are the per-session renderer's
+ * (renderSessionParts) run over the day's already-redacted messages; only the slug, id and frontmatter differ.
+ * transcript_import stays on every day page (audit-secrets and Dream select on it) with layout: day, and
+ * transcript_archive records the day, the pinned timezone and what the day holds.
+ */
+
+import { createHash } from 'node:crypto';
+import { slugifySegment } from '../sync.ts';
+import type { TranscriptFormat, TranscriptMessage } from './types.ts';
+import { transcriptFullId, transcriptSlugId } from './types.ts';
+import { renderPartContent, renderSessionParts, type RedactedSession, type RenderedPart } from './render.ts';
+
+export type ArchiveChatType = 'direct' | 'group' | 'room';
+
+export interface ArchiveChannel {
+ /** From the harness's own session row; only a verified value can widen a page to a group. */
+ chatType?: ArchiveChatType;
+ verified: boolean;
+ cronJob?: string;
+}
+
+/** YYYY-MM-DD of an instant in timeZone. */
+export function localDay(iso: string, timeZone: string): string {
+ const parts = new Intl.DateTimeFormat('en-CA', { timeZone, year: 'numeric', month: '2-digit', day: '2-digit' }).formatToParts(new Date(iso));
+ const get = (type: string) => parts.find(p => p.type === type)?.value ?? '';
+ return get('year') + '-' + get('month') + '-' + get('day');
+}
+
+/** Throws a RangeError for a zone this runtime does not know. */
+export function assertTimeZone(timeZone: string): string {
+ new Intl.DateTimeFormat('en-CA', { timeZone });
+ return timeZone;
+}
+
+/** day plus delta calendar days. */
+export function addDays(day: string, delta: number): string {
+ const d = new Date(day + 'T00:00:00.000Z');
+ d.setUTCDate(d.getUTCDate() + delta);
+ return d.toISOString().slice(0, 10);
+}
+
+export interface DaySlice { day: string; messages: TranscriptMessage[]; indexes: number[] }
+
+/**
+ * Group messages by local day, oldest day first. A message without a timestamp carries the previous message's day
+ * (the renderer carries its anchor the same way); one before any timestamp joins the first dated day.
+ */
+export function sliceSessionByDay(messages: readonly TranscriptMessage[], timeZone: string): DaySlice[] {
+ const byDay = new Map<string, DaySlice>();
+ let current: string | null = null;
+ const pending: number[] = [];
+ messages.forEach((m, i) => {
+ if (m.timestamp && !Number.isNaN(Date.parse(m.timestamp))) current = localDay(m.timestamp, timeZone);
+ if (current === null) { pending.push(i); return; }
+ let slice = byDay.get(current);
+ if (!slice) byDay.set(current, slice = { day: current, messages: [], indexes: [] });
+ for (const p of pending.splice(0)) { slice.messages.push(messages[p]!); slice.indexes.push(p); }
+ slice.messages.push(m);
+ slice.indexes.push(i);
+ });
+ return [...byDay.values()].sort((a, b) => a.day.localeCompare(b.day));
+}
+
+/** 8 hex chars naming a conversation inside a label or a group principal; never the raw key. */
+export function conversationHash8(harness: TranscriptFormat, conversationKey: string): string {
+ return createHash('sha256').update(harness + ':' + conversationKey).digest('hex').slice(0, 8);
+}
+
+export function conversationLabel(harness: TranscriptFormat, conversationKey: string, channel?: ArchiveChannel): string {
+ if (channel?.cronJob) return 'cron-' + (slugifySegment(channel.cronJob).slice(0, 24) || conversationHash8(harness, conversationKey));
+ if (channel?.verified && (channel.chatType === 'group' || channel.chatType === 'room')) return channel.chatType + '-' + conversationHash8(harness, conversationKey);
+ if (channel?.verified && channel.chatType === 'direct') return 'dm';
+ return harness;
+}
+
+export function archiveDayBaseSlug(harness: TranscriptFormat, day: string, label: string, conversationKey: string): string {
+ return 'conversations/' + harness + '/' + day + '/' + label + '-' + transcriptSlugId(conversationKey).slice(0, 8);
+}
+
+export function archiveDayId(harness: TranscriptFormat, conversationKey: string, day: string, part: number): string {
+ return harness + '-' + transcriptFullId(conversationKey) + '-d' + day.replace(/-/g, '') + '-p' + part;
+}
+
+/** Counts of <REDACTED:rule> markers per rule in the day's text: the per-day receipt (never values). */
+export function redactionCounts(messages: readonly TranscriptMessage[]): Record<string, number> {
+ const out: Record<string, number> = {};
+ for (const m of messages) {
+ for (const hit of (m.text + '\n' + (m.speaker ?? '')).matchAll(/<REDACTED:([A-Za-z0-9_.-]+)>/g)) out[hit[1]!] = (out[hit[1]!] ?? 0) + 1;
+ }
+ return Object.fromEntries(Object.entries(out).sort(([a], [b]) => a.localeCompare(b)));
+}
+
+export interface ArchiveDayInput {
+ harness: TranscriptFormat;
+ conversationKey: string;
+ /** The harness's own key (OpenClaw session key) the group label hashes; defaults to conversationKey. */
+ scopeKey?: string;
+ channel?: ArchiveChannel;
+ day: string;
+ timezone: string;
+ /** The whole session, redacted (redaction always runs over every day of it). */
+ redacted: RedactedSession;
+ slice: DaySlice;
+ complete: boolean;
+ partTargetBytes: number;
+ readableBy?: string[];
+ /** The embedding hold the page carries from its first write (archive_backfill, archive_open_day), so no embed runs before the hold row exists. */
+ embeddingHold?: string;
+}
+
+export interface ArchiveDayRender { baseSlug: string; parts: RenderedPart[]; messages: number }
+
+export function renderArchiveDay(input: ArchiveDayInput): ArchiveDayRender {
+ const { harness, conversationKey, day, slice, redacted } = input;
+ const flags = redacted.imperativeFlags;
+ const sub: RedactedSession = {
+ session: { meta: { ...redacted.session.meta, startedAt: slice.messages.find(m => m.timestamp)?.timestamp }, messages: slice.messages },
+ redactionCount: 0,
+ imperativesFlagged: flags ? slice.indexes.filter(i => flags[i]).length : 0,
+ ...(flags ? { imperativeFlags: slice.indexes.map(i => flags[i] === true) } : {}),
+ };
+ const rendered = renderSessionParts(sub, { sourcePath: '', layout: 'single', partTargetBytes: input.partTargetBytes });
+ const label = conversationLabel(harness, input.scopeKey ?? conversationKey, input.channel);
+ const baseSlug = archiveDayBaseSlug(harness, day, label, conversationKey);
+ const timestamps = slice.messages.map(m => m.timestamp).filter(Boolean);
+ const single = rendered.parts.length === 1;
+ const archive = {
+ layout: 'day',
+ harness,
+ conversation_key_hash: transcriptFullId(conversationKey),
+ day,
+ timezone: input.timezone,
+ ...(input.channel?.verified && input.channel.chatType ? { chat_type: input.channel.chatType } : {}),
+ first_ts: timestamps[0] ?? null,
+ last_ts: timestamps[timestamps.length - 1] ?? null,
+ messages: slice.messages.length,
+ redactions: redactionCounts(slice.messages),
+ complete: input.complete,
+ ...(input.embeddingHold ? { embedding_hold: input.embeddingHold } : {}),
+ };
+ const parts = rendered.parts.map((part): RenderedPart => {
+ const slug = part.part === 1 ? baseSlug : baseSlug + '-p' + part.part;
+ const frontmatterId = archiveDayId(harness, conversationKey, day, part.part);
+ const imported = part.frontmatter.transcript_import as Record<string, unknown>;
+ const frontmatter: Record<string, unknown> = {
+ type: 'conversation',
+ title: harness + ' ' + label + ' ' + day + (single ? '' : ' (part ' + part.part + ')'),
+ date: day,
+ id: frontmatterId,
+ ...(input.readableBy ? { readable_by: input.readableBy } : {}),
+ transcript_import: { ...imported, layout: 'day', day },
+ transcript_archive: archive,
+ };
+ return { ...part, slug, frontmatterId, frontmatter, content: renderPartContent(frontmatter, part.body) };
+ });
+ return { baseSlug, parts, messages: slice.messages.length };
+}
+
+export interface DayIndexEntry { slug: string; harness: string; messages: number }
+
+/** conversations/days/<day>: the day's conversation pages with message counts, so a dated question has one place to land. */
+export function renderDayIndex(day: string, timezone: string, entries: readonly DayIndexEntry[], readableBy?: string[]): RenderedPart {
+ const sorted = [...entries].sort((a, b) => a.harness.localeCompare(b.harness) || a.slug.localeCompare(b.slug));
+ const frontmatter: Record<string, unknown> = {
+ type: 'conversation',
+ title: 'Conversations on ' + day,
+ date: day,
+ id: 'archive-day-index-d' + day.replace(/-/g, ''),
+ ...(readableBy ? { readable_by: readableBy } : {}),
+ transcript_archive_index: { day, timezone, conversations: sorted.length },
+ };
+ const lines = sorted.map(e => '- [[' + e.slug + ']] (' + e.harness + ', ' + e.messages + ' message' + (e.messages === 1 ? '' : 's') + ')');
+ const body = ['# Conversations on ' + day, '', ...lines].join('\n');
+ return { slug: 'conversations/days/' + day, content: renderPartContent(frontmatter, body), frontmatterId: String(frontmatter.id), part: 1, frontmatter, body };
+}
diff --git a/src/core/transcripts/archive-readers.ts b/src/core/transcripts/archive-readers.ts
new file mode 100644
index 0000000000..d444b69ee9
--- /dev/null
+++ b/src/core/transcripts/archive-readers.ts
@@ -0,0 +1,104 @@
+/**
+ * archive-readers.ts — where the conversation archive reads sessions on this machine (GST-2768).
+ *
+ * One reader per harness root from harnessRoots() (Claude Code, Codex, OpenClaw files including the archived
+ * .jsonl.reset.* / .jsonl.deleted.* variants, Hermes, Grok), parsed by the existing adapters, plus the OpenClaw
+ * Gateway reader when the openclaw CLI is on PATH (or GBRAIN_OPENCLAW_GATEWAY_URL names a Gateway). A harness with
+ * nothing on this machine is an informational line, never an error. Consumer chat apps have no local store: their
+ * history arrives through gbrain connectors or an export.
+ */
+
+import { existsSync, lstatSync } from 'node:fs';
+import { detectAdapter, harnessRoots, type HarnessRoot } from './detect.ts';
+import { discoverTranscriptFiles } from './discover.ts';
+import { openclawGatewayReader } from './openclaw-gateway.ts';
+import { archiveMessageId, type ArchiveConversation, type ArchiveReader } from './archive.ts';
+import type { ParsedSession, TranscriptFormat, TranscriptMessage } from './types.ts';
+
+const NO_SESSIONS = (harness: string) => 'no readable sessions for ' + harness + '; use gbrain connectors or an export with gbrain transcripts ingest';
+
+export function fileReader(root: HarnessRoot): ArchiveReader {
+ const present = existsSync(root.root);
+ return {
+ harness: root.format,
+ name: root.format + '-files',
+ ...(present ? {} : { unavailable: NO_SESSIONS(root.format) }),
+ async *scan({ sinceMs, minMtimeMs, dayOf }) {
+ const floor = minMtimeMs ?? sinceMs;
+ const found = new Map<string, { harness: TranscriptFormat; sessionId: string; dayIds: Map<string, Set<string>>; newest?: string; paths: string[] }>();
+ for (const file of discoverTranscriptFiles([root], { includeArchived: true })) {
+ if (floor !== undefined) {
+ try { if (lstatSync(file.path).mtimeMs < floor) continue; } catch { continue; }
+ }
+ for await (const session of parseFile(file.path, file.format)) {
+ const key = session.meta.harness + '\u0000' + session.meta.sessionId;
+ const seen = found.get(key) ?? { harness: session.meta.harness, sessionId: session.meta.sessionId, dayIds: new Map<string, Set<string>>(), paths: [] };
+ found.set(key, seen);
+ if (!seen.paths.includes(file.path)) seen.paths.push(file.path);
+ for (const m of session.messages) {
+ if (!m.timestamp) continue;
+ const day = dayOf(m.timestamp);
+ const ids = seen.dayIds.get(day) ?? new Set<string>();
+ seen.dayIds.set(day, ids.add(archiveMessageId(m)));
+ if (!seen.newest || m.timestamp > seen.newest) seen.newest = m.timestamp;
+ }
+ }
+ }
+ for (const s of found.values()) {
+ yield {
+ harness: s.harness, conversationKey: s.sessionId, sessionId: s.sessionId, via: 'file' as const,
+ days: [...s.dayIds.keys()], dayIds: Object.fromEntries([...s.dayIds].map(([d, ids]) => [d, [...ids]])), ...(s.newest ? { newest: s.newest } : {}),
+ load: () => loadSession(s.harness, s.sessionId, s.paths),
+ };
+ }
+ },
+ };
+}
+
+async function* parseFile(path: string, format: TranscriptFormat): AsyncGenerator<ParsedSession> {
+ const detected = detectAdapter(path, { explicitFormat: format });
+ if (!detected.ok) return;
+ const gen = detected.adapter.parse(path);
+ try {
+ for (let step = await gen.next(); !step.done; step = await gen.next()) {
+ if (step.value.messages.length) yield step.value;
+ }
+ } catch {
+ // One unreadable or oversized file never stops the harness; ingest's drift report names it.
+ }
+}
+
+/** One session read again from every file that holds it (live, rotated, reset, deleted), messages merged once each, oldest first. */
+async function loadSession(harness: TranscriptFormat, sessionId: string, paths: string[]): Promise<ArchiveConversation | null> {
+ let meta: ParsedSession['meta'] | undefined;
+ let metaNewest = '';
+ const seen = new Set<string>();
+ const messages: TranscriptMessage[] = [];
+ for (const path of paths) {
+ for await (const session of parseFile(path, harness)) {
+ if (session.meta.harness !== harness || session.meta.sessionId !== sessionId) continue;
+ const last = session.messages[session.messages.length - 1]?.timestamp ?? '';
+ if (!meta || last > metaNewest) { meta = session.meta; metaNewest = last; }
+ for (const m of session.messages) {
+ const id = archiveMessageId(m);
+ if (seen.has(id)) continue;
+ seen.add(id);
+ messages.push(m);
+ }
+ }
+ }
+ if (!meta || messages.length === 0) return null;
+ messages.sort((a, b) => (a.timestamp < b.timestamp ? -1 : a.timestamp > b.timestamp ? 1 : 0));
+ return { harness, conversationKey: sessionId, session: { meta, messages }, via: 'file' };
+}
+
+export function fileArchiveReaders(roots?: HarnessRoot[]): ArchiveReader[] {
+ const readers = harnessRoots(roots).map(fileReader);
+ const bun = (globalThis as { Bun?: { which?: (bin: string) => string | null } }).Bun;
+ const gateway = !!process.env.GBRAIN_OPENCLAW_GATEWAY_URL || !!bun?.which?.('openclaw');
+ readers.push(gateway ? openclawGatewayReader() : {
+ harness: 'openclaw', name: 'openclaw-gateway', list: async () => [],
+ unavailable: 'no OpenClaw Gateway on this machine (openclaw is not on PATH); OpenClaw conversations come only from its session files',
+ });
+ return readers;
+}
diff --git a/src/core/transcripts/archive.ts b/src/core/transcripts/archive.ts
new file mode 100644
index 0000000000..8546156e22
--- /dev/null
+++ b/src/core/transcripts/archive.ts
@@ -0,0 +1,949 @@
+/**
+ * archive.ts — the built-in conversation archive (GST-2768, PR 2: core and legacy adoption; off unless turned on).
+ *
+ * gbrain transcripts archive run | backfill --from --to | status | adopt-legacy | on | off.
+ *
+ * Each run reads every readable harness (the file adapters through harnessRoots(), OpenClaw archived session files,
+ * and the OpenClaw Gateway), redacts each session as a whole, cuts it into local days in the source's pinned
+ * timezone, and writes one page per (harness, conversation, day) plus a day index page (archive-day.ts). It never
+ * touches a per-session page: day pages have their own slug and id namespace.
+ *
+ * Ledger: one config row per (source, harness, conversation_key_hash, day) under transcripts.archive.ledger.*, holding
+ * the content hash, the complete flag, the slug and part count, and a held error code. The config table never
+ * expires rows (op_checkpoint does, after 7 days). A day is rewritten only when its rendered content changed, so a
+ * rerun, an overlapping backfill and a crash retry all converge on the same pages; a closed day whose redaction
+ * changed is rewritten because its hash changed. A day whose source now has fewer messages than the ledger recorded
+ * (an OpenClaw reset in place drops the earlier messages from chat.history) is kept as written.
+ *
+ * Timezone: pinned per source at the first run (transcripts.archive.pinned_timezone.<source>) from
+ * transcripts.archive.timezone or the machine's zone, and stamped into every page. A later change to the setting
+ * never re-slices.
+ *
+ * Privacy: a personal brain writes no readable_by (every caller is the owner). A shared brain (brain.audience
+ * shared, unknown, or two or more enabled members) gives every page readable_by [owner]; a verified group or room
+ * adds group:chat-<hash8> only when that group principal already exists. With no owner member set, every day is
+ * held with archive_owner_unset: no archive page is ever written without readable_by on a shared brain. A shared
+ * brain whose archive source sits in a repository archives into <source>-owner, a database-only source, so nothing
+ * is committed where repository readers could read it.
+ *
+ * Write paths: managed persistence goes through the coordinator; an unmanaged repository source gets the redacted
+ * files written, committed with gbrain git commit-push -- <paths>, then imported by sync (git stays the parent); any
+ * other brain writes the database directly. Nothing refuses for lack of a repository.
+ *
+ * Embedding: pages are written keyword-only. Backfilled pages are held in page_embedding_budget (reason
+ * archive_backfill) until the owner approves the previewed cost with gbrain embed --held. The open day is held
+ * (archive_open_day) until it closes or has been quiet for transcripts.archive.quiet_minutes. No fact extraction.
+ */
+
+import { createHash } from 'node:crypto';
+import { existsSync, lstatSync, mkdirSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from 'node:fs';
+import { dirname, join, relative, sep } from 'node:path';
+import type { BrainEngine } from '../engine.ts';
+import type { Action } from '../agent-output.ts';
+import type { OperationContext } from '../ops/contract.ts';
+import { importFromContent } from '../import-file.ts';
+import { managedPersistenceEnabled } from '../persistence/ownership.ts';
+import { classifyItemFailure } from '../item-failure.ts';
+import { classifyBrainAudience } from '../facts/writeback-audience.ts';
+import { listGroups, listMembers } from '../access/members.ts';
+import { ownerView } from '../access/owner.ts';
+import { holdPageEmbedding } from '../page-embedding-budget.ts';
+import { PAGE_MAX_BYTES } from '../page-size-policy.ts';
+import type { ParsedSession, TranscriptFormat } from './types.ts';
+import { transcriptFullId } from './types.ts';
+import { loadImportRedactionPatterns, PAGE_CEILING_FILL, redactSession, type RedactedSession, type RenderedPart } from './render.ts';
+import { deleteStaleTranscriptPart, submitTranscriptPart, transcriptPageCeiling } from './ingest.ts';
+import {
+ addDays, assertTimeZone, localDay, renderArchiveDay, renderDayIndex, sliceSessionByDay, conversationHash8,
+ type ArchiveChannel, type DayIndexEntry,
+} from './archive-day.ts';
+
+export type { ArchiveChannel } from './archive-day.ts';
+
+export const ARCHIVE_ENABLED_KEY = 'transcripts.archive.enabled';
+export const ARCHIVE_TIMEZONE_KEY = 'transcripts.archive.timezone';
+export const ARCHIVE_SOURCE_KEY = 'transcripts.archive.source';
+export const ARCHIVE_BACKFILL_CAP_KEY = 'transcripts.archive.backfill_days_per_run';
+export const ARCHIVE_RUN_WINDOW_KEY = 'transcripts.archive.run_window_days';
+export const ARCHIVE_QUIET_MINUTES_KEY = 'transcripts.archive.quiet_minutes';
+export const ARCHIVE_BACKFILL_HOLD_REASON = 'archive_backfill';
+export const ARCHIVE_OPEN_DAY_HOLD_REASON = 'archive_open_day';
+export const DEFAULT_ARCHIVE_SOURCE = 'conversations';
+export const DEFAULT_BACKFILL_DAYS_PER_RUN = 30;
+export const DEFAULT_RUN_WINDOW_DAYS = 2;
+export const DEFAULT_QUIET_MINUTES = 180;
+const LEDGER_PREFIX = 'transcripts.archive.ledger.';
+const PIN_PREFIX = 'transcripts.archive.pinned_timezone.';
+const LEGACY_PREFIX = 'transcripts.archive.legacy.';
+/** The audience (personal or shared) the archive last ran under, per configured source. */
+const AUDIENCE_PREFIX = 'transcripts.archive.audience.';
+/** Days whose index page still has to be written (a failed or interrupted index write), per source. */
+const INDEX_PENDING_PREFIX = 'transcripts.archive.index_pending.';
+const INGESTED_VIA = 'cli:transcripts-archive';
+const DAY_RE = /^\d{4}-\d{2}-\d{2}$/;
+
+export interface ArchiveConversation {
+ harness: TranscriptFormat;
+ /** The conversation's identity, the same whichever path read it: the session id. */
+ conversationKey: string;
+ /** The harness's own key for the conversation when it differs (the OpenClaw session key): it names the group principal and label, never the page. */
+ scopeKey?: string;
+ session: ParsedSession;
+ channel?: ArchiveChannel;
+ /** Where it was read; a Gateway conversation wins over an archived file of the same OpenClaw session. */
+ via?: 'gateway' | 'file';
+}
+
+/** One conversation found by a scan: its days and how to read it again; no message is held between the two. */
+export interface ArchiveScanEntry {
+ harness: TranscriptFormat;
+ conversationKey: string;
+ sessionId: string;
+ via?: 'gateway' | 'file';
+ /** Local days (dayOf) that hold at least one message. */
+ days: string[];
+ /** Per day, the identities (archiveMessageId) of its messages, so backfill can tell a finished day from a changed one. */
+ dayIds?: Record<string, string[]>;
+ newest?: string;
+ load(): Promise<ArchiveConversation | null>;
+}
+
+export interface ArchiveScanOpts { sinceMs?: number; minMtimeMs?: number; dayOf: (ts: string) => string }
+
+export interface ArchiveReader {
+ harness: TranscriptFormat;
+ name: string;
+ /** Every readable conversation (sinceMs narrows by activity). Throws a coded error; yields nothing partial. */
+ list?(opts: { sinceMs?: number }): Promise<ArchiveConversation[]>;
+ /** Streaming form for large stores: one entry per conversation, parsed one file at a time and loaded again only for the days a run writes. */
+ scan?(opts: ArchiveScanOpts): AsyncIterable<ArchiveScanEntry>;
+ /** Set when the harness has nothing to read on this machine: an informational line, never an error. */
+ unavailable?: string;
+}
+
+export interface ArchiveHeld { code: string; message: string }
+
+export interface ArchiveLedgerEntry {
+ source_id: string;
+ harness: string;
+ conversation_key_hash: string;
+ day: string;
+ hash?: string;
+ complete?: boolean;
+ slug?: string;
+ parts?: number;
+ messages?: number;
+ /** Identities (archiveMessageId) of the messages the page holds; a rewrite that would drop one is kept instead. */
+ ids?: string[];
+ /** Signature of the source's message identities for the day when it was written; backfill skips a finished day whose source still matches. */
+ source?: string;
+ /** Version of the redaction rules (built-in detectors and the user's patterns) the day was written under; backfill re-checks a day written under other rules. */
+ redaction?: string;
+ held?: ArchiveHeld;
+ updated_at?: string;
+}
+
+export type ArchiveDayStatus = 'written' | 'unchanged' | 'held' | 'planned' | 'skipped_legacy' | 'kept' | 'error';
+
+export interface ArchiveDayOutcome {
+ harness: string;
+ conversation_key_hash: string;
+ day: string;
+ status: ArchiveDayStatus;
+ slug?: string;
+ parts?: number;
+ messages?: number;
+ readable_by?: string[];
+ redactions?: number;
+ code?: string;
+ message?: string;
+}
+
+export interface ArchiveReaderError { harness: string; reader: string; code: string; message: string; fix?: Action }
+
+export interface ArchiveRunResult {
+ mode: 'run' | 'backfill';
+ enabled: boolean;
+ dry_run: boolean;
+ source_id: string;
+ timezone: string;
+ write_mode: 'database' | 'repo' | 'coordinator';
+ privacy: 'personal' | 'shared';
+ from: string;
+ to: string;
+ days: ArchiveDayOutcome[];
+ written: number;
+ unchanged: number;
+ held: number;
+ planned: number;
+ skipped_legacy: number;
+ index_pages: string[];
+ deferred_days: string[];
+ info: string[];
+ errors: ArchiveReaderError[];
+ message?: string;
+ fix?: Action;
+}
+
+export interface ArchiveRunOpts {
+ mode: 'run' | 'backfill';
+ from?: string;
+ to?: string;
+ dryRun?: boolean;
+ sourceId?: string;
+ readers?: ArchiveReader[];
+ now?: Date;
+ /** The CLI's trusted-local context: managed brains write through the coordinator with it. */
+ context?: OperationContext;
+ /** Builds that context for the archive source once it exists (the CLI passes its makeContext); called only on a managed brain. */
+ makeContext?: (sourceId: string) => Promise<OperationContext>;
+ /** Test seam: the redaction pass (default redactSession over the whole session). */
+ redact?: (session: ParsedSession) => RedactedSession;
+ /** Repository path: commit the written files (default gbrain git commit-push -- <paths>). */
+ commitPaths?: (root: string, paths: string[], message: string) => Promise<void>;
+ /** Repository path: import the committed files (default an in-process sync of the source). */
+ syncSource?: (engine: BrainEngine, sourceId: string, root: string) => Promise<void>;
+}
+
+const sha256 = (s: string) => createHash('sha256').update(s).digest('hex');
+const truthy = (v: string | null | undefined) => ['1', 'true', 'on', 'yes'].includes(String(v ?? '').trim().toLowerCase());
+
+/** The group principal a group or room conversation may be shared with, once the operator creates it: group:chat-<hash8>. */
+export function archiveGroupPrincipal(harness: TranscriptFormat, conversationKey: string): string {
+ return 'chat-' + conversationHash8(harness, conversationKey);
+}
+
+export async function archiveEnabled(engine: Pick<BrainEngine, 'getConfig'>): Promise<boolean> {
+ return truthy(await engine.getConfig(ARCHIVE_ENABLED_KEY));
+}
+
+export async function setArchiveEnabled(engine: BrainEngine, on: boolean): Promise<void> {
+ await engine.setConfig(ARCHIVE_ENABLED_KEY, on ? 'true' : 'false');
+}
+
+async function configuredSource(engine: BrainEngine, sourceId?: string): Promise<string> {
+ return sourceId ?? ((await engine.getConfig(ARCHIVE_SOURCE_KEY))?.trim() || DEFAULT_ARCHIVE_SOURCE);
+}
+
+/** The source run, status and adopt-legacy all use: on a shared brain whose archive source is a checkout, its owner-only twin. */
+async function effectiveSource(engine: BrainEngine, sourceId?: string): Promise<{ sourceId: string; privacy: Privacy }> {
+ const configured = await configuredSource(engine, sourceId);
+ const privacy = await resolvePrivacy(engine);
+ const [existing] = await engine.executeRaw<{ local_path: string | null }>('SELECT local_path FROM sources WHERE id = $1', [configured]);
+ return { sourceId: privacy.shared && existing?.local_path ? configured + '-owner' : configured, privacy };
+}
+
+/** One message's identity across readers and runs: role, timestamp and text, before redaction. */
+export function archiveMessageId(m: { role: string; timestamp: string; text: string }): string {
+ return createHash('sha256').update(m.role + '\u0000' + m.timestamp + '\u0000' + m.text).digest('hex').slice(0, 16);
+}
+
+const daySignature = (ids: Iterable<string>) => sha256([...new Set(ids)].sort().join(','));
+
+function positiveInt(value: string | null, fallback: number): number {
+ const n = Number(value?.trim());
+ return Number.isSafeInteger(n) && n > 0 ? n : fallback;
+}
+
+function ledgerKey(sourceId: string, harness: string, hash: string, day: string): string {
+ return LEDGER_PREFIX + sourceId + '.' + harness + '.' + hash + '.' + day;
+}
+
+export async function readArchiveLedger(engine: BrainEngine, sourceId: string): Promise<ArchiveLedgerEntry[]> {
+ const prefix = LEDGER_PREFIX + sourceId + '.';
+ const rows = await engine.executeRaw<{ key: string; value: string }>('SELECT key, value FROM config WHERE key LIKE $1 ORDER BY key', [prefix + '%']);
+ const out: ArchiveLedgerEntry[] = [];
+ for (const row of rows) {
+ const [harness, hash, day] = row.key.slice(prefix.length).split('.');
+ if (!harness || !hash || !day) continue;
+ let value: Record<string, unknown> = {};
+ try { value = JSON.parse(row.value) as Record<string, unknown>; } catch { /* an unreadable row reads as empty and is rewritten */ }
+ out.push({ ...value, source_id: sourceId, harness, conversation_key_hash: hash, day } as ArchiveLedgerEntry);
+ }
+ return out;
+}
+
+async function writeLedger(engine: BrainEngine, entry: ArchiveLedgerEntry): Promise<void> {
+ const { source_id, harness, conversation_key_hash, day, ...value } = entry;
+ await engine.setConfig(ledgerKey(source_id, harness, conversation_key_hash, day), JSON.stringify({ ...value, updated_at: new Date().toISOString() }));
+}
+
+async function legacyCovered(engine: BrainEngine, sourceId: string): Promise<Set<string>> {
+ const prefix = LEGACY_PREFIX + sourceId + '.';
+ const rows = await engine.executeRaw<{ key: string }>('SELECT key FROM config WHERE key LIKE $1', [prefix + '%']);
+ return new Set(rows.map(r => r.key.slice(prefix.length)));
+}
+
+function machineTimeZone(): string {
+ return Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC';
+}
+
+/** The pinned timezone, or the one the first run will pin. */
+async function archiveTimeZone(engine: BrainEngine, sourceId: string): Promise<{ timezone: string; pinned: boolean; configured: string }> {
+ const configured = (await engine.getConfig(ARCHIVE_TIMEZONE_KEY))?.trim() || machineTimeZone();
+ const pinned = (await engine.getConfig(`${PIN_PREFIX}${sourceId}`))?.trim();
+ return pinned ? { timezone: pinned, pinned: true, configured } : { timezone: assertTimeZone(configured), pinned: false, configured };
+}
+
+/** unproven: the brain reads as personal only because its client-usage check failed, so it may still be shared. */
+interface Privacy { shared: boolean; owner: string | null; unproven?: boolean }
+
+async function resolvePrivacy(engine: BrainEngine): Promise<Privacy> {
+ const audience = await classifyBrainAudience(engine);
+ const enabled = (await listMembers(engine)).filter(m => m.status === 'enabled').length;
+ const shared = audience.audience !== 'personal' || enabled >= 2;
+ return { shared, owner: shared ? (await ownerView(engine)).owner : null, ...(!shared && audience.unproven ? { unproven: true } : {}) };
+}
+
+async function readersFor(engine: BrainEngine, privacy: Privacy, conv: ArchiveConversation, groups: Map<string, boolean>): Promise<string[] | undefined> {
+ if (!privacy.shared) return undefined;
+ const readers = [privacy.owner!];
+ const ch = conv.channel;
+ if (ch?.verified && (ch.chatType === 'group' || ch.chatType === 'room')) {
+ const principal = archiveGroupPrincipal(conv.harness, conv.scopeKey ?? conv.conversationKey);
+ if (!groups.has(principal)) groups.set(principal, (await listGroups(engine, principal).catch(() => [])).length > 0);
+ if (groups.get(principal)) readers.push('group:' + principal);
+ }
+ return readers;
+}
+
+/**
+ * Pages written while the brain was personal name no readers. When the brain has become shared, the source's whole archive
+ * folder is kept to the owner by a folder rule (a narrowing, so it applies and restamps at once) before the run writes
+ * anything, then the new audience is recorded. A source whose rules live in git is left to its access.yml.
+ */
+async function reconcileAudience(engine: BrainEngine, sourceId: string, privacy: Privacy): Promise<void> {
+ const now = privacy.shared ? 'shared' : 'personal';
+ const recorded = await engine.getConfig(`${AUDIENCE_PREFIX}${sourceId}`);
+ if (recorded === now) return;
+ if (privacy.shared && !privacy.owner) return;
+ if (privacy.shared && recorded === 'personal' && (await engine.executeRaw('SELECT 1 FROM sources WHERE id = $1', [sourceId])).length) {
+ const policy = await import('../access/policy.ts');
+ const { normalizeRule } = await import('../access/folder-rules.ts');
+ if (await policy.readAccessStore(engine, sourceId) === 'git') return;
+ const normalized = normalizeRule({ prefix: 'conversations/', readable_by: [privacy.owner] });
+ if ('problem' in normalized) throw new Error('the owner-only archive rule is invalid: ' + normalized.problem);
+ const rule = normalized.rule;
+ const current = await policy.appliedRules(engine, sourceId);
+ const kept = current.find(r => r.prefix === rule.prefix);
+ if (!kept || kept.readable_by.join() !== rule.readable_by.join()) {
+ await policy.proposeRevision(engine, sourceId, [...current.filter(r => r.prefix !== rule.prefix), rule], { origin: 'cli', store: 'database', proposedBy: 'transcripts-archive' });
+ }
+ }
+ await engine.setConfig(`${AUDIENCE_PREFIX}${sourceId}`, now);
+}
+
+function isGitRepo(path: string | null): boolean {
+ return !!path && existsSync(join(path, '.git'));
+}
+
+async function ensureSource(engine: BrainEngine, sourceId: string): Promise<{ local_path: string | null }> {
+ const [row] = await engine.executeRaw<{ local_path: string | null }>('SELECT local_path FROM sources WHERE id = $1', [sourceId]);
+ if (row) return row;
+ const { addSource } = await import('../sources-ops.ts');
+ await addSource(engine, { id: sourceId, name: 'Conversation archive' });
+ return { local_path: null };
+}
+
+const offFix: Action = { argv: ['gbrain', 'transcripts', 'archive', 'on'], consent: [], actor: 'user', why: 'Turns the conversation archive on for this brain; it is off unless the owner turns it on.', requires_exclusive: false };
+const ownerFix: Action = {
+ argv: ['gbrain', 'access', 'owner', 'set', '<MEMBER>'], consent: [], actor: 'user', ask_user: true, requires_exclusive: false,
+ inputs: [{ name: 'MEMBER', how: 'Ask the user which enabled member is the brain owner (gbrain access member list).' }],
+ why: 'On a shared brain every archive page is kept to the owner member; with none named the archive holds every day and writes nothing.',
+} as Action;
+
+function emptyResult(mode: 'run' | 'backfill', base: Partial<ArchiveRunResult>): ArchiveRunResult {
+ return {
+ mode, enabled: false, dry_run: false, source_id: DEFAULT_ARCHIVE_SOURCE, timezone: 'UTC', write_mode: 'database', privacy: 'personal',
+ from: '', to: '', days: [], written: 0, unchanged: 0, held: 0, planned: 0, skipped_legacy: 0, index_pages: [], deferred_days: [], info: [], errors: [],
+ ...base,
+ };
+}
+
+/** Readers this machine has: file adapters for every harness root, OpenClaw archived files, and the OpenClaw Gateway. */
+export async function defaultArchiveReaders(): Promise<ArchiveReader[]> {
+ const { fileArchiveReaders } = await import('./archive-readers.ts');
+ return fileArchiveReaders();
+}
+
+/** A list reader's conversations as scan entries (in memory already); a scan reader as it is. A list reader that throws yields nothing. */
+export async function* scanReader(reader: ArchiveReader, opts: ArchiveScanOpts): AsyncGenerator<ArchiveScanEntry> {
+ if (reader.scan) { yield* reader.scan(opts); return; }
+ const convs = reader.list ? await reader.list({ sinceMs: opts.sinceMs }) : [];
+ for (const conv of convs) {
+ if (conv.session.messages.length === 0) continue;
+ const stamps = conv.session.messages.map(m => m.timestamp).filter(Boolean);
+ const dayIds: Record<string, string[]> = {};
+ for (const m of conv.session.messages) if (m.timestamp) (dayIds[opts.dayOf(m.timestamp)] ??= []).push(archiveMessageId(m));
+ yield {
+ harness: conv.harness, conversationKey: conv.conversationKey, sessionId: conv.session.meta.sessionId, ...(conv.via ? { via: conv.via } : {}),
+ days: Object.keys(dayIds), dayIds, newest: stamps.sort().pop(), load: async () => conv,
+ };
+ }
+}
+
+/** A Gateway conversation with the session files of the same session id: its key and channel, every message once, oldest first. */
+function mergeScanEntries(gateway: ArchiveScanEntry, files: ArchiveScanEntry[]): ArchiveScanEntry {
+ const dayIds: Record<string, string[]> = {};
+ for (const e of [gateway, ...files]) for (const [day, ids] of Object.entries(e.dayIds ?? {})) dayIds[day] = [...new Set([...(dayIds[day] ?? []), ...ids])];
+ const days = [...new Set([gateway, ...files].flatMap(e => e.days))];
+ const newest = [gateway, ...files].map(e => e.newest).filter((s): s is string => !!s).sort().pop();
+ return {
+ ...gateway, days, dayIds, ...(newest ? { newest } : {}),
+ async load() {
+ const base = await gateway.load();
+ if (!base) return null;
+ const seen = new Set(base.session.messages.map(archiveMessageId));
+ const messages = [...base.session.messages];
+ for (const f of files) {
+ const conv = await f.load();
+ for (const m of conv?.session.messages ?? []) {
+ const id = archiveMessageId(m);
+ if (seen.has(id)) continue;
+ seen.add(id);
+ messages.push(m);
+ }
+ }
+ messages.sort((a, b) => (a.timestamp < b.timestamp ? -1 : a.timestamp > b.timestamp ? 1 : 0));
+ return { ...base, session: { ...base.session, messages } };
+ },
+ };
+}
+
+/** A run writes its pages in batches of this many conversation-days or bytes, so memory stays bounded on a large store. */
+const FLUSH_DAYS = 200;
+/** Bump when the built-in redaction detectors change, so backfill re-redacts finished days written under the old ones. */
+const ARCHIVE_REDACTION_RULES = 'archive-redaction-1';
+const ARCHIVE_LOCK_PREFIX = 'transcripts-archive:';
+const FLUSH_BYTES = 8 * 1024 * 1024;
+
+interface DayWrite {
+ outcome: ArchiveDayOutcome;
+ parts: RenderedPart[];
+ harness: string;
+ raw?: Record<string, unknown>;
+ ledger?: ArchiveLedgerEntry;
+ hold: 'backfill' | 'open' | 'release' | null;
+ staleSlugs: string[];
+}
+
+export async function runTranscriptArchive(engine: BrainEngine, opts: ArchiveRunOpts): Promise<ArchiveRunResult> {
+ const dryRun = opts.dryRun === true;
+ const enabled = await archiveEnabled(engine);
+ if (!enabled && !dryRun) {
+ return emptyResult(opts.mode, {
+ source_id: await configuredSource(engine, opts.sourceId),
+ message: 'The conversation archive is off on this brain, so nothing was read or written. It stays off until the owner turns it on.',
+ fix: offFix,
+ });
+ }
+ const { sourceId, privacy } = await effectiveSource(engine, opts.sourceId);
+ if (dryRun) return archiveSource(engine, opts, { enabled, sourceId, privacy });
+ if (privacy.unproven) {
+ const { opError } = await import('../ops/contract.ts');
+ throw opError('archive_audience_unproven', 'The brain\'s audience could not be checked (its client-usage read failed), so the conversation archive wrote nothing: a page written as personal could be open to other members.',
+ 'Rerun gbrain transcripts archive run; if it repeats, declare the audience with gbrain config set brain.audience personal|shared.');
+ }
+ // One run per source at a time: a run decides from the ledger it read at its start, so an overlapping run could publish
+ // older history over what the other run just wrote.
+ const { withRefreshingLock, LockUnavailableError } = await import('../db-lock.ts');
+ try {
+ return await withRefreshingLock(engine, ARCHIVE_LOCK_PREFIX + sourceId, signal => archiveSource(engine, opts, { enabled, sourceId, privacy, signal }));
+ } catch (err) {
+ if (!(err instanceof LockUnavailableError)) throw err;
+ const { opError } = await import('../ops/contract.ts');
+ throw opError('archive_run_busy', 'Another conversation archive run holds source ' + sourceId + ', so this run read and wrote nothing.',
+ 'Wait for the other run to finish, then rerun; gbrain status --section locks --json shows the holder.');
+ }
+}
+
+async function archiveSource(engine: BrainEngine, opts: ArchiveRunOpts, ctx: { enabled: boolean; sourceId: string; privacy: Privacy; signal?: AbortSignal }): Promise<ArchiveRunResult> {
+ const dryRun = opts.dryRun === true;
+ const now = opts.now ?? new Date();
+ const { enabled, sourceId, privacy } = ctx;
+ const source = dryRun
+ ? (await engine.executeRaw<{ local_path: string | null }>('SELECT local_path FROM sources WHERE id = $1', [sourceId]))[0] ?? { local_path: null }
+ : await ensureSource(engine, sourceId);
+ const context = dryRun ? undefined
+ : opts.context ?? (opts.makeContext && await managedPersistenceEnabled(engine) ? await opts.makeContext(sourceId) : undefined);
+ const managed = !!context && await managedPersistenceEnabled(engine);
+ const writeMode: ArchiveRunResult['write_mode'] = managed ? 'coordinator' : (!privacy.shared && isGitRepo(source.local_path) ? 'repo' : 'database');
+ if (!dryRun) await reconcileAudience(engine, await configuredSource(engine, opts.sourceId), privacy);
+ const tz = await archiveTimeZone(engine, sourceId);
+ if (!tz.pinned && !dryRun) await engine.setConfig(PIN_PREFIX + sourceId, tz.timezone);
+ const today = localDay(now.toISOString(), tz.timezone);
+
+ let from: string;
+ let to: string;
+ if (opts.mode === 'run') {
+ to = today;
+ from = addDays(today, -(positiveInt(await engine.getConfig(ARCHIVE_RUN_WINDOW_KEY), DEFAULT_RUN_WINDOW_DAYS)));
+ } else {
+ if (!opts.from || !DAY_RE.test(opts.from) || (opts.to && !DAY_RE.test(opts.to))) {
+ const { opError } = await import('../ops/contract.ts');
+ throw opError('invalid_params', 'backfill needs --from YYYY-MM-DD (and an optional --to YYYY-MM-DD); nothing was read.',
+ 'Run gbrain transcripts archive backfill --from 2026-01-01 --to 2026-01-31.');
+ }
+ from = opts.from;
+ to = opts.to ?? today;
+ }
+
+ const result = emptyResult(opts.mode, {
+ enabled, dry_run: dryRun, source_id: sourceId, timezone: tz.timezone, write_mode: writeMode,
+ privacy: privacy.shared ? 'shared' : 'personal', from, to,
+ });
+ if (privacy.shared && !privacy.owner) {
+ result.message = 'This brain is shared and names no owner member, so every archive day is held (archive_owner_unset) and nothing was written.';
+ result.fix = ownerFix;
+ }
+
+ const readers = opts.readers ?? await defaultArchiveReaders();
+ const sinceMs = opts.mode === 'run' ? Date.parse(from + 'T00:00:00.000Z') - 2 * 86_400_000 : undefined;
+ const minMtimeMs = opts.mode === 'backfill' ? Date.parse(from + 'T00:00:00.000Z') - 2 * 86_400_000 : sinceMs;
+ const dayOf = (ts: string) => localDay(ts, tz.timezone);
+ const entries: ArchiveScanEntry[] = [];
+ for (const reader of readers) {
+ if (reader.unavailable) { result.info.push(reader.unavailable); continue; }
+ try {
+ let found = 0;
+ for await (const entry of scanReader(reader, { sinceMs, minMtimeMs, dayOf })) { entries.push(entry); found++; }
+ if (found === 0) result.info.push('no readable sessions for ' + reader.harness + '; use gbrain connectors or an export with gbrain transcripts ingest');
+ } catch (err) {
+ if (classifyItemFailure(err).scope === 'run') throw err;
+ const e = err as { code?: string; message?: string; fix?: Action };
+ result.errors.push({ harness: reader.harness, reader: reader.name, code: e.code ?? 'archive_reader_failed', message: e.message ?? String(err), ...(e.fix ? { fix: e.fix } : {}) });
+ }
+ }
+ // A Gateway conversation and the session files of the same session id are one conversation: a reset in place leaves
+ // the earlier messages only in the .jsonl.reset.* file, so the file messages merge in, each message once.
+ const filesBySession = new Map<string, ArchiveScanEntry[]>();
+ for (const e of entries) if (e.harness === 'openclaw' && e.via === 'file') filesBySession.set(e.sessionId, [...(filesBySession.get(e.sessionId) ?? []), e]);
+ const merged = new Set<ArchiveScanEntry>();
+ const scanned: ArchiveScanEntry[] = [];
+ for (const e of entries) {
+ if (e.harness !== 'openclaw' || e.via !== 'gateway' || !filesBySession.has(e.sessionId)) continue;
+ const files = filesBySession.get(e.sessionId)!;
+ files.forEach(f => merged.add(f));
+ merged.add(e);
+ scanned.push(mergeScanEntries(e, files));
+ }
+ scanned.push(...entries.filter(e => !merged.has(e)));
+
+ const patterns = loadImportRedactionPatterns();
+ const redactionVersion = sha256(ARCHIVE_REDACTION_RULES + '\u0000' + patterns.map(p => p.source).join('\u0000'));
+ const ledger = new Map((await readArchiveLedger(engine, sourceId)).map(e => [e.harness + '.' + e.conversation_key_hash + '.' + e.day, e]));
+ const legacy = await legacyCovered(engine, sourceId);
+ const covered = (harness: string, day: string) => legacy.has(harness + '.' + day) || legacy.has('*.' + day);
+
+ interface Item { entry: ArchiveScanEntry; hash: string; days: string[] }
+ const items: Item[] = [];
+ const allDays = new Set<string>();
+ for (const entry of scanned) {
+ const hash = transcriptFullId(entry.conversationKey);
+ const live: string[] = [];
+ for (const day of entry.days.filter(d => d >= from && d <= to)) {
+ if (covered(entry.harness, day)) {
+ result.days.push({ harness: entry.harness, conversation_key_hash: hash, day, status: 'skipped_legacy' });
+ result.skipped_legacy++;
+ continue;
+ }
+ const done = ledger.get(entry.harness + '.' + hash + '.' + day);
+ const sig = entry.dayIds?.[day] ? daySignature(entry.dayIds[day]!) : undefined;
+ if (opts.mode === 'backfill' && done?.hash && !done.held && done.complete && day < today && sig && done.source === sig && done.redaction === redactionVersion) { result.unchanged++; continue; }
+ live.push(day);
+ allDays.add(day);
+ }
+ if (live.length) items.push({ entry, hash, days: live });
+ }
+ // A day held in the ledger for every conversation on it is retried on every backfill run without counting against
+ // the cap, and a day that fails during this run hands its slot to the next older day.
+ const heldDays = new Set([...allDays].filter(day => items.every(it => !it.days.includes(day) || !!ledger.get(it.entry.harness + '.' + it.hash + '.' + day)?.held)));
+ const cap = opts.mode === 'backfill' ? positiveInt(await engine.getConfig(ARCHIVE_BACKFILL_CAP_KEY), DEFAULT_BACKFILL_DAYS_PER_RUN) : 0;
+ const ordered = [...allDays].filter(d => !heldDays.has(d)).sort().reverse();
+
+ const redact = opts.redact ?? ((s: ParsedSession) => redactSession(s, { patterns }));
+ const pageCeiling = dryRun ? Math.floor(PAGE_MAX_BYTES * PAGE_CEILING_FILL) : await transcriptPageCeiling(engine);
+ const quietMs = positiveInt(await engine.getConfig(ARCHIVE_QUIET_MINUTES_KEY), DEFAULT_QUIET_MINUTES) * 60_000;
+ const groups = new Map<string, boolean>();
+ const pending: ArchiveDayOutcome[] = [];
+ const pendingKey = INDEX_PENDING_PREFIX + sourceId;
+ const touched = new Set<string>(dryRun ? [] : readDays(await engine.getConfig(`${INDEX_PENDING_PREFIX}${sourceId}`)));
+ let batch: DayWrite[] = [];
+ let batchBytes = 0;
+ const flushBatch = async () => {
+ if (!batch.length) return;
+ if (ctx.signal?.aborted) throw ctx.signal.reason;
+ await flushWrites(engine, opts, { sourceId, writeMode, root: source.local_path, context }, batch);
+ for (const w of batch) {
+ if (w.outcome.status !== 'written' || !w.ledger) continue;
+ ledger.set(w.ledger.harness + '.' + w.ledger.conversation_key_hash + '.' + w.ledger.day, w.ledger);
+ touched.add(w.outcome.day);
+ }
+ if (!dryRun && touched.size) await engine.setConfig(pendingKey, JSON.stringify([...touched].sort()));
+ batch = [];
+ batchBytes = 0;
+ };
+
+ const hold = async (outcome: ArchiveDayOutcome, held: ArchiveHeld, prior: ArchiveLedgerEntry | undefined) => {
+ Object.assign(outcome, { status: 'held', code: held.code, message: held.message });
+ if (!dryRun) await writeLedger(engine, { ...(prior ?? {}), source_id: sourceId, harness: outcome.harness, conversation_key_hash: outcome.conversation_key_hash, day: outcome.day, held });
+ };
+
+ const processRound = async (allowed: Set<string> | null) => { for (const { entry, hash, days } of items) {
+ const todo = [...(allowed ? days.filter(d => allowed!.has(d)) : days)].sort().reverse();
+ if (todo.length === 0) continue;
+ const outcomes = todo.map(day => ({ harness: entry.harness, conversation_key_hash: hash, day, status: 'planned' as ArchiveDayStatus } as ArchiveDayOutcome));
+ pending.push(...outcomes);
+ const prior = (day: string) => ledger.get(entry.harness + '.' + hash + '.' + day);
+ if (privacy.shared && !privacy.owner) {
+ for (const o of outcomes) await hold(o, { code: 'archive_owner_unset', message: 'The brain is shared and names no owner member.' }, prior(o.day));
+ continue;
+ }
+ let conv: ArchiveConversation | null;
+ try {
+ conv = await entry.load();
+ } catch (err) {
+ conv = null;
+ for (const o of outcomes) Object.assign(o, { status: 'error', code: 'archive_reader_failed', message: (err instanceof Error ? err.message : String(err)).slice(0, 300) });
+ continue;
+ }
+ if (!conv) {
+ for (const o of outcomes) Object.assign(o, { status: 'error', code: 'archive_day_missing', message: 'the conversation could not be read again for this run' });
+ continue;
+ }
+ let redacted: RedactedSession;
+ try {
+ redacted = redact(conv.session);
+ } catch (err) {
+ const message = 'Redaction failed for this session, so its days were held, not written: ' + (err instanceof Error ? err.message : String(err)).slice(0, 200);
+ for (const o of outcomes) await hold(o, { code: 'archive_redaction_failed', message }, prior(o.day));
+ continue;
+ }
+ const readableBy = await readersFor(engine, privacy, conv, groups);
+ const slices = new Map(sliceSessionByDay(redacted.session.messages, tz.timezone).map(s => [s.day, s]));
+ const raw = conv.session.messages.length === redacted.session.messages.length ? conv.session.messages : redacted.session.messages;
+ for (const outcome of outcomes) {
+ const slice = slices.get(outcome.day);
+ if (!slice) { Object.assign(outcome, { status: 'error', code: 'archive_day_missing', message: 'the redacted session has no messages on this day' }); continue; }
+ const complete = outcome.day < today;
+ const ids = [...new Set(slice.indexes.map(i => archiveMessageId(raw[i]!)))];
+ const lastTs = slice.messages[slice.messages.length - 1]?.timestamp;
+ const quiet = !!lastTs && now.getTime() - Date.parse(lastTs) >= quietMs;
+ const hold: DayWrite['hold'] = opts.mode === 'backfill' ? 'backfill' : (!complete && !quiet ? 'open' : 'release');
+ const dayInput = {
+ harness: conv.harness, conversationKey: conv.conversationKey, ...(conv.scopeKey ? { scopeKey: conv.scopeKey } : {}), channel: conv.channel, day: outcome.day, timezone: tz.timezone,
+ redacted, slice, complete, partTargetBytes: pageCeiling, readableBy,
+ };
+ let rendered = renderArchiveDay(dayInput);
+ const contentHash = sha256(rendered.parts.map(p => p.content).join('\u0000'));
+ Object.assign(outcome, {
+ slug: rendered.baseSlug, parts: rendered.parts.length, messages: rendered.messages,
+ redactions: Object.values((rendered.parts[0]!.frontmatter.transcript_archive as { redactions: Record<string, number> }).redactions).reduce((a, b) => a + b, 0),
+ ...(readableBy ? { readable_by: readableBy } : {}),
+ });
+ const before = prior(outcome.day);
+ const lost = before?.ids ? before.ids.some(id => !ids.includes(id)) : (before?.messages ?? 0) > rendered.messages;
+ if (before?.hash && lost) {
+ outcome.status = 'kept';
+ outcome.message = 'the source no longer holds every message the archive wrote for this day (a reset or compaction); the archived page is kept';
+ continue;
+ }
+ if (dryRun) { outcome.status = 'planned'; continue; }
+ if (before?.hash === contentHash && !before.held && await pageExists(engine, sourceId, rendered.baseSlug)) {
+ outcome.status = 'unchanged';
+ if (before.redaction !== redactionVersion) await writeLedger(engine, { ...before, redaction: redactionVersion });
+ if (hold === 'release') await releaseOpenDayHold(engine, sourceId, rendered.parts.map(p => p.slug));
+ continue;
+ }
+ const embeddingHold = hold === 'backfill' ? ARCHIVE_BACKFILL_HOLD_REASON : hold === 'open' ? ARCHIVE_OPEN_DAY_HOLD_REASON : undefined;
+ if (embeddingHold) rendered = renderArchiveDay({ ...dayInput, embeddingHold });
+ const renamed = before?.slug && before.slug !== rendered.baseSlug
+ ? [before.slug, ...Array.from({ length: Math.max(0, (before.parts ?? 1) - 1) }, (_, i) => before.slug + '-p' + (i + 2))]
+ : [];
+ batch.push({
+ outcome,
+ parts: rendered.parts,
+ harness: conv.harness,
+ ...(redacted.session.meta.raw ? { raw: redacted.session.meta.raw } : {}),
+ ledger: { source_id: sourceId, harness: conv.harness, conversation_key_hash: hash, day: outcome.day, hash: contentHash, complete, slug: rendered.baseSlug, parts: rendered.parts.length, messages: rendered.messages, ids, source: daySignature(ids), redaction: redactionVersion },
+ hold,
+ staleSlugs: renamed.length ? renamed : Array.from({ length: Math.max(0, (before?.parts ?? 0) - rendered.parts.length) }, (_, i) => rendered.baseSlug + '-p' + (rendered.parts.length + i + 1)),
+ });
+ batchBytes += rendered.parts.reduce((n, p) => n + p.content.length, 0);
+ }
+ if (batch.length >= FLUSH_DAYS || batchBytes >= FLUSH_BYTES) await flushBatch();
+ } };
+ if (opts.mode !== 'backfill') await processRound(null);
+ else {
+ let next = 0;
+ let good = 0;
+ let round = new Set([...heldDays, ...ordered.slice(0, cap)]);
+ next = Math.min(cap, ordered.length);
+ while (round.size) {
+ const start = pending.length;
+ await processRound(round);
+ const ok = new Set(pending.slice(start).filter(o => o.status !== 'held' && o.status !== 'error').map(o => o.day));
+ good += [...round].filter(d => !heldDays.has(d) && ok.has(d)).length;
+ round = new Set(ordered.slice(next, next + Math.max(0, cap - good)));
+ next += round.size;
+ }
+ result.deferred_days = ordered.slice(next).sort();
+ }
+ const indexFor = (days: Set<string>, view: Map<string, ArchiveLedgerEntry>): DayWrite[] => [...days].sort().map(day => {
+ const dayEntries: DayIndexEntry[] = [];
+ for (const e of view.values()) {
+ if (e.day !== day || !e.slug || e.held) continue;
+ dayEntries.push({ slug: e.slug, harness: e.harness, messages: e.messages ?? 0 });
+ }
+ const page = renderDayIndex(day, tz.timezone, dayEntries, privacy.shared ? [privacy.owner!] : undefined);
+ return { outcome: { harness: 'index', conversation_key_hash: '', day, status: 'planned', slug: page.slug }, parts: [page], harness: 'index', hold: null, staleSlugs: [] };
+ });
+ let indexWrites: DayWrite[] = [];
+ if (writeMode === 'repo') {
+ // A repo commit carries the last batch's days with their index pages, so a small run is one commit.
+ const view = new Map(ledger);
+ const days = new Set(touched);
+ for (const w of batch) if (w.ledger) { view.set(w.ledger.harness + '.' + w.ledger.conversation_key_hash + '.' + w.ledger.day, w.ledger); days.add(w.outcome.day); }
+ indexWrites = indexFor(days, view);
+ batch.push(...indexWrites);
+ await flushBatch();
+ } else {
+ await flushBatch();
+ if (touched.size) {
+ indexWrites = indexFor(touched, ledger);
+ await flushWrites(engine, opts, { sourceId, writeMode, root: source.local_path, context }, indexWrites);
+ }
+ }
+ if (!dryRun && touched.size) await engine.setConfig(pendingKey, JSON.stringify(indexWrites.filter(w => w.outcome.status !== 'written' && w.outcome.status !== 'unchanged').map(w => w.outcome.day).sort()));
+ for (const w of indexWrites) {
+ if (w.outcome.status === 'written' || w.outcome.status === 'unchanged') result.index_pages.push(w.outcome.slug!);
+ else if (w.outcome.status === 'error') result.errors.push({ harness: 'index', reader: 'day-index', code: w.outcome.code ?? 'archive_commit_failed', message: 'the day index ' + w.outcome.slug + ' was not written: ' + (w.outcome.message ?? 'no detail') });
+ }
+
+ pending.sort((a, b) => b.day.localeCompare(a.day));
+ result.days.push(...pending);
+ for (const d of result.days) {
+ if (d.status === 'written') result.written++;
+ else if (d.status === 'unchanged') result.unchanged++;
+ else if (d.status === 'held') result.held++;
+ else if (d.status === 'planned') result.planned++;
+ }
+ if (result.deferred_days.length) result.info.push(result.deferred_days.length + ' older day(s) wait for the next backfill run (' + ARCHIVE_BACKFILL_CAP_KEY + ' caps one run)');
+ return result;
+}
+
+function readDays(value: string | null): string[] {
+ try {
+ const days = JSON.parse(value ?? '[]') as unknown;
+ return Array.isArray(days) ? days.filter((d): d is string => typeof d === 'string' && DAY_RE.test(d)) : [];
+ } catch {
+ return [];
+ }
+}
+
+async function pageExists(engine: BrainEngine, sourceId: string, slug: string): Promise<boolean> {
+ return (await engine.executeRaw('SELECT 1 FROM pages WHERE source_id = $1 AND slug = $2 AND deleted_at IS NULL', [sourceId, slug])).length > 0;
+}
+
+async function defaultCommit(root: string, paths: string[], message: string): Promise<void> {
+ const { commitPushChild } = await import('../git/reconcile.ts');
+ await commitPushChild(root, ['-m', message], undefined, paths);
+}
+
+async function defaultSync(engine: BrainEngine, sourceId: string, root: string): Promise<void> {
+ const { performSync } = await import('../../commands/sync/perform.ts');
+ await performSync(engine, { sourceId, repoPath: root, noPull: true, noEmbed: true, noExtract: true });
+}
+
+/** Writes every rendered page through the brain's one write path, then records raw data, embedding holds and the ledger. */
+/** A closed or quiet day's open-day hold is opened, not deleted, so the page's frontmatter marker never holds it again. */
+async function releaseOpenDayHold(engine: BrainEngine, sourceId: string | null, slugs: string[], pageId?: number): Promise<void> {
+ const ids = pageId !== undefined ? [pageId]
+ : (await engine.executeRaw<{ id: number }>('SELECT id FROM pages WHERE source_id = $1 AND slug = ANY($2::text[]) AND deleted_at IS NULL', [sourceId, slugs])).map(r => r.id);
+ if (!ids.length) return;
+ await engine.executeRaw("UPDATE page_embedding_budget SET state = 'open', reason = NULL, updated_at = now() WHERE page_id = ANY($1::int[]) AND state = 'held' AND reason = $2", [ids, ARCHIVE_OPEN_DAY_HOLD_REASON]);
+}
+
+async function flushWrites(engine: BrainEngine, opts: ArchiveRunOpts, target: { sourceId: string; writeMode: ArchiveRunResult['write_mode']; root: string | null; context?: OperationContext }, writes: DayWrite[]): Promise<void> {
+ const { sourceId, writeMode } = target;
+ const provenance = (harness: string) => ({ source_kind: 'transcript:' + harness, source_uri: 'archive:' + harness, ingested_via: INGESTED_VIA });
+ if (writeMode === 'repo') {
+ const root = target.root!;
+ const paths: string[] = [];
+ for (const w of writes) {
+ for (const part of w.parts) {
+ const file = join(root, part.slug + '.md');
+ mkdirSync(dirname(file), { recursive: true });
+ writeFileSync(file, part.content);
+ paths.push(relative(root, file).split(sep).join('/'));
+ }
+ for (const stale of w.staleSlugs) {
+ const file = join(root, stale + '.md');
+ if (existsSync(file)) { rmSync(file); paths.push(relative(root, file).split(sep).join('/')); }
+ }
+ }
+ try {
+ await (opts.commitPaths ?? defaultCommit)(root, paths, 'gbrain: archive ' + writes.filter(w => w.ledger).length + ' conversation day(s)');
+ await (opts.syncSource ?? defaultSync)(engine, sourceId, root);
+ } catch (err) {
+ const e = err as { code?: string; message?: string };
+ for (const w of writes) Object.assign(w.outcome, { status: 'error', code: e.code ?? 'archive_commit_failed', message: (e.message ?? String(err)).slice(0, 300) });
+ return;
+ }
+ for (const w of writes) w.outcome.status = 'written';
+ } else {
+ for (const w of writes) {
+ try {
+ for (const part of w.parts) {
+ const r = writeMode === 'coordinator'
+ ? await submitTranscriptPart(target.context!, engine, sourceId, part, provenance(w.harness))
+ : await importFromContent(engine, part.slug, part.content, { sourceId, noEmbed: true, ...provenance(w.harness) });
+ if (r.status === 'error') throw new Error('the page write for ' + part.slug + ' returned an error');
+ }
+ for (const stale of w.staleSlugs) {
+ if (writeMode === 'coordinator') await deleteStaleTranscriptPart(target.context!, engine, sourceId, stale);
+ else await engine.deletePages([stale], { sourceId });
+ }
+ w.outcome.status = 'written';
+ } catch (err) {
+ if (classifyItemFailure(err).scope === 'run') throw err;
+ const e = err as { code?: string; message?: string };
+ Object.assign(w.outcome, { status: 'error', code: e.code ?? classifyItemFailure(err).code, message: (e.message ?? String(err)).slice(0, 300) });
+ }
+ }
+ }
+ for (const w of writes) {
+ if (w.outcome.status !== 'written') continue;
+ if (w.raw) await engine.putRawData(w.parts[0]!.slug, 'transcript:' + w.harness, w.raw, { sourceId });
+ if (w.hold) {
+ const ids = await engine.executeRaw<{ id: number }>('SELECT id FROM pages WHERE source_id = $1 AND slug = ANY($2::text[]) AND deleted_at IS NULL', [sourceId, w.parts.map(p => p.slug)]);
+ for (const { id } of ids) {
+ if (w.hold === 'backfill') await holdPageEmbedding(engine, id, ARCHIVE_BACKFILL_HOLD_REASON);
+ else if (w.hold === 'open') await holdPageEmbedding(engine, id, ARCHIVE_OPEN_DAY_HOLD_REASON);
+ else await releaseOpenDayHold(engine, null, [], id);
+ }
+ }
+ if (w.ledger) await writeLedger(engine, w.ledger);
+ }
+}
+
+// ── status ──────────────────────────────────────────────────────────────────
+
+export interface ArchiveStatus {
+ enabled: boolean;
+ source_id: string;
+ timezone: string | null;
+ configured_timezone: string;
+ harnesses: Array<{ harness: string; reader: string; conversations: number; newest_session_day: string | null; newest_archived_day: string | null; info?: string }>;
+ archived_days: number;
+ held: Array<{ harness: string; day: string; code: string }>;
+ legacy_days: number;
+ errors: ArchiveReaderError[];
+}
+
+export async function archiveStatus(engine: BrainEngine, opts: { sourceId?: string; readers?: ArchiveReader[] } = {}): Promise<ArchiveStatus> {
+ const { sourceId } = await effectiveSource(engine, opts.sourceId);
+ const pinned = (await engine.getConfig(`${PIN_PREFIX}${sourceId}`))?.trim() || null;
+ const configured = (await engine.getConfig(ARCHIVE_TIMEZONE_KEY))?.trim() || machineTimeZone();
+ const entries = await readArchiveLedger(engine, sourceId);
+ const out: ArchiveStatus = {
+ enabled: await archiveEnabled(engine), source_id: sourceId, timezone: pinned, configured_timezone: configured, harnesses: [],
+ archived_days: entries.filter(e => e.hash && !e.held).length,
+ held: entries.filter(e => e.held).map(e => ({ harness: e.harness, day: e.day, code: e.held!.code })),
+ legacy_days: (await legacyCovered(engine, sourceId)).size, errors: [],
+ };
+ for (const reader of opts.readers ?? await defaultArchiveReaders()) {
+ const archived = entries.filter(e => e.harness === reader.harness && e.hash).map(e => e.day).sort();
+ const row = { harness: reader.harness, reader: reader.name, conversations: 0, newest_session_day: null as string | null, newest_archived_day: archived[archived.length - 1] ?? null };
+ if (reader.unavailable) { out.harnesses.push({ ...row, info: reader.unavailable }); continue; }
+ try {
+ const zone = pinned ?? configured;
+ let conversations = 0;
+ let newest: string | undefined;
+ for await (const entry of scanReader(reader, { dayOf: ts => localDay(ts, zone) })) {
+ conversations++;
+ if (entry.newest && (!newest || entry.newest > newest)) newest = entry.newest;
+ }
+ out.harnesses.push({ ...row, conversations, newest_session_day: newest ? localDay(newest, zone) : null,
+ ...(conversations ? {} : { info: 'no readable sessions for ' + reader.harness + '; use gbrain connectors or an export with gbrain transcripts ingest' }) });
+ } catch (err) {
+ const e = err as { code?: string; message?: string; fix?: Action };
+ out.errors.push({ harness: reader.harness, reader: reader.name, code: e.code ?? 'archive_reader_failed', message: e.message ?? String(err), ...(e.fix ? { fix: e.fix } : {}) });
+ out.harnesses.push(row);
+ }
+ }
+ return out;
+}
+
+// ── adopt-legacy ────────────────────────────────────────────────────────────
+
+const HARNESSES: readonly string[] = ['claude-code', 'codex', 'openclaw', 'hermes', 'grok', 'chatgpt', 'claude-export'];
+
+export interface LegacyDay { harness: string; day: string; file: string }
+export interface AdoptLegacyResult { dir: string; source_id: string; applied: boolean; covered: LegacyDay[]; by_harness: Record<string, number> }
+
+function legacyFiles(dir: string, base = dir, depth = 0, out: string[] = []): string[] {
+ if (depth > 6) return out;
+ let names: string[];
+ try { names = readdirSync(dir).sort(); } catch { return out; }
+ for (const name of names) {
+ if (name.startsWith('.')) continue;
+ const path = join(dir, name);
+ const st = lstatSync(path);
+ if (st.isSymbolicLink()) continue;
+ if (st.isDirectory()) legacyFiles(path, base, depth + 1, out);
+ else if (st.isFile()) out.push(path);
+ }
+ return out;
+}
+
+function frontmatterField(head: string, field: string): string | null {
+ const m = /^---\r?\n([\s\S]*?)\r?\n---/.exec(head);
+ if (!m) return null;
+ for (const line of m[1]!.split(/\r?\n/)) {
+ if (!line.startsWith(field + ':')) continue;
+ const value = line.slice(field.length + 1).trim().replace(/^["']/, '');
+ const end = value.search(/["']/);
+ const out = (end < 0 ? value : value.slice(0, end)).trim();
+ return out || null;
+ }
+ return null;
+}
+
+/**
+ * Reads an old archive folder's day files (a YYYY-MM-DD in the file name, else a date: in the frontmatter) and lists
+ * the days they cover per harness (from a harness: field, a folder or file name naming one, else every harness). With
+ * apply it records each day as covered-by-legacy, so run and backfill skip it. Never edits, moves or deletes a file.
+ */
+export async function adoptLegacyArchive(engine: BrainEngine, opts: { dir: string; apply: boolean; sourceId?: string }): Promise<AdoptLegacyResult> {
+ const { sourceId } = await effectiveSource(engine, opts.sourceId);
+ if (!existsSync(opts.dir) || !statSync(opts.dir).isDirectory()) {
+ const { opError } = await import('../ops/contract.ts');
+ throw opError('invalid_params', opts.dir + ' is not a folder, so nothing was adopted.', 'Pass the old archive folder with --dir <folder>.');
+ }
+ const covered = new Map<string, LegacyDay>();
+ for (const path of legacyFiles(opts.dir)) {
+ const rel = relative(opts.dir, path).split(sep).join('/');
+ let day = /(\d{4}-\d{2}-\d{2})/.exec(rel.split('/').pop()!)?.[1] ?? null;
+ let head = '';
+ if (/\.(md|markdown|txt)$/i.test(path)) {
+ try { head = readFileSync(path, 'utf8').slice(0, 4096); } catch { head = ''; }
+ }
+ if (!day) day = frontmatterField(head, 'date')?.slice(0, 10) ?? null;
+ if (!day || !DAY_RE.test(day) || Number.isNaN(Date.parse(day))) continue;
+ const named = frontmatterField(head, 'harness');
+ const harness = (named && HARNESSES.includes(named) ? named : null)
+ ?? HARNESSES.find(h => rel.split('/').slice(0, -1).includes(h))
+ ?? HARNESSES.find(h => rel.split('/').pop()!.includes(h))
+ ?? '*';
+ const key = harness + '.' + day;
+ if (!covered.has(key)) covered.set(key, { harness, day, file: rel });
+ }
+ const list = [...covered.values()].sort((a, b) => a.day.localeCompare(b.day) || a.harness.localeCompare(b.harness));
+ if (opts.apply) {
+ for (const d of list) await engine.setConfig(LEGACY_PREFIX + sourceId + '.' + d.harness + '.' + d.day, JSON.stringify({ file: d.file, adopted_at: new Date().toISOString() }));
+ }
+ const by_harness: Record<string, number> = {};
+ for (const d of list) by_harness[d.harness] = (by_harness[d.harness] ?? 0) + 1;
+ return { dir: opts.dir, source_id: sourceId, applied: opts.apply, covered: list, by_harness };
+}
diff --git a/src/core/transcripts/discover.ts b/src/core/transcripts/discover.ts
index e9ae43d970..794014bc65 100644
--- a/src/core/transcripts/discover.ts
+++ b/src/core/transcripts/discover.ts
@@ -21,7 +21,7 @@ import { join } from 'node:path';
import type { BrainEngine } from '../engine.ts';
import type { TranscriptFormat } from './types.ts';
import { harnessRoots, type HarnessRoot } from './detect.ts';
-import { isOpenclawCheckpointFile } from './openclaw.ts';
+import { isOpenclawArchivedFile, isOpenclawCheckpointFile, isOpenclawSkippedArtifact } from './openclaw.ts';
import { isGrokChatHistoryFile } from './grok.ts';
import { isClaudeCodeSubagentFile, isClaudeCodeWorkflowArtifactFile } from './claude-code.ts';
import { isClaudeCliSelfTranscriptPath } from '../ai/providers/claude-cli-scratch.ts';
@@ -44,6 +44,12 @@ export interface DiscoverOpts {
* escape hatch for deliberately auditing the provider's own calls.
*/
includeSelf?: boolean;
+ /**
+ * GST-2768: also list OpenClaw's retained session files (`.jsonl.reset.*`, `.jsonl.deleted.*.zst`, `.jsonl.bak.*`).
+ * They share a session id with the live file, so only the conversation archive, which merges every file of one
+ * session, asks for them; ingest keeps one file per session.
+ */
+ includeArchived?: boolean;
}
/** Recursively list regular files under root (lstat: symlinks are skipped). */
@@ -85,8 +91,8 @@ export function discoverTranscriptFiles(roots?: HarnessRoot[], opts: DiscoverOpt
const files: string[] = [];
walk(root, files);
for (const p of files) {
- if (!p.endsWith(extension)) continue;
- if (isOpenclawCheckpointFile(p)) continue;
+ if (!p.endsWith(extension) && !(opts.includeArchived && format === 'openclaw' && isOpenclawArchivedFile(p))) continue;
+ if (isOpenclawCheckpointFile(p) || (format === 'openclaw' && isOpenclawSkippedArtifact(p))) continue;
// Grok: only chat_history.jsonl is a session; updates/events/rewind
// JSONL next to it would otherwise inflate found-counts and freeze
// the watermark as host-format drift. SCOPED to the grok root — a
diff --git a/src/core/transcripts/ingest.ts b/src/core/transcripts/ingest.ts
index 5ed29a1d7a..cbbfd47dcc 100644
--- a/src/core/transcripts/ingest.ts
+++ b/src/core/transcripts/ingest.ts
@@ -550,7 +550,7 @@ async function importSession(engine: BrainEngine, opts: TranscriptsIngestOpts, r
* still publishing waits on that write (transcriptPartRequest) instead of
* admitting a second one.
*/
-async function submitTranscriptPart(
+export async function submitTranscriptPart(
ctx: OperationContext,
engine: BrainEngine,
sourceId: string,
@@ -612,7 +612,7 @@ async function transcriptPartRequest(ctx: OperationContext, sourceId: string, sl
* That refusal (owner_unavailable) pauses background work only, so it is this session's error, never the run's:
* foreground parts of every other session are still admitted.
*/
-async function deleteStaleTranscriptPart(ctx: OperationContext, engine: BrainEngine, sourceId: string, slug: string): Promise<void> {
+export async function deleteStaleTranscriptPart(ctx: OperationContext, engine: BrainEngine, sourceId: string, slug: string): Promise<void> {
const snapshot = await engine.readPageSnapshot(slug, { sourceId });
if (!snapshot) return;
const receipt = await submitPageMutation({ ...ctx, admissionOrigin: 'background' }, { operation: 'delete_page', params: { slug, source_id: sourceId, expected_revision: snapshot.revision, request_id: randomUUID() } })
@@ -661,7 +661,7 @@ function adoptExistingBaseSlug(rendered: RenderSessionResult, existingSlug: stri
* line would land with embedding skipped), times PAGE_CEILING_FILL for the
* frontmatter. Read once per run.
*/
-async function transcriptPageCeiling(engine: BrainEngine): Promise<number> {
+export async function transcriptPageCeiling(engine: BrainEngine): Promise<number> {
const [policy, block] = await Promise.all([readPageBodyPolicy(engine), effectiveBytesBlock(engine)]);
return Math.floor(Math.min(policy.maxBytes, block) * PAGE_CEILING_FILL);
}
diff --git a/src/core/transcripts/openclaw-gateway.ts b/src/core/transcripts/openclaw-gateway.ts
new file mode 100644
index 0000000000..d85dbfaa1b
--- /dev/null
+++ b/src/core/transcripts/openclaw-gateway.ts
@@ -0,0 +1,243 @@
+/**
+ * openclaw-gateway.ts — the conversation archive's reader for current OpenClaw (GST-2768).
+ *
+ * Current OpenClaw keeps live session rows and transcripts in per-agent SQLite owned by its Gateway; gbrain never
+ * opens that file. It reads through the Gateway's RPCs, called with the documented CLI
+ * (openclaw gateway call <method> --json --params <json>), which also owns the Gateway credential and device
+ * pairing. Verified against OpenClaw 2026.11.3 (test/fixtures/openclaw-gateway/README.md):
+ *
+ * - sessions.list {limit, offset}: rows with key, sessionId, agentId, chatType (direct | group | room), createdAt,
+ * updatedAt (epoch ms); paged by hasMore / nextOffset. Archived sessions are not listed, and the call takes no
+ * parameter that lists them.
+ * - chat.history {sessionKey, limit, offset}: messages {role, content (string or [{type: text, text}]),
+ * timestamp (epoch ms), __openclaw {id, seq}}; pages walk backward from the newest (hasMore / nextOffset), each
+ * page oldest first. It still reads an archived session by key. A reset in place keeps the sessionId and drops
+ * the earlier messages from this read.
+ * - Errors come back as a JSON envelope {ok: false, error: {type, code?, message, details?}}: a closed port is
+ * gateway_transport_error, a missing credential gateway_credentials_required, a wrong one gateway_request_error
+ * with details.code AUTH_*.
+ *
+ * Channel metadata: only the row's chatType counts, and only as one of the three documented values. The key's shape
+ * and the derived kind / classification fields never do (a session created with a group-shaped key reads kind
+ * group and chatType direct). Anything else is unknown, which the archive resolves owner-only.
+ *
+ * Failures throw archive_gateway_unreachable, archive_gateway_auth or archive_gateway_failed with the Gateway's
+ * message redacted; the reader yields nothing before every read finished, so a failed run changes no ledger row.
+ */
+
+import { execFileBounded } from '../bounded-child-exec.ts';
+import { opError } from '../ops/contract.ts';
+import { applyRedaction, planRedaction } from '../secret-scan.ts';
+import { sanitizeForJsonb } from '../batch-rows.ts';
+import type { TranscriptMessage } from './types.ts';
+import { utcTimestamp } from './types.ts';
+import type { ArchiveChannel } from './archive-day.ts';
+import { conversationHash8 } from './archive-day.ts';
+
+export interface GatewayTransport {
+ call(method: string, params: Record<string, unknown>): Promise<unknown>;
+}
+
+export interface GatewayConversation {
+ harness: 'openclaw';
+ /** The session id: one identity whether the Gateway or a session file read it. */
+ conversationKey: string;
+ /** The session key, for the group principal and label only. */
+ scopeKey: string;
+ via: 'gateway';
+ channel: ArchiveChannel;
+ session: { meta: { harness: 'openclaw'; sessionId: string; startedAt?: string; raw: Record<string, unknown> }; messages: TranscriptMessage[] };
+}
+
+const LIST_PAGE = 100;
+const HISTORY_PAGE = 200;
+const MAX_PAGES = 10_000;
+
+/** Redact and bound text that came from the Gateway before it reaches a message, a log or a receipt. */
+export function redactGatewayText(text: string): string {
+ const clean = applyRedaction(planRedaction(sanitizeForJsonb(text), { highEntropy: true, labeledCredentials: true }));
+ return clean.length > 500 ? clean.slice(0, 500) + '...' : clean;
+}
+
+function gatewayError(kind: 'unreachable' | 'auth' | 'failed', detail: string): Error {
+ const message = redactGatewayText(detail);
+ if (kind === 'unreachable') {
+ return opError('archive_gateway_unreachable', 'The OpenClaw Gateway did not answer, so no OpenClaw conversation was archived this run: ' + message,
+ 'Start the Gateway (openclaw gateway) or point gbrain at it (GBRAIN_OPENCLAW_GATEWAY_URL), then rerun gbrain transcripts archive run.',
+ { retryable: true, fix: { argv: ['openclaw', 'health', '--json'], consent: [], actor: 'agent', why: 'Shows whether the local OpenClaw Gateway is running, read-only.', requires_exclusive: false } });
+ }
+ if (kind === 'auth') {
+ return opError('archive_gateway_auth', 'The OpenClaw Gateway refused gbrain\'s credential, so no OpenClaw conversation was archived this run: ' + message,
+ 'Ask the user to give gbrain the Gateway token (GBRAIN_OPENCLAW_GATEWAY_TOKEN, or gateway.auth in the OpenClaw config on this machine), then rerun gbrain transcripts archive run.',
+ { fix: { argv: ['openclaw', 'gateway', 'call', 'health', '--json'], consent: [], actor: 'user', why: 'Checks that the Gateway accepts the configured credential, read-only.', requires_exclusive: false } });
+ }
+ return opError('archive_gateway_failed', 'The OpenClaw Gateway returned an error, so no OpenClaw conversation was archived this run: ' + message,
+ 'Rerun gbrain transcripts archive run; if it repeats, run gbrain transcripts archive status --json and report the message.', { retryable: true });
+}
+
+/** One CLI result: the parsed document, or the stable archive error for its error envelope. */
+export function parseGatewayCliOutput(stdout: string, exitCode: number | null): unknown {
+ let doc: unknown;
+ try {
+ doc = JSON.parse(stdout.trim());
+ } catch {
+ throw gatewayError('unreachable', 'openclaw gateway call exited ' + String(exitCode) + ' without a JSON result: ' + stdout.trim().slice(0, 300));
+ }
+ const rec = doc && typeof doc === 'object' ? doc as Record<string, unknown> : null;
+ if ((!rec || rec.ok !== false) && (exitCode === 0 || exitCode === null)) return doc;
+ if (!rec || rec.ok !== false) throw gatewayError('failed', 'openclaw gateway call exited ' + String(exitCode) + ': ' + stdout.trim().slice(0, 300));
+ const err = (rec.error && typeof rec.error === 'object' ? rec.error : {}) as Record<string, unknown>;
+ const type = String(err.type ?? '');
+ const message = String(err.message ?? 'no detail');
+ const details = (err.details && typeof err.details === 'object' ? err.details : {}) as Record<string, unknown>;
+ if (type === 'gateway_transport_error') throw gatewayError('unreachable', message);
+ if (type === 'gateway_credentials_required' || /^AUTH_/.test(String(details.code ?? '')) || /unauthori[sz]ed|forbidden/i.test(message)) throw gatewayError('auth', message);
+ throw gatewayError('failed', message);
+}
+
+const DEFAULT_MAX_OUTPUT_BYTES = 256 * 1024 * 1024;
+
+/** The parent variables the openclaw child gets: where binaries and the home folder are, and the locale. */
+const CHILD_ENV_ALLOW = new Set(['PATH', 'HOME', 'USERPROFILE', 'SYSTEMROOT', 'LANG', 'LC_ALL', 'LC_CTYPE', 'LC_MESSAGES']);
+
+/** The default transport: the documented CLI on this machine. The token, when gbrain holds one, goes through the environment, never argv. */
+export function cliGatewayTransport(opts: {
+ bin?: string; url?: string; token?: string; timeoutMs?: number;
+ /** Largest stdout the child may write before it is stopped (default 256 MiB). Internal: tests pass a small bound. */
+ maxOutputBytes?: number;
+ /** Test seam: the child launcher (default execFileBounded: array arguments, no shell, SIGKILL after the TERM grace). */
+ spawn?: typeof execFileBounded;
+} = {}): GatewayTransport {
+ const bin = opts.bin ?? 'openclaw';
+ const maxOutputBytes = opts.maxOutputBytes ?? DEFAULT_MAX_OUTPUT_BYTES;
+ const launch = opts.spawn ?? execFileBounded;
+ // OpenClaw's own OPENCLAW_GATEWAY_URL/_TOKEN count when gbrain's are unset: a working-folder .env cannot set them (env-trust).
+ const { GBRAIN_OPENCLAW_GATEWAY_URL, GBRAIN_OPENCLAW_GATEWAY_TOKEN, OPENCLAW_GATEWAY_URL, OPENCLAW_GATEWAY_TOKEN } = process.env;
+ const url = opts.url ?? GBRAIN_OPENCLAW_GATEWAY_URL ?? OPENCLAW_GATEWAY_URL;
+ const urlVar = opts.url !== undefined || GBRAIN_OPENCLAW_GATEWAY_URL !== undefined ? 'GBRAIN_OPENCLAW_GATEWAY_URL' : 'OPENCLAW_GATEWAY_URL';
+ const token = opts.token ?? GBRAIN_OPENCLAW_GATEWAY_TOKEN ?? OPENCLAW_GATEWAY_TOKEN;
+ const timeoutMs = opts.timeoutMs ?? 60_000;
+ return {
+ call(method, params) {
+ if (url && !/^(wss?|https?):\/\/[^\s]+$/i.test(url)) {
+ return Promise.reject(gatewayError('failed', urlVar + ' must use the ws, wss, http or https scheme; nothing was run'));
+ }
+ const args = ['gateway', 'call', method, '--json', '--params', JSON.stringify(params), '--timeout', String(timeoutMs)];
+ // The URL goes through OPENCLAW_GATEWAY_URL: OpenClaw refuses a --url override whose credential comes only from
+ // the environment (gateway_credentials_required, 2026.10.9 and 2026.11.3), and accepts the environment pair.
+ // Only the allow-listed parent variables pass, so no inherited OPENCLAW_* value (a cwd .env's included) redirects the child.
+ const env: NodeJS.ProcessEnv = {};
+ for (const [k, v] of Object.entries(process.env)) if (v !== undefined && CHILD_ENV_ALLOW.has(k)) env[k] = v;
+ if (url) env.OPENCLAW_GATEWAY_URL = url;
+ if (token) env.OPENCLAW_GATEWAY_TOKEN = token;
+ return launch(bin, args, { env, timeout: timeoutMs + 10_000, maxBuffer: maxOutputBytes, killTree: true }).then(({ error, stdout }) => {
+ const code = error && typeof (error as NodeJS.ErrnoException).code === 'string' ? (error as NodeJS.ErrnoException).code : null;
+ if (code === 'ENOENT') throw gatewayError('unreachable', bin + ' is not on PATH');
+ if (code === 'ERR_CHILD_PROCESS_STDIO_MAXBUFFER') throw gatewayError('failed', 'openclaw gateway call wrote more than ' + maxOutputBytes + ' bytes, so it was stopped and nothing it wrote was read');
+ return parseGatewayCliOutput(String(stdout), error ? (typeof error.code === 'number' ? error.code : 1) : 0);
+ });
+ },
+ };
+}
+
+/** The verified channel of one sessions.list row. */
+export function gatewayChannel(row: Record<string, unknown>): ArchiveChannel {
+ const chatType = row.chatType;
+ if (chatType !== 'direct' && chatType !== 'group' && chatType !== 'room') return { verified: false };
+ const key = String(row.key ?? '');
+ const cron = /:cron:([^:]+)/.exec(key);
+ return { chatType, verified: true, ...(cron ? { cronJob: cron[1] } : {}) };
+}
+
+function messageText(content: unknown): string {
+ if (typeof content === 'string') return content.trim();
+ if (!Array.isArray(content)) return '';
+ return content
+ .filter((b): b is Record<string, unknown> => !!b && typeof b === 'object' && (b as Record<string, unknown>).type === 'text' && typeof (b as Record<string, unknown>).text === 'string')
+ .map(b => String(b.text))
+ .filter(t => t.trim())
+ .join('\n')
+ .trim();
+}
+
+function messageTimestamp(value: unknown): string {
+ if (typeof value === 'number' && Number.isFinite(value)) return new Date(value).toISOString();
+ return typeof value === 'string' ? utcTimestamp(value) : '';
+}
+
+const asRecord = (v: unknown): Record<string, unknown> => (v && typeof v === 'object' ? v as Record<string, unknown> : {});
+
+async function readHistory(transport: GatewayTransport, sessionKey: string): Promise<{ sessionId?: string; messages: TranscriptMessage[] }> {
+ const pages: Array<Record<string, unknown>> = [];
+ let offset = 0;
+ for (let i = 0; i < MAX_PAGES; i++) {
+ const page = asRecord(await transport.call('chat.history', { sessionKey, limit: HISTORY_PAGE, ...(offset ? { offset } : {}) }));
+ pages.push(page);
+ if (page.hasMore !== true || typeof page.nextOffset !== 'number' || page.nextOffset <= offset) break;
+ offset = page.nextOffset;
+ }
+ const seen = new Set<string>();
+ const messages: TranscriptMessage[] = [];
+ for (const page of pages.reverse()) {
+ for (const raw of Array.isArray(page.messages) ? page.messages : []) {
+ const m = asRecord(raw);
+ const id = asRecord(m.__openclaw).id;
+ if (typeof id === 'string') {
+ if (seen.has(id)) continue;
+ seen.add(id);
+ }
+ if (m.role !== 'user' && m.role !== 'assistant') continue;
+ const text = messageText(m.content);
+ const timestamp = messageTimestamp(m.timestamp);
+ if (!text || !timestamp) continue;
+ messages.push({ role: m.role, timestamp, text });
+ }
+ }
+ const sessionId = pages.map(p => p.sessionId).find((v): v is string => typeof v === 'string');
+ return { ...(sessionId ? { sessionId } : {}), messages };
+}
+
+export function openclawGatewayReader(opts: { transport?: GatewayTransport } = {}) {
+ const transport = opts.transport ?? cliGatewayTransport();
+ return {
+ harness: 'openclaw' as const,
+ name: 'openclaw-gateway',
+ async list(listOpts: { sinceMs?: number } = {}): Promise<GatewayConversation[]> {
+ const rows: Array<Record<string, unknown>> = [];
+ let offset = 0;
+ for (let i = 0; i < MAX_PAGES; i++) {
+ const page = asRecord(await transport.call('sessions.list', { limit: LIST_PAGE, ...(offset ? { offset } : {}) }));
+ for (const row of Array.isArray(page.sessions) ? page.sessions : []) rows.push(asRecord(row));
+ if (page.hasMore !== true || typeof page.nextOffset !== 'number' || page.nextOffset <= offset) break;
+ offset = page.nextOffset;
+ }
+ const out: GatewayConversation[] = [];
+ for (const row of rows) {
+ const key = typeof row.key === 'string' ? row.key : '';
+ if (!key) continue;
+ const updated = typeof row.updatedAt === 'number' ? row.updatedAt : null;
+ if (listOpts.sinceMs !== undefined && updated !== null && updated < listOpts.sinceMs) continue;
+ const history = await readHistory(transport, key);
+ if (history.messages.length === 0) continue;
+ const sessionId = typeof row.sessionId === 'string' ? row.sessionId : history.sessionId ?? key;
+ out.push({
+ harness: 'openclaw',
+ conversationKey: sessionId,
+ scopeKey: key,
+ via: 'gateway',
+ channel: gatewayChannel(row),
+ session: {
+ meta: {
+ harness: 'openclaw',
+ sessionId,
+ startedAt: history.messages[0]!.timestamp,
+ raw: { session_id: sessionId, conversation: conversationHash8('openclaw', key), agent_id: typeof row.agentId === 'string' ? row.agentId : null, chat_type: typeof row.chatType === 'string' ? row.chatType : null },
+ },
+ messages: history.messages,
+ },
+ });
+ }
+ return out;
+ },
+ };
+}
diff --git a/src/core/transcripts/openclaw.ts b/src/core/transcripts/openclaw.ts
index 71aa55bdbf..c104381a10 100644
--- a/src/core/transcripts/openclaw.ts
+++ b/src/core/transcripts/openclaw.ts
@@ -48,6 +48,42 @@ export function isOpenclawCheckpointFile(path: string): boolean {
return CHECKPOINT_RE.test(path);
}
+/**
+ * GST-2768: the retained transcript archives OpenClaw keeps under sessions/ (its isRetainedSessionTranscriptArchiveName,
+ * OpenClaw 2026.10.9 and 2026.11.3): `<id>.jsonl.reset.<stamp>`, `<id>.jsonl.deleted.<stamp>` and the legacy
+ * pre-compaction backup `<id>.jsonl.bak.<stamp>`, each optionally `.<32 hex>` and zstd `.zst` (recorded on both
+ * versions: `<id>.jsonl.deleted.2026-11-15T06-38-21.539Z.<hex>.zst`). Same line format as a live session file, so they
+ * are read; two files of one session merge in the archive reader.
+ */
+const ARCHIVED_RE = /\.jsonl\.(?:reset|deleted|bak)\.\d{4}-\d{2}-\d{2}T\d{2}-\d{2}-\d{2}(?:\.\d{3})?Z(?:\.[0-9a-f]{32})?(?:\.zst)?$/;
+
+export function isOpenclawArchivedFile(path: string): boolean {
+ return ARCHIVED_RE.test(path);
+}
+
+/**
+ * Files OpenClaw keeps beside sessions that are never read as conversations: migration rollback copies
+ * (`*.migrated`, `*.migrated.<n>` and `<id>.jsonl.pre-doctor-<kind>-repair-<stamp>.bak`, its
+ * isMigrationArchiveArtifactName: the original of a transcript already imported into the session database, which
+ * the Gateway serves) and Doctor's recovery copies of a zero-byte transcript (`<id>.jsonl.bak-<pid>-<timestamp>`).
+ * Reading them would count one conversation twice.
+ */
+const SKIPPED_RE = /(?:\.migrated(?:\.\d+)?|\.jsonl\.pre-doctor-[a-z-]+-repair-[0-9TZ:.-]+\.bak|\.jsonl\.bak-\d+-[0-9TZ:.-]+)$/;
+
+export function isOpenclawSkippedArtifact(path: string): boolean {
+ return SKIPPED_RE.test(path);
+}
+
+async function readSessionText(path: string): Promise<string> {
+ const bytes = readFileSync(path);
+ if (!path.endsWith('.zst')) return bytes.toString('utf8');
+ const zlib = (await import('node:zlib')) as unknown as { zstdDecompressSync?: (b: Buffer) => Buffer };
+ const bun = (globalThis as { Bun?: { zstdDecompressSync?: (b: Buffer) => Uint8Array } }).Bun;
+ const out = zlib.zstdDecompressSync ? zlib.zstdDecompressSync(bytes) : bun?.zstdDecompressSync?.(bytes);
+ if (!out) throw new Error('openclaw archived session is zstd-compressed and this runtime has no zstd decoder');
+ return Buffer.from(out).toString('utf8');
+}
+
/**
* One parsed openclaw session line, classified (cathedral-5). Exported so the
* compaction-boundary tail reader (corpus-segments) reuses the SAME
@@ -123,7 +159,9 @@ export const openclawAdapter: TranscriptAdapter = {
specTarget: OPENCLAW_SPEC_TARGET,
detect(path: string, sample: Buffer): boolean {
- if (!path.endsWith('.jsonl') || isOpenclawCheckpointFile(path)) return false;
+ if (isOpenclawCheckpointFile(path)) return false;
+ if (isOpenclawArchivedFile(path) && path.endsWith('.zst')) return true;
+ if (!path.endsWith('.jsonl') && !isOpenclawArchivedFile(path)) return false;
const firstLine = sample.toString('utf8').split('\n', 1)[0]?.trim();
if (!firstLine) return false;
try {
@@ -140,7 +178,10 @@ export const openclawAdapter: TranscriptAdapter = {
if (size > cap) {
throw new Error(`openclaw session too large for import: ${size} bytes (cap ${cap})`);
}
- const raw = readFileSync(path, 'utf8');
+ const raw = await readSessionText(path);
+ if (raw.length > cap) {
+ throw new Error(`openclaw session too large for import: ${raw.length} characters decompressed (cap ${cap})`);
+ }
let skippedLines = 0;
let sessionId = '';
let cwd: string | undefined;
@@ -171,7 +212,7 @@ export const openclawAdapter: TranscriptAdapter = {
let sessions = 0;
if (messages.length > 0) {
sessions = 1;
- const sid = sessionId || basename(path, '.jsonl');
+ const sid = sessionId || basename(path).split('.jsonl')[0]!;
yield {
meta: {
harness: 'openclaw',
diff --git a/test/cli-socket.test.ts b/test/cli-socket.test.ts
index 48eefbd108..61777cb53e 100644
--- a/test/cli-socket.test.ts
+++ b/test/cli-socket.test.ts
@@ -107,6 +107,7 @@ const NOT_READ_BEHAVIOR = new Set([
'GBRAIN_CLIENT_HEADER', 'GBRAIN_STDIN_TIMEOUT_MS', 'GBRAIN_STDOUT_DRAIN_DEADLINE_MS', 'GBRAIN_TEARDOWN_DEADLINE_MS', 'GBRAIN_PID1_REAP', 'GBRAIN_NICE',
'GBRAIN_REMOTE_CLIENT_ID', 'GBRAIN_REMOTE_CLIENT_SECRET', 'GBRAIN_REMOTE_ISSUER_URL', 'GBRAIN_REMOTE_MCP_URL', 'GBRAIN_REMOTE_TOKEN', 'GBRAIN_REMOTE_TOKEN_ENV',
'GBRAIN_THIN_CLIENT_NAME', 'GBRAIN_TAILSCALE_LOGIN_TIMEOUT_MS', 'GBRAIN_ADMIN_BOOTSTRAP_TOKEN',
+ 'GBRAIN_OPENCLAW_GATEWAY_URL', 'GBRAIN_OPENCLAW_GATEWAY_TOKEN',
'GBRAIN_SCHEMA_AUDIT_VERBOSE', 'GBRAIN_SCHEMA_LOCK_TIMEOUT_SECONDS', 'GBRAIN_SCHEMA_PACK_MAX_BYTES', 'GBRAIN_SCHEMA_PACK_WARN_BYTES',
'GBRAIN_INITSCHEMA_LOCK_TIMEOUT_SECONDS', 'GBRAIN_INIT_SKIP_EMBED_CHECK', 'GBRAIN_IN_AGENT_SETUP', 'GBRAIN_INTERACTIVE', 'GBRAIN_NON_INTERACTIVE',
'GBRAIN_BOOTSTRAP_ABORT_AFTER', 'GBRAIN_SEAT', 'GBRAIN_CORE', 'GBRAIN_PRESSURE', 'GBRAIN_PAYLOAD', 'GBRAIN_DESCRIPTION', 'GBRAIN_HARNESS_MARKER_VALUE',
diff --git a/test/fixtures/goldens/exports/types.json b/test/fixtures/goldens/exports/types.json
index 742c0062a2..07196cf176 100644
--- a/test/fixtures/goldens/exports/types.json
+++ b/test/fixtures/goldens/exports/types.json
@@ -6875,7 +6875,7 @@
{
"kind": "class",
"members": [
- "canonical?: \"unknown\" | \"timeout\" | \"keyword_candidates_incomplete\" | \"projection_pending\" | \"rate_limited\" | \"embedding_disabled\" | \"writer_lock_unavailable\" | \"writer_pool_capacity\" | \"owner_unavailable\" | \"recovery_required\" | \"queue_capacity\" | \"write_quota_exhausted\" | \"write_quota_insufficient\" | \"write_capacity_backpressure\" | \"revision_required\" | \"revision_conflict\" | \"idempotency_conflict\" | \"source_changed\" | \"write_pending\" | \"storage_error\" | \"cancelled\" | \"permission_denied\" | \"scope_denied\" | \"invalid_params\" | \"not_found\" | \"page_not_found\" | \"page_identity_changed\" | \"write_claim_lost\" | \"request_too_large\" | \"response_too_large\" | \"writer_registration_required\" | \"writer_identity_invalid\" | \"writer_not_initialized\" | \"writer_coordinator_required\" | \"fact_already_expired\" | \"source_writeback_required\" | \"unsupported_mutation_protocol\" | \"writer_upgrade_required\" | \"writer_not_quiesced\" | \"skill_bundle_required\" | \"take_row_collision\" | \"invalid_source_uri\" | \"writer_admin_locked\" | \"connector_account_changed\" | \"connector_intent_outdated\" | \"checkpoint_validation_timeout\" | \"invalid_connector_text\" | \"connector_holds_exhausted\" | \"connector_fence_below_timeline\" | \"core_budget_exceeded\" | \"core_mark_owner_only\" | \"core_remote_edit_refused\" | \"core_delete_owner_only\" | \"timeline_rows_would_be_removed\" | \"purged_content\" | \"preparation_stalled\" | \"fence_unrenderable\" | \"page_body_segmented\" | \"page_body_rows_unreadable\" | \"revision_changed\" | \"tail_changed\" | \"write_outcome_unknown\" | \"write_interrupted\" | \"connection_inherited_read_only\" | \"owner_marker_missing\" | \"embedding_failed\" | \"bucket_not_found\" | \"database_error\" | \"unknown_transport\" | \"extraction_failed\" | \"fact_not_found\" | \"provenance_required\" | \"unavailable\" | \"budget_unsatisfiable\" | \"embedding_budget_below_worst_case\" | \"absent_from_worktree\" | \"access_command_failed\" | \"access_identity_unavailable\" | \"access_member_disabled\" | \"autopilot_no_repo\" | \"autopilot_pglite_daemon\" | \"cycle_already_running\" | \"engine_degraded\" | \"maintenance_disabled\" | \"maintenance_owned_by_serve\" | \"schedule_unsupported\" | \"schedule_install_failed\" | \"maintenance_owned_by_autopilot\" | \"on_behalf_of_invalid_audience\" | \"on_behalf_of_unknown_member\" | \"on_behalf_of_not_permitted\" | \"actor_not_attested\" | \"actor_unlinked\" | \"on_behalf_of_in_arguments\" | \"access_policy_invalid\" | \"access_policy_pending\" | \"access_policy_widening_held\" | \"access_policy_history_unreadable\" | \"access_policy_store_mismatch\" | \"access_rule_invalid\" | \"requester_scope_required\" | \"requester_scope_unknown\" | \"credential_scope_required\" | \"access_serves_roster_empty\" | \"access_serves_key_dir_in_git\" | \"repair_propose_dir_in_git\" | \"access_member_id_retired\" | \"access_labelled_only_owner\" | \"access_labelled_only_unnamed\" | \"access_member_invalid\" | \"access_denied_test_user\" | \"access_env_missing\" | \"acknowledgment_unavailable\" | \"admin_policy_enforced\" | \"admission_paused\" | \"ambient_capture_off\" | \"ambiguous_id\" | \"ambiguous_ref\" | \"ambiguous_skill\" | \"answer_not_yours\" | \"answer_pending\" | \"answer_unavailable\" | \"api_not_enabled\" | \"append_links_fence\" | \"approval_required\" | \"atom_drain_incomplete\" | \"atom_drain_no_progress\" | \"atom_drain_provider_failure\" | \"autopilot_job_owned_by_other_brain\" | \"body_staging_expired\" | \"budget_below_recent_runtime\" | \"backup_dump_failed\" | \"backup_too_large\" | \"belief_coverage_required\" | \"belief_stale_preparation\" | \"benchmark_duplicate_task_id\" | \"benchmark_empty\" | \"benchmark_exists\" | \"benchmark_judge_llm_no_rubric\" | \"benchmark_judge_qrels_no_expected\" | \"benchmark_judge_rule_no_checks\" | \"benchmark_judge_unknown_kind\" | \"benchmark_malformed\" | \"benchmark_missing_judge\" | \"benchmark_missing_task\" | \"benchmark_missing_task_id\" | \"benchmark_not_found\" | \"benchmark_rule_check_bad_arg\" | \"benchmark_rule_check_malformed\" | \"benchmark_rule_check_unknown_op\" | \"bootstrap_cleanup_pending\" | \"bootstrap_empty\" | \"bootstrap_pending_review\" | \"bridge_ownership_conflict\" | \"valid_time_undeclared\" | \"coverage_from\" | \"coverage_gap\" | \"clock_mismatch\" | \"extract_write_lost\" | \"item_failures_systemic\" | \"known_as_of_unproven\" | \"cache_quota_exceeded\" | \"calendar_unreadable\" | \"capacity_paused\" | \"catalog_capacity_exceeded\" | \"catalog_unavailable\" | \"host_upgrade_required\" | \"change_reason_rejected\" | \"changed_since_preview\" | \"chronicle_skipped\" | \"no_pricing\" | \"checkout_not_a_source\" | \"checkpoint_missing\" | \"claim_already_used\" | \"client_behind\" | \"client_json_unreadable\" | \"client_json_wrong_type\" | \"client_not_found\" | \"client_shape_invalid\" | \"code_callees_requires_symbol\" | \"code_callers_requires_symbol\" | \"code_def_requires_symbol\" | \"code_refs_requires_symbol\" | \"code_reused\" | \"colon_slug_windows_write_through\" | \"command_failed\" | \"commit_lock_busy\" | \"commit_base_moved\" | \"commit_paths_invalid\" | \"commit_paths_skipped\" | \"commit_prestaged_paths\" | \"commit_outside_source_root\" | \"concurrent_write\" | \"config_error\" | \"confirmation_required\" | \"conflicting_source_scope\" | \"connection_lost\" | \"connector_access_denied\" | \"connector_auth_required\" | \"connector_binding_missing\" | \"connector_item_too_large\" | \"connector_response_too_large\" | \"connector_shape_drift\" | \"chunkless_pages_healable\" | \"consent_required\" | \"consent_timeout\" | \"consumers_without_heartbeat\" | \"content_repair_needs_human\" | \"content_filter_owned\" | \"content_null_bytes\" | \"content_rejected\" | \"continuation_unproven\" | \"corpus_chat_unavailable\" | \"corpus_window_backoff\" | \"corpus_window_dead_lettered\" | \"convergence_read_failed\" | \"cost_cap_exceeded\" | \"cost_preview_requires_yes\" | \"cursor_incarnation_retired\" | \"cycle_failed\" | \"d_sel_too_small\" | \"delta_cursor_upgrade_required\" | \"deleted_page_chunks_overdue\" | \"demo_graph_failed\" | \"downgrade_blocked_deferred_writes\" | \"demo_import_failed\" | \"demo_verification_failed\" | \"derived_cap_exhausted\" | \"destination_not_ready\" | \"drain_holder_stalled\" | \"drain_stalled\" | \"managed_sync_not_moving\" | \"dream_breaker_tripped\" | \"dirty_tree\" | \"doctor_alert_act\" | \"doctor_alert_unknown\" | \"doctor_tier_unavailable\" | \"effect_not_failed\" | \"embed_backfill_stalled\" | \"embedding_auth_failed\" | \"embedding_configuration\" | \"embedding_identity_unqualified\" | \"embedding_model_mismatch\" | \"embedding_plane_split\" | \"embedding_state_unknown\" | \"embedding_storage_unavailable\" | \"embedding_unavailable\" | \"embedding_unconfigured\" | \"embedding_width_mismatch\" | \"embedded_core_required\" | \"enforced_member_refused\" | \"engine_embedding_mismatch\" | \"embedded_engine_required\" | \"engine_closed\" | \"engine_host_missing\" | \"engine_scope_missing\" | \"engine_scope_required\" | \"engine_settings_invalid\" | \"engine_graduated\" | \"entity_types_unavailable\" | \"eval_suite_unwired\" | \"explicit_tasks_conflict\" | \"export_limit\" | \"extractor_identity_mismatch\" | \"fact_withdrawn\" | \"fact_rows_collide\" | \"fence_row_changed\" | \"facts_absorb_write_refused\" | \"facts_backstop_skipped\" | \"facts_drain_deferred\" | \"facts_drain_job_failed\" | \"facts_payload_expired\" | \"feedback_disabled\" | \"feedback_not_authorized\" | \"fetch_failed\" | \"file_database_drift\" | \"fix_not_writable\" | \"file_drift\" | \"file_drift_mixed_selection\" | \"file_drift_state_required\" | \"file_removed_during_scan\" | \"file_unreadable\" | \"file_wins_rewrite\" | \"file_too_large\" | \"follow_approval_required\" | \"foreign_table\" | \"forget_requires_owner\" | \"reserved_frontmatter_key\" | \"generation_fenced\" | \"generation_held\" | \"generation_live_conflict\" | \"frontmatter_slug_conflict\" | \"full_resync_required\" | \"gate_failed\" | \"ghost_mass_delete_refused\" | \"notifier_stalled\" | \"moved_pages_mass_fold_refused\" | \"origin_twin_mass_merge_refused\" | \"ghost_storage_unknown\" | \"git_branch_unborn\" | \"git_durability_not_enabled\" | \"git_checkout_required\" | \"git_worktree_link_broken\" | \"hosted_repo_refused\" | \"git_detached_head\" | \"git_index_conflict\" | \"git_index_locked\" | \"git_index_stale\" | \"git_target_pending\" | \"git_operation_in_progress\" | \"git_push_unavailable\" | \"git_target_changed\" | \"git_target_ignored\" | \"git_target_unsafe\" | \"git_unavailable\" | \"git_object_missing\" | \"git_conversion_blocked\" | \"git_conversion_delta\" | \"conversion_file_in_source_root\" | \"git_conversion_incomplete\" | \"git_home_required\" | \"git_private_commit_refused\" | \"git_protocol_required\" | \"git_lfs_not_enabled\" | \"gitlink_not_followed\" | \"graduation_drain_timeout\" | \"graduation_embedding_dimension_mismatch\" | \"graduation_foreign_host_binding\" | \"graduation_in_progress\" | \"graduation_interrupted\" | \"graduation_rollback_writes_lost\" | \"graduation_source_writer_held\" | \"graduation_split_brain\" | \"graduation_target_auth_failed\" | \"graduation_target_ddl_unreachable\" | \"graduation_target_not_empty\" | \"graduation_target_unsupported\" | \"graduation_unclassified_table\" | \"graduation_unsupported_platform\" | \"graduation_verify_failed\" | \"grant_scope\" | \"graph_incomplete\" | \"host_identity_mismatch\" | \"harness_hook_duplicates\" | \"harness_hook_unowned\" | \"head_blocked\" | \"held_failed\" | \"held_out_overlaps_benchmark\" | \"held_out_required_for_bundled\" | \"independent_principal_required\" | \"insufficient_scope\" | \"integrate_conflict\" | \"internal\" | \"internal_error\" | \"interrupted\" | \"invalid_acknowledgment\" | \"invalid_client\" | \"invalid_fence\" | \"unclosed_ambiguous_tail\" | \"tail_exposure_approval\" | \"sync_in_progress\" | \"invalid_frontmatter\" | \"invalid_grant_clock_skew\" | \"invalid_grant_revoked\" | \"invalid_grant_testing_expiry\" | \"invalid_rating\" | \"invalid_receipt\" | \"invalid_receipt_transition\" | \"invalid_request\" | \"invalid_source\" | \"invalid_source_pin\" | \"invalid_token\" | \"invalid_write_wait\" | \"ipc_path_too_long\" | \"legacy_twin_mass_merge_refused\" | \"legacy_twins_storage_unknown\" | \"legacy_job_selection_invalid\" | \"legacy_jobs_active\" | \"legacy_lock_removal_refused\" | \"link_cursor_invalid\" | \"link_dates_unattributed\" | \"local_conflict\" | \"lock_busy\" | \"loops_extraction_skipped\" | \"maintenance_backpressure\" | \"maintenance_lock_busy\" | \"managed_failure_unsettled\" | \"managed_remedy_unavailable\" | \"memory_setting_write_outcome_unknown\" | \"merge_recommended\" | \"movement_unknown\" | \"managed_sync_flag_refused\" | \"managed_pull_skipped\" | \"manual_only_skipped\" | \"membership_inactive\" | \"membership_not_found\" | \"method_not_allowed\" | \"migration_lease_lost\" | \"migration_declined\" | \"migration_failed\" | \"migration_host_work_pending\" | \"migrations_pending\" | \"migrations_running\" | \"missing_source_scope\" | \"missing_source_value\" | \"models_strict_violation\" | \"no_context_ceiling\" | \"no_refresh_token\" | \"no_routing_eval\" | \"no_skill_md\" | \"no_worker\" | \"no_worker_surface\" | \"not_a_git_repo\" | \"not_converged\" | \"not_owner_host\" | \"not_connected\" | \"no_brain\" | \"overlapping_path\" | \"page_projection_conflict\" | \"page_projection_stale\" | \"label_edit_refused\" | \"page_shrink_refused\" | \"page_too_large_for_inline\" | \"parse_error\" | \"part_splits_code_point\" | \"part_too_large\" | \"frontmatter_alias_limit\" | \"parser_regression\" | \"pasted_wrong_url\" | \"payload_too_large\" | \"path_not_utf8\" | \"pattern_claims_pending\" | \"patterns_min_interval\" | \"pending_verify\" | \"pending_writes\" | \"persistence_restart_required\" | \"pool_inflight_over_budget\" | \"deep_health_probe_failed\" | \"refresh_required\" | \"serve_self_exit_suppressed\" | \"serve_unhealthy_exit\" | \"persistence_write_stall\" | \"publish_secret_found\" | \"publish_scan_unreadable\" | \"push_unconfirmed\" | \"schema_lock_blocked\" | \"postgres_tls_required\" | \"pg_connection_recovery_failed\" | \"pg_connection_stuck\" | \"pg_pool_stalled\" | \"pg_statement_build_failed\" | \"pglite_busy\" | \"pglite_vector_index_too_large\" | \"plan_output_failed\" | \"plan_stale\" | \"port_in_use\" | \"preview_changed\" | \"profile_incompatible\" | \"projection_owner_resident\" | \"knock_on_exhausted\" | \"keyword_skipped_after_timeout\" | \"proposal_closed\" | \"proposal_expired\" | \"proposals_pending_limit\" | \"publication_pending\" | \"pull_blocked_local_changes\" | \"purge_blocked_pending_recovery\" | \"purge_lookup_timeout\" | \"purge_mirror_unwritable\" | \"question_not_found\" | \"question_owner_only\" | \"rank_entities_timeout\" | \"read_budget_exhausted\" | \"reason_purge_scan_capped\" | \"reason_purge_scan_failed\" | \"reason_scan_capped\" | \"reason_scan_failed\" | \"reason_trailer_secret\" | \"receipt_conflict\" | \"receipt_fence_changed\" | \"receipt_fence_required\" | \"receipt_identity_mismatch\" | \"pat_file_unreadable\" | \"reconcile_host_required\" | \"reconcile_read_incomplete\" | \"recovery_prior_stage_unavailable\" | \"recovery_version_unsupported\" | \"redirect_uri_mismatch\" | \"ref_not_in_answer\" | \"refresh_dirty\" | \"refresh_diverged\" | \"refresh_drain_timeout\" | \"refresh_lock_timeout\" | \"refresh_no_upstream\" | \"refresh_not_managed\" | \"refresh_not_owner\" | \"pending_import_expired\" | \"refresh_barrier_expired\" | \"refresh_source_changed\" | \"refresh_sync_timeout\" | \"relay_disabled\" | \"relay_session_expired\" | \"relay_unreachable\" | \"rename_held\" | \"rename_identity_unreadable\" | \"rename_reconcile_failed\" | \"repair_preview_timeout\" | \"request_already_submitted\" | \"requirements_changed\" | \"requires_local_engine\" | \"restart_refused\" | \"restart_unverified\" | \"runtime_build_skew\" | \"reset_upstream_refused\" | \"resume_spec_mismatch\" | \"result_too_large\" | \"revision_backfill_pending\" | \"revision_unavailable\" | \"schema_owner_required\" | \"scope_missing\" | \"search_cursor_requires_keyword\" | \"search_mode_local_only\" | \"scorecard_group_by_local_only\" | \"search_offset_over_cap\" | \"segmented_mirror_writer_gate\" | \"serve_build_drift\" | \"serve_port_in_use\" | \"serve_status_only\" | \"settle_mismatch\" | \"openclaw_not_found\" | \"setup_harness_unsupported\" | \"setup_hosted_connection\" | \"setup_owner_conflict\" | \"setup_serves_several\" | \"setup_serves_unreadable\" | \"hook_session_cwd\" | \"turn_hook_not_loaded\" | \"shared_skills_unavailable\" | \"shared_skills_unsupported\" | \"skill_asset_not_found\" | \"skill_candidate_invalid\" | \"skill_not_found\" | \"skill_catalog_mismatch\" | \"skill_catalog_too_large\" | \"skill_catalog_unknown\" | \"skill_index_incompatible\" | \"skill_index_locked\" | \"skill_index_missing\" | \"skill_index_removed\" | \"skill_resolve_invalid_input\" | \"skill_resolve_no_input\" | \"skill_resolve_timeout\" | \"skill_resolver_config_invalid\" | \"skill_resolver_config_unreadable\" | \"skill_resolver_setup_incomplete\" | \"skill_resolver_plugin_not_installed\" | \"skill_resolver_host_not_local\" | \"skill_resolver_selftest_failed\" | \"skill_resolver_pin_unknown\" | \"skill_resolver_library_missing\" | \"skill_resolver_library_mismatch\" | \"skill_resolver_library_skew\" | \"skill_resolver_worker_crash\" | \"skill_resolver_worker_stuck\" | \"skill_resolver_worker_unavailable\" | \"skill_resolver_state_missing\" | \"skill_resolver_host_load_cut\" | \"skill_resolver_internal\" | \"skill_source_invalid\" | \"skill_source_unreadable\" | \"skill_retention_capacity\" | \"skill_unavailable\" | \"skillpack_backup_corrupt\" | \"skillpack_journal_not_found\" | \"skillpack_restore_conflict\" | \"skillpack_state_locked\" | \"skillpack_update_incomplete\" | \"skillpack_workspace_locked\" | \"slug_page_mismatch\" | \"slug_root_mode_mismatch\" | \"source_binding_required\" | \"source_id_taken\" | \"source_mismatch\" | \"source_not_ready\" | \"source_profile_no_backfill\" | \"source_pages_remain\" | \"source_referenced\" | \"source_retire_superseded\" | \"source_root_conflict\" | \"source_root_unreadable\" | \"sources_not_readable\" | \"split_bad_ratio\" | \"split_unparseable\" | \"stable_channel_unresolved\" | \"stable_invalid\" | \"stable_missing\" | \"stable_unreadable\" | \"stale_unavailable\" | \"state_mismatch\" | \"store_unavailable\" | \"storage_adopt_field_refused\" | \"storage_adopt_unverified\" | \"storage_card_exists\" | \"storage_manifest_missing\" | \"storage_record_invalid\" | \"storage_record_secret\" | \"storage_revision_conflict\" | \"storage_source_managed\" | \"strict_requires_managed\" | \"sync_anchor_relative\" | \"sync_not_applicable\" | \"sync_admission_restricted\" | \"sync_deadline_stalled\" | \"sync_drain_lease_held\" | \"content_receipt_stale\" | \"sync_first_run_in_progress\" | \"sync_no_successful_sync\" | \"sync_cursor_replaced\" | \"sync_pass_superseded\" | \"sync_parser_regression\" | \"tag_too_long\" | \"target_escape\" | \"timeline_comment_markup\" | \"source_writer_restricted\" | \"guest_page_immutable\" | \"guest_page_promoted\" | \"guest_attribution_missing\" | \"guest_frontmatter_refused\" | \"guest_readers_invalid\" | \"guest_promotion_owner_only\" | \"guest_mark_immutable\" | \"guest_source_database_only\" | \"readable_by_invalid\" | \"topology_change_required\" | \"transcript_secret_exposure\" | \"transcript_source_mismatch\" | \"triage_unreliable_backoff\" | \"trust_raise_refused\" | \"two_consumers_on_host\" | \"trusted_local_only\" | \"unexpected_file_bytes\" | \"unexpected_staging_bytes\" | \"unknown_command\" | \"unknown_flag\" | \"unknown_method\" | \"unknown_source\" | \"unknown_tool\" | \"unpriced_model\" | \"unmanaged_database_path\" | \"unmanaged_path\" | \"unsupported_export_data\" | \"unsupported_path\" | \"untrack_not_db_only\" | \"upload_expired\" | \"upload_not_found\" | \"upload_protected_content\" | \"upload_too_many_parts\" | \"upstream\" | \"upstream_diverged\" | \"upstream_rewritten\" | \"upstream_unknown\" | \"vector_prewarm_extension_missing\" | \"vector_prewarm_unsupported_engine\" | \"vector_prewarm_wont_fit\" | \"verification_failed\" | \"visibility_noncanonical\" | \"visibility_not_applied\" | \"wakeshift_key_invalid\" | \"wakeshift_needs_conversation\" | \"wakeshift_not_enabled\" | \"wakeshift_refused\" | \"wakeshift_shared_brain\" | \"wakeshift_unavailable\" | \"withdrawal_capacity\" | \"withdrawal_mirror_failed\" | \"withdrawal_mirror_pending\" | \"withdrawal_provenance\" | \"write_capacity_low\" | \"write_gate_rejected\" | \"write_held\" | \"worktree_dirty\" | \"worktree_refreshing\" | \"writer_admin_intent_required\" | \"writer_admin_state_changed\" | \"writer_busy\" | \"writer_identity_publish_failed\" | \"writer_manifest_mismatch\" | \"writer_manifest_rescope_required\" | \"writer_manifest_unsafe\" | \"writer_regrant_required\" | \"writer_transfer_conflict\" | \"writer_transfer_required\" | \"wrong_account_consented\" | \"schema_pack_too_large\" | \"schema_pack_emit_mismatch\" | undefined",
+ "canonical?: \"unknown\" | \"timeout\" | \"keyword_candidates_incomplete\" | \"projection_pending\" | \"rate_limited\" | \"embedding_disabled\" | \"writer_lock_unavailable\" | \"writer_pool_capacity\" | \"owner_unavailable\" | \"recovery_required\" | \"queue_capacity\" | \"write_quota_exhausted\" | \"write_quota_insufficient\" | \"write_capacity_backpressure\" | \"revision_required\" | \"revision_conflict\" | \"idempotency_conflict\" | \"source_changed\" | \"write_pending\" | \"storage_error\" | \"cancelled\" | \"permission_denied\" | \"scope_denied\" | \"invalid_params\" | \"not_found\" | \"page_not_found\" | \"page_identity_changed\" | \"write_claim_lost\" | \"request_too_large\" | \"response_too_large\" | \"writer_registration_required\" | \"writer_identity_invalid\" | \"writer_not_initialized\" | \"writer_coordinator_required\" | \"fact_already_expired\" | \"source_writeback_required\" | \"unsupported_mutation_protocol\" | \"writer_upgrade_required\" | \"writer_not_quiesced\" | \"skill_bundle_required\" | \"take_row_collision\" | \"invalid_source_uri\" | \"writer_admin_locked\" | \"connector_account_changed\" | \"connector_intent_outdated\" | \"checkpoint_validation_timeout\" | \"invalid_connector_text\" | \"connector_holds_exhausted\" | \"connector_fence_below_timeline\" | \"core_budget_exceeded\" | \"core_mark_owner_only\" | \"core_remote_edit_refused\" | \"core_delete_owner_only\" | \"timeline_rows_would_be_removed\" | \"purged_content\" | \"preparation_stalled\" | \"fence_unrenderable\" | \"page_body_segmented\" | \"page_body_rows_unreadable\" | \"revision_changed\" | \"tail_changed\" | \"write_outcome_unknown\" | \"write_interrupted\" | \"connection_inherited_read_only\" | \"owner_marker_missing\" | \"embedding_failed\" | \"bucket_not_found\" | \"database_error\" | \"unknown_transport\" | \"extraction_failed\" | \"fact_not_found\" | \"provenance_required\" | \"unavailable\" | \"budget_unsatisfiable\" | \"embedding_budget_below_worst_case\" | \"absent_from_worktree\" | \"access_command_failed\" | \"access_identity_unavailable\" | \"access_member_disabled\" | \"archive_audience_unproven\" | \"archive_gateway_unreachable\" | \"archive_gateway_auth\" | \"archive_gateway_failed\" | \"archive_owner_unset\" | \"archive_redaction_failed\" | \"archive_run_busy\" | \"autopilot_no_repo\" | \"autopilot_pglite_daemon\" | \"cycle_already_running\" | \"engine_degraded\" | \"maintenance_disabled\" | \"maintenance_owned_by_serve\" | \"schedule_unsupported\" | \"schedule_install_failed\" | \"maintenance_owned_by_autopilot\" | \"on_behalf_of_invalid_audience\" | \"on_behalf_of_unknown_member\" | \"on_behalf_of_not_permitted\" | \"actor_not_attested\" | \"actor_unlinked\" | \"on_behalf_of_in_arguments\" | \"access_policy_invalid\" | \"access_policy_pending\" | \"access_policy_widening_held\" | \"access_policy_history_unreadable\" | \"access_policy_store_mismatch\" | \"access_rule_invalid\" | \"requester_scope_required\" | \"requester_scope_unknown\" | \"credential_scope_required\" | \"access_serves_roster_empty\" | \"access_serves_key_dir_in_git\" | \"repair_propose_dir_in_git\" | \"access_member_id_retired\" | \"access_labelled_only_owner\" | \"access_labelled_only_unnamed\" | \"access_member_invalid\" | \"access_denied_test_user\" | \"access_env_missing\" | \"acknowledgment_unavailable\" | \"admin_policy_enforced\" | \"admission_paused\" | \"ambient_capture_off\" | \"ambiguous_id\" | \"ambiguous_ref\" | \"ambiguous_skill\" | \"answer_not_yours\" | \"answer_pending\" | \"answer_unavailable\" | \"api_not_enabled\" | \"append_links_fence\" | \"approval_required\" | \"atom_drain_incomplete\" | \"atom_drain_no_progress\" | \"atom_drain_provider_failure\" | \"autopilot_job_owned_by_other_brain\" | \"body_staging_expired\" | \"budget_below_recent_runtime\" | \"backup_dump_failed\" | \"backup_too_large\" | \"belief_coverage_required\" | \"belief_stale_preparation\" | \"benchmark_duplicate_task_id\" | \"benchmark_empty\" | \"benchmark_exists\" | \"benchmark_judge_llm_no_rubric\" | \"benchmark_judge_qrels_no_expected\" | \"benchmark_judge_rule_no_checks\" | \"benchmark_judge_unknown_kind\" | \"benchmark_malformed\" | \"benchmark_missing_judge\" | \"benchmark_missing_task\" | \"benchmark_missing_task_id\" | \"benchmark_not_found\" | \"benchmark_rule_check_bad_arg\" | \"benchmark_rule_check_malformed\" | \"benchmark_rule_check_unknown_op\" | \"bootstrap_cleanup_pending\" | \"bootstrap_empty\" | \"bootstrap_pending_review\" | \"bridge_ownership_conflict\" | \"valid_time_undeclared\" | \"coverage_from\" | \"coverage_gap\" | \"clock_mismatch\" | \"extract_write_lost\" | \"item_failures_systemic\" | \"known_as_of_unproven\" | \"cache_quota_exceeded\" | \"calendar_unreadable\" | \"capacity_paused\" | \"catalog_capacity_exceeded\" | \"catalog_unavailable\" | \"host_upgrade_required\" | \"change_reason_rejected\" | \"changed_since_preview\" | \"chronicle_skipped\" | \"no_pricing\" | \"checkout_not_a_source\" | \"checkpoint_missing\" | \"claim_already_used\" | \"client_behind\" | \"client_json_unreadable\" | \"client_json_wrong_type\" | \"client_not_found\" | \"client_shape_invalid\" | \"code_callees_requires_symbol\" | \"code_callers_requires_symbol\" | \"code_def_requires_symbol\" | \"code_refs_requires_symbol\" | \"code_reused\" | \"colon_slug_windows_write_through\" | \"command_failed\" | \"commit_lock_busy\" | \"commit_base_moved\" | \"commit_paths_invalid\" | \"commit_paths_skipped\" | \"commit_prestaged_paths\" | \"commit_outside_source_root\" | \"concurrent_write\" | \"config_error\" | \"confirmation_required\" | \"conflicting_source_scope\" | \"connection_lost\" | \"connector_access_denied\" | \"connector_auth_required\" | \"connector_binding_missing\" | \"connector_item_too_large\" | \"connector_response_too_large\" | \"connector_shape_drift\" | \"chunkless_pages_healable\" | \"consent_required\" | \"consent_timeout\" | \"consumers_without_heartbeat\" | \"content_repair_needs_human\" | \"content_filter_owned\" | \"content_null_bytes\" | \"content_rejected\" | \"continuation_unproven\" | \"corpus_chat_unavailable\" | \"corpus_window_backoff\" | \"corpus_window_dead_lettered\" | \"convergence_read_failed\" | \"cost_cap_exceeded\" | \"cost_preview_requires_yes\" | \"cursor_incarnation_retired\" | \"cycle_failed\" | \"d_sel_too_small\" | \"delta_cursor_upgrade_required\" | \"deleted_page_chunks_overdue\" | \"demo_graph_failed\" | \"downgrade_blocked_deferred_writes\" | \"demo_import_failed\" | \"demo_verification_failed\" | \"derived_cap_exhausted\" | \"destination_not_ready\" | \"drain_holder_stalled\" | \"drain_stalled\" | \"managed_sync_not_moving\" | \"dream_breaker_tripped\" | \"dirty_tree\" | \"doctor_alert_act\" | \"doctor_alert_unknown\" | \"doctor_tier_unavailable\" | \"effect_not_failed\" | \"embed_backfill_stalled\" | \"embedding_auth_failed\" | \"embedding_configuration\" | \"embedding_identity_unqualified\" | \"embedding_model_mismatch\" | \"embedding_plane_split\" | \"embedding_state_unknown\" | \"embedding_storage_unavailable\" | \"embedding_unavailable\" | \"embedding_unconfigured\" | \"embedding_width_mismatch\" | \"embedded_core_required\" | \"enforced_member_refused\" | \"engine_embedding_mismatch\" | \"embedded_engine_required\" | \"engine_closed\" | \"engine_host_missing\" | \"engine_scope_missing\" | \"engine_scope_required\" | \"engine_settings_invalid\" | \"engine_graduated\" | \"entity_types_unavailable\" | \"eval_suite_unwired\" | \"explicit_tasks_conflict\" | \"export_limit\" | \"extractor_identity_mismatch\" | \"fact_withdrawn\" | \"fact_rows_collide\" | \"fence_row_changed\" | \"facts_absorb_write_refused\" | \"facts_backstop_skipped\" | \"facts_drain_deferred\" | \"facts_drain_job_failed\" | \"facts_payload_expired\" | \"feedback_disabled\" | \"feedback_not_authorized\" | \"fetch_failed\" | \"file_database_drift\" | \"fix_not_writable\" | \"file_drift\" | \"file_drift_mixed_selection\" | \"file_drift_state_required\" | \"file_removed_during_scan\" | \"file_unreadable\" | \"file_wins_rewrite\" | \"file_too_large\" | \"follow_approval_required\" | \"foreign_table\" | \"forget_requires_owner\" | \"reserved_frontmatter_key\" | \"generation_fenced\" | \"generation_held\" | \"generation_live_conflict\" | \"frontmatter_slug_conflict\" | \"full_resync_required\" | \"gate_failed\" | \"ghost_mass_delete_refused\" | \"notifier_stalled\" | \"moved_pages_mass_fold_refused\" | \"origin_twin_mass_merge_refused\" | \"ghost_storage_unknown\" | \"git_branch_unborn\" | \"git_durability_not_enabled\" | \"git_checkout_required\" | \"git_worktree_link_broken\" | \"hosted_repo_refused\" | \"git_detached_head\" | \"git_index_conflict\" | \"git_index_locked\" | \"git_index_stale\" | \"git_target_pending\" | \"git_operation_in_progress\" | \"git_push_unavailable\" | \"git_target_changed\" | \"git_target_ignored\" | \"git_target_unsafe\" | \"git_unavailable\" | \"git_object_missing\" | \"git_conversion_blocked\" | \"git_conversion_delta\" | \"conversion_file_in_source_root\" | \"git_conversion_incomplete\" | \"git_home_required\" | \"git_private_commit_refused\" | \"git_protocol_required\" | \"git_lfs_not_enabled\" | \"gitlink_not_followed\" | \"graduation_drain_timeout\" | \"graduation_embedding_dimension_mismatch\" | \"graduation_foreign_host_binding\" | \"graduation_in_progress\" | \"graduation_interrupted\" | \"graduation_rollback_writes_lost\" | \"graduation_source_writer_held\" | \"graduation_split_brain\" | \"graduation_target_auth_failed\" | \"graduation_target_ddl_unreachable\" | \"graduation_target_not_empty\" | \"graduation_target_unsupported\" | \"graduation_unclassified_table\" | \"graduation_unsupported_platform\" | \"graduation_verify_failed\" | \"grant_scope\" | \"graph_incomplete\" | \"host_identity_mismatch\" | \"harness_hook_duplicates\" | \"harness_hook_unowned\" | \"head_blocked\" | \"held_failed\" | \"held_out_overlaps_benchmark\" | \"held_out_required_for_bundled\" | \"independent_principal_required\" | \"insufficient_scope\" | \"integrate_conflict\" | \"internal\" | \"internal_error\" | \"interrupted\" | \"invalid_acknowledgment\" | \"invalid_client\" | \"invalid_fence\" | \"unclosed_ambiguous_tail\" | \"tail_exposure_approval\" | \"sync_in_progress\" | \"invalid_frontmatter\" | \"invalid_grant_clock_skew\" | \"invalid_grant_revoked\" | \"invalid_grant_testing_expiry\" | \"invalid_rating\" | \"invalid_receipt\" | \"invalid_receipt_transition\" | \"invalid_request\" | \"invalid_source\" | \"invalid_source_pin\" | \"invalid_token\" | \"invalid_write_wait\" | \"ipc_path_too_long\" | \"legacy_twin_mass_merge_refused\" | \"legacy_twins_storage_unknown\" | \"legacy_job_selection_invalid\" | \"legacy_jobs_active\" | \"legacy_lock_removal_refused\" | \"link_cursor_invalid\" | \"link_dates_unattributed\" | \"local_conflict\" | \"lock_busy\" | \"loops_extraction_skipped\" | \"maintenance_backpressure\" | \"maintenance_lock_busy\" | \"managed_failure_unsettled\" | \"managed_remedy_unavailable\" | \"memory_setting_write_outcome_unknown\" | \"merge_recommended\" | \"movement_unknown\" | \"managed_sync_flag_refused\" | \"managed_pull_skipped\" | \"manual_only_skipped\" | \"membership_inactive\" | \"membership_not_found\" | \"method_not_allowed\" | \"migration_lease_lost\" | \"migration_declined\" | \"migration_failed\" | \"migration_host_work_pending\" | \"migrations_pending\" | \"migrations_running\" | \"missing_source_scope\" | \"missing_source_value\" | \"models_strict_violation\" | \"no_context_ceiling\" | \"no_refresh_token\" | \"no_routing_eval\" | \"no_skill_md\" | \"no_worker\" | \"no_worker_surface\" | \"not_a_git_repo\" | \"not_converged\" | \"not_owner_host\" | \"not_connected\" | \"no_brain\" | \"overlapping_path\" | \"page_projection_conflict\" | \"page_projection_stale\" | \"label_edit_refused\" | \"page_shrink_refused\" | \"page_too_large_for_inline\" | \"parse_error\" | \"part_splits_code_point\" | \"part_too_large\" | \"frontmatter_alias_limit\" | \"parser_regression\" | \"pasted_wrong_url\" | \"payload_too_large\" | \"path_not_utf8\" | \"pattern_claims_pending\" | \"patterns_min_interval\" | \"pending_verify\" | \"pending_writes\" | \"persistence_restart_required\" | \"pool_inflight_over_budget\" | \"deep_health_probe_failed\" | \"refresh_required\" | \"serve_self_exit_suppressed\" | \"serve_unhealthy_exit\" | \"persistence_write_stall\" | \"publish_secret_found\" | \"publish_scan_unreadable\" | \"push_unconfirmed\" | \"schema_lock_blocked\" | \"postgres_tls_required\" | \"pg_connection_recovery_failed\" | \"pg_connection_stuck\" | \"pg_pool_stalled\" | \"pg_statement_build_failed\" | \"pglite_busy\" | \"pglite_vector_index_too_large\" | \"plan_output_failed\" | \"plan_stale\" | \"port_in_use\" | \"preview_changed\" | \"profile_incompatible\" | \"projection_owner_resident\" | \"knock_on_exhausted\" | \"keyword_skipped_after_timeout\" | \"proposal_closed\" | \"proposal_expired\" | \"proposals_pending_limit\" | \"publication_pending\" | \"pull_blocked_local_changes\" | \"purge_blocked_pending_recovery\" | \"purge_lookup_timeout\" | \"purge_mirror_unwritable\" | \"question_not_found\" | \"question_owner_only\" | \"rank_entities_timeout\" | \"read_budget_exhausted\" | \"reason_purge_scan_capped\" | \"reason_purge_scan_failed\" | \"reason_scan_capped\" | \"reason_scan_failed\" | \"reason_trailer_secret\" | \"receipt_conflict\" | \"receipt_fence_changed\" | \"receipt_fence_required\" | \"receipt_identity_mismatch\" | \"pat_file_unreadable\" | \"reconcile_host_required\" | \"reconcile_read_incomplete\" | \"recovery_prior_stage_unavailable\" | \"recovery_version_unsupported\" | \"redirect_uri_mismatch\" | \"ref_not_in_answer\" | \"refresh_dirty\" | \"refresh_diverged\" | \"refresh_drain_timeout\" | \"refresh_lock_timeout\" | \"refresh_no_upstream\" | \"refresh_not_managed\" | \"refresh_not_owner\" | \"pending_import_expired\" | \"refresh_barrier_expired\" | \"refresh_source_changed\" | \"refresh_sync_timeout\" | \"relay_disabled\" | \"relay_session_expired\" | \"relay_unreachable\" | \"rename_held\" | \"rename_identity_unreadable\" | \"rename_reconcile_failed\" | \"repair_preview_timeout\" | \"request_already_submitted\" | \"requirements_changed\" | \"requires_local_engine\" | \"restart_refused\" | \"restart_unverified\" | \"runtime_build_skew\" | \"reset_upstream_refused\" | \"resume_spec_mismatch\" | \"result_too_large\" | \"revision_backfill_pending\" | \"revision_unavailable\" | \"schema_owner_required\" | \"scope_missing\" | \"search_cursor_requires_keyword\" | \"search_mode_local_only\" | \"scorecard_group_by_local_only\" | \"search_offset_over_cap\" | \"segmented_mirror_writer_gate\" | \"serve_build_drift\" | \"serve_port_in_use\" | \"serve_status_only\" | \"settle_mismatch\" | \"openclaw_not_found\" | \"setup_harness_unsupported\" | \"setup_hosted_connection\" | \"setup_owner_conflict\" | \"setup_serves_several\" | \"setup_serves_unreadable\" | \"hook_session_cwd\" | \"turn_hook_not_loaded\" | \"shared_skills_unavailable\" | \"shared_skills_unsupported\" | \"skill_asset_not_found\" | \"skill_candidate_invalid\" | \"skill_not_found\" | \"skill_catalog_mismatch\" | \"skill_catalog_too_large\" | \"skill_catalog_unknown\" | \"skill_index_incompatible\" | \"skill_index_locked\" | \"skill_index_missing\" | \"skill_index_removed\" | \"skill_resolve_invalid_input\" | \"skill_resolve_no_input\" | \"skill_resolve_timeout\" | \"skill_resolver_config_invalid\" | \"skill_resolver_config_unreadable\" | \"skill_resolver_setup_incomplete\" | \"skill_resolver_plugin_not_installed\" | \"skill_resolver_host_not_local\" | \"skill_resolver_selftest_failed\" | \"skill_resolver_pin_unknown\" | \"skill_resolver_library_missing\" | \"skill_resolver_library_mismatch\" | \"skill_resolver_library_skew\" | \"skill_resolver_worker_crash\" | \"skill_resolver_worker_stuck\" | \"skill_resolver_worker_unavailable\" | \"skill_resolver_state_missing\" | \"skill_resolver_host_load_cut\" | \"skill_resolver_internal\" | \"skill_source_invalid\" | \"skill_source_unreadable\" | \"skill_retention_capacity\" | \"skill_unavailable\" | \"skillpack_backup_corrupt\" | \"skillpack_journal_not_found\" | \"skillpack_restore_conflict\" | \"skillpack_state_locked\" | \"skillpack_update_incomplete\" | \"skillpack_workspace_locked\" | \"slug_page_mismatch\" | \"slug_root_mode_mismatch\" | \"source_binding_required\" | \"source_id_taken\" | \"source_mismatch\" | \"source_not_ready\" | \"source_profile_no_backfill\" | \"source_pages_remain\" | \"source_referenced\" | \"source_retire_superseded\" | \"source_root_conflict\" | \"source_root_unreadable\" | \"sources_not_readable\" | \"split_bad_ratio\" | \"split_unparseable\" | \"stable_channel_unresolved\" | \"stable_invalid\" | \"stable_missing\" | \"stable_unreadable\" | \"stale_unavailable\" | \"state_mismatch\" | \"store_unavailable\" | \"storage_adopt_field_refused\" | \"storage_adopt_unverified\" | \"storage_card_exists\" | \"storage_manifest_missing\" | \"storage_record_invalid\" | \"storage_record_secret\" | \"storage_revision_conflict\" | \"storage_source_managed\" | \"strict_requires_managed\" | \"sync_anchor_relative\" | \"sync_not_applicable\" | \"sync_admission_restricted\" | \"sync_deadline_stalled\" | \"sync_drain_lease_held\" | \"content_receipt_stale\" | \"sync_first_run_in_progress\" | \"sync_no_successful_sync\" | \"sync_cursor_replaced\" | \"sync_pass_superseded\" | \"sync_parser_regression\" | \"tag_too_long\" | \"target_escape\" | \"timeline_comment_markup\" | \"source_writer_restricted\" | \"guest_page_immutable\" | \"guest_page_promoted\" | \"guest_attribution_missing\" | \"guest_frontmatter_refused\" | \"guest_readers_invalid\" | \"guest_promotion_owner_only\" | \"guest_mark_immutable\" | \"guest_source_database_only\" | \"readable_by_invalid\" | \"topology_change_required\" | \"transcript_secret_exposure\" | \"transcript_source_mismatch\" | \"triage_unreliable_backoff\" | \"trust_raise_refused\" | \"two_consumers_on_host\" | \"trusted_local_only\" | \"unexpected_file_bytes\" | \"unexpected_staging_bytes\" | \"unknown_command\" | \"unknown_flag\" | \"unknown_method\" | \"unknown_source\" | \"unknown_tool\" | \"unpriced_model\" | \"unmanaged_database_path\" | \"unmanaged_path\" | \"unsupported_export_data\" | \"unsupported_path\" | \"untrack_not_db_only\" | \"upload_expired\" | \"upload_not_found\" | \"upload_protected_content\" | \"upload_too_many_parts\" | \"upstream\" | \"upstream_diverged\" | \"upstream_rewritten\" | \"upstream_unknown\" | \"vector_prewarm_extension_missing\" | \"vector_prewarm_unsupported_engine\" | \"vector_prewarm_wont_fit\" | \"verification_failed\" | \"visibility_noncanonical\" | \"visibility_not_applied\" | \"wakeshift_key_invalid\" | \"wakeshift_needs_conversation\" | \"wakeshift_not_enabled\" | \"wakeshift_refused\" | \"wakeshift_shared_brain\" | \"wakeshift_unavailable\" | \"withdrawal_capacity\" | \"withdrawal_mirror_failed\" | \"withdrawal_mirror_pending\" | \"withdrawal_provenance\" | \"write_capacity_low\" | \"write_gate_rejected\" | \"write_held\" | \"worktree_dirty\" | \"worktree_refreshing\" | \"writer_admin_intent_required\" | \"writer_admin_state_changed\" | \"writer_busy\" | \"writer_identity_publish_failed\" | \"writer_manifest_mismatch\" | \"writer_manifest_rescope_required\" | \"writer_manifest_unsafe\" | \"writer_regrant_required\" | \"writer_transfer_conflict\" | \"writer_transfer_required\" | \"wrong_account_consented\" | \"schema_pack_too_large\" | \"schema_pack_emit_mismatch\" | undefined",
"canonicalCode: string",
"cause?: unknown",
"code: ErrorCode",
diff --git a/test/fixtures/goldens/sql-text/chunks.json b/test/fixtures/goldens/sql-text/chunks.json
index 637110d3c9..82c5fc378f 100644
--- a/test/fixtures/goldens/sql-text/chunks.json
+++ b/test/fixtures/goldens/sql-text/chunks.json
@@ -33,8 +33,8 @@
{
"lane": "pool",
"params": [],
- "sql": "SELECT count(*)::int AS count FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND cc.\"embedding\" IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576)",
- "textSha256": "e83227b9f8a2a11848819a683ff9a14d87c35b36b55c2581cf4fab097b8f5cdf"
+ "sql": "SELECT count(*)::int AS count FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND cc.\"embedding\" IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day')))",
+ "textSha256": "015e51a8ca10ded927bee5668092ab805f640a595a1fbe855dbd59d92216d9ee"
}
],
"signature+includeNullSignature": [
@@ -49,8 +49,8 @@
"params": [
"string"
],
- "sql": "SELECT count(*)::int AS count FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND (cc.\"embedding\" IS NULL OR p.embedding_signature IS NULL OR p.embedding_signature <> $1) AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576)",
- "textSha256": "0601a60050db58018495d7d45f55002a4e6b591c17305f7f3de4b1450b5c4c2b"
+ "sql": "SELECT count(*)::int AS count FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND (cc.\"embedding\" IS NULL OR p.embedding_signature IS NULL OR p.embedding_signature <> $1) AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day')))",
+ "textSha256": "57e533d34617f6186517a738198fb71b8b96255252ae8403813386061480a16f"
}
],
"signature+sourceId": [
@@ -66,8 +66,8 @@
"string",
"string"
],
- "sql": "SELECT count(*)::int AS count FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND (cc.\"embedding\" IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $1)) AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576) AND p.source_id = $2",
- "textSha256": "5c75ef9176fa2316ed9a82fa49c8edb8ab2006794e95e6f39153c621fb41004e"
+ "sql": "SELECT count(*)::int AS count FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND (cc.\"embedding\" IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $1)) AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day'))) AND p.source_id = $2",
+ "textSha256": "cd7f5c8b2b3e27b383966bcd8ffb75a15ac5808700da532207fd8bdcb6c8f105"
}
]
},
@@ -414,8 +414,8 @@
"params": [
"array<>"
],
- "sql": "WITH b AS (SELECT id FROM content_chunks WHERE id > 0 ORDER BY id LIMIT 5000), w AS (UPDATE content_chunks cc SET \"embedding\" = NULL, embedded_at = NULL, embedded_text_hash = NULL, embedding_pending_since = now() FROM b, pages p WHERE cc.id = b.id AND cc.page_id = p.id AND p.source_id=ANY($1::text[]) AND EXISTS (SELECT 1 FROM sources s WHERE s.id=p.source_id AND NOT s.archived) AND p.deleted_at IS NULL AND cc.\"embedding\" IS NOT NULL AND cc.embedded_text_hash IS NOT NULL AND cc.embedded_text_hash <> md5(cc.chunk_text) AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576) RETURNING 1) SELECT (SELECT max(id) FROM b) AS last_id, (SELECT count(*) FROM b)::int AS checked, (SELECT count(*) FROM w)::int AS drifted",
- "textSha256": "e4ab4d9f62a62d4acbe361aee22049a5f8dbe556ce2eb5822c5eba6c8ee4b262"
+ "sql": "WITH b AS (SELECT id FROM content_chunks WHERE id > 0 ORDER BY id LIMIT 5000), w AS (UPDATE content_chunks cc SET \"embedding\" = NULL, embedded_at = NULL, embedded_text_hash = NULL, embedding_pending_since = now() FROM b, pages p WHERE cc.id = b.id AND cc.page_id = p.id AND p.source_id=ANY($1::text[]) AND EXISTS (SELECT 1 FROM sources s WHERE s.id=p.source_id AND NOT s.archived) AND p.deleted_at IS NULL AND cc.\"embedding\" IS NOT NULL AND cc.embedded_text_hash IS NOT NULL AND cc.embedded_text_hash <> md5(cc.chunk_text) AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day'))) RETURNING 1) SELECT (SELECT max(id) FROM b) AS last_id, (SELECT count(*) FROM b)::int AS checked, (SELECT count(*) FROM w)::int AS drifted",
+ "textSha256": "ac05675d4616689d3a83b47a484d97ca0adeefa087c60c1a4e3f2de782ae8b75"
},
{
"event": "commit",
@@ -448,8 +448,8 @@
"array<>",
"string"
],
- "sql": "WITH b AS (SELECT id FROM content_chunks WHERE id > 0 ORDER BY id LIMIT 5000), w AS (UPDATE content_chunks cc SET \"embedding\" = NULL, embedded_at = NULL, embedded_text_hash = NULL, embedding_pending_since = now() FROM b, pages p WHERE cc.id = b.id AND cc.page_id = p.id AND p.source_id=ANY($1::text[]) AND EXISTS (SELECT 1 FROM sources s WHERE s.id=p.source_id AND NOT s.archived) AND p.deleted_at IS NULL AND cc.\"embedding\" IS NOT NULL AND cc.embedded_text_hash IS NOT NULL AND cc.embedded_text_hash <> md5(cc.chunk_text) AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576) AND p.source_id = $2 RETURNING 1) SELECT (SELECT max(id) FROM b) AS last_id, (SELECT count(*) FROM b)::int AS checked, (SELECT count(*) FROM w)::int AS drifted",
- "textSha256": "364dd0839c4cecd030d0a136025343116458e060b6f36c9ba9d099736c249858"
+ "sql": "WITH b AS (SELECT id FROM content_chunks WHERE id > 0 ORDER BY id LIMIT 5000), w AS (UPDATE content_chunks cc SET \"embedding\" = NULL, embedded_at = NULL, embedded_text_hash = NULL, embedding_pending_since = now() FROM b, pages p WHERE cc.id = b.id AND cc.page_id = p.id AND p.source_id=ANY($1::text[]) AND EXISTS (SELECT 1 FROM sources s WHERE s.id=p.source_id AND NOT s.archived) AND p.deleted_at IS NULL AND cc.\"embedding\" IS NOT NULL AND cc.embedded_text_hash IS NOT NULL AND cc.embedded_text_hash <> md5(cc.chunk_text) AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day'))) AND p.source_id = $2 RETURNING 1) SELECT (SELECT max(id) FROM b) AS last_id, (SELECT count(*) FROM b)::int AS checked, (SELECT count(*) FROM w)::int AS drifted",
+ "textSha256": "100146b42933adf362f4e85559488eb79967fcfd1981f51133af1f0021b982ee"
},
{
"event": "commit",
@@ -571,8 +571,8 @@
"array<string>",
"number"
],
- "sql": "SELECT r.slug, r.source_id, r.state, r.chunks, r.bytes, r.embed_skip_reason, r.marker_bytes, r.quarantine_reason, r.deleted_at, r.overdue FROM ( SELECT p.slug, p.source_id, CASE WHEN p.deleted_at IS NOT NULL THEN 'deleted_page_chunks' WHEN NOT NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') THEN 'quarantined' WHEN cc.\"embedding\" IS NOT NULL THEN 'embedded' WHEN p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND cc.\"embedding\" IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576) THEN 'pending' ELSE 'keyword_only' END AS state, count(*)::bigint AS chunks, COALESCE(sum(octet_length(cc.chunk_text)), 0)::bigint AS bytes, p.frontmatter -> 'embed_skip' ->> 'reason' AS embed_skip_reason, CASE WHEN jsonb_typeof(p.frontmatter -> 'embed_skip' -> 'bytes') = 'number' THEN (p.frontmatter -> 'embed_skip' ->> 'bytes')::numeric::bigint END AS marker_bytes, p.frontmatter -> 'quarantine' ->> 'reason' AS quarantine_reason, p.deleted_at, COALESCE(p.deleted_at < now() - make_interval(hours => $1::int), false) AS overdue FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) GROUP BY p.id, 3 UNION ALL SELECT p.slug, p.source_id, CASE WHEN NOT NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') THEN 'quarantined' WHEN p.deleted_at IS NULL AND (p.compiled_truth <> '' OR p.timeline <> '' OR (p.body_storage = 'segmented' AND p.body_bytes > 0)) AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND (p.text_projection_revision IS DISTINCT FROM p.knowledge_revision OR COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM content_chunks cc WHERE cc.page_id = p.id) THEN 'chunkless_healable' ELSE 'empty' END, 0::bigint, (octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')))::bigint, p.frontmatter -> 'embed_skip' ->> 'reason', CASE WHEN jsonb_typeof(p.frontmatter -> 'embed_skip' -> 'bytes') = 'number' THEN (p.frontmatter -> 'embed_skip' ->> 'bytes')::numeric::bigint END, p.frontmatter -> 'quarantine' ->> 'reason', p.deleted_at, false FROM pages p WHERE p.deleted_at IS NULL AND NOT EXISTS (SELECT 1 FROM content_chunks cc WHERE cc.page_id = p.id) ) r WHERE r.state = ANY($2::text[]) ORDER BY r.state, r.source_id, r.slug LIMIT $3",
- "textSha256": "169d2f37cf96b16e58ec6a0686f0676ff5bd9c296ecbe792de58080bc36fc9b7"
+ "sql": "SELECT r.slug, r.source_id, r.state, r.chunks, r.bytes, r.embed_skip_reason, r.marker_bytes, r.quarantine_reason, r.deleted_at, r.overdue FROM ( SELECT p.slug, p.source_id, CASE WHEN p.deleted_at IS NOT NULL THEN 'deleted_page_chunks' WHEN NOT NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') THEN 'quarantined' WHEN cc.\"embedding\" IS NOT NULL THEN 'embedded' WHEN p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND cc.\"embedding\" IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day'))) THEN 'pending' ELSE 'keyword_only' END AS state, count(*)::bigint AS chunks, COALESCE(sum(octet_length(cc.chunk_text)), 0)::bigint AS bytes, p.frontmatter -> 'embed_skip' ->> 'reason' AS embed_skip_reason, CASE WHEN jsonb_typeof(p.frontmatter -> 'embed_skip' -> 'bytes') = 'number' THEN (p.frontmatter -> 'embed_skip' ->> 'bytes')::numeric::bigint END AS marker_bytes, p.frontmatter -> 'quarantine' ->> 'reason' AS quarantine_reason, p.deleted_at, COALESCE(p.deleted_at < now() - make_interval(hours => $1::int), false) AS overdue FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) GROUP BY p.id, 3 UNION ALL SELECT p.slug, p.source_id, CASE WHEN NOT NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') THEN 'quarantined' WHEN p.deleted_at IS NULL AND (p.compiled_truth <> '' OR p.timeline <> '' OR (p.body_storage = 'segmented' AND p.body_bytes > 0)) AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND (p.text_projection_revision IS DISTINCT FROM p.knowledge_revision OR COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM content_chunks cc WHERE cc.page_id = p.id) THEN 'chunkless_healable' ELSE 'empty' END, 0::bigint, (octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')))::bigint, p.frontmatter -> 'embed_skip' ->> 'reason', CASE WHEN jsonb_typeof(p.frontmatter -> 'embed_skip' -> 'bytes') = 'number' THEN (p.frontmatter -> 'embed_skip' ->> 'bytes')::numeric::bigint END, p.frontmatter -> 'quarantine' ->> 'reason', p.deleted_at, false FROM pages p WHERE p.deleted_at IS NULL AND NOT EXISTS (SELECT 1 FROM content_chunks cc WHERE cc.page_id = p.id) ) r WHERE r.state = ANY($2::text[]) ORDER BY r.state, r.source_id, r.slug LIMIT $3",
+ "textSha256": "f478914714700c6ef4c75ac988da7b889ce83263aa2b106c86f297a9c5a04479"
}
],
"sourceIds": [
@@ -591,8 +591,8 @@
"array<string>",
"number"
],
- "sql": "SELECT r.slug, r.source_id, r.state, r.chunks, r.bytes, r.embed_skip_reason, r.marker_bytes, r.quarantine_reason, r.deleted_at, r.overdue FROM ( SELECT p.slug, p.source_id, CASE WHEN p.deleted_at IS NOT NULL THEN 'deleted_page_chunks' WHEN NOT NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') THEN 'quarantined' WHEN cc.\"embedding\" IS NOT NULL THEN 'embedded' WHEN p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND cc.\"embedding\" IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576) THEN 'pending' ELSE 'keyword_only' END AS state, count(*)::bigint AS chunks, COALESCE(sum(octet_length(cc.chunk_text)), 0)::bigint AS bytes, p.frontmatter -> 'embed_skip' ->> 'reason' AS embed_skip_reason, CASE WHEN jsonb_typeof(p.frontmatter -> 'embed_skip' -> 'bytes') = 'number' THEN (p.frontmatter -> 'embed_skip' ->> 'bytes')::numeric::bigint END AS marker_bytes, p.frontmatter -> 'quarantine' ->> 'reason' AS quarantine_reason, p.deleted_at, COALESCE(p.deleted_at < now() - make_interval(hours => $1::int), false) AS overdue FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND p.source_id = ANY($2::text[]) GROUP BY p.id, 3 UNION ALL SELECT p.slug, p.source_id, CASE WHEN NOT NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') THEN 'quarantined' WHEN p.deleted_at IS NULL AND (p.compiled_truth <> '' OR p.timeline <> '' OR (p.body_storage = 'segmented' AND p.body_bytes > 0)) AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND (p.text_projection_revision IS DISTINCT FROM p.knowledge_revision OR COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM content_chunks cc WHERE cc.page_id = p.id) THEN 'chunkless_healable' ELSE 'empty' END, 0::bigint, (octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')))::bigint, p.frontmatter -> 'embed_skip' ->> 'reason', CASE WHEN jsonb_typeof(p.frontmatter -> 'embed_skip' -> 'bytes') = 'number' THEN (p.frontmatter -> 'embed_skip' ->> 'bytes')::numeric::bigint END, p.frontmatter -> 'quarantine' ->> 'reason', p.deleted_at, false FROM pages p WHERE p.deleted_at IS NULL AND NOT EXISTS (SELECT 1 FROM content_chunks cc WHERE cc.page_id = p.id) AND p.source_id = ANY($3::text[]) ) r WHERE r.state = ANY($4::text[]) ORDER BY r.state, r.source_id, r.slug LIMIT $5",
- "textSha256": "5e65cf4ecea50baaed7cf17985e5dfa8d29e4b1f6c61e85095518fbfd3b386c3"
+ "sql": "SELECT r.slug, r.source_id, r.state, r.chunks, r.bytes, r.embed_skip_reason, r.marker_bytes, r.quarantine_reason, r.deleted_at, r.overdue FROM ( SELECT p.slug, p.source_id, CASE WHEN p.deleted_at IS NOT NULL THEN 'deleted_page_chunks' WHEN NOT NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') THEN 'quarantined' WHEN cc.\"embedding\" IS NOT NULL THEN 'embedded' WHEN p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND cc.\"embedding\" IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day'))) THEN 'pending' ELSE 'keyword_only' END AS state, count(*)::bigint AS chunks, COALESCE(sum(octet_length(cc.chunk_text)), 0)::bigint AS bytes, p.frontmatter -> 'embed_skip' ->> 'reason' AS embed_skip_reason, CASE WHEN jsonb_typeof(p.frontmatter -> 'embed_skip' -> 'bytes') = 'number' THEN (p.frontmatter -> 'embed_skip' ->> 'bytes')::numeric::bigint END AS marker_bytes, p.frontmatter -> 'quarantine' ->> 'reason' AS quarantine_reason, p.deleted_at, COALESCE(p.deleted_at < now() - make_interval(hours => $1::int), false) AS overdue FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND p.source_id = ANY($2::text[]) GROUP BY p.id, 3 UNION ALL SELECT p.slug, p.source_id, CASE WHEN NOT NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') THEN 'quarantined' WHEN p.deleted_at IS NULL AND (p.compiled_truth <> '' OR p.timeline <> '' OR (p.body_storage = 'segmented' AND p.body_bytes > 0)) AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND (p.text_projection_revision IS DISTINCT FROM p.knowledge_revision OR COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM content_chunks cc WHERE cc.page_id = p.id) THEN 'chunkless_healable' ELSE 'empty' END, 0::bigint, (octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')))::bigint, p.frontmatter -> 'embed_skip' ->> 'reason', CASE WHEN jsonb_typeof(p.frontmatter -> 'embed_skip' -> 'bytes') = 'number' THEN (p.frontmatter -> 'embed_skip' ->> 'bytes')::numeric::bigint END, p.frontmatter -> 'quarantine' ->> 'reason', p.deleted_at, false FROM pages p WHERE p.deleted_at IS NULL AND NOT EXISTS (SELECT 1 FROM content_chunks cc WHERE cc.page_id = p.id) AND p.source_id = ANY($3::text[]) ) r WHERE r.state = ANY($4::text[]) ORDER BY r.state, r.source_id, r.slug LIMIT $5",
+ "textSha256": "0cf94bb28cbf6fd1783b69ef51662416c561a56e9383e7f829ae53802757153b"
}
]
},
@@ -611,8 +611,8 @@
"number",
"number"
],
- "sql": "SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, p.source_id, cc.page_id FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE cc.\"embedding\" IS NULL AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576) AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND (cc.page_id, cc.chunk_index) > ($1, $2) ORDER BY cc.page_id, cc.chunk_index LIMIT $3",
- "textSha256": "43e1e9a1f41696ea88ef821dd7d8bb2739e86d2f2246212bed3c7e10be339752"
+ "sql": "SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, p.source_id, cc.page_id FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE cc.\"embedding\" IS NULL AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day'))) AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND (cc.page_id, cc.chunk_index) > ($1, $2) ORDER BY cc.page_id, cc.chunk_index LIMIT $3",
+ "textSha256": "4f366ddb0cb5375db52db9848a68ee86ae47a507423190fa6b5da4d113676d38"
}
],
"sourceId": [
@@ -630,8 +630,8 @@
"number",
"number"
],
- "sql": "SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, p.source_id, cc.page_id FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE cc.\"embedding\" IS NULL AND p.deleted_at IS NULL AND p.source_id = $1 AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576) AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND (cc.page_id, cc.chunk_index) > ($2, $3) ORDER BY cc.page_id, cc.chunk_index LIMIT $4",
- "textSha256": "feb2f91ac34cce170f5a25e79a6fba43959dbd07a502a946a56bf80fd156e164"
+ "sql": "SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, p.source_id, cc.page_id FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE cc.\"embedding\" IS NULL AND p.deleted_at IS NULL AND p.source_id = $1 AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day'))) AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND (cc.page_id, cc.chunk_index) > ($2, $3) ORDER BY cc.page_id, cc.chunk_index LIMIT $4",
+ "textSha256": "dc90c85af7d570cd7f06818d3e6d7c4dfc59bfb650ed638b4b30d0ec36d27bbd"
}
],
"updated_desc": [
@@ -646,8 +646,8 @@
"params": [
"number"
],
- "sql": "SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, p.source_id, cc.page_id, p.updated_at FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE cc.\"embedding\" IS NULL AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576) AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC LIMIT $1",
- "textSha256": "0b9d9d00a2d02ba20abd83cd85b7dc6b2ba8607ea17d69be6b8c8a1b3bcd6034"
+ "sql": "SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, p.source_id, cc.page_id, p.updated_at FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE cc.\"embedding\" IS NULL AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day'))) AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC LIMIT $1",
+ "textSha256": "456bc381607a7ff5de75acaa1539f6b1b7d9f31d3b96bb10019137d555eb795c"
}
],
"updated_desc+cursor": [
@@ -668,8 +668,8 @@
"number",
"number"
],
- "sql": "SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, p.source_id, cc.page_id, p.updated_at FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE cc.\"embedding\" IS NULL AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576) AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND ( p.updated_at < $1::timestamptz OR (p.updated_at = $2::timestamptz AND p.id > $3) OR (p.updated_at = $4::timestamptz AND p.id = $5 AND cc.chunk_index > $6) ) ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC LIMIT $7",
- "textSha256": "14e05e5362b3f96c021dee659ad6c04a656915c6d81aa862fd08312e9b12a8bc"
+ "sql": "SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, p.source_id, cc.page_id, p.updated_at FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE cc.\"embedding\" IS NULL AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day'))) AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND ( p.updated_at < $1::timestamptz OR (p.updated_at = $2::timestamptz AND p.id > $3) OR (p.updated_at = $4::timestamptz AND p.id = $5 AND cc.chunk_index > $6) ) ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC LIMIT $7",
+ "textSha256": "f1cf43d12e7078f9f0154081c928c7a77089262ef960e8274c2a203ddd022fb5"
}
],
"updated_desc+sourceId": [
@@ -685,8 +685,8 @@
"string",
"number"
],
- "sql": "SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, p.source_id, cc.page_id, p.updated_at FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE cc.\"embedding\" IS NULL AND p.deleted_at IS NULL AND p.source_id = $1 AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576) AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC LIMIT $2",
- "textSha256": "1f308ec9b71a2d430dffa6c0746917dcbead9aed5daed00914a7fac0cc2d43cf"
+ "sql": "SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, p.source_id, cc.page_id, p.updated_at FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE cc.\"embedding\" IS NULL AND p.deleted_at IS NULL AND p.source_id = $1 AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day'))) AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC LIMIT $2",
+ "textSha256": "8b6584cacd80ae18d63de7ce9eb92b76becbb22683c60ff9a1058754e2a9802d"
}
],
"updated_desc+sourceId+cursor": [
@@ -708,8 +708,8 @@
"number",
"number"
],
- "sql": "SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, p.source_id, cc.page_id, p.updated_at FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE cc.\"embedding\" IS NULL AND p.deleted_at IS NULL AND p.source_id = $1 AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576) AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND ( p.updated_at < $2::timestamptz OR (p.updated_at = $3::timestamptz AND p.id > $4) OR (p.updated_at = $5::timestamptz AND p.id = $6 AND cc.chunk_index > $7) ) ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC LIMIT $8",
- "textSha256": "30a408c8494070d912f5b0e1bafec94a93e64a52633de47abbf9e4b9b8996f95"
+ "sql": "SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, p.source_id, cc.page_id, p.updated_at FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE cc.\"embedding\" IS NULL AND p.deleted_at IS NULL AND p.source_id = $1 AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day'))) AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND ( p.updated_at < $2::timestamptz OR (p.updated_at = $3::timestamptz AND p.id > $4) OR (p.updated_at = $5::timestamptz AND p.id = $6 AND cc.chunk_index > $7) ) ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC LIMIT $8",
+ "textSha256": "abe659f4d029c2a83bd8eadf25841a8326fd84e5a26a717e928e93f6c61cb399"
}
]
},
@@ -726,8 +726,8 @@
"params": [
"number"
],
- "sql": "SELECT s.state, count(*)::bigint AS n, count(*) FILTER (WHERE s.eligible)::bigint AS eligible, count(*) FILTER (WHERE s.overdue)::bigint AS overdue, count(*) FILTER (WHERE s.unstamped)::bigint AS unstamped, min(s.created_at) FILTER (WHERE s.unstamped) AS unstamped_min FROM (SELECT CASE WHEN p.deleted_at IS NOT NULL THEN 'deleted_page_chunks' WHEN NOT NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') THEN 'quarantined' WHEN cc.\"embedding\" IS NOT NULL THEN 'embedded' WHEN p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND cc.\"embedding\" IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576) THEN 'pending' ELSE 'keyword_only' END AS state, NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AS eligible, COALESCE((p.deleted_at < now() - make_interval(hours => $1::int)), false) AS overdue, cc.embedding_pending_since IS NULL AS unstamped, cc.created_at FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE TRUE AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation))))) s GROUP BY s.state",
- "textSha256": "b1fc6d17a408d2b5d43f85a7317983d19470bbe6145a6d63c64864127a7bcc9c"
+ "sql": "SELECT s.state, count(*)::bigint AS n, count(*) FILTER (WHERE s.eligible)::bigint AS eligible, count(*) FILTER (WHERE s.overdue)::bigint AS overdue, count(*) FILTER (WHERE s.unstamped)::bigint AS unstamped, min(s.created_at) FILTER (WHERE s.unstamped) AS unstamped_min FROM (SELECT CASE WHEN p.deleted_at IS NOT NULL THEN 'deleted_page_chunks' WHEN NOT NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') THEN 'quarantined' WHEN cc.\"embedding\" IS NOT NULL THEN 'embedded' WHEN p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND cc.\"embedding\" IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day'))) THEN 'pending' ELSE 'keyword_only' END AS state, NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AS eligible, COALESCE((p.deleted_at < now() - make_interval(hours => $1::int)), false) AS overdue, cc.embedding_pending_since IS NULL AS unstamped, cc.created_at FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE TRUE AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation))))) s GROUP BY s.state",
+ "textSha256": "ddb89f41cb9a37812467b2f20c9176687e9b0b043340f74d62cb92a3ed179f68"
},
{
"lane": "pool",
@@ -755,8 +755,8 @@
"number",
"array<string>"
],
- "sql": "SELECT s.state, count(*)::bigint AS n, count(*) FILTER (WHERE s.eligible)::bigint AS eligible, count(*) FILTER (WHERE s.overdue)::bigint AS overdue, count(*) FILTER (WHERE s.unstamped)::bigint AS unstamped, min(s.created_at) FILTER (WHERE s.unstamped) AS unstamped_min FROM (SELECT CASE WHEN p.deleted_at IS NOT NULL THEN 'deleted_page_chunks' WHEN NOT NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') THEN 'quarantined' WHEN cc.\"embedding\" IS NOT NULL THEN 'embedded' WHEN p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND cc.\"embedding\" IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576) THEN 'pending' ELSE 'keyword_only' END AS state, NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AS eligible, COALESCE((p.deleted_at < now() - make_interval(hours => $1::int)), false) AS overdue, cc.embedding_pending_since IS NULL AS unstamped, cc.created_at FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE TRUE AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND p.source_id = ANY($2::text[])) s GROUP BY s.state",
- "textSha256": "089182bfd064f4f1c211169a74c9fffc8c942fbd1fe2e0e8f072aca02959c74c"
+ "sql": "SELECT s.state, count(*)::bigint AS n, count(*) FILTER (WHERE s.eligible)::bigint AS eligible, count(*) FILTER (WHERE s.overdue)::bigint AS overdue, count(*) FILTER (WHERE s.unstamped)::bigint AS unstamped, min(s.created_at) FILTER (WHERE s.unstamped) AS unstamped_min FROM (SELECT CASE WHEN p.deleted_at IS NOT NULL THEN 'deleted_page_chunks' WHEN NOT NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') THEN 'quarantined' WHEN cc.\"embedding\" IS NOT NULL THEN 'embedded' WHEN p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND cc.\"embedding\" IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day'))) THEN 'pending' ELSE 'keyword_only' END AS state, NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AS eligible, COALESCE((p.deleted_at < now() - make_interval(hours => $1::int)), false) AS overdue, cc.embedding_pending_since IS NULL AS unstamped, cc.created_at FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE TRUE AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND p.source_id = ANY($2::text[])) s GROUP BY s.state",
+ "textSha256": "ea7342850de4b013bf3221c8e89bbafa30a9f62d751f99cb0db7318945c55fae"
},
{
"lane": "pool",
@@ -817,8 +817,8 @@
{
"lane": "pool",
"params": [],
- "sql": "SELECT COALESCE(SUM(chars), 0)::bigint AS chars FROM (SELECT LENGTH(cc.chunk_text) AS chars FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND cc.\"embedding\" IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576)) stale",
- "textSha256": "bc6b4e9fceb1436d43b371361914ce8ada5eec09495d58aded7305eaa67a28fb"
+ "sql": "SELECT COALESCE(SUM(chars), 0)::bigint AS chars FROM (SELECT LENGTH(cc.chunk_text) AS chars FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND cc.\"embedding\" IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day')))) stale",
+ "textSha256": "349bd8fa1be77137951bc1425da68b823145e2a2da20327922749e5c56bb039c"
}
],
"signature": [
@@ -833,8 +833,8 @@
"params": [
"string"
],
- "sql": "SELECT COALESCE(SUM(chars), 0)::bigint AS chars FROM (SELECT LENGTH(cc.chunk_text) AS chars FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND cc.\"embedding\" IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576) UNION ALL SELECT LENGTH(cc.chunk_text) AS chars FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND cc.\"embedding\" IS NOT NULL AND p.embedding_signature <> $1 AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND (CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576)) stale",
- "textSha256": "0f2993ee80433b3f1165186851e6a4e5c124828e997ef551c84ef988346d1bb3"
+ "sql": "SELECT COALESCE(SUM(chars), 0)::bigint AS chars FROM (SELECT LENGTH(cc.chunk_text) AS chars FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND cc.\"embedding\" IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day'))) UNION ALL SELECT LENGTH(cc.chunk_text) AS chars FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.deleted_at IS NULL AND ((cc.body_generation IS NULL AND p.body_storage <> 'segmented') OR (cc.body_generation <= p.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= p.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > p.body_generation)))) AND cc.\"embedding\" IS NOT NULL AND p.embedding_signature <> $1 AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND NOT EXISTS (SELECT 1 FROM page_embedding_budget peb WHERE peb.page_id = p.id AND peb.state <> 'open' AND ((CASE WHEN p.body_storage = 'segmented' THEN p.body_bytes ELSE octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) END) > 1048576 OR peb.reason IN ('archive_backfill', 'archive_open_day')))) stale",
+ "textSha256": "3124e504b8f95f784713ef345561b78e5fa065b928cbb75efe157c8254062cdc"
}
]
},
diff --git a/test/fixtures/openclaw-gateway/README.md b/test/fixtures/openclaw-gateway/README.md
new file mode 100644
index 0000000000..706efbd5d7
--- /dev/null
+++ b/test/fixtures/openclaw-gateway/README.md
@@ -0,0 +1,23 @@
+# OpenClaw Gateway fixtures
+
+Recorded on 2026-11-15 from OpenClaw 2026.11.3 running `openclaw gateway --dev --auth token` on loopback with no
+model credentials and no channels, through `openclaw gateway call <method> --json --params <json>` (the documented
+CLI for Gateway RPCs). Paths were rewritten to `/home/alice-example`; session ids are the random ids the dev Gateway
+minted. Rows and messages are trimmed to the fields the archive reader reads.
+
+| File | Call |
+| --- | --- |
+| sessions-list-page1.json, sessions-list-page2.json | `sessions.list {limit:2}`, then `{limit:2, offset:2}` |
+| chat-history-scratch-alpha-page1/2.json | `chat.history {sessionKey, limit:2}`, then `offset:2` (pages walk backward; each page is oldest first) |
+| chat-history-archived.json | `chat.history` on a session archived with `sessions.patch {archived:true}` |
+| chat-history-scratch-3.json | `chat.history` after `sessions.reset` (reset in place: earlier messages are gone from the page) |
+| chat-history-group-shaped-key.json | a session created with a group-shaped key: `kind` says group, `chatType` stays `direct` |
+| error-auth.json, error-unreachable.json, error-credentials-required.json | wrong token, closed port, no credential |
+
+Doc-shaped (not recorded, marked `_note`): sessions-list-group-docshape.json, chat-history-group-docshape.json and
+chat-history-legacy-row.json. A channel-created group row needs a real messaging account, which the recording run never
+connected.
+
+2026.10.9 shape (synthetic, built from a 2026-11-15 probe of a 2026.10.9 dev Gateway): sessions-list-1009.json,
+chat-history-1009-main.json and chat-history-1009-probe-a.json. On 10.9 a session an operator created with
+sessions.create has no chatType (classification custom), and history rows carry __openclaw.transcriptPosition.
diff --git a/test/fixtures/openclaw-gateway/chat-history-1009-main.json b/test/fixtures/openclaw-gateway/chat-history-1009-main.json
new file mode 100644
index 0000000000..a5fbd9dc49
--- /dev/null
+++ b/test/fixtures/openclaw-gateway/chat-history-1009-main.json
@@ -0,0 +1,48 @@
+{
+ "sessionKey": "agent:dev:main",
+ "sessionId": "00000000-0000-4000-8000-000000001009",
+ "messages": [
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "text",
+ "text": "alice-example asks for the weekly summary"
+ }
+ ],
+ "timestamp": 1794724680000,
+ "__openclaw": {
+ "id": "00000000-0000-4000-8000-0000000011a1",
+ "recordTimestampMs": 1794724680000,
+ "transcriptPosition": {
+ "source": "example-transcript-source",
+ "rawSeq": 1
+ },
+ "seq": 1
+ }
+ },
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "text",
+ "text": "the weekly summary is in acme-example/notes"
+ }
+ ],
+ "timestamp": 1794724681000,
+ "__openclaw": {
+ "id": "00000000-0000-4000-8000-0000000011a2",
+ "recordTimestampMs": 1794724681000,
+ "transcriptPosition": {
+ "source": "example-transcript-source",
+ "rawSeq": 2
+ },
+ "seq": 2
+ },
+ "stopReason": "stop",
+ "provider": "openclaw",
+ "model": "gateway-injected"
+ }
+ ],
+ "hasMore": false
+}
diff --git a/test/fixtures/openclaw-gateway/chat-history-1009-probe-a.json b/test/fixtures/openclaw-gateway/chat-history-1009-probe-a.json
new file mode 100644
index 0000000000..20cad3ad59
--- /dev/null
+++ b/test/fixtures/openclaw-gateway/chat-history-1009-probe-a.json
@@ -0,0 +1,51 @@
+{
+ "sessionKey": "agent:dev:example-probe-a",
+ "sessionId": "00000000-0000-4000-8000-000000001010",
+ "messages": [
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "text",
+ "text": "note 1 for probe a"
+ }
+ ],
+ "timestamp": 1794724689369,
+ "__openclaw": {
+ "id": "00000000-0000-4000-8000-0000000012a1",
+ "recordTimestampMs": 1794724689369,
+ "transcriptPosition": {
+ "source": "example-transcript-source",
+ "rawSeq": 1
+ },
+ "seq": 1
+ },
+ "stopReason": "stop",
+ "provider": "openclaw",
+ "model": "gateway-injected"
+ },
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "text",
+ "text": "note 2 for probe a"
+ }
+ ],
+ "timestamp": 1794724691183,
+ "__openclaw": {
+ "id": "00000000-0000-4000-8000-0000000012a2",
+ "recordTimestampMs": 1794724691183,
+ "transcriptPosition": {
+ "source": "example-transcript-source",
+ "rawSeq": 2
+ },
+ "seq": 2
+ },
+ "stopReason": "stop",
+ "provider": "openclaw",
+ "model": "gateway-injected"
+ }
+ ],
+ "hasMore": false
+}
diff --git a/test/fixtures/openclaw-gateway/chat-history-archived.json b/test/fixtures/openclaw-gateway/chat-history-archived.json
new file mode 100644
index 0000000000..4dbc8b0f47
--- /dev/null
+++ b/test/fixtures/openclaw-gateway/chat-history-archived.json
@@ -0,0 +1,23 @@
+{
+ "sessionKey": "agent:dev:example-scratch-1",
+ "sessionId": "4afa4e52-4b37-451c-8337-f0c92ebab497",
+ "hasMore": false,
+ "totalMessages": 1,
+ "messages": [
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "text",
+ "text": "note 1"
+ }
+ ],
+ "timestamp": 1794721340452,
+ "__openclaw": {
+ "id": "d0fb7bd2-9d73-4237-8c5b-7ba393d71c11",
+ "seq": 1,
+ "recordTimestampMs": 1794721340452
+ }
+ }
+ ]
+}
diff --git a/test/fixtures/openclaw-gateway/chat-history-group-docshape.json b/test/fixtures/openclaw-gateway/chat-history-group-docshape.json
new file mode 100644
index 0000000000..6a550e708e
--- /dev/null
+++ b/test/fixtures/openclaw-gateway/chat-history-group-docshape.json
@@ -0,0 +1,32 @@
+{
+ "_note": "DOC-SHAPED history for the doc-shaped group row: same envelope as the recorded chat.history pages.",
+ "sessionKey": "agent:main:telegram:group:example-group-1",
+ "sessionId": "00000000-0000-4000-8000-000000000101",
+ "hasMore": false,
+ "totalMessages": 2,
+ "messages": [
+ {
+ "role": "user",
+ "content": "standup notes for acme-example",
+ "timestamp": 1794700100000,
+ "__openclaw": {
+ "id": "00000000-0000-4000-8000-0000000001a1",
+ "seq": 1
+ }
+ },
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "text",
+ "text": "noted the standup"
+ }
+ ],
+ "timestamp": 1794700160000,
+ "__openclaw": {
+ "id": "00000000-0000-4000-8000-0000000001a2",
+ "seq": 2
+ }
+ }
+ ]
+}
diff --git a/test/fixtures/openclaw-gateway/chat-history-group-shaped-key.json b/test/fixtures/openclaw-gateway/chat-history-group-shaped-key.json
new file mode 100644
index 0000000000..71f8cc92b3
--- /dev/null
+++ b/test/fixtures/openclaw-gateway/chat-history-group-shaped-key.json
@@ -0,0 +1,23 @@
+{
+ "sessionKey": "agent:dev:telegram:group:example-group-9",
+ "sessionId": "534a6bcb-e7fa-433e-99ad-086680bcec6f",
+ "hasMore": false,
+ "totalMessages": 1,
+ "messages": [
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "text",
+ "text": "group-shaped note"
+ }
+ ],
+ "timestamp": 1794721729670,
+ "__openclaw": {
+ "id": "3c55702f-3242-492c-99e4-85ca5eb14d66",
+ "seq": 1,
+ "recordTimestampMs": 1794721729670
+ }
+ }
+ ]
+}
diff --git a/test/fixtures/openclaw-gateway/chat-history-legacy-row.json b/test/fixtures/openclaw-gateway/chat-history-legacy-row.json
new file mode 100644
index 0000000000..f2dd53c155
--- /dev/null
+++ b/test/fixtures/openclaw-gateway/chat-history-legacy-row.json
@@ -0,0 +1,18 @@
+{
+ "_note": "DOC-SHAPED history for the row without chatType.",
+ "sessionKey": "agent:main:webhook:example-legacy-row",
+ "sessionId": "00000000-0000-4000-8000-000000000102",
+ "hasMore": false,
+ "totalMessages": 1,
+ "messages": [
+ {
+ "role": "user",
+ "content": "webhook ping from source-a",
+ "timestamp": 1794700200000,
+ "__openclaw": {
+ "id": "00000000-0000-4000-8000-0000000001b1",
+ "seq": 1
+ }
+ }
+ ]
+}
diff --git a/test/fixtures/openclaw-gateway/chat-history-scratch-3.json b/test/fixtures/openclaw-gateway/chat-history-scratch-3.json
new file mode 100644
index 0000000000..1b1ba39faa
--- /dev/null
+++ b/test/fixtures/openclaw-gateway/chat-history-scratch-3.json
@@ -0,0 +1,23 @@
+{
+ "sessionKey": "agent:dev:example-scratch-3",
+ "sessionId": "9eb7e494-36f2-4762-8537-70b6a1a97c7d",
+ "hasMore": false,
+ "totalMessages": 1,
+ "messages": [
+ {
+ "role": "system",
+ "content": [
+ {
+ "type": "text",
+ "text": "Reset"
+ }
+ ],
+ "timestamp": 1794721432564,
+ "__openclaw": {
+ "id": "cffc6095-a427-4cdb-b8b8-e213de55588b",
+ "seq": 1,
+ "kind": "reset"
+ }
+ }
+ ]
+}
diff --git a/test/fixtures/openclaw-gateway/chat-history-scratch-alpha-page1.json b/test/fixtures/openclaw-gateway/chat-history-scratch-alpha-page1.json
new file mode 100644
index 0000000000..0268b9eb1f
--- /dev/null
+++ b/test/fixtures/openclaw-gateway/chat-history-scratch-alpha-page1.json
@@ -0,0 +1,33 @@
+{
+ "sessionKey": "agent:dev:example-scratch-alpha",
+ "sessionId": "318c6a69-4cfa-4aba-bda5-8babbb465c4e",
+ "hasMore": true,
+ "nextOffset": 2,
+ "totalMessages": 3,
+ "messages": [
+ {
+ "role": "user",
+ "content": "what is the capital of acme-example land",
+ "timestamp": 1794721320954,
+ "provenance": {
+ "kind": "external_user"
+ },
+ "__openclaw": {
+ "id": "7e7446df-a66b-422d-81e3-83d4bcaa5aa9",
+ "seq": 2,
+ "recordTimestampMs": 1794721321423
+ }
+ },
+ {
+ "role": "custom",
+ "content": "Your request couldn't be completed: No route-compatible authentication source is configured for openai.",
+ "timestamp": 1794721322162,
+ "customType": "run-failed-before-reply",
+ "__openclaw": {
+ "id": "2db6c3a7-e0d9-4bbe-8b4c-26967c2d315a",
+ "seq": 3,
+ "recordTimestampMs": 1794721322162
+ }
+ }
+ ]
+}
diff --git a/test/fixtures/openclaw-gateway/chat-history-scratch-alpha-page2.json b/test/fixtures/openclaw-gateway/chat-history-scratch-alpha-page2.json
new file mode 100644
index 0000000000..847a84e3b2
--- /dev/null
+++ b/test/fixtures/openclaw-gateway/chat-history-scratch-alpha-page2.json
@@ -0,0 +1,23 @@
+{
+ "sessionKey": "agent:dev:example-scratch-alpha",
+ "sessionId": "318c6a69-4cfa-4aba-bda5-8babbb465c4e",
+ "hasMore": false,
+ "totalMessages": 3,
+ "messages": [
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "text",
+ "text": "hello from synthetic user"
+ }
+ ],
+ "timestamp": 1794721281750,
+ "__openclaw": {
+ "id": "44f47e7a-6b14-4125-bd59-4fc4432647b0",
+ "seq": 1,
+ "recordTimestampMs": 1794721281750
+ }
+ }
+ ]
+}
diff --git a/test/fixtures/openclaw-gateway/error-auth.json b/test/fixtures/openclaw-gateway/error-auth.json
new file mode 100644
index 0000000000..1d809fbecc
--- /dev/null
+++ b/test/fixtures/openclaw-gateway/error-auth.json
@@ -0,0 +1,15 @@
+{
+ "ok": false,
+ "error": {
+ "type": "gateway_request_error",
+ "code": "INVALID_REQUEST",
+ "message": "unauthorized: gateway token mismatch (use this gateway's gateway.auth.token or pair the device)",
+ "details": {
+ "code": "AUTH_TOKEN_MISMATCH",
+ "authReason": "token_mismatch",
+ "canRetryWithDeviceToken": false,
+ "recommendedNextStep": "update_auth_credentials"
+ },
+ "retryable": false
+ }
+}
diff --git a/test/fixtures/openclaw-gateway/error-credentials-required.json b/test/fixtures/openclaw-gateway/error-credentials-required.json
new file mode 100644
index 0000000000..41d6fd951c
--- /dev/null
+++ b/test/fixtures/openclaw-gateway/error-credentials-required.json
@@ -0,0 +1,7 @@
+{
+ "ok": false,
+ "error": {
+ "type": "gateway_credentials_required",
+ "message": "gateway sessions.list requires credentials before opening a websocket\nFix: configure gateway.auth token/password, pair this device, or pass --token/--password.\nConfig: /home/alice-example/.openclaw/openclaw.json"
+ }
+}
diff --git a/test/fixtures/openclaw-gateway/error-unreachable.json b/test/fixtures/openclaw-gateway/error-unreachable.json
new file mode 100644
index 0000000000..9b1b7ea295
--- /dev/null
+++ b/test/fixtures/openclaw-gateway/error-unreachable.json
@@ -0,0 +1,14 @@
+{
+ "ok": false,
+ "error": {
+ "type": "gateway_transport_error",
+ "kind": "closed",
+ "message": "Gateway not reachable at ws://127.0.0.1:18799 (ECONNREFUSED).",
+ "reason": "connect ECONNREFUSED 127.0.0.1:18799"
+ },
+ "gateway": {
+ "url": "ws://127.0.0.1:18799",
+ "urlSource": "local loopback",
+ "bindDetail": "Bind: loopback"
+ }
+}
diff --git a/test/fixtures/openclaw-gateway/sessions-list-1009.json b/test/fixtures/openclaw-gateway/sessions-list-1009.json
new file mode 100644
index 0000000000..03ec3583c4
--- /dev/null
+++ b/test/fixtures/openclaw-gateway/sessions-list-1009.json
@@ -0,0 +1,45 @@
+{
+ "_note": "Synthetic, in the shape OpenClaw 2026.10.9 returned for sessions.list on a dev Gateway (probe 2026-11-15): the main session carries chatType direct; a session an operator created with sessions.create has no chatType and classification custom. Values are placeholders; rows are trimmed to the fields the reader reads plus the 10.9 markers.",
+ "ts": 1794724694256,
+ "path": "/home/alice-example/.openclaw/agents/dev/agent/openclaw-agent.sqlite",
+ "count": 2,
+ "totalCount": 2,
+ "limitApplied": 200,
+ "nextOffset": null,
+ "hasMore": false,
+ "owners": [],
+ "sessions": [
+ {
+ "key": "agent:dev:main",
+ "sessionId": "00000000-0000-4000-8000-000000001009",
+ "agentId": "dev",
+ "kind": "direct",
+ "classification": "main",
+ "chatType": "direct",
+ "createdVia": "system",
+ "isMain": true,
+ "archived": false,
+ "visibility": "shared",
+ "updatedAt": 1794724693000,
+ "lifecycleRevision": "00000000-0000-4000-8000-0000000010a1"
+ },
+ {
+ "key": "agent:dev:example-probe-a",
+ "sessionId": "00000000-0000-4000-8000-000000001010",
+ "agentId": "dev",
+ "kind": "direct",
+ "classification": "custom",
+ "label": "probe a",
+ "displayName": "probe a",
+ "createdVia": "operator",
+ "isMain": false,
+ "isDock": false,
+ "isBackground": false,
+ "archived": false,
+ "pinned": false,
+ "visibility": "shared",
+ "updatedAt": 1794724693268,
+ "lifecycleRevision": "00000000-0000-4000-8000-0000000010a2"
+ }
+ ]
+}
diff --git a/test/fixtures/openclaw-gateway/sessions-list-group-docshape.json b/test/fixtures/openclaw-gateway/sessions-list-group-docshape.json
new file mode 100644
index 0000000000..654edb0e77
--- /dev/null
+++ b/test/fixtures/openclaw-gateway/sessions-list-group-docshape.json
@@ -0,0 +1,39 @@
+{
+ "_note": "DOC-SHAPED, not recorded: a channel-created group row cannot be produced without a real messaging account. Fields follow the recorded sessions.list row shape plus SessionEntry chatType/groupChannel/subject from docs/reference/session-management-compaction/schema.md (OpenClaw 2026.11.3).",
+ "count": 2,
+ "totalCount": 2,
+ "limitApplied": 100,
+ "nextOffset": null,
+ "hasMore": false,
+ "sessions": [
+ {
+ "key": "agent:main:telegram:group:example-group-1",
+ "sessionId": "00000000-0000-4000-8000-000000000101",
+ "agentId": "main",
+ "kind": "group",
+ "classification": "group",
+ "chatType": "group",
+ "groupChannel": "telegram",
+ "subject": "acme-example team",
+ "displayName": "acme-example team",
+ "createdVia": "channel",
+ "createdAt": 1794700000000,
+ "updatedAt": 1794710000000,
+ "archived": false,
+ "isMain": false,
+ "spawnDepth": 0,
+ "visibility": "shared"
+ },
+ {
+ "key": "agent:main:webhook:example-legacy-row",
+ "sessionId": "00000000-0000-4000-8000-000000000102",
+ "agentId": "main",
+ "displayName": "legacy row without chatType",
+ "createdAt": 1794700000000,
+ "updatedAt": 1794710000000,
+ "archived": false,
+ "isMain": false,
+ "spawnDepth": 0
+ }
+ ]
+}
diff --git a/test/fixtures/openclaw-gateway/sessions-list-page1.json b/test/fixtures/openclaw-gateway/sessions-list-page1.json
new file mode 100644
index 0000000000..d45ea11c90
--- /dev/null
+++ b/test/fixtures/openclaw-gateway/sessions-list-page1.json
@@ -0,0 +1,49 @@
+{
+ "ts": 1794721489994,
+ "path": "/home/alice-example/.openclaw/agents/dev/agent/openclaw-agent.sqlite",
+ "count": 2,
+ "totalCount": 3,
+ "limitApplied": 2,
+ "nextOffset": 2,
+ "hasMore": true,
+ "sessions": [
+ {
+ "key": "agent:dev:example-scratch-3",
+ "sessionId": "9eb7e494-36f2-4762-8537-70b6a1a97c7d",
+ "agentId": "dev",
+ "kind": "direct",
+ "classification": "direct",
+ "chatType": "direct",
+ "label": "scratch 3",
+ "displayName": "scratch 3",
+ "createdVia": "operator",
+ "createdAt": 1794721343940,
+ "updatedAt": 1794721432561,
+ "archived": false,
+ "isMain": false,
+ "parentSessionKey": "agent:dev:example-main",
+ "spawnDepth": 0,
+ "visibility": "shared"
+ },
+ {
+ "key": "agent:dev:example-scratch-alpha",
+ "sessionId": "318c6a69-4cfa-4aba-bda5-8babbb465c4e",
+ "agentId": "dev",
+ "kind": "direct",
+ "classification": "direct",
+ "chatType": "direct",
+ "label": "scratch alpha",
+ "displayName": "scratch alpha",
+ "createdVia": "operator",
+ "createdAt": 1794721280138,
+ "updatedAt": 1794721322173,
+ "archived": false,
+ "isMain": false,
+ "parentSessionKey": "agent:dev:example-main",
+ "spawnDepth": 0,
+ "visibility": "shared",
+ "lastInteractionAt": 1794721320980,
+ "lastActivityAt": 1794721322120
+ }
+ ]
+}
diff --git a/test/fixtures/openclaw-gateway/sessions-list-page2.json b/test/fixtures/openclaw-gateway/sessions-list-page2.json
new file mode 100644
index 0000000000..e87cf86286
--- /dev/null
+++ b/test/fixtures/openclaw-gateway/sessions-list-page2.json
@@ -0,0 +1,29 @@
+{
+ "ts": 1794721491075,
+ "path": "/home/alice-example/.openclaw/agents/dev/agent/openclaw-agent.sqlite",
+ "count": 1,
+ "totalCount": 3,
+ "limitApplied": 2,
+ "nextOffset": null,
+ "hasMore": false,
+ "sessions": [
+ {
+ "key": "agent:dev:telegram:group:example-group-9",
+ "sessionId": "534a6bcb-e7fa-433e-99ad-086680bcec6f",
+ "agentId": "dev",
+ "kind": "group",
+ "classification": "group",
+ "chatType": "direct",
+ "label": "group test",
+ "displayName": "group test",
+ "createdVia": "operator",
+ "createdAt": 1794721319385,
+ "updatedAt": 1794721319384,
+ "archived": false,
+ "isMain": false,
+ "parentSessionKey": "agent:dev:example-main",
+ "spawnDepth": 0,
+ "visibility": "shared"
+ }
+ ]
+}
diff --git a/test/helpers/config-reader-registry.ts b/test/helpers/config-reader-registry.ts
index c0e2fa015d..bc722d025d 100644
--- a/test/helpers/config-reader-registry.ts
+++ b/test/helpers/config-reader-registry.ts
@@ -25,6 +25,9 @@ export interface InternalConfigState {
export const INTERNAL_CONFIG_STATE: readonly InternalConfigState[] = [
{ key: 'version', reason: 'schema version stamp written by migrate.ts' },
+ { key: 'transcripts.archive.pinned_timezone.', prefix: true, reason: 'the timezone the conversation archive pinned for a source on its first run (transcripts/archive.ts); a hand edit would re-slice every archived day' },
+ { key: 'transcripts.archive.audience.', prefix: true, reason: 'the audience (personal or shared) the conversation archive last ran under per source (transcripts/archive.ts); it decides when earlier pages are kept to the owner' },
+ { key: 'transcripts.archive.index_pending.', prefix: true, reason: 'days whose conversation archive day index still has to be written (transcripts/archive.ts); the next run retries them' },
{ key: 'maintenance.last_run', reason: 'the free-upkeep run record (steps, cursor, parked items, next due time) that maintenance/run.ts writes after each gbrain maintain --unattended run; maintain --status and doctor read it, so a hand edit would misreport upkeep' },
{ key: 'maintenance.embed_spend', reason: 'the per-day dollar total the capped upkeep embed step has spent (maintenance/run.ts); the daily cap reads it, so a hand edit would let upkeep spend past maintenance.embed_daily_usd' },
{ key: 'heal.audit', reason: 'the bounded self-heal audit log (last 200 rows, 30 days) that heal-audit.ts writes for each heal; doctor reads it to count a heal\'s consecutive failures, so a hand edit would hide an escalation' },
diff --git a/test/helpers/transcript-archive.ts b/test/helpers/transcript-archive.ts
new file mode 100644
index 0000000000..90fbac8c22
--- /dev/null
+++ b/test/helpers/transcript-archive.ts
@@ -0,0 +1,42 @@
+/**
+ * Shared builders for the conversation-archive suites (GST-2768): synthetic sessions, a static reader, and readers of
+ * what the archive wrote. Every value is synthetic.
+ */
+import type { BrainEngine } from '../../src/core/engine.ts';
+import type { ArchiveChannel, ArchiveConversation, ArchiveReader } from '../../src/core/transcripts/archive.ts';
+import type { TranscriptFormat } from '../../src/core/transcripts/types.ts';
+
+export type Turn = ['user' | 'assistant', string, string];
+
+export function conversation(harness: TranscriptFormat, key: string, turns: Turn[], channel?: ArchiveChannel, raw?: Record<string, unknown>): ArchiveConversation {
+ return {
+ harness,
+ conversationKey: key,
+ ...(channel ? { channel } : {}),
+ session: {
+ meta: { harness, sessionId: key, startedAt: turns[0]?.[1], ...(raw ? { raw } : {}) },
+ messages: turns.map(([role, timestamp, text]) => ({ role, timestamp, text })),
+ },
+ };
+}
+
+export function staticReader(harness: TranscriptFormat, convs: ArchiveConversation[] | (() => ArchiveConversation[])): ArchiveReader {
+ return { harness, name: harness, list: async () => (typeof convs === 'function' ? convs() : convs) };
+}
+
+export interface ArchivedPage { slug: string; revision: number; frontmatter: Record<string, unknown>; body: string }
+
+export async function archivedPages(engine: BrainEngine, sourceId = 'conversations'): Promise<ArchivedPage[]> {
+ const rows = await engine.executeRaw<{ slug: string; type: string; revision: number | string; frontmatter: unknown; compiled_truth: string }>(
+ 'SELECT slug, type, generation AS revision, frontmatter, compiled_truth FROM pages WHERE source_id = $1 AND deleted_at IS NULL ORDER BY slug', [sourceId]);
+ return rows.map(r => ({
+ slug: r.slug,
+ revision: Number(r.revision),
+ frontmatter: { type: r.type, ...((typeof r.frontmatter === 'string' ? JSON.parse(r.frontmatter) : r.frontmatter) as Record<string, unknown>) },
+ body: r.compiled_truth,
+ }));
+}
+
+export async function ledgerRows(engine: BrainEngine): Promise<Array<{ key: string; value: string }>> {
+ return engine.executeRaw<{ key: string; value: string }>("SELECT key, value FROM config WHERE key LIKE 'transcripts.archive.%' ORDER BY key");
+}
diff --git a/test/import-private-hold-callers-7897.test.ts b/test/import-private-hold-callers-7897.test.ts
index 76827c6ac7..a9ad7b128e 100644
--- a/test/import-private-hold-callers-7897.test.ts
+++ b/test/import-private-hold-callers-7897.test.ts
@@ -37,6 +37,7 @@ const AUDITED: Record<string, Audit> = {
'src/core/persistence/sync-prepare.ts': { calls: { importFromContent: 2, screenImportContent: 2 }, disposition: 'held',
why: 'screenSyncImport screens with the hold first; the two unflagged screens are code files, which carry no frontmatter' },
'src/core/transcripts/ingest.ts': { calls: { importFromContent: 1 }, disposition: 'derived', why: 'render.ts writes fixed keys; other keys come from the stored page' },
+ 'src/core/transcripts/archive.ts': { calls: { importFromContent: 1 }, disposition: 'derived', why: 'archive-day.ts writes fixed keys; readable_by is the owner (and a verified group) the archive resolves, never source text' },
'src/core/trust/page-handlers.ts': { calls: { importFromContent: 1 }, disposition: 'derived', why: 'a stored version\'s markdown' },
'src/eval/brainbench/metrics/write-back.ts': { calls: { importFromContent: 1 }, disposition: 'eval', why: 'benchmark brain' },
'src/eval/brainbench/seed.ts': { calls: { importFromContent: 1 }, disposition: 'eval', why: 'benchmark brain' },
diff --git a/test/page-row-deleters-guard-6712.test.ts b/test/page-row-deleters-guard-6712.test.ts
index ca6b5dab62..0acc7ed614 100644
--- a/test/page-row-deleters-guard-6712.test.ts
+++ b/test/page-row-deleters-guard-6712.test.ts
@@ -34,6 +34,7 @@ const REVIEWED: Record<string, { count: number; why: string }> = {
'src/core/engine-constants.ts': { count: 1, why: 'doc comment of the batch bound' },
'src/core/bootstrap/verify.ts': { count: 1, why: 'unmanaged brains only; a managed brain removes its probe pages through delete_page --purge' },
'src/core/transcripts/ingest.ts': { count: 1, why: 'unmanaged brains only; a managed brain deletes stale parts through the coordinator' },
+ 'src/core/transcripts/archive.ts': { count: 1, why: 'repo-less database mode only; a managed brain deletes stale day parts through the coordinator and a repo brain removes the files and syncs' },
'src/core/persistence/purge-deleted.ts': { count: 1, why: 'unmanaged retention (expireRecordedTombstones); a managed brain purges through delete_page --purge' },
'src/commands/migrate-engine.ts': { count: 1, why: 'engine migration copy, outside any source write' },
'src/core/github-source.ts': { count: 2, why: 'GitHub connector source, not a managed Git checkout' },
diff --git a/test/transcripts-archive-adopt-legacy.test.ts b/test/transcripts-archive-adopt-legacy.test.ts
new file mode 100644
index 0000000000..075b9756e1
--- /dev/null
+++ b/test/transcripts-archive-adopt-legacy.test.ts
@@ -0,0 +1,68 @@
+/**
+ * GST-2768 PR 2 (planned PR 3 test, moved here by the review split): adopt-legacy reads an old archive folder's day
+ * files, lists the days it covers per harness, and with --apply marks them covered so backfill writes only the
+ * missing days. The dry run writes nothing, and the old files are never changed. Repo-less PGLite.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { ARCHIVE_TIMEZONE_KEY, adoptLegacyArchive, runTranscriptArchive, setArchiveEnabled } from '../src/core/transcripts/archive.ts';
+import { archivedPages, conversation, ledgerRows, staticReader } from './helpers/transcript-archive.ts';
+
+let engine: PGLiteEngine;
+let dir: string;
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+});
+
+afterAll(async () => {
+ await engine.disconnect();
+});
+
+beforeEach(async () => {
+ await resetPgliteState(engine);
+ dir = mkdtempSync(join(tmpdir(), 'archive-adopt-'));
+ mkdirSync(join(dir, 'claude-code'));
+ writeFileSync(join(dir, '2026-10-01.md'), '# chats of the day\n');
+ writeFileSync(join(dir, 'claude-code', '2026-10-02-session.md'), '# a claude code day\n');
+ writeFileSync(join(dir, 'notes.md'), '---\ndate: 2026-10-03\nharness: claude-code\n---\n# dated by frontmatter\n');
+ writeFileSync(join(dir, 'readme.txt'), 'not a day file');
+});
+
+const sessions = () => staticReader('claude-code', ['01', '02', '03', '04', '05'].map(d =>
+ conversation('claude-code', `session-${d}`, [['user', `2026-10-${d}T12:00:00.000Z`, `work on the ${d}th`]])));
+
+describe('adopt-legacy (pglite)', () => {
+ test('dry run lists covered days and writes nothing; apply marks them; backfill skips them and writes the rest', async () => {
+ try {
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await setArchiveEnabled(engine, true);
+ const mtime = statSync(join(dir, 'notes.md')).mtimeMs;
+ const configBefore = await ledgerRows(engine);
+ const preview = await adoptLegacyArchive(engine, { dir, apply: false });
+ expect(preview.applied).toBe(false);
+ expect(preview.covered).toEqual([
+ { harness: '*', day: '2026-10-01', file: '2026-10-01.md' },
+ { harness: 'claude-code', day: '2026-10-02', file: 'claude-code/2026-10-02-session.md' },
+ { harness: 'claude-code', day: '2026-10-03', file: 'notes.md' },
+ ]);
+ expect(await ledgerRows(engine)).toEqual(configBefore);
+
+ const applied = await adoptLegacyArchive(engine, { dir, apply: true });
+ expect(applied.applied).toBe(true);
+ const result = await runTranscriptArchive(engine, { mode: 'backfill', from: '2026-10-01', to: '2026-10-05', readers: [sessions()], now: new Date('2026-10-20T00:00:00Z') });
+ expect(result.days.filter(d => d.status === 'skipped_legacy').map(d => d.day).sort()).toEqual(['2026-10-01', '2026-10-02', '2026-10-03']);
+ expect((await archivedPages(engine)).filter(p => p.slug.startsWith('conversations/claude-code/')).map(p => p.slug.split('/')[2])).toEqual(['2026-10-04', '2026-10-05']);
+ expect(readFileSync(join(dir, 'notes.md'), 'utf8')).toContain('dated by frontmatter');
+ expect(statSync(join(dir, 'notes.md')).mtimeMs).toBe(mtime);
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ }, 60_000);
+});
diff --git a/test/transcripts-archive-audience.test.ts b/test/transcripts-archive-audience.test.ts
new file mode 100644
index 0000000000..9f7f6f4f0e
--- /dev/null
+++ b/test/transcripts-archive-audience.test.ts
@@ -0,0 +1,72 @@
+/**
+ * GST-2768 PR 2: the archive and a brain's audience. A brain archived while personal and then made shared keeps no
+ * earlier archive page open to other members: the next run re-scopes the finished days first, checked with remote=true
+ * credentials bound to members through get_page and search. A run whose audience lookup failed publishes nothing.
+ * Repo-less PGLite; every value is synthetic.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { GBrainOAuthProvider } from '../src/core/oauth-provider.ts';
+import { sqlQueryForEngine } from '../src/core/sql-query.ts';
+import { mintLegacyToken } from '../src/core/token-mint.ts';
+import { dispatchToolCall } from '../src/mcp/dispatch.ts';
+import { addMember, bindCredential } from '../src/core/access/members.ts';
+import { setOwner } from '../src/core/access/owner.ts';
+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
+import { ARCHIVE_TIMEZONE_KEY, runTranscriptArchive, setArchiveEnabled } from '../src/core/transcripts/archive.ts';
+import type { BrainEngine } from '../src/core/engine.ts';
+import type { AuthInfo } from '../src/core/ops/contract.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { archivedPages, conversation, staticReader } from './helpers/transcript-archive.ts';
+
+const SRC = 'conversations';
+const CANARY = 'canary-personal-era-0007';
+let engine: PGLiteEngine;
+const provider = () => new GBrainOAuthProvider({ sql: sqlQueryForEngine(engine), transaction: fn => engine.transaction(tx => fn(sqlQueryForEngine(tx))) });
+
+beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); }, 120_000);
+afterAll(async () => { await disposePersistenceConsumer(engine); await engine.disconnect(); }, 60_000);
+beforeEach(async () => {
+ await resetPgliteState(engine);
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await setArchiveEnabled(engine, true);
+});
+
+async function token(who: string): Promise<AuthInfo> {
+ const minted = await mintLegacyToken(engine, { name: 'tok-' + who, scopes: ['read'], takesHolders: ['world'], sourceGrant: [SRC] });
+ await bindCredential(engine, { kind: 'token', id: minted.id }, { member: who });
+ return await provider().verifyAccessToken(minted.token) as unknown as AuthInfo;
+}
+const text = (out: { content: Array<{ text?: string }> }) => out.content.map(c => c.text ?? '').join('\n');
+
+describe('archive and the brain audience (pglite, remote=true)', () => {
+ test('a day archived while personal is re-scoped when the brain becomes shared: a non-owner member reads it through neither get_page nor search', async () => {
+ await engine.setConfig('brain.audience', 'personal');
+ const reader = staticReader('openclaw', [conversation('openclaw', 'agent:main:main', [['user', '2026-10-01T10:00:00.000Z', CANARY + ' private note']], { chatType: 'direct', verified: true })]);
+ const first = await runTranscriptArchive(engine, { mode: 'backfill', from: '2026-10-01', to: '2026-10-01', readers: [reader], now: new Date('2026-10-05T00:00:00Z') });
+ expect([first.privacy, first.written]).toEqual(['personal', 1]);
+ await engine.setConfig('brain.audience', 'shared');
+ for (const id of ['alice-example', 'bob-example']) await addMember(engine, id);
+ await setOwner(engine, 'alice-example');
+ const next = await runTranscriptArchive(engine, { mode: 'run', readers: [staticReader('openclaw', [])], now: new Date('2026-11-05T00:00:00Z') });
+ expect(next.privacy).toBe('shared');
+ const slug = (await archivedPages(engine, SRC)).find(p => p.body.includes(CANARY))!.slug;
+ const [owner, other] = [await token('alice-example'), await token('bob-example')];
+ const ctx = (auth: AuthInfo) => ({ remote: true, transport: 'http' as const, auth, sourceId: SRC, takesHoldersAllowList: ['world'] });
+ expect(text(await dispatchToolCall(engine, 'get_page', { slug }, ctx(other)))).not.toContain(CANARY);
+ expect(text(await dispatchToolCall(engine, 'search', { query: CANARY }, ctx(other)))).not.toContain(slug);
+ expect(text(await dispatchToolCall(engine, 'get_page', { slug }, ctx(owner)))).toContain(CANARY);
+ }, 120_000);
+
+ test('a run whose audience lookup failed writes no day page and ends archive_audience_unproven', async () => {
+ const failing = new Proxy(engine, { get(target, prop) {
+ const value = Reflect.get(target, prop, target);
+ if (prop !== 'executeRaw') return typeof value === 'function' ? value.bind(target) : value;
+ return (sql: string, params?: unknown[]) => /mcp_request_log/.test(sql) ? Promise.reject(new Error('synthetic usage read failure')) : target.executeRaw(sql, params);
+ } }) as unknown as BrainEngine;
+ const reader = staticReader('claude-code', [conversation('claude-code', 'session-unproven', [['user', '2026-10-01T10:00:00.000Z', 'unproven audience note']])]);
+ const err = await runTranscriptArchive(failing, { mode: 'backfill', from: '2026-10-01', to: '2026-10-01', readers: [reader], now: new Date('2026-10-05T00:00:00Z') }).then(() => null, (e: { code?: string }) => e);
+ expect(err?.code).toBe('archive_audience_unproven');
+ expect(await archivedPages(engine, SRC)).toEqual([]);
+ }, 120_000);
+});
diff --git a/test/transcripts-archive-backfill-cap-held.test.ts b/test/transcripts-archive-backfill-cap-held.test.ts
new file mode 100644
index 0000000000..85930fb96d
--- /dev/null
+++ b/test/transcripts-archive-backfill-cap-held.test.ts
@@ -0,0 +1,44 @@
+/**
+ * GST-2768 PR 2: days that keep failing never use up transcripts.archive.backfill_days_per_run. With the five newest
+ * days held (redaction fails for their sessions) and a cap of 5, one backfill run still archives five good days, and
+ * the held days are retried on the next run without counting against its cap. Synthetic values. Repo-less PGLite.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { redactSession } from '../src/core/transcripts/render.ts';
+import { ARCHIVE_BACKFILL_CAP_KEY, ARCHIVE_TIMEZONE_KEY, runTranscriptArchive, setArchiveEnabled } from '../src/core/transcripts/archive.ts';
+import type { ParsedSession } from '../src/core/transcripts/types.ts';
+import { conversation, staticReader } from './helpers/transcript-archive.ts';
+
+let engine: PGLiteEngine;
+beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); });
+afterAll(async () => { await engine.disconnect(); });
+beforeEach(async () => {
+ await resetPgliteState(engine);
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await engine.setConfig(ARCHIVE_BACKFILL_CAP_KEY, '5');
+ await setArchiveEnabled(engine, true);
+});
+
+const day = (n: number) => '2026-11-' + String(n).padStart(2, '0');
+const convs = Array.from({ length: 12 }, (_, i) => conversation('codex', 'rollout-' + day(i + 1), [['user', day(i + 1) + 'T09:00:00.000Z', 'work on ' + day(i + 1)]]));
+const FAILING = new Set([8, 9, 10, 11, 12].map(n => 'rollout-' + day(n)));
+const redact = (s: ParsedSession) => {
+ if (FAILING.has(s.meta.sessionId)) throw new Error('synthetic redaction failure');
+ return redactSession(s);
+};
+const backfill = () => runTranscriptArchive(engine, { mode: 'backfill', from: '2026-11-01', readers: [staticReader('codex', convs)], now: new Date('2026-11-14T12:00:00Z'), redact });
+
+describe('backfill cap with days that keep failing (pglite)', () => {
+ test('five held days and a cap of 5: the run still archives five good days, and the next run retries the held days', async () => {
+ const first = await backfill();
+ expect(first.days.filter(d => d.status === 'held').map(d => d.day).sort()).toEqual([8, 9, 10, 11, 12].map(day));
+ expect(first.days.filter(d => d.status === 'written').map(d => d.day).sort()).toEqual([3, 4, 5, 6, 7].map(day));
+ expect(first.deferred_days).toEqual([1, 2].map(day));
+ const second = await backfill();
+ expect(second.days.filter(d => d.status === 'held').map(d => d.day).sort()).toEqual([8, 9, 10, 11, 12].map(day));
+ expect(second.days.filter(d => d.status === 'written').map(d => d.day).sort()).toEqual([1, 2].map(day));
+ expect(second.deferred_days).toEqual([]);
+ }, 60_000);
+});
diff --git a/test/transcripts-archive-backfill-held.test.ts b/test/transcripts-archive-backfill-held.test.ts
new file mode 100644
index 0000000000..1b0ece6fbc
--- /dev/null
+++ b/test/transcripts-archive-backfill-held.test.ts
@@ -0,0 +1,83 @@
+/**
+ * GST-2768 PR 2: backfilled archive pages are keyword-only with their embedding held (reason archive_backfill) in the
+ * existing page_embedding_budget ledger, whatever their size, so the embed sweep sends nothing until the owner approves
+ * the previewed cost (gbrain embed --held). Pages the forward run writes for closed days are not held. Repo-less PGLite.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { importFromContent } from '../src/core/import-file.ts';
+import { planPageEmbedding } from '../src/core/page-embedding-budget.ts';
+import { readBudgetLimited } from '../src/core/held-embedding.ts';
+import { ARCHIVE_BACKFILL_HOLD_REASON, ARCHIVE_TIMEZONE_KEY, runTranscriptArchive, setArchiveEnabled } from '../src/core/transcripts/archive.ts';
+import { conversation, staticReader } from './helpers/transcript-archive.ts';
+
+let engine: PGLiteEngine;
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+});
+
+afterAll(async () => {
+ await engine.disconnect();
+});
+
+beforeEach(async () => {
+ await resetPgliteState(engine);
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await setArchiveEnabled(engine, true);
+});
+
+async function pageRow(slug: string, sourceId = 'conversations') {
+ const [row] = await engine.executeRaw<{ id: number; compiled_truth: string; body_storage: 'inline' | 'segmented' | null }>(
+ 'SELECT id, compiled_truth, body_storage FROM pages WHERE slug = $1 AND source_id = $2', [slug, sourceId]);
+ return row!;
+}
+const stale = [{ chunk_index: 0, chunk_text: 'some chunk text' }];
+
+describe('backfill embedding hold (pglite)', () => {
+ test('backfilled pages carry a held embedding and nothing is granted before approval', async () => {
+ const result = await runTranscriptArchive(engine, {
+ mode: 'backfill', from: '2026-09-01', to: '2026-09-02',
+ readers: [staticReader('codex', [conversation('codex', 'rollout-old', [['user', '2026-09-01T10:00:00.000Z', 'old work'], ['user', '2026-09-02T10:00:00.000Z', 'more old work']])])],
+ now: new Date('2026-11-01T00:00:00Z'),
+ });
+ const slugs = result.days.filter(d => d.status === 'written').map(d => d.slug!);
+ expect(slugs.length).toBe(2);
+ const holds = await engine.executeRaw<{ slug: string; state: string; reason: string }>(
+ 'SELECT p.slug, b.state, b.reason FROM page_embedding_budget b JOIN pages p ON p.id = b.page_id ORDER BY p.slug');
+ expect(holds.filter(h => slugs.includes(h.slug)).map(h => [h.state, h.reason])).toEqual([['held', ARCHIVE_BACKFILL_HOLD_REASON], ['held', ARCHIVE_BACKFILL_HOLD_REASON]]);
+ expect((await readBudgetLimited(engine, { sourceId: 'conversations' })).held_reasons[ARCHIVE_BACKFILL_HOLD_REASON]).toBe(2);
+ for (const slug of slugs) {
+ expect((await planPageEmbedding(engine, await pageRow(slug), stale)).kind).toBe('held');
+ expect((await planPageEmbedding(engine, await pageRow(slug), stale, { approvedCapUsd: 0.01 })).kind).not.toBe('held');
+ expect((await planPageEmbedding(engine, await pageRow(slug), stale)).kind).toBe('unmetered');
+ }
+ }, 60_000);
+
+ test('a backfilled page published before its hold row is still held on its first embedding plan', async () => {
+ const result = await runTranscriptArchive(engine, {
+ mode: 'backfill', from: '2026-09-01', to: '2026-09-01',
+ readers: [staticReader('codex', [conversation('codex', 'rollout-race', [['user', '2026-09-01T10:00:00.000Z', 'raced work']])])],
+ now: new Date('2026-11-01T00:00:00Z'),
+ });
+ const slug = result.days.find(d => d.status === 'written')!.slug!;
+ const page = await pageRow(slug);
+ await engine.executeRaw('DELETE FROM page_embedding_budget WHERE page_id = $1', [page.id]);
+ expect(await planPageEmbedding(engine, page, stale)).toEqual({ kind: 'held', reason: ARCHIVE_BACKFILL_HOLD_REASON });
+ expect((await readBudgetLimited(engine, { sourceId: 'conversations' })).held_reasons[ARCHIVE_BACKFILL_HOLD_REASON]).toBe(1);
+ }, 60_000);
+
+ test('an ordinary small page and a forward-run closed day stay unmetered', async () => {
+ await importFromContent(engine, 'notes/plain', '---\ntype: note\ntitle: Plain\n---\nplain body', { noEmbed: true });
+ expect((await planPageEmbedding(engine, await pageRow('notes/plain', 'default'), stale)).kind).toBe('unmetered');
+ const result = await runTranscriptArchive(engine, {
+ mode: 'run', readers: [staticReader('codex', [conversation('codex', 'rollout-new', [['user', '2026-11-09T10:00:00.000Z', 'yesterday']])])],
+ now: new Date('2026-11-10T12:00:00Z'),
+ });
+ const slug = result.days.find(d => d.status === 'written')!.slug!;
+ expect((await planPageEmbedding(engine, await pageRow(slug), stale)).kind).toBe('unmetered');
+ }, 60_000);
+});
diff --git a/test/transcripts-archive-day-layout.test.ts b/test/transcripts-archive-day-layout.test.ts
new file mode 100644
index 0000000000..d379b3ccd6
--- /dev/null
+++ b/test/transcripts-archive-day-layout.test.ts
@@ -0,0 +1,87 @@
+/**
+ * GST-2768 PR 2: the day layout. One page per (harness, conversation, local day) in the pinned timezone, with its own
+ * slug and id namespace, the archive stamp beside transcript_import, UTC anchors, and the DST fall-back day kept as one
+ * day. Repo-less PGLite, keyless.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { localDay, sliceSessionByDay } from '../src/core/transcripts/archive-day.ts';
+import { ARCHIVE_TIMEZONE_KEY, runTranscriptArchive, setArchiveEnabled } from '../src/core/transcripts/archive.ts';
+import { transcriptFullId } from '../src/core/transcripts/types.ts';
+import { archivedPages, conversation, staticReader } from './helpers/transcript-archive.ts';
+
+let engine: PGLiteEngine;
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+});
+
+afterAll(async () => {
+ await engine.disconnect();
+});
+
+beforeEach(async () => {
+ await resetPgliteState(engine);
+});
+
+const LA = 'America/Los_Angeles';
+
+describe('local day slicing', () => {
+ test('localDay uses the zone, not UTC', () => {
+ expect(localDay('2026-11-03T06:00:00.000Z', LA)).toBe('2026-11-02');
+ expect(localDay('2026-11-03T06:00:00.000Z', 'UTC')).toBe('2026-11-03');
+ });
+
+ test('the DST fall-back day (2026-11-01 in Los Angeles, 25 hours long) is one day', () => {
+ const c = conversation('claude-code', 'session-dst', [
+ ['user', '2026-11-01T06:30:00.000Z', 'late on Oct 31 PDT'],
+ ['assistant', '2026-11-01T08:30:00.000Z', '01:30 PDT on Nov 1'],
+ ['user', '2026-11-01T09:30:00.000Z', '01:30 PST on Nov 1, the repeated hour'],
+ ['assistant', '2026-11-02T07:30:00.000Z', '23:30 PST on Nov 1'],
+ ['user', '2026-11-02T08:30:00.000Z', '00:30 PST on Nov 2'],
+ ]);
+ const days = sliceSessionByDay(c.session.messages, LA);
+ expect(days.map(d => [d.day, d.messages.length])).toEqual([['2026-10-31', 1], ['2026-11-01', 3], ['2026-11-02', 1]]);
+ });
+});
+
+describe('archive day pages (pglite, repo-less)', () => {
+ test('a session spanning three local days writes three day pages and three day index pages', async () => {
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, LA);
+ await setArchiveEnabled(engine, true);
+ const c = conversation('claude-code', 'session-three-days', [
+ ['user', '2026-11-03T17:00:00.000Z', 'morning on the 3rd'],
+ ['assistant', '2026-11-04T07:30:00.000Z', 'late evening on the 3rd (Los Angeles)'],
+ ['user', '2026-11-04T18:00:00.000Z', 'the 4th'],
+ ['assistant', '2026-11-05T19:00:00.000Z', 'the 5th'],
+ ]);
+ const result = await runTranscriptArchive(engine, {
+ mode: 'backfill', from: '2026-11-01', to: '2026-11-06', readers: [staticReader('claude-code', [c])], now: new Date('2026-11-10T12:00:00Z'),
+ });
+ expect(result.held).toBe(0);
+ const pages = await archivedPages(engine);
+ const dayPages = pages.filter(p => !p.slug.startsWith('conversations/days/'));
+ expect(dayPages.map(p => p.slug.split('/').slice(0, 3).join('/'))).toEqual([
+ 'conversations/claude-code/2026-11-03', 'conversations/claude-code/2026-11-04', 'conversations/claude-code/2026-11-05']);
+ const hash = transcriptFullId('session-three-days');
+ expect(dayPages.map(p => p.frontmatter.id)).toEqual([
+ `claude-code-${hash}-d20261103-p1`, `claude-code-${hash}-d20261104-p1`, `claude-code-${hash}-d20261105-p1`]);
+ const first = dayPages[0]!;
+ expect(first.frontmatter.type).toBe('conversation');
+ expect(first.frontmatter.date).toBe('2026-11-03');
+ expect(first.frontmatter.transcript_import).toMatchObject({ harness: 'claude-code', session_id: 'session-three-days', layout: 'day', day: '2026-11-03' });
+ expect(first.frontmatter.transcript_archive).toMatchObject({
+ layout: 'day', harness: 'claude-code', conversation_key_hash: hash, day: '2026-11-03', timezone: LA, messages: 2, complete: true,
+ first_ts: '2026-11-03T17:00:00.000Z', last_ts: '2026-11-04T07:30:00.000Z',
+ });
+ expect(first.frontmatter.readable_by).toBeUndefined();
+ expect(first.body).toContain('(2026-11-04 7:30 AM)');
+ expect(pages.filter(p => p.slug.startsWith('conversations/days/')).map(p => p.slug)).toEqual([
+ 'conversations/days/2026-11-03', 'conversations/days/2026-11-04', 'conversations/days/2026-11-05']);
+ const index = pages.find(p => p.slug === 'conversations/days/2026-11-04')!;
+ expect(index.body).toContain(dayPages[1]!.slug);
+ }, 60_000);
+});
diff --git a/test/transcripts-archive-idempotent.test.ts b/test/transcripts-archive-idempotent.test.ts
new file mode 100644
index 0000000000..6b96c5e36e
--- /dev/null
+++ b/test/transcripts-archive-idempotent.test.ts
@@ -0,0 +1,94 @@
+/**
+ * GST-2768 PR 2: idempotency. Two runs and an overlapping backfill give the same pages and revisions; the open day
+ * is re-rendered as it grows, and closed days stay byte-identical. Repo-less PGLite.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { ARCHIVE_TIMEZONE_KEY, runTranscriptArchive, setArchiveEnabled } from '../src/core/transcripts/archive.ts';
+import { archivedPages, conversation, staticReader, type Turn } from './helpers/transcript-archive.ts';
+
+let engine: PGLiteEngine;
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+});
+
+afterAll(async () => {
+ await engine.disconnect();
+});
+
+beforeEach(async () => {
+ await resetPgliteState(engine);
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await setArchiveEnabled(engine, true);
+});
+
+describe('archive idempotency (pglite)', () => {
+ test('run, run again, then an overlapping backfill: same pages, same revisions; the open day grows in place', async () => {
+ const turns: Turn[] = [
+ ['user', '2026-11-08T09:00:00.000Z', 'two days ago'],
+ ['user', '2026-11-09T09:00:00.000Z', 'yesterday'],
+ ['user', '2026-11-10T09:00:00.000Z', 'today, first message'],
+ ];
+ const reader = staticReader('hermes', () => [conversation('hermes', 'hermes-session-1', turns)]);
+ const now = new Date('2026-11-10T12:00:00Z');
+ const first = await runTranscriptArchive(engine, { mode: 'run', readers: [reader], now });
+ expect(first.written).toBe(3);
+ const pages1 = await archivedPages(engine);
+ const today = pages1.find(p => p.slug.startsWith('conversations/hermes/2026-11-10/'))!;
+ expect((today.frontmatter.transcript_archive as Record<string, unknown>).complete).toBe(false);
+ expect((pages1.find(p => p.slug.startsWith('conversations/hermes/2026-11-09/'))!.frontmatter.transcript_archive as Record<string, unknown>).complete).toBe(true);
+
+ const second = await runTranscriptArchive(engine, { mode: 'run', readers: [reader], now });
+ expect(second.written).toBe(0);
+ expect(await archivedPages(engine)).toEqual(pages1);
+
+ const backfill = await runTranscriptArchive(engine, { mode: 'backfill', from: '2026-11-01', to: '2026-11-10', readers: [reader], now });
+ expect(backfill.written).toBe(0);
+ expect(await archivedPages(engine)).toEqual(pages1);
+
+ turns.push(['assistant', '2026-11-10T13:00:00.000Z', 'today, second message']);
+ const third = await runTranscriptArchive(engine, { mode: 'run', readers: [reader], now: new Date('2026-11-10T14:00:00Z') });
+ expect(third.written).toBe(1);
+ const pages3 = await archivedPages(engine);
+ const today3 = pages3.find(p => p.slug === today.slug)!;
+ expect(today3.revision).toBeGreaterThan(today.revision);
+ expect(today3.body).toContain('today, second message');
+ for (const p of pages1.filter(p => !p.slug.includes('2026-11-10'))) expect(pages3.find(q => q.slug === p.slug)).toEqual(p);
+ }, 60_000);
+});
+
+describe('archive day index retry (pglite)', () => {
+ test('a day index whose write failed is written by the next run', async () => {
+ let failing = true;
+ const flaky = new Proxy(engine, { get(target, prop) {
+ const value = Reflect.get(target, prop, target);
+ if (typeof value !== 'function') return value;
+ if (prop === 'transaction') return (fn: (tx: unknown) => Promise<unknown>) => target.transaction(tx => fn(new Proxy(tx, { get(t, p) {
+ const v = Reflect.get(t, p, t);
+ if (p === 'putPage') return (slug: string, ...rest: unknown[]) => {
+ if (failing && slug.startsWith('conversations/days/')) return Promise.reject(new Error('synthetic index write failure'));
+ return (v as (...a: unknown[]) => unknown).call(t, slug, ...rest);
+ };
+ return typeof v === 'function' ? v.bind(t) : v;
+ } })));
+ if (prop === 'putPage') return (slug: string, ...rest: unknown[]) => {
+ if (failing && slug.startsWith('conversations/days/')) return Promise.reject(new Error('synthetic index write failure'));
+ return (value as (...a: unknown[]) => unknown).call(target, slug, ...rest);
+ };
+ return value.bind(target);
+ } }) as unknown as PGLiteEngine;
+ const reader = () => staticReader('claude-code', [conversation('claude-code', 'session-index', [['user', '2026-10-01T10:00:00.000Z', 'index retry note']])]);
+ const opts = { mode: 'backfill' as const, from: '2026-10-01', to: '2026-10-01', now: new Date('2026-10-05T00:00:00Z') };
+ const first = await runTranscriptArchive(flaky, { ...opts, readers: [reader()] });
+ expect(first.errors.map(e => e.reader)).toEqual(['day-index']);
+ expect((await archivedPages(engine)).some(p => p.slug.startsWith('conversations/days/'))).toBe(false);
+ failing = false;
+ const second = await runTranscriptArchive(engine, { ...opts, readers: [reader()] });
+ expect(second.errors).toEqual([]);
+ expect((await archivedPages(engine)).filter(p => p.slug.startsWith('conversations/days/')).map(p => p.slug)).toEqual(['conversations/days/2026-10-01']);
+ }, 60_000);
+});
diff --git a/test/transcripts-archive-legacy-collision.test.ts b/test/transcripts-archive-legacy-collision.test.ts
new file mode 100644
index 0000000000..1155335105
--- /dev/null
+++ b/test/transcripts-archive-legacy-collision.test.ts
@@ -0,0 +1,79 @@
+/**
+ * GST-2768 PR 2, Critical review fix 1: a brain that already holds per-session pages for a session gets day pages in
+ * their own slug and id namespace, and every legacy page keeps its slug, revision and content. A later per-session
+ * ingest of the same session is unaffected too. Repo-less PGLite.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { mkdtempSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { runTranscriptsIngest } from '../src/core/transcripts/ingest.ts';
+import { ARCHIVE_TIMEZONE_KEY, runTranscriptArchive, setArchiveEnabled } from '../src/core/transcripts/archive.ts';
+import { archivedPages, conversation, staticReader } from './helpers/transcript-archive.ts';
+
+let engine: PGLiteEngine;
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+});
+
+afterAll(async () => {
+ await engine.disconnect();
+});
+
+beforeEach(async () => {
+ await resetPgliteState(engine);
+});
+
+const SID = 'legacy-session-0001';
+const TURNS: Array<['user' | 'assistant', string, string]> = [
+ ['user', '2026-10-20T10:00:00.000Z', 'first day question'],
+ ['assistant', '2026-10-20T10:01:00.000Z', 'first day answer'],
+ ['user', '2026-10-21T10:00:00.000Z', 'second day question'],
+];
+
+function sessionFile(dir: string): string {
+ const path = join(dir, `${SID}.jsonl`);
+ const lines = [{ type: 'session', version: 4, id: SID, timestamp: TURNS[0]![1], cwd: '/home/alice-example/project-a' },
+ ...TURNS.map(([role, timestamp, text]) => ({ type: 'message', timestamp, message: { role, content: [{ type: 'text', text }] } }))];
+ writeFileSync(path, lines.map(l => JSON.stringify(l)).join('\n') + '\n');
+ return path;
+}
+
+describe('day pages beside legacy per-session pages (pglite)', () => {
+ test('archive writes day pages and leaves every legacy page untouched, in either order of runs', async () => {
+ const dir = mkdtempSync(join(tmpdir(), 'archive-legacy-'));
+ try {
+ await engine.executeRaw("INSERT INTO sources (id, name) VALUES ('conversations', 'conversations') ON CONFLICT (id) DO NOTHING");
+ const path = sessionFile(dir);
+ const ingest = await runTranscriptsIngest(engine, { paths: [path], sourceId: 'conversations' });
+ expect(ingest.sessionsImported).toBe(1);
+ const legacyBefore = (await archivedPages(engine)).filter(p => p.slug.startsWith('conversations/sessions/'));
+ expect(legacyBefore.length).toBe(1);
+
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await setArchiveEnabled(engine, true);
+ const result = await runTranscriptArchive(engine, {
+ mode: 'backfill', from: '2026-10-19', to: '2026-10-22', readers: [staticReader('openclaw', [conversation('openclaw', SID, TURNS)])],
+ now: new Date('2026-10-30T00:00:00Z'),
+ });
+ expect(result.written).toBe(2);
+ const after = await archivedPages(engine);
+ const legacyAfter = after.filter(p => p.slug.startsWith('conversations/sessions/'));
+ expect(legacyAfter).toEqual(legacyBefore);
+ const dayIds = after.filter(p => p.slug.startsWith('conversations/openclaw/')).map(p => String(p.frontmatter.id));
+ expect(dayIds.length).toBe(2);
+ for (const id of dayIds) expect(legacyBefore.map(p => p.frontmatter.id)).not.toContain(id);
+
+ const again = await runTranscriptsIngest(engine, { paths: [path], sourceId: 'conversations' });
+ expect(again.pages.imported).toBe(0);
+ expect(await archivedPages(engine)).toEqual(after);
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ }, 60_000);
+});
diff --git a/test/transcripts-archive-managed.test.ts b/test/transcripts-archive-managed.test.ts
new file mode 100644
index 0000000000..7a6c2d40ad
--- /dev/null
+++ b/test/transcripts-archive-managed.test.ts
@@ -0,0 +1,96 @@
+/**
+ * GST-2768 PR 2: the conversation archive on a writer-claimed (managed) brain. The direct importer is refused there,
+ * so every day page and day index is submitted as a put_page write request through the CLI's trusted-local context
+ * (makeContext, never a hand-built remote: false), built only after the archive source exists: on a fresh brain the
+ * source does not exist until the first run creates it, and the CLI used to build the context first and fail. A rerun on the same input submits nothing, and on a shared brain
+ * the committed pages still name their readers. Same harness as test/transcripts-managed-ingest.test.ts.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { makeContext } from '../src/cli/main.ts';
+import { disposePersistenceConsumer, stopPersistenceConsumer } from '../src/core/persistence/service.ts';
+import { addMember } from '../src/core/access/members.ts';
+import { setOwner } from '../src/core/access/owner.ts';
+import { ARCHIVE_TIMEZONE_KEY, runTranscriptArchive, setArchiveEnabled } from '../src/core/transcripts/archive.ts';
+import { registerLocalWriter } from '../src/core/persistence/identity.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { withEnv } from './helpers/with-env.ts';
+import { archivedPages, conversation, staticReader } from './helpers/transcript-archive.ts';
+
+let engine: PGLiteEngine;
+let dir: string;
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+}, 120_000);
+
+afterAll(async () => {
+ await disposePersistenceConsumer(engine);
+ await engine.disconnect();
+});
+
+beforeEach(async () => {
+ await stopPersistenceConsumer(engine);
+ await resetPgliteState(engine);
+ dir = mkdtempSync(join(tmpdir(), 'gb-managed-archive-'));
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await setArchiveEnabled(engine, true);
+ await engine.executeRaw('UPDATE persistence_brain SET enabled = true WHERE singleton = 1');
+ await withEnv({ GBRAIN_HOME: dir }, () => registerLocalWriter(engine, 'cli'));
+ return () => rmSync(dir, { recursive: true, force: true });
+});
+
+const readers = () => [staticReader('codex', [conversation('codex', 'rollout-managed', [
+ ['user', '2026-10-01T10:00:00.000Z', 'day one question'], ['assistant', '2026-10-01T10:01:00.000Z', 'day one answer'],
+ ['user', '2026-10-02T09:00:00.000Z', 'day two question'],
+])])];
+
+async function archive() {
+ return withEnv({ GBRAIN_HOME: dir, DATABASE_URL: undefined, GBRAIN_DATABASE_URL: undefined }, async () => {
+ return runTranscriptArchive(engine, { mode: 'backfill', from: '2026-10-01', to: '2026-10-03', readers: readers(),
+ makeContext: source => makeContext(engine, { source, dry_run: false }), now: new Date('2026-10-10T00:00:00Z') });
+ });
+}
+
+const pageRequests = () => engine.executeRaw<{ slug: string; state: string; principal_kind: string }>(
+ "SELECT slug, state, principal_kind FROM persistence_requests WHERE operation = 'put_page' ORDER BY slug");
+
+describe('conversation archive on a writer-claimed brain', () => {
+ test('day pages commit as local put_page write requests and a rerun submits nothing', async () => {
+ const first = await archive();
+ expect(first.errors).toEqual([]);
+ expect(first.write_mode).toBe('coordinator');
+ expect(first.held).toBe(0);
+ const pages = await archivedPages(engine);
+ expect(pages.map(p => p.slug.split('/').slice(0, 3).join('/'))).toEqual([
+ 'conversations/codex/2026-10-01', 'conversations/codex/2026-10-02', 'conversations/days/2026-10-01', 'conversations/days/2026-10-02']);
+ const requests = await pageRequests();
+ expect(requests).toEqual(pages.map(p => ({ slug: p.slug, state: 'committed', principal_kind: 'local_cli' })));
+ const revisions = pages.map(p => p.revision);
+
+ const again = await archive();
+ expect(again.errors).toEqual([]);
+ expect(again.written).toBe(0);
+ expect(await pageRequests()).toHaveLength(requests.length);
+ expect((await archivedPages(engine)).map(p => p.revision)).toEqual(revisions);
+ }, 120_000);
+
+ test('on a shared brain the committed pages name the owner in readable_by', async () => {
+ await engine.setConfig('brain.audience', 'shared');
+ await addMember(engine, 'alice-example');
+ await addMember(engine, 'bob-example');
+ await setOwner(engine, 'alice-example');
+ const result = await archive();
+ expect(result.errors).toEqual([]);
+ expect(result.write_mode).toBe('coordinator');
+ const pages = await archivedPages(engine);
+ expect(pages.length).toBe(4);
+ for (const p of pages) expect(p.frontmatter.readable_by).toEqual(['alice-example']);
+ expect((await pageRequests()).every(r => r.state === 'committed')).toBe(true);
+ }, 120_000);
+});
diff --git a/test/transcripts-archive-never-shrinks.test.ts b/test/transcripts-archive-never-shrinks.test.ts
new file mode 100644
index 0000000000..302113c14a
--- /dev/null
+++ b/test/transcripts-archive-never-shrinks.test.ts
@@ -0,0 +1,140 @@
+/**
+ * GST-2768 PR 2: the archive never shrinks or replaces a day page because the source now shows less. Covered here:
+ * an OpenClaw reset in place (the same conversation now reads only the messages after the reset), a deleted session
+ * (gone from the source), a Gateway that is unreachable for a run, and a Gateway that returns partial history (only
+ * the newest messages). In every case each day page already written keeps its body and revision. Repo-less PGLite.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { ARCHIVE_TIMEZONE_KEY, runTranscriptArchive, setArchiveEnabled, type ArchiveReader } from '../src/core/transcripts/archive.ts';
+import { openclawGatewayReader, parseGatewayCliOutput } from '../src/core/transcripts/openclaw-gateway.ts';
+import { archivedPages, conversation, staticReader, type Turn } from './helpers/transcript-archive.ts';
+
+let engine: PGLiteEngine;
+beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); });
+afterAll(async () => { await engine.disconnect(); });
+beforeEach(async () => {
+ await resetPgliteState(engine);
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await setArchiveEnabled(engine, true);
+});
+
+const KEY = 'agent:main:scratch-reset';
+const FULL: Turn[] = [
+ ['user', '2026-11-09T08:00:00.000Z', 'monday one'], ['assistant', '2026-11-09T08:01:00.000Z', 'monday two'],
+ ['user', '2026-11-10T08:00:00.000Z', 'tuesday one'], ['assistant', '2026-11-10T08:01:00.000Z', 'tuesday two'],
+ ['user', '2026-11-10T09:00:00.000Z', 'tuesday three'],
+];
+const NOW = new Date('2026-11-10T23:00:00Z');
+const run = (readers: ArchiveReader[]) => runTranscriptArchive(engine, { mode: 'run', readers, now: NOW });
+const snapshot = async () => (await archivedPages(engine)).map(p => ({ slug: p.slug, revision: p.revision, body: p.body }));
+
+async function seed() {
+ const first = await run([staticReader('openclaw', [conversation('openclaw', KEY, FULL)])]);
+ expect(first.errors).toEqual([]);
+ const pages = await snapshot();
+ expect(pages.filter(p => p.slug.startsWith('conversations/openclaw/')).length).toBe(2);
+ return pages;
+}
+
+describe('the archive never shrinks a day page (pglite)', () => {
+ test('reset in place: the conversation now shows only the last message, and both day pages are kept unchanged', async () => {
+ const before = await seed();
+ const result = await run([staticReader('openclaw', [conversation('openclaw', KEY, [FULL[4]!])])]);
+ expect(result.errors).toEqual([]);
+ expect(result.days.map(d => [d.day, d.status])).toEqual([['2026-11-10', 'kept']]);
+ expect(await snapshot()).toEqual(before);
+ }, 60_000);
+
+ test('deleted session: the conversation is gone from the source, and its pages stay', async () => {
+ const before = await seed();
+ const result = await run([staticReader('openclaw', [])]);
+ expect(result.errors).toEqual([]);
+ expect(result.written).toBe(0);
+ expect(await snapshot()).toEqual(before);
+ }, 60_000);
+
+ test('Gateway unreachable for a run: the reader fails, nothing is written or removed, and the next run is unchanged', async () => {
+ const before = await seed();
+ const down = openclawGatewayReader({ transport: { async call() {
+ return parseGatewayCliOutput(JSON.stringify({ ok: false, error: { type: 'gateway_transport_error', message: 'connect ECONNREFUSED 127.0.0.1:18789', details: { kind: 'closed' } } }), 1);
+ } } }) as unknown as ArchiveReader;
+ const result = await run([down]);
+ expect(result.errors.map(e => e.code)).toEqual(['archive_gateway_unreachable']);
+ expect(result.written).toBe(0);
+ expect(await snapshot()).toEqual(before);
+ const again = await run([staticReader('openclaw', [conversation('openclaw', KEY, FULL)])]);
+ expect(again.written).toBe(0);
+ expect(await snapshot()).toEqual(before);
+ }, 60_000);
+
+ test('reset in place, then as many new messages the same day: the day page keeps the earlier messages', async () => {
+ const before = await seed();
+ const after: Turn[] = [
+ ['user', '2026-11-10T20:00:00.000Z', 'new a'], ['assistant', '2026-11-10T20:01:00.000Z', 'new b'],
+ ['user', '2026-11-10T20:02:00.000Z', 'new c'], ['assistant', '2026-11-10T20:03:00.000Z', 'new d'],
+ ];
+ const result = await run([staticReader('openclaw', [conversation('openclaw', KEY, after)])]);
+ expect(result.days.map(d => [d.day, d.status])).toEqual([['2026-11-10', 'kept']]);
+ expect(await snapshot()).toEqual(before);
+ }, 60_000);
+
+ test('the same reset with the reset file still on disk: the run merges both and the day grows', async () => {
+ await seed();
+ const after: Turn[] = [['user', '2026-11-10T20:00:00.000Z', 'new a'], ['assistant', '2026-11-10T20:01:00.000Z', 'new b']];
+ const gateway = { ...staticReader('openclaw', [{ ...conversation('openclaw', KEY, after), via: 'gateway' as const }]), name: 'openclaw-gateway' };
+ const file = staticReader('openclaw', [{ ...conversation('openclaw', KEY, FULL), via: 'file' as const }]);
+ const result = await run([gateway, file]);
+ expect(result.errors).toEqual([]);
+ expect(result.days.filter(d => d.status === 'written').map(d => d.day)).toEqual(['2026-11-10']);
+ const tuesday = (await snapshot()).find(p => p.slug.startsWith('conversations/openclaw/2026-11-10'))!;
+ for (const text of ['tuesday one', 'tuesday three', 'new a', 'new b']) expect(tuesday.body).toContain(text);
+ expect(tuesday.body.split('tuesday one').length - 1).toBe(1);
+ }, 60_000);
+
+ test('partial history: the Gateway returns only the newest messages, and each shorter day is kept', async () => {
+ const before = await seed();
+ const result = await run([staticReader('openclaw', [conversation('openclaw', KEY, FULL.slice(3))])]);
+ expect(result.errors).toEqual([]);
+ expect(result.days.map(d => [d.day, d.status])).toEqual([['2026-11-10', 'kept']]);
+ expect(await snapshot()).toEqual(before);
+ }, 60_000);
+});
+
+describe('never shrinks across holds and overlapping runs (pglite)', () => {
+ test('a day written, then held, then read shorter after a reset keeps its page', async () => {
+ const before = await seed();
+ const held = await runTranscriptArchive(engine, { mode: 'run', readers: [staticReader('openclaw', [conversation('openclaw', KEY, FULL)])], now: NOW, redact: () => { throw new Error('bad pattern'); } });
+ expect(held.days.map(d => d.status)).toEqual(['held', 'held']);
+ const result = await run([staticReader('openclaw', [conversation('openclaw', KEY, [FULL[4]!])])]);
+ expect(result.days.map(d => [d.day, d.status])).toEqual([['2026-11-10', 'kept']]);
+ expect(await snapshot()).toEqual(before);
+ }, 60_000);
+
+ test('two overlapping runs never publish older history over newer', async () => {
+ const OLD = FULL.slice(2, 4);
+ const NEW = FULL.slice(2);
+ let release!: () => void;
+ const gate = new Promise<void>(r => { release = r; });
+ let started!: () => void;
+ const loading = new Promise<void>(r => { started = r; });
+ const slow: ArchiveReader = {
+ harness: 'openclaw', name: 'openclaw',
+ async *scan() {
+ const conv = conversation('openclaw', KEY, OLD);
+ yield { harness: 'openclaw', conversationKey: KEY, sessionId: KEY, days: ['2026-11-10'], load: async () => { started(); await gate; return conv; } };
+ },
+ };
+ const first = run([slow]).then(r => r, (e: unknown) => e as Error);
+ await loading;
+ const second = await run([staticReader('openclaw', [conversation('openclaw', KEY, NEW)])]).then(r => r, (e: unknown) => e as Error);
+ release();
+ const older = await first;
+ const tuesday = (await snapshot()).find(p => p.slug.startsWith('conversations/openclaw/2026-11-10'));
+ const wrote = (r: unknown) => !(r instanceof Error) && (r as { written: number }).written > 0;
+ if (wrote(second)) expect(tuesday?.body ?? '').toContain('tuesday three');
+ else expect((second as { code?: string }).code).toBe('archive_run_busy');
+ expect(wrote(older) || wrote(second)).toBe(true);
+ }, 60_000);
+});
diff --git a/test/transcripts-archive-openclaw-identity.test.ts b/test/transcripts-archive-openclaw-identity.test.ts
new file mode 100644
index 0000000000..d0a1c9d551
--- /dev/null
+++ b/test/transcripts-archive-openclaw-identity.test.ts
@@ -0,0 +1,61 @@
+/**
+ * GST-2768 PR 2: one OpenClaw conversation has one identity, its sessionId, whichever path read it. A backfill from
+ * the session files while the Gateway is down, then a run with the Gateway up over the same days, leaves one page per
+ * conversation-day: the Gateway's session key only labels and scopes the page. Synthetic values. Repo-less PGLite.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { mkdtempSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { ARCHIVE_TIMEZONE_KEY, runTranscriptArchive, setArchiveEnabled, type ArchiveReader } from '../src/core/transcripts/archive.ts';
+import { fileReader } from '../src/core/transcripts/archive-readers.ts';
+import { openclawGatewayReader, parseGatewayCliOutput } from '../src/core/transcripts/openclaw-gateway.ts';
+import { archivedPages } from './helpers/transcript-archive.ts';
+
+const root = mkdtempSync(join(tmpdir(), 'openclaw-identity-'));
+let engine: PGLiteEngine;
+beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); });
+afterAll(async () => { await engine.disconnect(); rmSync(root, { recursive: true, force: true }); });
+beforeEach(async () => {
+ await resetPgliteState(engine);
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await setArchiveEnabled(engine, true);
+});
+
+const SID = 'example-session-id-0001';
+const KEY = 'agent:main:telegram:group:example-group-7';
+const TURNS: Array<[string, string, string]> = [
+ ['user', '2026-11-09T08:00:00.000Z', 'monday question'], ['assistant', '2026-11-09T08:01:00.000Z', 'monday answer'],
+ ['user', '2026-11-10T08:00:00.000Z', 'tuesday question'], ['assistant', '2026-11-10T08:01:00.000Z', 'tuesday answer'],
+];
+const lines = [{ type: 'session', version: 3, id: SID, timestamp: TURNS[0]![1], cwd: '/home/alice-example/agent-workspace' },
+ ...TURNS.map(([role, timestamp, text], i) => ({ type: 'message', id: 'm' + i, parentId: i ? 'm' + (i - 1) : null, timestamp, message: { role, timestamp, content: [{ type: 'text', text }] } }))];
+writeFileSync(join(root, SID + '.jsonl'), lines.map(l => JSON.stringify(l)).join('\n') + '\n');
+
+const down = openclawGatewayReader({ transport: { async call() {
+ return parseGatewayCliOutput(JSON.stringify({ ok: false, error: { type: 'gateway_transport_error', message: 'connect ECONNREFUSED 127.0.0.1:18789', details: { kind: 'closed' } } }), 1);
+} } }) as unknown as ArchiveReader;
+const up = openclawGatewayReader({ transport: { async call(method: string) {
+ if (method === 'sessions.list') return { sessions: [{ key: KEY, sessionId: SID, chatType: 'group', updatedAt: Date.parse('2026-11-10T08:01:00.000Z') }], hasMore: false };
+ return { sessionKey: KEY, sessionId: SID, messages: TURNS.map(([role, ts, text], i) => ({ role, content: [{ type: 'text', text }], timestamp: Date.parse(ts), __openclaw: { id: 'm' + i, seq: i } })), hasMore: false };
+} } }) as unknown as ArchiveReader;
+const files = () => fileReader({ format: 'openclaw', root, extension: '.jsonl' });
+const NOW = new Date('2026-11-10T23:00:00Z');
+const dayPages = async () => (await archivedPages(engine)).filter(p => p.slug.startsWith('conversations/openclaw/')).map(p => p.slug);
+
+describe('one identity for an OpenClaw conversation (pglite)', () => {
+ test('files-only backfill with the Gateway down, then a Gateway run over the same days: one page per day, no duplicates', async () => {
+ const backfill = await runTranscriptArchive(engine, { mode: 'backfill', from: '2026-11-01', readers: [files(), down], now: NOW });
+ expect(backfill.days.filter(d => d.status === 'written').map(d => d.day).sort()).toEqual(['2026-11-09', '2026-11-10']);
+ const run = await runTranscriptArchive(engine, { mode: 'run', readers: [files(), up], now: NOW });
+ expect(run.errors).toEqual([]);
+ const slugs = await dayPages();
+ expect(slugs.length).toBe(2);
+ expect(slugs.map(s => s.split('/')[2]).sort()).toEqual(['2026-11-09', '2026-11-10']);
+ const again = await runTranscriptArchive(engine, { mode: 'run', readers: [files(), up], now: NOW });
+ expect(again.written).toBe(0);
+ expect(await dayPages()).toEqual(slugs);
+ }, 60_000);
+});
diff --git a/test/transcripts-archive-owner-unset.test.ts b/test/transcripts-archive-owner-unset.test.ts
new file mode 100644
index 0000000000..f7869f792a
--- /dev/null
+++ b/test/transcripts-archive-owner-unset.test.ts
@@ -0,0 +1,65 @@
+/**
+ * GST-2768 PR 2, Critical review fix 2: on a shared brain with no owner member the archive holds every day with
+ * archive_owner_unset and a fix command, and writes nothing; with an owner every page carries readable_by. Repo-less
+ * PGLite.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { addMember } from '../src/core/access/members.ts';
+import { setOwner } from '../src/core/access/owner.ts';
+import { ARCHIVE_TIMEZONE_KEY, readArchiveLedger, runTranscriptArchive, setArchiveEnabled } from '../src/core/transcripts/archive.ts';
+import { archivedPages, conversation, staticReader } from './helpers/transcript-archive.ts';
+
+let engine: PGLiteEngine;
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+});
+
+afterAll(async () => {
+ await engine.disconnect();
+});
+
+beforeEach(async () => {
+ await resetPgliteState(engine);
+ await engine.setConfig('brain.audience', 'shared');
+ await addMember(engine, 'alice-example');
+ await addMember(engine, 'bob-example');
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await setArchiveEnabled(engine, true);
+});
+
+const reader = () => staticReader('codex', [
+ conversation('codex', 'rollout-a', [['user', '2026-10-01T10:00:00.000Z', 'day one'], ['user', '2026-10-02T10:00:00.000Z', 'day two']]),
+ conversation('codex', 'rollout-b', [['user', '2026-10-02T11:00:00.000Z', 'other session']]),
+]);
+const run = () => runTranscriptArchive(engine, { mode: 'backfill', from: '2026-10-01', to: '2026-10-03', readers: [reader()], now: new Date('2026-10-10T00:00:00Z') });
+
+describe('shared brain without an owner member (pglite)', () => {
+ test('every day is held with archive_owner_unset, nothing is written and the fix names the owner command', async () => {
+ const result = await run();
+ expect(result.written).toBe(0);
+ expect(result.held).toBe(3);
+ expect(result.days.every(d => d.status === 'held' && d.code === 'archive_owner_unset')).toBe(true);
+ expect(result.fix?.argv?.slice(0, 4)).toEqual(['gbrain', 'access', 'owner', 'set']);
+ expect(await archivedPages(engine)).toEqual([]);
+ expect((await engine.executeRaw('SELECT 1 FROM pages')).length).toBe(0);
+ const ledger = await readArchiveLedger(engine, 'conversations');
+ expect(ledger.length).toBe(3);
+ expect(ledger.every(e => e.held?.code === 'archive_owner_unset' && !e.hash)).toBe(true);
+ }, 60_000);
+
+ test('once the owner is set, the next run writes every held day, each with readable_by naming the owner', async () => {
+ await run();
+ await setOwner(engine, 'alice-example');
+ const result = await run();
+ expect(result.held).toBe(0);
+ const pages = await archivedPages(engine);
+ expect(pages.length).toBe(5);
+ for (const p of pages) expect(p.frontmatter.readable_by).toEqual(['alice-example']);
+ expect((await readArchiveLedger(engine, 'conversations')).some(e => e.held)).toBe(false);
+ }, 60_000);
+});
diff --git a/test/transcripts-archive-redaction.test.ts b/test/transcripts-archive-redaction.test.ts
new file mode 100644
index 0000000000..e0b4e3d3e8
--- /dev/null
+++ b/test/transcripts-archive-redaction.test.ts
@@ -0,0 +1,94 @@
+/**
+ * GST-2768 PR 2, review change 7: redaction runs over the whole session on every run, so a credential labelled on one
+ * day is also scrubbed where it is echoed bare on another day, on the page and in its raw data. A redaction failure
+ * holds the session's days (never drops them) and the next run writes them. Repo-less PGLite. The credential is
+ * synthetic.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { ARCHIVE_TIMEZONE_KEY, readArchiveLedger, runTranscriptArchive, setArchiveEnabled } from '../src/core/transcripts/archive.ts';
+import { archivedPages, conversation, staticReader } from './helpers/transcript-archive.ts';
+import { withEnv } from './helpers/with-env.ts';
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+
+let engine: PGLiteEngine;
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+});
+
+afterAll(async () => {
+ await engine.disconnect();
+});
+
+beforeEach(async () => {
+ await resetPgliteState(engine);
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await setArchiveEnabled(engine, true);
+});
+
+const SECRET = ['Zq7mPq', 'synthetic', '0001'].join('-');
+const conv = () => conversation('claude-code', 'session-secret', [
+ ['user', '2026-10-01T10:00:00.000Z', `the staging password: ${SECRET}`],
+ ['assistant', '2026-10-01T10:01:00.000Z', 'noted'],
+ ['user', '2026-10-02T10:00:00.000Z', `log in again with ${SECRET} please`],
+], undefined, { session_id: 'session-secret', cwd: `/home/alice-example/${SECRET}` });
+const opts = { mode: 'backfill' as const, from: '2026-10-01', to: '2026-10-03', now: new Date('2026-10-10T00:00:00Z') };
+
+describe('archive redaction (pglite)', () => {
+ test('a secret labelled on day 1 and echoed on day 2 is redacted on both pages and in raw data', async () => {
+ await runTranscriptArchive(engine, { ...opts, readers: [staticReader('claude-code', [conv()])] });
+ const pages = (await archivedPages(engine)).filter(p => p.slug.startsWith('conversations/claude-code/'));
+ expect(pages.length).toBe(2);
+ for (const p of pages) {
+ expect(p.body).not.toContain(SECRET);
+ expect(JSON.stringify(p.frontmatter)).not.toContain(SECRET);
+ }
+ const day2 = pages.find(p => p.slug.includes('/2026-10-02/'))!;
+ expect(day2.body).toContain('<REDACTED:');
+ const raw = await engine.getRawData(day2.slug, 'transcript:claude-code', { sourceId: 'conversations' });
+ expect(raw.length).toBe(1);
+ expect(JSON.stringify(raw[0]!.data)).not.toContain(SECRET);
+ }, 60_000);
+
+ test('a redaction throw holds every day of the session, writes nothing for it, and the next run writes them', async () => {
+ const failed = await runTranscriptArchive(engine, { ...opts, readers: [staticReader('claude-code', [conv()])], redact: () => { throw new Error('scanner failed'); } });
+ expect(failed.written).toBe(0);
+ expect(failed.days.map(d => [d.day, d.status, d.code])).toEqual([
+ ['2026-10-02', 'held', 'archive_redaction_failed'], ['2026-10-01', 'held', 'archive_redaction_failed']]);
+ expect(await archivedPages(engine)).toEqual([]);
+ expect((await readArchiveLedger(engine, 'conversations')).map(e => e.held?.code)).toEqual(['archive_redaction_failed', 'archive_redaction_failed']);
+ const retried = await runTranscriptArchive(engine, { ...opts, readers: [staticReader('claude-code', [conv()])] });
+ expect(retried.written).toBe(2);
+ expect((await readArchiveLedger(engine, 'conversations')).some(e => e.held)).toBe(false);
+ }, 60_000);
+});
+
+describe('backfill and redaction rule changes (pglite)', () => {
+ test('a finished day whose value only becomes redactable under newer redaction rules is re-redacted by backfill', async () => {
+ const home = mkdtempSync(join(tmpdir(), 'archive-rules-'));
+ try {
+ await withEnv({ GBRAIN_HOME: home }, async () => {
+ const MARK = 'BLUEFALCON-synthetic-77';
+ const c = () => conversation('claude-code', 'session-rules', [
+ ['user', '2026-10-01T10:00:00.000Z', 'the project codename is ' + MARK], ['assistant', '2026-10-01T10:01:00.000Z', 'noted'],
+ ]);
+ const first = await runTranscriptArchive(engine, { ...opts, readers: [staticReader('claude-code', [c()])] });
+ expect(first.written).toBe(1);
+ expect((await archivedPages(engine)).find(p => p.slug.startsWith('conversations/claude-code/'))!.body).toContain(MARK);
+ mkdirSync(join(home, '.gbrain'), { recursive: true });
+ writeFileSync(join(home, '.gbrain', 'harvest-private-patterns.txt'), 'BLUEFALCON-[a-z]+-[0-9]+\n');
+ await runTranscriptArchive(engine, { ...opts, readers: [staticReader('claude-code', [c()])] });
+ const page = (await archivedPages(engine)).find(p => p.slug.startsWith('conversations/claude-code/'))!;
+ expect(page.body).not.toContain(MARK);
+ });
+ } finally {
+ rmSync(home, { recursive: true, force: true });
+ }
+ }, 60_000);
+});
diff --git a/test/transcripts-archive-repo.test.ts b/test/transcripts-archive-repo.test.ts
new file mode 100644
index 0000000000..58abaf1c05
--- /dev/null
+++ b/test/transcripts-archive-repo.test.ts
@@ -0,0 +1,68 @@
+/**
+ * GST-2768 PR 2: on a personal brain whose archive source is a git checkout, the archive writes the redacted day
+ * pages as files, commits only those paths through `gbrain git commit-push` (a real child CLI, real bare remote),
+ * then syncs the checkout so the database converges on git. A second run on the same input changes nothing.
+ */
+import { afterAll, beforeAll, expect, test } from 'bun:test';
+import { execFileSync } from 'node:child_process';
+import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { ARCHIVE_TIMEZONE_KEY, runTranscriptArchive, setArchiveEnabled } from '../src/core/transcripts/archive.ts';
+import { withEnv } from './helpers/with-env.ts';
+import { archivedPages, conversation, staticReader } from './helpers/transcript-archive.ts';
+
+const dir = realpathSync(mkdtempSync(join(tmpdir(), 'transcripts-archive-repo-')));
+const identity = { GIT_AUTHOR_NAME: 'Example', GIT_AUTHOR_EMAIL: '[email protected]', GIT_COMMITTER_NAME: 'Example', GIT_COMMITTER_EMAIL: '[email protected]' };
+const git = (root: string, ...args: string[]) => execFileSync('git', ['-C', root, '-c', 'user.name=Example', '-c', '[email protected]',
+ '-c', 'commit.gpgsign=false', ...args], { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] }).trim();
+const cli = join(dir, 'gbrain');
+const remote = join(dir, 'brain.git');
+const root = join(dir, 'brain');
+let engine: PGLiteEngine;
+
+beforeAll(async () => {
+ writeFileSync(cli, `#!/bin/sh\nexec '${process.execPath}' '${join(import.meta.dir, '..', 'src', 'cli.ts')}' "$@"\n`);
+ chmodSync(cli, 0o755);
+ execFileSync('git', ['init', '-q', '--bare', '-b', 'main', remote]);
+ execFileSync('git', ['clone', '-q', remote, root], { stdio: 'ignore' });
+ git(root, 'checkout', '-q', '-b', 'main');
+ mkdirSync(join(root, 'notes'));
+ writeFileSync(join(root, 'notes', 'a.md'), '---\ntype: note\ntitle: A\n---\nstart\n');
+ git(root, 'add', '.'); git(root, 'commit', '-qm', 'init'); git(root, 'push', '-q', '-u', 'origin', 'main');
+ engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema();
+ await engine.executeRaw('INSERT INTO sources(id,name,local_path,config) VALUES($1,$1,$2,$3::jsonb)', ['conversations', root, JSON.stringify({ remote_url: remote })]);
+}, 120_000);
+afterAll(async () => { await engine.disconnect(); rmSync(dir, { recursive: true, force: true }); });
+
+test('archive commits only its day files, pushes them, and the synced database matches git', async () => {
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await setArchiveEnabled(engine, true);
+ writeFileSync(join(root, 'notes', 'draft.md'), 'someone else is still editing this\n');
+ const readers = [staticReader('codex', [conversation('codex', 'codex-repo-1', [
+ ['user', '2026-11-08T08:00:00.000Z', 'plan the garden'], ['assistant', '2026-11-08T08:01:00.000Z', 'tomatoes first'],
+ ['user', '2026-11-09T09:00:00.000Z', 'and the beans?'],
+ ])])];
+ const env = { ...identity, GBRAIN_JOB_CHILD_CLI: cli, GBRAIN_HOME: join(dir, 'home') };
+ const result = await withEnv(env, () => runTranscriptArchive(engine, { mode: 'backfill', from: '2026-11-08', to: '2026-11-09', readers, now: new Date('2026-11-12T00:00:00Z') }));
+ expect(result.errors).toEqual([]);
+ expect(result.write_mode).toBe('repo');
+ expect(result.written).toBeGreaterThanOrEqual(2);
+ const committed = git(root, 'show', '--name-only', '--format=', 'HEAD').split('\n').filter(Boolean).sort();
+ expect(committed.every(p => p.startsWith('conversations/'))).toBe(true);
+ expect(committed.filter(p => p.startsWith('conversations/codex/'))).toHaveLength(2);
+ expect(git(root, 'status', '--porcelain', '--', 'notes/draft.md')).toBe('?? notes/draft.md');
+ expect(git(remote, 'rev-parse', 'refs/heads/main')).toBe(git(root, 'rev-parse', 'HEAD'));
+ const pages = await archivedPages(engine);
+ for (const p of committed) {
+ expect(existsSync(join(root, p))).toBe(true);
+ expect(pages.map(x => x.slug)).toContain(p.replace(/\.md$/, ''));
+ }
+ expect(readFileSync(join(root, committed.find(p => p.includes('/codex/2026-11-08/'))!), 'utf8')).toContain('tomatoes first');
+ const head = git(root, 'rev-parse', 'HEAD');
+ const again = await withEnv(env, () => runTranscriptArchive(engine, { mode: 'backfill', from: '2026-11-08', to: '2026-11-09', readers, now: new Date('2026-11-12T00:00:00Z') }));
+ expect(again.errors).toEqual([]);
+ expect(again.written).toBe(0);
+ expect(git(root, 'rev-parse', 'HEAD')).toBe(head);
+}, 180_000);
diff --git a/test/transcripts-archive-repoless.test.ts b/test/transcripts-archive-repoless.test.ts
new file mode 100644
index 0000000000..d1eec97370
--- /dev/null
+++ b/test/transcripts-archive-repoless.test.ts
@@ -0,0 +1,54 @@
+/**
+ * GST-2768 PR 2: a brain with no repository archives into the database (repo-less brains are first class), and the
+ * archive source is created database-only when missing. PGLite, no git anywhere.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { ARCHIVE_TIMEZONE_KEY, runTranscriptArchive, setArchiveEnabled } from '../src/core/transcripts/archive.ts';
+import { archivedPages, conversation, staticReader } from './helpers/transcript-archive.ts';
+
+let engine: PGLiteEngine;
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+});
+
+afterAll(async () => {
+ await engine.disconnect();
+});
+
+beforeEach(async () => {
+ await resetPgliteState(engine);
+});
+
+describe('repo-less archive (pglite)', () => {
+ test('runs, creates the database-only source, and the database is the record', async () => {
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await setArchiveEnabled(engine, true);
+ const result = await runTranscriptArchive(engine, {
+ mode: 'run', readers: [staticReader('grok', [conversation('grok', 'grok-session-1', [['user', '2026-11-10T08:00:00.000Z', 'hello from grok']])])],
+ now: new Date('2026-11-10T09:00:00Z'),
+ });
+ expect(result.write_mode).toBe('database');
+ expect(result.written).toBe(1);
+ const [source] = await engine.executeRaw<{ local_path: string | null }>("SELECT local_path FROM sources WHERE id = 'conversations'");
+ expect(source).toEqual({ local_path: null });
+ expect((await archivedPages(engine)).map(p => p.slug.split('/').slice(0, 3).join('/'))).toEqual(['conversations/days/2026-11-10', 'conversations/grok/2026-11-10']);
+ }, 60_000);
+
+ test('off by default: an unset switch refuses to write and says how to turn it on; a dry run previews', async () => {
+ const readers = [staticReader('grok', [conversation('grok', 'grok-session-2', [['user', '2026-11-10T08:00:00.000Z', 'hello']])])];
+ const off = await runTranscriptArchive(engine, { mode: 'run', readers, now: new Date('2026-11-10T09:00:00Z') });
+ expect(off.enabled).toBe(false);
+ expect(off.written).toBe(0);
+ expect(off.fix?.argv).toEqual(['gbrain', 'transcripts', 'archive', 'on']);
+ expect((await engine.executeRaw('SELECT 1 FROM pages')).length).toBe(0);
+ const preview = await runTranscriptArchive(engine, { mode: 'run', readers, dryRun: true, now: new Date('2026-11-10T09:00:00Z') });
+ expect(preview.planned).toBe(1);
+ expect((await engine.executeRaw('SELECT 1 FROM pages')).length).toBe(0);
+ expect((await engine.executeRaw("SELECT 1 FROM config WHERE key LIKE 'transcripts.archive.%'")).length).toBe(0);
+ }, 60_000);
+});
diff --git a/test/transcripts-archive-scale.test.ts b/test/transcripts-archive-scale.test.ts
new file mode 100644
index 0000000000..9db764e2eb
--- /dev/null
+++ b/test/transcripts-archive-scale.test.ts
@@ -0,0 +1,90 @@
+/**
+ * GST-2768 PR 2: backfill over a large session folder stays bounded. Two thousand small synthetic OpenClaw session
+ * files over 100 days, with transcripts.archive.backfill_days_per_run set to 5: each run reads every file once to learn
+ * its days (holding no messages), loads again only the conversations on the 5 newest days still to do, writes them,
+ * and leaves the rest as deferred; the next run starts where the last stopped and counts the finished days as
+ * unchanged. Heap growth and wall time get loose bounds. Every value is synthetic. Repo-less PGLite.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { ARCHIVE_BACKFILL_CAP_KEY, ARCHIVE_TIMEZONE_KEY, runTranscriptArchive, setArchiveEnabled, type ArchiveReader } from '../src/core/transcripts/archive.ts';
+import { fileReader } from '../src/core/transcripts/archive-readers.ts';
+
+const FILES = 2000;
+const DAYS = 100;
+const PER_DAY = FILES / DAYS;
+const CAP = 5;
+const root = mkdtempSync(join(tmpdir(), 'openclaw-scale-'));
+let engine: PGLiteEngine;
+beforeAll(async () => {
+ engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema();
+ for (let i = 0; i < FILES; i++) {
+ const day = new Date(Date.UTC(2026, 7, 1) + (i % DAYS) * 86_400_000).toISOString().slice(0, 10);
+ const id = 'scale-' + String(i).padStart(4, '0');
+ const dir = join(root, 'agent-' + (i % 10));
+ mkdirSync(dir, { recursive: true });
+ const lines = [
+ { type: 'session', version: 3, id, timestamp: day + 'T09:00:00.000Z', cwd: '/home/alice-example/agent-workspace' },
+ { type: 'message', id: id + '-m0', parentId: null, timestamp: day + 'T09:00:00.000Z', message: { role: 'user', timestamp: day + 'T09:00:00.000Z', content: [{ type: 'text', text: 'question ' + i }] } },
+ { type: 'message', id: id + '-m1', parentId: id + '-m0', timestamp: day + 'T09:01:00.000Z', message: { role: 'assistant', timestamp: day + 'T09:01:00.000Z', content: [{ type: 'text', text: 'answer ' + i }] } },
+ ];
+ writeFileSync(join(dir, id + '.jsonl'), lines.map(l => JSON.stringify(l)).join('\n') + '\n');
+ }
+});
+afterAll(async () => { await engine.disconnect(); rmSync(root, { recursive: true, force: true }); });
+beforeEach(async () => {
+ await resetPgliteState(engine);
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await engine.setConfig(ARCHIVE_BACKFILL_CAP_KEY, String(CAP));
+ await setArchiveEnabled(engine, true);
+});
+
+/** The file reader, counting how many conversations each run loads again after the scan. */
+function countingReader(): { reader: ArchiveReader; loads: () => number; scanned: () => number } {
+ const inner = fileReader({ format: 'openclaw', root, extension: '.jsonl' });
+ let loads = 0;
+ let scanned = 0;
+ return {
+ reader: { ...inner, async *scan(o) {
+ for await (const e of inner.scan!(o)) {
+ scanned++;
+ expect(Object.keys(e)).not.toContain('session');
+ yield { ...e, load: () => { loads++; return e.load(); } };
+ }
+ } },
+ loads: () => loads,
+ scanned: () => scanned,
+ };
+}
+
+const NOW = new Date('2026-11-14T12:00:00Z');
+
+describe('backfill over a large session folder', () => {
+ test('each run writes only the capped days, loads only their conversations, and the next run resumes', async () => {
+ Bun.gc(true);
+ const heap0 = process.memoryUsage().heapUsed;
+ const t0 = performance.now();
+ const seen = new Set<string>();
+ for (let run = 1; run <= 2; run++) {
+ const c = countingReader();
+ const result = await runTranscriptArchive(engine, { mode: 'backfill', from: '2026-01-01', readers: [c.reader], now: NOW });
+ expect(result.errors).toEqual([]);
+ expect(c.scanned()).toBe(FILES);
+ expect(c.loads()).toBe(CAP * PER_DAY);
+ expect(result.written).toBe(CAP * PER_DAY);
+ expect(result.unchanged).toBe((run - 1) * CAP * PER_DAY);
+ expect(result.deferred_days.length).toBe(DAYS - run * CAP);
+ const days = [...new Set(result.days.filter(d => d.status === 'written').map(d => d.day))].sort();
+ expect(days.length).toBe(CAP);
+ for (const d of days) { expect(seen.has(d)).toBe(false); seen.add(d); }
+ expect(days[0]! > result.deferred_days[result.deferred_days.length - 1]!).toBe(true);
+ }
+ expect(performance.now() - t0).toBeLessThan(60_000);
+ Bun.gc(true);
+ expect(process.memoryUsage().heapUsed - heap0).toBeLessThan(256 * 1024 * 1024);
+ }, 240_000);
+});
diff --git a/test/transcripts-archive-scope.test.ts b/test/transcripts-archive-scope.test.ts
new file mode 100644
index 0000000000..0fda29b304
--- /dev/null
+++ b/test/transcripts-archive-scope.test.ts
@@ -0,0 +1,82 @@
+/**
+ * GST-2768 PR 2, review change 3: read scope of archive pages on a shared brain, read with remote=true credentials
+ * bound to members, with access.owner_reads_restricted off. The owner reads every archive page (group pages list the
+ * owner beside the group); a group member reads only that group's pages; another member reads none. Repo-less PGLite.
+ */
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { GBrainOAuthProvider } from '../src/core/oauth-provider.ts';
+import { sqlQueryForEngine } from '../src/core/sql-query.ts';
+import { mintLegacyToken } from '../src/core/token-mint.ts';
+import { dispatchToolCall } from '../src/mcp/dispatch.ts';
+import { addGroup, addMember, bindCredential } from '../src/core/access/members.ts';
+import { ownerReadsRestrictedOn, setOwner } from '../src/core/access/owner.ts';
+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
+import { ARCHIVE_TIMEZONE_KEY, archiveGroupPrincipal, runTranscriptArchive, setArchiveEnabled } from '../src/core/transcripts/archive.ts';
+import type { AuthInfo } from '../src/core/ops/contract.ts';
+import { archivedPages, conversation, staticReader } from './helpers/transcript-archive.ts';
+
+const SRC = 'conversations';
+let engine: PGLiteEngine;
+const tokens: Record<string, string> = {};
+const provider = () => new GBrainOAuthProvider({ sql: sqlQueryForEngine(engine), transaction: fn => engine.transaction(tx => fn(sqlQueryForEngine(tx))) });
+
+async function readable(who: string, slug: string, canary: string): Promise<boolean> {
+ const auth = await provider().verifyAccessToken(tokens[who]!) as unknown as AuthInfo;
+ const out = await dispatchToolCall(engine, 'get_page', { slug }, { remote: true, transport: 'http', auth, sourceId: SRC, takesHoldersAllowList: ['world'] });
+ return out.content.map(c => c.text).join('\n').includes(canary);
+}
+
+const GROUP_KEY = 'agent:main:telegram:group:example-group-2';
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+ await engine.setConfig('brain.audience', 'shared');
+ for (const id of ['alice-example', 'bob-example', 'carol-example']) await addMember(engine, id);
+ await setOwner(engine, 'alice-example');
+ await addGroup(engine, archiveGroupPrincipal('openclaw', GROUP_KEY), ['bob-example']);
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await setArchiveEnabled(engine, true);
+ const reader = staticReader('openclaw', [
+ conversation('openclaw', 'agent:main:main', [['user', '2026-10-01T10:00:00.000Z', 'canary-dm-0001 private note']], { chatType: 'direct', verified: true }),
+ conversation('openclaw', GROUP_KEY, [['user', '2026-10-01T11:00:00.000Z', 'canary-group-0002 team note']], { chatType: 'group', verified: true }),
+ conversation('openclaw', 'agent:main:legacy-unknown', [['user', '2026-10-01T12:00:00.000Z', 'canary-unknown-0003']], { verified: false }),
+ ]);
+ const result = await runTranscriptArchive(engine, { mode: 'backfill', from: '2026-10-01', to: '2026-10-01', readers: [reader], now: new Date('2026-10-05T00:00:00Z') });
+ expect(result.written).toBe(3);
+ for (const who of ['alice-example', 'bob-example', 'carol-example']) {
+ const minted = await mintLegacyToken(engine, { name: `tok-${who}`, scopes: ['read'], takesHolders: ['world'], sourceGrant: [SRC] });
+ await bindCredential(engine, { kind: 'token', id: minted.id }, { member: who });
+ tokens[who] = minted.token;
+ }
+}, 240_000);
+afterAll(async () => { await disposePersistenceConsumer(engine); await engine.disconnect(); }, 60_000);
+
+describe('archive read scope on a shared brain (pglite, remote=true)', () => {
+ test('owner reads all, the group member reads only the group page, another member reads none', async () => {
+ expect(await ownerReadsRestrictedOn(engine)).toBe(false);
+ const pages = (await archivedPages(engine, SRC)).filter(p => p.slug.startsWith('conversations/openclaw/'));
+ const slugOf = (canary: string) => pages.find(p => p.body.includes(canary))!.slug;
+ const cases: Array<[string, string]> = [['canary-dm-0001', 'dm'], ['canary-group-0002', 'group'], ['canary-unknown-0003', 'unknown']];
+ for (const [canary] of cases) expect(pages.find(p => p.body.includes(canary))!.frontmatter.readable_by).toContain('alice-example');
+ const seen = async (who: string) => {
+ const out: string[] = [];
+ for (const [canary, label] of cases) if (await readable(who, slugOf(canary), canary)) out.push(label);
+ return out;
+ };
+ expect(await seen('alice-example')).toEqual(['dm', 'group', 'unknown']);
+ expect(await seen('bob-example')).toEqual(['group']);
+ expect(await seen('carol-example')).toEqual([]);
+ }, 120_000);
+
+ test('no archive page is open: every page names readers, and none names more than the owner and one group', async () => {
+ for (const p of await archivedPages(engine, SRC)) {
+ const readers = p.frontmatter.readable_by as string[];
+ expect(readers[0]).toBe('alice-example');
+ expect(readers.length).toBeLessThanOrEqual(2);
+ if (readers.length === 2) expect(readers[1]).toMatch(/^group:chat-[0-9a-f]{8}$/);
+ }
+ });
+});
diff --git a/test/transcripts-archive-timezone-pin.test.ts b/test/transcripts-archive-timezone-pin.test.ts
new file mode 100644
index 0000000000..6a8184e8c8
--- /dev/null
+++ b/test/transcripts-archive-timezone-pin.test.ts
@@ -0,0 +1,53 @@
+/**
+ * GST-2768 PR 2: the archive timezone is pinned per source at the first run (ledger and frontmatter). Changing
+ * transcripts.archive.timezone afterwards never re-slices existing or new days. Repo-less PGLite.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { ARCHIVE_TIMEZONE_KEY, archiveStatus, runTranscriptArchive, setArchiveEnabled } from '../src/core/transcripts/archive.ts';
+import { archivedPages, conversation, staticReader } from './helpers/transcript-archive.ts';
+
+let engine: PGLiteEngine;
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+});
+
+afterAll(async () => {
+ await engine.disconnect();
+});
+
+beforeEach(async () => {
+ await resetPgliteState(engine);
+});
+
+describe('timezone pin (pglite)', () => {
+ test('a timezone change after the first run never re-slices', async () => {
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await setArchiveEnabled(engine, true);
+ const reader = staticReader('codex', [conversation('codex', 'rollout-tz', [
+ ['user', '2026-10-05T20:00:00.000Z', 'evening UTC, next morning in Tokyo'],
+ ['user', '2026-10-06T20:00:00.000Z', 'the next evening'],
+ ])]);
+ const opts = { mode: 'backfill' as const, from: '2026-10-01', to: '2026-10-09', readers: [reader], now: new Date('2026-10-20T00:00:00Z') };
+ await runTranscriptArchive(engine, opts);
+ const before = await archivedPages(engine);
+ expect(before.filter(p => p.slug.startsWith('conversations/codex/')).map(p => p.slug.split('/')[2])).toEqual(['2026-10-05', '2026-10-06']);
+
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'Asia/Tokyo');
+ const result = await runTranscriptArchive(engine, opts);
+ expect(result.timezone).toBe('UTC');
+ expect(result.written).toBe(0);
+ const after = await archivedPages(engine);
+ expect(after).toEqual(before);
+ for (const p of after.filter(p => p.slug.startsWith('conversations/codex/'))) {
+ expect((p.frontmatter.transcript_archive as Record<string, unknown>).timezone).toBe('UTC');
+ }
+ const status = await archiveStatus(engine, { readers: [reader] });
+ expect(status.timezone).toBe('UTC');
+ expect(status.configured_timezone).toBe('Asia/Tokyo');
+ }, 60_000);
+});
diff --git a/test/transcripts-openclaw-archived-files.test.ts b/test/transcripts-openclaw-archived-files.test.ts
new file mode 100644
index 0000000000..39b2245229
--- /dev/null
+++ b/test/transcripts-openclaw-archived-files.test.ts
@@ -0,0 +1,61 @@
+/**
+ * GST-2768 PR 2: OpenClaw archives a session file when it is reset (`<id>.jsonl.reset.<stamp>`, plain JSONL) or
+ * deleted (`<id>.jsonl.deleted.<stamp>.<hex>.zst`, zstd-compressed JSONL; recorded from OpenClaw 2026.11.3). Both
+ * keep the live file's line format. Discovery lists them and the adapter parses them, so the file reader archives
+ * a conversation whose live file is gone. Checkpoint files stay excluded.
+ */
+import { afterAll, describe, expect, test } from 'bun:test';
+import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { discoverTranscriptFiles } from '../src/core/transcripts/discover.ts';
+import { openclawAdapter } from '../src/core/transcripts/openclaw.ts';
+import type { FileDiagnostics, ParsedSession } from '../src/core/transcripts/types.ts';
+
+const AGENT_FIXTURE = join(import.meta.dir, 'fixtures', 'transcripts', 'agent-session.jsonl');
+const root = mkdtempSync(join(tmpdir(), 'openclaw-archived-'));
+afterAll(() => rmSync(root, { recursive: true, force: true }));
+
+async function drain(gen: AsyncGenerator<ParsedSession, FileDiagnostics>): Promise<ParsedSession[]> {
+ const out: ParsedSession[] = [];
+ for (let r = await gen.next(); !r.done; r = await gen.next()) out.push(r.value);
+ return out;
+}
+
+describe('openclaw archived session files', () => {
+ const text = readFileSync(AGENT_FIXTURE, 'utf8');
+ const live = join(root, 'live-1.jsonl');
+ const reset = join(root, 'reset-1.jsonl.reset.2026-11-15T05-40-00.000Z');
+ const deleted = join(root, 'deleted-1.jsonl.deleted.2026-11-15T05-43-50.473Z.9f3a1c0d4e5b6a7f8091a2b3c4d5e6f7.zst');
+ const checkpoint = join(root, 'live-1.checkpoint.11111111-aaaa-bbbb-cccc-222222222222.jsonl');
+ writeFileSync(live, text);
+ writeFileSync(reset, text);
+ writeFileSync(deleted, Bun.zstdCompressSync(Buffer.from(text)));
+ writeFileSync(checkpoint, text);
+
+ test('archive discovery lists the live, reset and deleted files and skips the checkpoint', () => {
+ const found = discoverTranscriptFiles([{ format: 'openclaw', root, extension: '.jsonl' }], { includeArchived: true }).map(d => d.path).sort();
+ expect(found).toEqual([deleted, live, reset].sort());
+ });
+
+ test('plain discovery (transcripts ingest) lists only the live file, so one session never imports twice', () => {
+ const found = discoverTranscriptFiles([{ format: 'openclaw', root, extension: '.jsonl' }]).map(d => d.path);
+ expect(found).toEqual([live]);
+ });
+
+ test('detect accepts both archived variants', () => {
+ expect(openclawAdapter.detect(reset, Buffer.from(text.slice(0, 4096)))).toBe(true);
+ expect(openclawAdapter.detect(deleted, readFileSync(deleted).subarray(0, 4096))).toBe(true);
+ expect(openclawAdapter.detect(checkpoint, Buffer.from(text.slice(0, 4096)))).toBe(false);
+ });
+
+ test('the reset and the zstd deleted file parse to the same messages as the live file', async () => {
+ const [base] = await drain(openclawAdapter.parse(live));
+ expect(base?.meta.sessionId).toBeTruthy();
+ for (const path of [reset, deleted]) {
+ const [s] = await drain(openclawAdapter.parse(path));
+ expect(s?.meta.sessionId).toBe(base!.meta.sessionId);
+ expect(s?.messages).toEqual(base!.messages);
+ }
+ });
+});
diff --git a/test/transcripts-openclaw-files-only.test.ts b/test/transcripts-openclaw-files-only.test.ts
new file mode 100644
index 0000000000..9d6c698a99
--- /dev/null
+++ b/test/transcripts-openclaw-files-only.test.ts
@@ -0,0 +1,99 @@
+/**
+ * GST-2768 PR 2: an OpenClaw session folder with no live transcript, read while the Gateway is unreachable, still
+ * backfills. The folder holds what OpenClaw 2026.10.9 and 2026.11.3 leave beside sessions:
+ *
+ * - read: `<id>.jsonl.reset.<stamp>`, `<id>.jsonl.deleted.<stamp>.<32 hex>.zst` and `<id>.jsonl.bak.<stamp>`. OpenClaw's
+ * isRetainedSessionTranscriptArchiveName (2026.10.9 dist, transcript-file-names) names these three as "retained
+ * archives and disposable legacy compact backups" of one session, in the live file's line format. Two files of one
+ * session merge into one conversation, each message once.
+ * - skipped: `*.migrated`, `*.migrated.<n>` and `<id>.jsonl.pre-doctor-<kind>-repair-<stamp>.bak` (its
+ * isMigrationArchiveArtifactName: "migration rollback archives retained beside their legacy source", the original of a
+ * transcript already imported into the session database the Gateway serves), and `<id>.jsonl.bak-<pid>-<timestamp>`
+ * (docs/cli/doctor/sqlite-maintenance.md: Doctor's recovery copy of a zero-byte transcript). Reading them would
+ * count a conversation twice; the Gateway reads the database copy when it is back.
+ *
+ * Every value is synthetic. Repo-less PGLite.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { mkdtempSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { ARCHIVE_TIMEZONE_KEY, runTranscriptArchive, setArchiveEnabled, type ArchiveReader } from '../src/core/transcripts/archive.ts';
+import { fileReader } from '../src/core/transcripts/archive-readers.ts';
+import { discoverTranscriptFiles } from '../src/core/transcripts/discover.ts';
+import { openclawGatewayReader, parseGatewayCliOutput } from '../src/core/transcripts/openclaw-gateway.ts';
+import { archivedPages, type Turn } from './helpers/transcript-archive.ts';
+
+const root = mkdtempSync(join(tmpdir(), 'openclaw-files-only-'));
+let engine: PGLiteEngine;
+beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); });
+afterAll(async () => { await engine.disconnect(); rmSync(root, { recursive: true, force: true }); });
+beforeEach(async () => {
+ await resetPgliteState(engine);
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await setArchiveEnabled(engine, true);
+});
+
+function sessionText(id: string, turns: Turn[]): string {
+ const lines: unknown[] = [{ type: 'session', version: 3, id, timestamp: turns[0]![1], cwd: '/home/alice-example/agent-workspace' }];
+ turns.forEach(([role, timestamp, text], i) => lines.push({ type: 'message', id: id + '-m' + i, parentId: i ? id + '-m' + (i - 1) : null, timestamp, message: { role, timestamp, content: [{ type: 'text', text }] } }));
+ return lines.map(l => JSON.stringify(l)).join('\n') + '\n';
+}
+const put = (name: string, text: string) => { const p = join(root, name); writeFileSync(p, name.endsWith('.zst') ? Bun.zstdCompressSync(Buffer.from(text)) : text); return p; };
+const HEX = '0123456789abcdef0123456789abcdef';
+const SPLIT_A: Turn[] = [['user', '2026-11-05T09:00:00.000Z', 'split one'], ['assistant', '2026-11-05T09:01:00.000Z', 'split two']];
+const SPLIT_B: Turn[] = [...SPLIT_A, ['user', '2026-11-06T09:00:00.000Z', 'split three']];
+
+const read = [
+ put('s-reset.jsonl.reset.2026-11-03T10-00-00.000Z', sessionText('s-reset', [['user', '2026-11-02T08:00:00.000Z', 'reset day text'], ['assistant', '2026-11-02T08:01:00.000Z', 'reset reply']])),
+ put('s-deleted.jsonl.deleted.2026-11-04T10-00-00.000Z.' + HEX + '.zst', sessionText('s-deleted', [['user', '2026-11-03T08:00:00.000Z', 'deleted day text']])),
+ put('s-bak.jsonl.bak.2026-11-05T10-00-00Z', sessionText('s-bak', [['user', '2026-11-04T08:00:00.000Z', 'compact backup text']])),
+ put('s-split.jsonl.reset.2026-11-05T12-00-00.000Z', sessionText('s-split', SPLIT_A)),
+ put('s-split.jsonl.deleted.2026-11-06T12-00-00.000Z.' + HEX + '.zst', sessionText('s-split', SPLIT_B)),
+];
+const SKIPPED_TEXT = 'SKIPPED-ARTIFACT-TEXT';
+const skipped = [
+ put('s-mig.jsonl.migrated', sessionText('s-mig', [['user', '2026-11-04T09:00:00.000Z', SKIPPED_TEXT + ' migrated']])),
+ put('s-mig2.jsonl.migrated.2', sessionText('s-mig2', [['user', '2026-11-04T09:00:00.000Z', SKIPPED_TEXT + ' migrated 2']])),
+ put('s-pre.jsonl.pre-doctor-branch-repair-2026-11-04T09-00-00Z.bak', sessionText('s-pre', [['user', '2026-11-04T09:00:00.000Z', SKIPPED_TEXT + ' pre-doctor']])),
+ put('s-zero.jsonl.bak-4242-2026-11-04T09-00-00Z', sessionText('s-zero', [['user', '2026-11-04T09:00:00.000Z', SKIPPED_TEXT + ' doctor recovery']])),
+];
+
+const gatewayDown = openclawGatewayReader({ transport: { async call() {
+ return parseGatewayCliOutput(JSON.stringify({ ok: false, error: { type: 'gateway_transport_error', message: 'connect ECONNREFUSED 127.0.0.1:18789', details: { kind: 'closed' } } }), 1);
+} } }) as unknown as ArchiveReader;
+const readers = () => [fileReader({ format: 'openclaw', root, extension: '.jsonl' }), gatewayDown];
+const NOW = new Date('2026-11-10T12:00:00Z');
+const backfill = () => runTranscriptArchive(engine, { mode: 'backfill', from: '2026-11-01', readers: readers(), now: NOW });
+
+describe('openclaw files-only folder with the Gateway unreachable', () => {
+ test('discovery reads the reset, deleted and bak.<stamp> files and skips migration and Doctor copies', () => {
+ const found = discoverTranscriptFiles([{ format: 'openclaw', root, extension: '.jsonl' }], { includeArchived: true }).map(d => d.path).sort();
+ expect(found).toEqual([...read].sort());
+ for (const p of skipped) expect(found).not.toContain(p);
+ });
+
+ test('backfill writes every day from the archived files, reports the Gateway, a rerun writes nothing, and a finished day whose source grew is written again', async () => {
+ const first = await backfill();
+ expect(first.errors.map(e => e.code)).toEqual(['archive_gateway_unreachable']);
+ expect(first.days.filter(d => d.status === 'written').map(d => d.day).sort()).toEqual(['2026-11-02', '2026-11-03', '2026-11-04', '2026-11-05', '2026-11-06']);
+ const pages = (await archivedPages(engine)).filter(p => p.slug.startsWith('conversations/openclaw/'));
+ const all = pages.map(p => p.body).join('\n');
+ for (const text of ['reset day text', 'reset reply', 'deleted day text', 'compact backup text', 'split three']) expect(all).toContain(text);
+ expect(all).not.toContain(SKIPPED_TEXT);
+ expect(all.split('split one').length - 1).toBe(1);
+ expect(all.split('split two').length - 1).toBe(1);
+
+ const before = await archivedPages(engine);
+ const again = await backfill();
+ expect(again.written).toBe(0);
+ expect(await archivedPages(engine)).toEqual(before);
+
+ put('s-reset.jsonl.reset.2026-11-03T10-00-00.000Z', sessionText('s-reset', [['user', '2026-11-02T08:00:00.000Z', 'reset day text'], ['assistant', '2026-11-02T08:01:00.000Z', 'reset reply'], ['user', '2026-11-02T08:02:00.000Z', 'recovered later']]));
+ const refreshed = await backfill();
+ expect(refreshed.days.filter(d => d.status === 'written').map(d => d.day)).toEqual(['2026-11-02']);
+ expect((await archivedPages(engine)).map(p => p.body).join('\n')).toContain('recovered later');
+ }, 120_000);
+});
diff --git a/test/transcripts-openclaw-gateway-argv.test.ts b/test/transcripts-openclaw-gateway-argv.test.ts
new file mode 100644
index 0000000000..371997c771
--- /dev/null
+++ b/test/transcripts-openclaw-gateway-argv.test.ts
@@ -0,0 +1,190 @@
+/**
+ * GST-2768 PR 2: the "openclaw gateway call" child process. Arguments go as an array with no shell, the binary is a
+ * fixed name, the method is fixed, and every Gateway-supplied value (a session key that looks like an option, shell
+ * metacharacters) reaches the child only inside the JSON value of --params, so it can never become an option or change
+ * what runs. The token and the Gateway URL go through the environment, never argv. A fake openclaw records its argv.
+ */
+import { afterAll, describe, expect, test } from 'bun:test';
+import { chmodSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { isCwdDotenvProtectedKey, quarantineCwdDotenv } from '../src/core/env-trust.ts';
+import { cliGatewayTransport, openclawGatewayReader } from '../src/core/transcripts/openclaw-gateway.ts';
+import { withEnv } from './helpers/with-env.ts';
+
+const syntheticToken = (n: string) => ['tok', 'synthetic', n].join('-');
+
+const dir = mkdtempSync(join(tmpdir(), 'openclaw-argv-'));
+afterAll(() => rmSync(dir, { recursive: true, force: true }));
+const log = join(dir, 'calls.jsonl');
+const bin = join(dir, 'openclaw');
+const HOSTILE = ['--url=ws://attacker.invalid', '-rf', '$(touch ' + join(dir, 'pwned') + ')', 'a;b|c`d`'];
+writeFileSync(bin, `#!${process.execPath}
+const fs = require('node:fs');
+const argv = process.argv.slice(2);
+fs.appendFileSync(${JSON.stringify(log)}, JSON.stringify({ argv, token: process.env.OPENCLAW_GATEWAY_TOKEN ?? null, url: process.env.OPENCLAW_GATEWAY_URL ?? null }) + '\\n');
+const method = argv[2];
+const params = JSON.parse(argv[argv.indexOf('--params') + 1]);
+if (method === 'sessions.list') {
+ console.log(JSON.stringify({ sessions: ${JSON.stringify(HOSTILE)}.map((key, i) => ({ key, sessionId: 's' + i, chatType: 'direct', updatedAt: 1794700000000 })), hasMore: false }));
+} else {
+ console.log(JSON.stringify({ sessionKey: params.sessionKey, messages: [{ role: 'user', content: 'hello', timestamp: 1794700000000, __openclaw: { id: params.sessionKey } }], hasMore: false }));
+}
+`);
+chmodSync(bin, 0o755);
+
+describe('openclaw gateway call child process', () => {
+ test('hostile session keys stay inside --params; argv has a fixed shape; the token is in env only', async () => {
+ const reader = openclawGatewayReader({ transport: cliGatewayTransport({ bin, url: 'ws://127.0.0.1:18789', token: syntheticToken('0001'), timeoutMs: 5_000 }) });
+ const convs = await reader.list();
+ expect(convs.map(c => c.scopeKey)).toEqual(HOSTILE);
+ const calls = readFileSync(log, 'utf8').trim().split('\n').map(l => JSON.parse(l) as { argv: string[]; token: string | null; url: string | null });
+ expect(calls.length).toBe(1 + HOSTILE.length);
+ for (const { argv, token, url } of calls) {
+ expect(token).toBe(syntheticToken('0001'));
+ expect(url).toBe('ws://127.0.0.1:18789');
+ expect(argv.join(' ')).not.toContain(syntheticToken('0001'));
+ expect(argv.length).toBe(8);
+ expect(argv.slice(0, 2)).toEqual(['gateway', 'call']);
+ expect(['sessions.list', 'chat.history']).toContain(argv[2]);
+ expect(argv[3]).toBe('--json');
+ expect(argv[4]).toBe('--params');
+ expect(argv[5]!.startsWith('{')).toBe(true);
+ expect(argv.slice(6)).toEqual(['--timeout', '5000']);
+ }
+ const historyKeys = calls.filter(c => c.argv[2] === 'chat.history').map(c => (JSON.parse(c.argv[5]!) as { sessionKey: string }).sessionKey);
+ expect(historyKeys).toEqual(HOSTILE);
+ expect(() => readFileSync(join(dir, 'pwned'))).toThrow();
+ }, 30_000);
+
+ test('a Gateway URL that is not a ws, wss, http or https URL is refused before anything runs', async () => {
+ const before = readFileSync(log, 'utf8');
+ await expect(cliGatewayTransport({ bin, url: '--config=/tmp/evil' }).call('sessions.list', {})).rejects.toThrow('GBRAIN_OPENCLAW_GATEWAY_URL must use the ws, wss, http or https scheme; nothing was run');
+ expect(readFileSync(log, 'utf8')).toBe(before);
+ });
+
+ test('a bad OpenClaw OPENCLAW_GATEWAY_URL is refused by that name before anything runs', async () => {
+ const before = readFileSync(log, 'utf8');
+ await withEnv({ OPENCLAW_GATEWAY_URL: '--config=/tmp/evil', GBRAIN_OPENCLAW_GATEWAY_URL: undefined }, async () => {
+ const err = await cliGatewayTransport({ bin }).call('sessions.list', {}).then(() => null, (e: Error) => e);
+ expect(err?.message).toContain('OPENCLAW_GATEWAY_URL must use the ws, wss, http or https scheme; nothing was run');
+ expect(err?.message).not.toContain('GBRAIN_OPENCLAW_GATEWAY_URL');
+ });
+ expect(readFileSync(log, 'utf8')).toBe(before);
+ });
+
+ test('a child that hangs past the timeout is killed and reported as unreachable', async () => {
+ const slow = join(dir, 'openclaw-slow');
+ writeFileSync(slow, '#!/bin/sh\nsleep 30\n');
+ chmodSync(slow, 0o755);
+ const started = Date.now();
+ await expect(cliGatewayTransport({ bin: slow, timeoutMs: 200 }).call('sessions.list', {})).rejects.toMatchObject({ code: 'archive_gateway_unreachable' });
+ // gstack-timing-ok: the kill bound is the contract under test (timeoutMs + 10 s grace), with 10 s of slack
+ expect(Date.now() - started).toBeLessThan(20_000);
+ }, 30_000);
+
+ test('a child that writes more than the output bound is stopped and reported as archive_gateway_failed', async () => {
+ const loud = join(dir, 'openclaw-loud');
+ writeFileSync(loud, '#!/bin/sh\nhead -c 65536 /dev/zero | tr "\\\\0" a\nsleep 5\n');
+ chmodSync(loud, 0o755);
+ const err = await cliGatewayTransport({ bin: loud, timeoutMs: 20_000, maxOutputBytes: 1024 }).call('sessions.list', {}).then(() => null, e => e as { code?: string; message?: string });
+ expect(err?.code).toBe('archive_gateway_failed');
+ expect(err?.message).toContain('more than 1024 bytes');
+ }, 30_000);
+
+ test('with no override the child is the openclaw binary, launched with an argument array', async () => {
+ const launched: Array<{ file: string; args: unknown }> = [];
+ const spawn = (async (file: string, args: unknown) => {
+ launched.push({ file, args });
+ return { error: null, stdout: JSON.stringify({ sessions: [], hasMore: false }), stderr: '' };
+ }) as unknown as NonNullable<Parameters<typeof cliGatewayTransport>[0]>['spawn'];
+ await cliGatewayTransport({ spawn }).call('sessions.list', { limit: 1 });
+ expect(launched).toHaveLength(1);
+ expect(launched[0]!.file).toBe('openclaw');
+ expect(Array.isArray(launched[0]!.args)).toBe(true);
+ expect((launched[0]!.args as string[]).slice(0, 3)).toEqual(['gateway', 'call', 'sessions.list']);
+ });
+});
+
+describe('openclaw gateway call child environment', () => {
+ const envLog = join(dir, 'env.jsonl');
+ const envBin = join(dir, 'openclaw-env');
+ writeFileSync(envBin, '#!' + process.execPath + '\nrequire("node:fs").appendFileSync(' + JSON.stringify(envLog) + ', JSON.stringify(process.env) + "\\n");\nconsole.log(JSON.stringify({ sessions: [], hasMore: false }));\n');
+ chmodSync(envBin, 0o755);
+ const lastEnv = () => JSON.parse(readFileSync(envLog, 'utf8').trim().split('\n').pop()!) as Record<string, string>;
+
+ const planted = (line: string) => {
+ const project = mkdtempSync(join(dir, 'project-'));
+ writeFileSync(join(project, '.env'), line + '\n');
+ return () => quarantineCwdDotenv(process.env, project, { warn: () => {} });
+ };
+
+ test('a planted OPENCLAW_GATEWAY_URL in a working-folder .env never reaches the child', async () => {
+ const quarantine = planted('OPENCLAW_GATEWAY_URL=wss://planted.invalid');
+ await withEnv({ OPENCLAW_GATEWAY_URL: 'wss://planted.invalid', GBRAIN_OPENCLAW_GATEWAY_URL: undefined, GBRAIN_OPENCLAW_GATEWAY_TOKEN: syntheticToken('0002') }, async () => {
+ quarantine();
+ await cliGatewayTransport({ bin: envBin }).call('sessions.list', {});
+ });
+ const env = lastEnv();
+ expect(env.OPENCLAW_GATEWAY_URL).toBeUndefined();
+ expect(env.OPENCLAW_GATEWAY_TOKEN).toBe(syntheticToken('0002'));
+ }, 30_000);
+
+ test('a planted OPENCLAW_GATEWAY_TOKEN in a working-folder .env never reaches the child', async () => {
+ const quarantine = planted('OPENCLAW_GATEWAY_TOKEN=tok-planted-0003');
+ await withEnv({ OPENCLAW_GATEWAY_TOKEN: 'tok-planted-0003', GBRAIN_OPENCLAW_GATEWAY_TOKEN: undefined, GBRAIN_OPENCLAW_GATEWAY_URL: 'ws://127.0.0.1:18789' }, async () => {
+ quarantine();
+ await cliGatewayTransport({ bin: envBin }).call('sessions.list', {});
+ });
+ const env = lastEnv();
+ expect(env.OPENCLAW_GATEWAY_TOKEN).toBeUndefined();
+ expect(env.OPENCLAW_GATEWAY_URL).toBe('ws://127.0.0.1:18789');
+ }, 30_000);
+
+ test('a stock OpenClaw environment with no GBRAIN_ variables reaches the Gateway', async () => {
+ await withEnv({ OPENCLAW_GATEWAY_URL: 'ws://127.0.0.1:18790', OPENCLAW_GATEWAY_TOKEN: 'tok-stock-0005', GBRAIN_OPENCLAW_GATEWAY_URL: undefined, GBRAIN_OPENCLAW_GATEWAY_TOKEN: undefined }, async () => {
+ await cliGatewayTransport({ bin: envBin }).call('sessions.list', {});
+ });
+ const env = lastEnv();
+ expect(env.OPENCLAW_GATEWAY_URL).toBe('ws://127.0.0.1:18790');
+ expect(env.OPENCLAW_GATEWAY_TOKEN).toBe('tok-stock-0005');
+ }, 30_000);
+
+ test('the child gets only PATH, HOME, locale and the resolved Gateway URL and token from the parent', async () => {
+ await withEnv({ OPENCLAW_STATE_DIR: '/planted/state', OPENCLAW_CONFIG_PATH: '/planted/config.json', GBRAIN_SYNTHETIC_MARKER: 'x', LANG: 'C.UTF-8' }, async () => {
+ await cliGatewayTransport({ bin: envBin, url: 'ws://127.0.0.1:18789', token: syntheticToken('0004') }).call('sessions.list', {});
+ });
+ const env = lastEnv();
+ for (const k of ['OPENCLAW_STATE_DIR', 'OPENCLAW_CONFIG_PATH', 'GBRAIN_SYNTHETIC_MARKER']) expect(env[k]).toBeUndefined();
+ expect(env.LANG).toBe('C.UTF-8');
+ expect(env.PATH).toBe(process.env.PATH!);
+ expect(env.OPENCLAW_GATEWAY_TOKEN).toBe(syntheticToken('0004'));
+ }, 30_000);
+
+ test('every OPENCLAW_* name and the gbrain Gateway names are protected from a working-folder .env', () => {
+ for (const k of ['OPENCLAW_GATEWAY_URL', 'OPENCLAW_GATEWAY_TOKEN', 'OPENCLAW_GATEWAY_PASSWORD', 'OPENCLAW_STATE_DIR', 'OPENCLAW_CONFIG_PATH', 'OPENCLAW_HOME', 'OPENCLAW_PROFILE', 'GBRAIN_OPENCLAW_GATEWAY_URL', 'GBRAIN_OPENCLAW_GATEWAY_TOKEN']) {
+ expect([k, isCwdDotenvProtectedKey(k)]).toEqual([k, true]);
+ }
+ });
+
+ test('a child that ignores SIGTERM, and its own child, are gone at the limit', async () => {
+ const pids = join(dir, 'stubborn.pids');
+ const stubborn = join(dir, 'openclaw-stubborn');
+ writeFileSync(stubborn, "#!/bin/sh\ntrap '' TERM\nsleep 40 &\necho $$ $! > " + pids + "\nwait\nsleep 40\n");
+ chmodSync(stubborn, 0o755);
+ const started = Date.now();
+ await expect(cliGatewayTransport({ bin: stubborn, timeoutMs: 200 }).call('sessions.list', {})).rejects.toMatchObject({ code: 'archive_gateway_unreachable' });
+ // gstack-timing-ok: the kill bound is the contract under test (timeoutMs + 10 s, then the TERM grace), with slack
+ expect(Date.now() - started).toBeLessThan(20_000);
+ for (const pid of readFileSync(pids, 'utf8').trim().split(/\s+/).map(Number)) expect([pid, gone(pid)]).toEqual([pid, true]);
+ }, 60_000);
+});
+
+function gone(pid: number): boolean {
+ try {
+ const stat = readFileSync('/proc/' + pid + '/stat', 'utf8');
+ return stat.slice(stat.lastIndexOf(')') + 2).startsWith('Z');
+ } catch {
+ try { process.kill(pid, 0); return false; } catch { return true; }
+ }
+}
diff --git a/test/transcripts-openclaw-gateway.test.ts b/test/transcripts-openclaw-gateway.test.ts
new file mode 100644
index 0000000000..dda7f3f4d7
--- /dev/null
+++ b/test/transcripts-openclaw-gateway.test.ts
@@ -0,0 +1,183 @@
+/**
+ * GST-2768 PR 2: the OpenClaw Gateway reader against responses recorded from OpenClaw 2026.11.3 (sessions.list and
+ * chat.history over `openclaw gateway call`), plus doc-shaped group rows. Covers paging, chatType mapping (only a
+ * verified chatType counts; the key's shape never does), the unknown channel resolving owner-only, the CLI error
+ * envelopes mapped to archive_gateway_auth / archive_gateway_unreachable, redacted error bodies, and a failed reader
+ * leaving the ledger unchanged. Repo-less PGLite for the archive cases.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { readFileSync } from 'node:fs';
+import { join } from 'node:path';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { addGroup, addMember } from '../src/core/access/members.ts';
+import { setOwner } from '../src/core/access/owner.ts';
+import { gatewayChannel, openclawGatewayReader, parseGatewayCliOutput, type GatewayTransport } from '../src/core/transcripts/openclaw-gateway.ts';
+import { ARCHIVE_TIMEZONE_KEY, archiveGroupPrincipal, runTranscriptArchive, setArchiveEnabled } from '../src/core/transcripts/archive.ts';
+import { archivedPages, conversation, ledgerRows, staticReader } from './helpers/transcript-archive.ts';
+
+const SYNTHETIC_ANTHROPIC = ['sk', 'ant', 'api03', 'SYNTHETIC'.repeat(4) + '0000000000-AAAAAAAA'].join('-');
+const FIX = join(import.meta.dir, 'fixtures', 'openclaw-gateway');
+const load = (name: string) => JSON.parse(readFileSync(join(FIX, name), 'utf8')) as Record<string, unknown>;
+
+/** Serves the recorded pages by method, offset and session key. */
+function recordedTransport(opts: { list: Array<Record<string, unknown>>; history: Record<string, Array<Record<string, unknown>>> }): GatewayTransport & { calls: Array<[string, Record<string, unknown>]> } {
+ const calls: Array<[string, Record<string, unknown>]> = [];
+ return {
+ calls,
+ async call(method, params) {
+ calls.push([method, params]);
+ if (method === 'sessions.list') {
+ const offset = Number(params.offset ?? 0);
+ return opts.list.find(p => (p.__offset ?? 0) === offset) ?? opts.list[0];
+ }
+ if (method === 'chat.history') {
+ const pages = opts.history[String(params.sessionKey)] ?? [];
+ const offset = Number(params.offset ?? 0);
+ return pages.find(p => (p.__offset ?? 0) === offset) ?? { sessionKey: params.sessionKey, messages: [], hasMore: false };
+ }
+ throw new Error(`unexpected method ${method}`);
+ },
+ };
+}
+
+const recorded = () => recordedTransport({
+ list: [load('sessions-list-page1.json'), { ...load('sessions-list-page2.json'), __offset: 2 }],
+ history: {
+ 'agent:dev:example-scratch-alpha': [load('chat-history-scratch-alpha-page1.json'), { ...load('chat-history-scratch-alpha-page2.json'), __offset: 2 }],
+ 'agent:dev:example-scratch-3': [load('chat-history-scratch-3.json')],
+ 'agent:dev:telegram:group:example-group-9': [load('chat-history-group-shaped-key.json')],
+ },
+});
+
+const docShaped = () => recordedTransport({
+ list: [load('sessions-list-group-docshape.json')],
+ history: {
+ 'agent:main:telegram:group:example-group-1': [load('chat-history-group-docshape.json')],
+ 'agent:main:webhook:example-legacy-row': [load('chat-history-legacy-row.json')],
+ },
+});
+
+const recorded1009 = () => recordedTransport({
+ list: [load('sessions-list-1009.json')],
+ history: {
+ 'agent:dev:main': [load('chat-history-1009-main.json')],
+ 'agent:dev:example-probe-a': [load('chat-history-1009-probe-a.json')],
+ },
+});
+
+describe('openclaw gateway reader (recorded fixtures)', () => {
+ test('pages sessions.list by nextOffset and chat.history backward, oldest first, user and assistant text only', async () => {
+ const transport = recorded();
+ const convs = await openclawGatewayReader({ transport }).list({});
+ expect(transport.calls.filter(([m]) => m === 'sessions.list').map(([, p]) => p.offset ?? 0)).toEqual([0, 2]);
+ expect(convs.map(c => c.scopeKey)).toEqual(['agent:dev:example-scratch-alpha', 'agent:dev:telegram:group:example-group-9']);
+ expect(convs.map(c => c.conversationKey)).toEqual(convs.map(c => c.session.meta.sessionId));
+ const alpha = convs[0]!;
+ expect(alpha.harness).toBe('openclaw');
+ expect(alpha.session.meta.sessionId).toBe('318c6a69-4cfa-4aba-bda5-8babbb465c4e');
+ expect(alpha.session.messages.map(m => [m.role, m.text])).toEqual([
+ ['assistant', 'hello from synthetic user'], ['user', 'what is the capital of acme-example land']]);
+ expect(alpha.session.messages[0]!.timestamp).toBe(new Date(1794721281750).toISOString());
+ });
+
+ test('OpenClaw 2026.10.9 shape: pages and maps the list, and an operator-made session with no chatType is unverified', async () => {
+ const transport = recorded1009();
+ const convs = await openclawGatewayReader({ transport }).list({});
+ expect(transport.calls.filter(([m]) => m === 'sessions.list')).toHaveLength(1);
+ expect(convs.map(c => c.scopeKey)).toEqual(['agent:dev:main', 'agent:dev:example-probe-a']);
+ expect(convs.map(c => c.conversationKey)).toEqual(['00000000-0000-4000-8000-000000001009', '00000000-0000-4000-8000-000000001010']);
+ expect(convs.map(c => c.channel)).toEqual([{ chatType: 'direct', verified: true }, { verified: false }]);
+ expect(convs[1]!.session.messages.map(m => [m.role, m.text])).toEqual([['assistant', 'note 1 for probe a'], ['assistant', 'note 2 for probe a']]);
+ expect(convs[1]!.session.messages[0]!.timestamp).toBe(new Date(1794724689369).toISOString());
+ });
+ test('chatType mapping: only the row chatType counts, never the key shape or kind', async () => {
+ const [, groupShaped] = await openclawGatewayReader({ transport: recorded() }).list({});
+ expect(groupShaped!.channel).toEqual({ chatType: 'direct', verified: true });
+ const doc = await openclawGatewayReader({ transport: docShaped() }).list({});
+ expect(doc.map(c => c.channel)).toEqual([{ chatType: 'group', verified: true }, { verified: false }]);
+ expect(gatewayChannel({ key: 'agent:main:cron:nightly-digest', chatType: 'direct' })).toEqual({ chatType: 'direct', verified: true, cronJob: 'nightly-digest' });
+ expect(gatewayChannel({ key: 'x', chatType: 'channel' })).toEqual({ verified: false });
+ });
+
+ test('the CLI error envelopes map to the stable codes, and an error body is redacted before it is shown', () => {
+ const codeOf = (name: string) => { try { parseGatewayCliOutput(JSON.stringify(load(name)), 1); return null; } catch (e) { return (e as { code?: string }).code; } };
+ expect(codeOf('error-auth.json')).toBe('archive_gateway_auth');
+ expect(codeOf('error-credentials-required.json')).toBe('archive_gateway_auth');
+ expect(codeOf('error-unreachable.json')).toBe('archive_gateway_unreachable');
+ const leaky = JSON.stringify({ ok: false, error: { type: 'gateway_request_error', code: 'INVALID_REQUEST', message: 'unauthorized: token ' + SYNTHETIC_ANTHROPIC + ' rejected', details: { code: 'AUTH_TOKEN_MISMATCH' } } });
+ let message = '';
+ try { parseGatewayCliOutput(leaky, 1); } catch (e) { message = (e as Error).message; }
+ expect(message).toContain('<REDACTED:');
+ expect(message).not.toContain('SYNTHETICSYNTHETIC');
+ expect(parseGatewayCliOutput(JSON.stringify(load('sessions-list-page2.json')), 0)).toMatchObject({ hasMore: false });
+ });
+});
+
+describe('gateway reader inside the archive (pglite)', () => {
+ let engine: PGLiteEngine;
+
+ beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+ });
+
+ afterAll(async () => {
+ await engine.disconnect();
+ });
+
+ beforeEach(async () => {
+ await resetPgliteState(engine);
+ await engine.setConfig(ARCHIVE_TIMEZONE_KEY, 'UTC');
+ await setArchiveEnabled(engine, true);
+ });
+
+ test('OpenClaw 2026.10.9 on a shared brain: the operator-made session with no chatType is owner-only, even when a group named for its key exists', async () => {
+ await engine.setConfig('brain.audience', 'shared');
+ await addMember(engine, 'alice-example');
+ await addMember(engine, 'bob-example');
+ await setOwner(engine, 'alice-example');
+ await addGroup(engine, archiveGroupPrincipal('openclaw', 'agent:dev:example-probe-a'), ['bob-example']);
+ const result = await runTranscriptArchive(engine, { mode: 'backfill', from: '2026-11-14', to: '2026-11-15', readers: [openclawGatewayReader({ transport: recorded1009() })], now: new Date('2026-11-20T00:00:00Z') });
+ expect(result.errors).toEqual([]);
+ const pages = (await archivedPages(engine)).filter(p => p.slug.startsWith('conversations/openclaw/'));
+ expect(pages).toHaveLength(2);
+ for (const page of pages) expect(page.frontmatter.readable_by).toEqual(['alice-example']);
+ const probe = pages.find(p => p.body.includes('note 1 for probe a'))!;
+ expect((probe.frontmatter.transcript_archive as Record<string, unknown>).chat_type).toBeUndefined();
+ }, 60_000);
+ test('on a shared brain the verified group gets owner plus its existing group; the unknown channel resolves owner only', async () => {
+ await engine.setConfig('brain.audience', 'shared');
+ await addMember(engine, 'alice-example');
+ await addMember(engine, 'bob-example');
+ await setOwner(engine, 'alice-example');
+ const groupKey = 'agent:main:telegram:group:example-group-1';
+ await addGroup(engine, archiveGroupPrincipal('openclaw', groupKey), ['bob-example']);
+ const result = await runTranscriptArchive(engine, { mode: 'backfill', from: '2026-11-14', to: '2026-11-15', readers: [openclawGatewayReader({ transport: docShaped() })], now: new Date('2026-11-20T00:00:00Z') });
+ expect(result.held).toBe(0);
+ const pages = (await archivedPages(engine)).filter(p => p.slug.startsWith('conversations/openclaw/'));
+ const group = pages.find(p => (p.frontmatter.transcript_archive as Record<string, unknown>).chat_type === 'group')!;
+ expect(group.slug).toMatch(/\/group-[0-9a-f]{8}-[0-9a-f]{8}$/);
+ expect(group.frontmatter.readable_by).toEqual(['alice-example', `group:${archiveGroupPrincipal('openclaw', groupKey)}`]);
+ const unknown = pages.find(p => p !== group)!;
+ expect(unknown.frontmatter.readable_by).toEqual(['alice-example']);
+ expect(unknown.slug).not.toContain('legacy-row');
+ }, 60_000);
+
+ test('auth and unreachable failures carry their codes, leave the ledger unchanged and do not stop other readers', async () => {
+ const failing = (code: 'archive_gateway_auth' | 'archive_gateway_unreachable') => openclawGatewayReader({ transport: {
+ async call() { return parseGatewayCliOutput(JSON.stringify(load(code === 'archive_gateway_auth' ? 'error-auth.json' : 'error-unreachable.json')), 1); },
+ } });
+ const other = staticReader('codex', [conversation('codex', 'rollout-ok', [['user', '2026-11-14T10:00:00.000Z', 'still archived']])]);
+ const opts = { mode: 'backfill' as const, from: '2026-11-13', to: '2026-11-15', now: new Date('2026-11-20T00:00:00Z') };
+ await runTranscriptArchive(engine, { ...opts, readers: [other] });
+ const before = await ledgerRows(engine);
+ for (const code of ['archive_gateway_auth', 'archive_gateway_unreachable'] as const) {
+ const result = await runTranscriptArchive(engine, { ...opts, readers: [failing(code), other] });
+ expect(result.errors.map(e => [e.harness, e.code])).toEqual([['openclaw', code]]);
+ expect(result.errors[0]!.fix?.argv?.[0]).toBe('openclaw');
+ expect(await ledgerRows(engine)).toEqual(before);
+ }
+ }, 60_000);
+});