Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(transcripts): ingest --facts reports a budget stop and skipped pages

garrytanb1b142eedab3
diff --git a/docs/architecture/key-files/commands-6.md b/docs/architecture/key-files/commands-6.md
index d1df7f91b3..3caf28d71c 100644
--- a/docs/architecture/key-files/commands-6.md
+++ b/docs/architecture/key-files/commands-6.md
@@ -10,7 +10,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/commands/takes-mutation.ts` — one parser and operation adapter for local, resident PGLite, and HTTP take mutations. Allocates a request UUID before delivery; retries retain that UUID and caller intent. `--source` remains claim provenance, `--source-id` selects the logical source, and `--dir` becomes a trusted local canonical-root assertion. Default owner/resolver choices remain unset until admission freezes them. The CLI and thin-client router share this parser for all four mutations before connecting; `runTakesRebuild` (W9F item 8) is `gbrain takes rebuild`'s parser and renderer, running `takes_rebuild` through a resident owner or the local engine (thin clients refused); queued/error responses never print completed-mutation text. Remote failures use the shared typed reporter directly, retaining UUIDs, unknown delivery status and original receipt fields in JSON. `test/persistence-cli-delegation.test.ts` drives the real CLI through a held PGLite owner's socket, checks all four parameter mappings and pending receipts, and rejects malformed numeric/precondition input.
-- `src/commands/transcripts.ts` — the transcripts command family, all local-only (`ctx.remote=false` by construction). `recent`: raw `.txt` corpus reads via `listRecentTranscripts` (same library as the gated `get_recent_transcripts` op). `ingest <path-or-glob>`: the cross-harness session importer — resolves ONE source id (6-tier chain), threads activePack once, streams progress (phase `transcripts.ingest`), and calls `runTranscriptsIngest` with the CLI's `makeContext` context, so on a writer-claimed brain each rendered part is submitted as a `put_page` write request naming the page's current revision (a part whose publication would change nothing is screened request-free first and takes no admission; unmanaged brains and dry runs keep the direct importer; `recover codex --apply` passes the same context; pinned by `test/transcripts-managed-ingest.test.ts`); embedding is OFF by default (embed backfill is the catch-up lane; the embed flag opts in); the max-bytes flag (validated size string, e.g. `4gb`) overrides the per-format file/store byte caps for oversized stores while omission preserves each adapter's native default; the since-last watermark is an op-checkpoint (op `transcripts-ingest`, fingerprint = source + pathspec + format + adapter version + any explicit byte cap via `ingestCheckpointFingerprintInput` — a checkpoint written under one cap is never silently reused under another, so a capped run's skipped tail can't read as scanned; the pathspec binds the user-stated paths resolved + sorted, and the no-arg/all lane substitutes hostname + sorted harness roots so DB-backed checkpoints shared across a brain's machines never let one machine inherit another's watermark and skip sessions it never scanned) advanced ONLY after a clean, untruncated, non-dry scan (a Codex sub-agent rollout, `session_meta` payload.source.subagent, has no user turn by design: the adapter reports `subagentRollout`, the scan counts it in `subagentRollouts` and prints `N sub-agent rollouts (no user turns by design)`, and it keeps the scan clean; an ordinary assistant-only rollout is still `userTurnsMissing` drift); no-arg = confined discovery table, the all flag imports the discovered set. `status`: found-vs-imported gap table (disk scan vs ONE conversation-pages frontmatter query — `executeRaw`, both engines — in `src/core/transcripts/discover.ts`) — the correctness surface that catches late-arriving sessions no watermark can. The facts flag hands EVERY touched slug (including hash-skipped) to `src/core/transcripts/ingest-facts.ts`; a page that fails extraction counts in `facts.pagesFailed`, stays unfinished, and makes the command exit 1. Registered in CLI_ONLY + CLI_ONLY_SELF_HELP + SELF_HELP_WITHOUT_ENGINE (engine-free help). `expandTilde(raw)` (bare `~`, leading `~/` or `~\\` → home dir; `~user` forms and a mid-string `~` stay literal) is the one tilde expansion `expandPaths` applies to every path spec. `recover codex` (`src/commands/transcripts-recover.ts`, logic `src/core/transcripts/recover.ts`; D15 for #5163): finds codex conversation pages in one source with no `**User**` anchor, re-parses the retained rollouts (discovery root + archived store, or given paths) and classifies each session recoverable / still userless / unrecoverable (rollout gone); preview by default, `--apply` re-imports only the recoverable rollouts through `runTranscriptsIngest` (content-hash dedup, no facts, no watermark write). Pinned by `test/transcripts-recover-codex.test.ts`. `audit-secrets [--source-id S] [--json]` (`src/commands/transcripts-audit.ts`, logic `src/core/transcripts/secret-audit.ts`): the read-only installed-base audit that streams imported conversation pages (`type = 'conversation'` with `transcript_import` frontmatter) in id-keyset batches of 25, scans title and body with the transcript-lane detectors, lists slug, source, hit counts per pattern and up to 20 line numbers (never values or previews), and caches the summary without slugs in config row `transcripts.secret_audit` for the doctor check `transcript_secret_exposure` (`src/commands/doctor/checks/transcript-secrets.ts`, brain category), which counts imported conversation pages and reads that summary (never scans): no audit, an older `TRANSCRIPT_SECRET_AUDIT_VERSION` or a source-scoped audit warns with the audit as fix; hits warn with the audit's age; a clean full audit is ok. Pinned by `test/transcripts-secret-audit.test.ts`.
+- `src/commands/transcripts.ts` — the transcripts command family, all local-only (`ctx.remote=false` by construction). `recent`: raw `.txt` corpus reads via `listRecentTranscripts` (same library as the gated `get_recent_transcripts` op). `ingest <path-or-glob>`: the cross-harness session importer — resolves ONE source id (6-tier chain), threads activePack once, streams progress (phase `transcripts.ingest`), and calls `runTranscriptsIngest` with the CLI's `makeContext` context, so on a writer-claimed brain each rendered part is submitted as a `put_page` write request naming the page's current revision (a part whose publication would change nothing is screened request-free first and takes no admission; unmanaged brains and dry runs keep the direct importer; `recover codex --apply` passes the same context; pinned by `test/transcripts-managed-ingest.test.ts`); embedding is OFF by default (embed backfill is the catch-up lane; the embed flag opts in); the max-bytes flag (validated size string, e.g. `4gb`) overrides the per-format file/store byte caps for oversized stores while omission preserves each adapter's native default; the since-last watermark is an op-checkpoint (op `transcripts-ingest`, fingerprint = source + pathspec + format + adapter version + any explicit byte cap via `ingestCheckpointFingerprintInput` — a checkpoint written under one cap is never silently reused under another, so a capped run's skipped tail can't read as scanned; the pathspec binds the user-stated paths resolved + sorted, and the no-arg/all lane substitutes hostname + sorted harness roots so DB-backed checkpoints shared across a brain's machines never let one machine inherit another's watermark and skip sessions it never scanned) advanced ONLY after a clean, untruncated, non-dry scan (a Codex sub-agent rollout, `session_meta` payload.source.subagent, has no user turn by design: the adapter reports `subagentRollout`, the scan counts it in `subagentRollouts` and prints `N sub-agent rollouts (no user turns by design)`, and it keeps the scan clean; an ordinary assistant-only rollout is still `userTurnsMissing` drift); no-arg = confined discovery table, the all flag imports the discovered set. `status`: found-vs-imported gap table (disk scan vs ONE conversation-pages frontmatter query — `executeRaw`, both engines — in `src/core/transcripts/discover.ts`) — the correctness surface that catches late-arriving sessions no watermark can. The facts flag hands EVERY touched slug (including hash-skipped) to `src/core/transcripts/ingest-facts.ts`; a page that fails extraction counts in `facts.pagesFailed`, stays unfinished, and makes the command exit 1; a pass that stopped at its budget reports `facts.budgetExhausted` and `facts.budgetReason`, and pages skipped as too large or held by another run count in `facts.pagesSkippedTooLarge` and `facts.pagesLockSkipped` (#8722). Registered in CLI_ONLY + CLI_ONLY_SELF_HELP + SELF_HELP_WITHOUT_ENGINE (engine-free help). `expandTilde(raw)` (bare `~`, leading `~/` or `~\\` → home dir; `~user` forms and a mid-string `~` stay literal) is the one tilde expansion `expandPaths` applies to every path spec. `recover codex` (`src/commands/transcripts-recover.ts`, logic `src/core/transcripts/recover.ts`; D15 for #5163): finds codex conversation pages in one source with no `**User**` anchor, re-parses the retained rollouts (discovery root + archived store, or given paths) and classifies each session recoverable / still userless / unrecoverable (rollout gone); preview by default, `--apply` re-imports only the recoverable rollouts through `runTranscriptsIngest` (content-hash dedup, no facts, no watermark write). Pinned by `test/transcripts-recover-codex.test.ts`. `audit-secrets [--source-id S] [--json]` (`src/commands/transcripts-audit.ts`, logic `src/core/transcripts/secret-audit.ts`): the read-only installed-base audit that streams imported conversation pages (`type = 'conversation'` with `transcript_import` frontmatter) in id-keyset batches of 25, scans title and body with the transcript-lane detectors, lists slug, source, hit counts per pattern and up to 20 line numbers (never values or previews), and caches the summary without slugs in config row `transcripts.secret_audit` for the doctor check `transcript_secret_exposure` (`src/commands/doctor/checks/transcript-secrets.ts`, brain category), which counts imported conversation pages and reads that summary (never scans): no audit, an older `TRANSCRIPT_SECRET_AUDIT_VERSION` or a source-scoped audit warns with the audit as fix; hits warn with the audit's age; a clean full audit is ok. Pinned by `test/transcripts-secret-audit.test.ts`.
- `src/commands/upgrade.ts` — self-update CLI. It sets `GBRAIN_UPGRADE_OWNS_MIGRATIONS=1` for the install step, so package postinstall (`scripts/postinstall.ts`) skips `apply-migrations`; post-upgrade owns them. An install step that fails or times out after `gbrain --version` already reports a newer version counts as installed. The run ends with a `Binary:` line (installed version, or upgrade failed) and a `Migrations:` line: complete; running, with the lock holder's host and pid when post-upgrade exits 75; or failed, with the recovery command (`--force-retry <v>` for wedged versions). `runPostUpgrade()` enumerates migrations from the TS registry (`src/commands/migrations/index.ts`) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally. After schema apply it prints the one-shot auto_chronicle default-on `[AGENT]` notice (`core/chronicle/upgrade-notice.ts`). Before the swap of a bun-link, package or ClawHub install it refuses (exit 78, nothing changed) a target whose Bun floor is above the host's Bun or unreadable (`--no-bun-floor-check` overrides, manual only; `src/core/bun-floor.ts`); after the swap it runs `gbrain --help` as a smoke that passes the startup Bun check and, on failure, exits 1 with per-method recovery (#5855). Everything the post-swap tail needs is imported statically, so the old process never loads new-release modules into its cached graph (#6604; the supervisor step constant lives in the import-free leaf `src/commands/movement-step.ts`; guard: `test/upgrade-post-swap-imports.test.ts`). A tag-pinned reinstall hint (`bunReinstallCommand`, `reinstallHintLines`) probes `git ls-remote --exit-code --tags … refs/tags/v<x>` before `bun remove -g gbrain`, prints a restore line, and on Windows prints one step per line; `planPinnedReinstall` probes the tag itself (published / not published / unknown) and falls back to the newest published `v<x>` tag (`latestPublishedVersion`) when the requested one is missing (#6662, D24; `test/upgrade-release-tag-hint.test.ts`). A verified version equal to the running one after the install step is no swap (#6828): it prints `No newer version was installed`, skips `last_upgrade`, `JUST_UPGRADED`, the cache reset and the smoke, and on a tag pin with no newer published tag the hint moves the install off the pin with `to_version: latest`, never a cached unpublished version (#6718). With `GBRAIN_SWAP_RECEIPT` set it writes a `SwapReceipt` (`installed` / `not_installed` / `installed_broken`) for the autopilot channel (`test/autopilot-unpublished-target-6828.test.ts`). Tests: `upgrade-migration-outcome.test.ts`, `upgrade-no-autopilot.serial.test.ts`, `upgrade-bun-link-arc.serial.test.ts`. `post-upgrade --json` (D2, `json: 'document'`): human lines go to stderr and stdout is one report `{status, apply_migrations:{exit_code}, schema, warnings[]}`; it calls `applyMigrations()` and still ends on any status apply-migrations returns (a failure is one envelope leading with the report). GBRA-175: `--channel stable|alpha` (`latest` is an alias of `alpha`; resolution order in the Release channels entry below): `resolveStableRelease` maps the commit `refs/tags/stable` peels to onto its `v<version>` tag through `git ls-remote`, and `installStableRelease` installs exactly that release (bun: remove then add `#v<version>`, restoring the previous release on failure; bun-link: fast-forward only; binary: `fetchReleaseByTag`), never downgrading. Before the swap `preflightHostRestart` refuses (`restart_refused`, exit 3) when a running process could not be restarted; after post-upgrade the new build runs `gbrain upgrade --restart-only` (`--no-restart` skips it). Tests: `upgrade-stable-channel.test.ts`. A global install pinned to a tag or commit (`bunGlobalExactTagPin(root, true)`) is replaced with the newest published release the same way instead of `bun update`, which keeps the pin. `parseChannelArg` and `resolveStableRelease` live in `src/core/stable-channel.ts` (re-exported).
- Release channels: `src/core/self-upgrade.ts` resolves the channel (`resolveReleaseChannel`: `--channel`, `GBRAIN_RELEASE_CHANNEL`, `self_upgrade.channel`, default `stable`; `latest` normalizes to `alpha`; an invalid value at the first present source is an error) and keeps the update cache (`update-check.<channel>.json`, tagged with the evidence policy; an older answer never overwrites a newer one) and the time-based snooze (`update-snoozed.<channel>.json`) per channel; `offeredVersion` never offers a downgrade except to move off a `known_bad` version. `src/core/stable-channel.ts` reads and strictly parses the evidence record (`stable.json` on `stable-record`; `requiredEvidenceKeys`: `soak_run`, plus `matrix_run` from `MATRIX_REQUIRED_FROM`), maps failures to `stable_missing` / `stable_invalid` / `stable_unreadable` with sanitized causes, and `resolveStableRelease` checks the `stable` and `v<version>` tags against the record commit. `src/commands/upgrade-channel.ts` plans an upgrade's target (`channelTarget`, `runUpgradeDryRun` for `gbrain upgrade --dry-run --json`) and refuses with the typed code. `src/commands/check-update.ts` `refreshUpdateCache` never reads a bare VERSION for stable and never falls back to alpha. `src/core/release-channel-notice.ts` prints the one-time `[gbrain notice release_channel]` line on stderr (marker file under `notices/`; not under `--json`, `serve` or `mcp`; only when no channel is set). `src/core/install-ref.ts` is the one ref every install surface names, linted by `scripts/check-install-ref.ts`. Tests: `test/release-channel-w1.serial.test.ts`, `test/upgrade-stable-channel.test.ts`, `test/install-ref.test.ts`. Docs: `docs/guides/upgrades-auto-update.md#stable-channel`.
diff --git a/src/commands/transcripts.ts b/src/commands/transcripts.ts
index 5f4d35baf5..b283abbfea 100644
--- a/src/commands/transcripts.ts
+++ b/src/commands/transcripts.ts
@@ -537,7 +537,7 @@ async function runIngest(engine: BrainEngine, args: string[], dispatch: Pick<Cli
// --facts: ONE extractor invocation over every touched slug (including
// hash-skipped pages — the extractor's version-token gate dedupes work).
- let factsSummary: { pages: number; pagesFailed: number; spentUsd?: number } | undefined;
+ let factsSummary: import('../core/transcripts/ingest-facts.ts').IngestFactsResult | undefined;
if (parsed.facts && !parsed.dryRun && result.slugsTouched.length > 0) {
const { runIngestFacts } = await import('../core/transcripts/ingest-facts.ts');
factsSummary = await runIngestFacts(engine, {
@@ -558,6 +558,11 @@ async function runIngest(engine: BrainEngine, args: string[], dispatch: Pick<Cli
(factsSummary.spentUsd !== undefined ? `, ~$${factsSummary.spentUsd.toFixed(2)} spent` : '') +
(factsSummary.pagesFailed > 0 ? ' (failed pages stay unfinished; stderr names each retry command)' : ''),
);
+ if (factsSummary.budgetExhausted) console.log(factsSummary.budgetReason === 'no_pricing'
+ ? 'facts: stopped because a model has no pricing; pages after the stop have no facts yet (raising --max-cost-usd cannot fix this)'
+ : `facts: stopped at the ${factsSummary.budgetReason ?? 'cost'} budget; pages after the stop have no facts yet (re-run with a higher --max-cost-usd to continue)`);
+ if (factsSummary.pagesSkippedTooLarge) console.log(`facts: ${factsSummary.pagesSkippedTooLarge} page(s) skipped as too large to extract`);
+ if (factsSummary.pagesLockSkipped) console.log(`facts: ${factsSummary.pagesLockSkipped} page(s) skipped while another run held them (re-run to finish them)`);
}
const firstImported = result.files.flatMap((f) => f.sessions).find((s) => !s.error && s.baseSlug);
if (firstImported && !parsed.dryRun) {
diff --git a/src/core/transcripts/ingest-facts.ts b/src/core/transcripts/ingest-facts.ts
index 7b2e6b7a88..a5c9ec80f1 100644
--- a/src/core/transcripts/ingest-facts.ts
+++ b/src/core/transcripts/ingest-facts.ts
@@ -17,7 +17,7 @@
import type { BrainEngine } from '../engine.ts';
import { isFactsExtractionEnabled } from '../facts/extract.ts';
-import { BudgetTracker, loadPricingOverrides } from '../budget/budget-tracker.ts';
+import { BudgetTracker, loadPricingOverrides, type BudgetReason } from '../budget/budget-tracker.ts';
import { withBudgetTracker } from '../ai/gateway.ts';
import { conversationFactsCostCap } from '../facts/conversation-budget.ts';
import {
@@ -30,6 +30,10 @@ export interface IngestFactsResult {
pagesFailed: number;
spentUsd?: number;
skippedDisabled?: boolean;
+ budgetExhausted?: boolean;
+ budgetReason?: BudgetReason;
+ pagesSkippedTooLarge?: number;
+ pagesLockSkipped?: number;
}
export async function runIngestFacts(
@@ -59,5 +63,7 @@ export async function runIngestFacts(
budgetTracker: tracker,
}),
);
- return { pages: opts.slugs.length, pagesFailed: result.pages_failed, spentUsd: tracker.totalSpent };
+ return { pages: opts.slugs.length, pagesFailed: result.pages_failed, spentUsd: tracker.totalSpent,
+ budgetExhausted: result.budget_exhausted === true, ...(result.budget_reason ? { budgetReason: result.budget_reason } : {}),
+ pagesSkippedTooLarge: result.pages_skipped_too_large, pagesLockSkipped: result.pages_lock_skipped };
}
diff --git a/test/extract-conversation-facts.test.ts b/test/extract-conversation-facts.test.ts
index a7c9ef0444..408f27dfe2 100644
--- a/test/extract-conversation-facts.test.ts
+++ b/test/extract-conversation-facts.test.ts
@@ -1663,6 +1663,16 @@ describe('runExtractConversationFactsCore', () => {
expect(result).toMatchObject({ pages: 1, pagesFailed: 1 });
});
+ test('transcript ingest reports a budget stop and skipped pages instead of reading complete (#8722)', async () => {
+ const slugs = ['conversations/batch-first', 'conversations/batch-last'];
+ await seedBatch(slugs);
+ chatFailure = new BudgetExhausted('reserve denied', { reason: 'cost', spent: 5, cap: 5 });
+ await withEnv({ ANTHROPIC_API_KEY: 'sk-test' }, async () => {
+ const result = await runIngestFacts(engine, { sourceId: 'default', slugs, quiet: true });
+ expect(result).toMatchObject({ pages: 2, pagesFailed: 0, budgetExhausted: true, budgetReason: 'cost', pagesSkippedTooLarge: 0, pagesLockSkipped: 0 });
+ });
+ });
+
test.each([
{ name: 'a failed page exits 1', malformed: true, exit: 1, failed: 1 },
{ name: 'no failed page exits 0', malformed: false, exit: 0, failed: 0 },