Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

refactor(extract_atoms): route session transcripts through the shared synthesize-source-routing helpers (#7846)

garrytan75a7675d55e2
diff --git a/docs/architecture/key-files/core-cycle.md b/docs/architecture/key-files/core-cycle.md
index a66355ad32..302da9e827 100644
--- a/docs/architecture/key-files/core-cycle.md
+++ b/docs/architecture/key-files/core-cycle.md
@@ -23,8 +23,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/cycle/extract-atoms-budget-notice.ts` — #6425: `drainBudgetNotice(result)` renders the `atom_drain_budget` notice a `budget`-stopped drain carries in `gbrain dream --drain --json` (spend, per-run cap, pages left; paid fix `gbrain config set cycle.extract_atoms.budget_usd <USD>`, verify `gbrain config get`), and `usd` formats the amounts for the stderr stop line. Dynamically imported by `dream.ts` so its notice-rendering imports stay out of the CLI flag scan. Tests: `test/dream-drain-failure-summary.serial.test.ts`.
- `src/core/cycle/extract-atoms-input.ts` — #6503 the input cap and the partial disposition. `resolveExtractAtomsInputCap` reads `cycle.extract_atoms.max_input_chars` (floor 1000; the older `max_source_chars`, floor 500, loses when both are set; default 50 000) before discovery. `extractAtomsInput(item, cap)` binds the cut once: paste-stripped for a transcript (#5812), UTF-8-safe, with `truncated` measured on the text the cap applied to. A page item also takes its stored `extract_atoms` watermark (`src/core/extraction-watermark.ts`, large pages A7): when the body still starts with the watermarked prefix the model reads only the appended text (its newest `cap` characters when longer), which is still partial input, so the same disposition below applies; `inputTruncatedLine` is null for a complete tail read (not counted, no stderr) and names the window otherwise, and the phase commits the plan once the item's writes committed (a pending managed batch keeps the old watermark). A cut item is extracted once from its head and gets a partial disposition instead of a completion: `input_truncated` counts it on the phase result (and the drain sums it into `input_truncated`, with one `[drain]` line), `inputTruncatedLine` is written to stderr naming the knob, the head atoms publish, no completion stamp is written and nothing is retired. `recordPartialDisposition` stores the cap under (content hash, cap) in `extract_atoms_page_state` / `extract_atoms_transcript_state.partial_input_chars` (migration v240; the page row is pinned to the identity discovery read), and on a managed brain the origin carries `partialInputChars`, which enters the run key, keeps the completion's checkpoint `partial`, skips the page-state stamp and the retirement, and marks the completion outcome `status: 'partial'`. Discovery and the backlog count skip a partial row like a tombstone (`PAGE_SCAN_STATE_EXCLUSION_SQL`, `tombstonedTranscriptsForHashes`, the partial checkpoint); `releasePartialDispositions` runs before discovery with the current cap and clears the rows and checkpoints a raised cap supersedes, so the item is eligible again with zero model calls in between; `countExtractAtomsBacklog` reads the same cap and counts a page whose partial row or checkpoint sits below it (`pageScanStateExclusionSql` / `managedAtomDiscoverySql` with the cap; #7391), so a drain, doctor and autopilot see it before any phase has released it, while counting itself writes nothing (`test/extract-atoms-raised-cap-drain-7391.test.ts`); a content change is a new row. Resumed and budget-skipped items are not counted. `stampTranscriptTombstone` (v146) lives here too. Tests: `test/extract-atoms-input-truncated-6503.test.ts` (PGLite and Postgres, page and transcript, managed and unmanaged).
-- `src/core/cycle/extract-atoms-source-routing.ts` — #7846 session source for extract_atoms. `keepAtomSourceTranscripts` resolves each session-corpus file through `resolveCorpusFileSource` (`context/corpus-source.ts`, the #6268 resolver the facts sweep uses): a file of the run's source is kept, a file of another source is counted under `corpus_source_other`, and a file with no usable evidence of its source is counted under the resolver's hold reason; nothing falls back to the run's source, and files outside the corpus pass. `withAtomSourceRouting` appends both counts to the phase summary and `details.source_routing` with the fix (`gbrain dream --source <id> --phase extract_atoms`, and `gbrain sweep --assign-corpus <source>` for unresolved and legacy holds, already-swept files included). Tests: `test/extract-atoms-session-source-7846.test.ts` (PGLite and Postgres).
-- `src/core/cycle/extract-atoms.ts` — the extract_atoms lens phase: mines eligible pages into `atoms/<date>/<stem>-<hash>` pages. The input cap, the cut and the partial disposition of a cut item are in `extract-atoms-input.ts` (#6503). Transcript input is paste-stripped before the cut (#5812); discovery skips self-captures (#5820) and keeps only the session-corpus files `extract-atoms-source-routing.ts` resolves to the run's source (#7846); the meeting transcripts dir is read only for `default`. Discovery and the backlog count append `connectorAtomExclusionSql` (`connector-atoms.ts`), so connector `email`/`meeting` pages stay out only when `cycle.extract_atoms.connector_pages` is false. The type allowlist is `unionExtractableTypes(packExtractable, packExcluded)`: the legacy floor (meeting, source, article, video, book, original) minus the floor types the active pack declares not extractable (`nonExtractableTypesFromPack`, #5756; a declared type without `extractable: true` is a pack decision, an undeclared type keeps the floor), plus the pack's `extractable: true` types, minus synthesis outputs; gbrain-base is unchanged, gbrain-recommended's `source: false` now holds for atoms as it did for facts. Pinned by `test/pack-extractable-floor-5756.test.ts`. A drain attempt passes `opts.attempt` (`budgetTracker`, `writeWait`) so its batches share one budget and one maintenance publish wait. Eligibility reads source-incarnation/page/full-hash-keyed derived scan state; scan and failure markers do not mutate canonical frontmatter. Managed extraction checks source-wide local authority and owner readiness before provider work, admits outputs plus a completion request together, and replays retained output without another model call. A failed managed batch is retried by the next run instead (#6325, `resumeManagedAtoms` in `persistence/atom-maintenance.ts`); an origin held after `MAX_DETERMINISTIC_FAILURES` (now defined in `extract-atoms-page-state.ts`, re-exported here) is skipped with no model call and counted in `details.held_failed`, never in `failures`. Managed atom files keep provisional source hashes; only the source-incarnation/page/full-hash durable completion receipt excludes an item from discovery and backlog counts. Unmanaged atom completion flips and their cardinality check share a transaction after provenance links persist, so a partial match rolls back and stays retryable. A page-derived atom upserts at its locator-folded slug, else a compatible legacy title-only slug; when both miss, the source page's live bound atoms are loaded once per page per run and exactly one JS-lowercased title match is adopted (a case-only title change never mints a duplicate; zero or several matches mint the new slug). Response parsing lives in `extract-atoms-parse.ts`. Atoms of one item that resolve to one slug (a repeated title) publish once, the first kept (#7514: two items at one key failed the generation, managed `generation_live_conflict`, unmanaged a primary-key violation). A managed item whose atom write did not commit is a write failure (`atomWriteFailure`, `persistence/atom-maintenance.ts`): its `failures` entry carries `request_id`, `code`, and its strike, never a provider classification. An undated source dates its atoms by the source page's creation date (`undated` for transcripts), never the run date. A page-derived atom's frontmatter `date` is the source page's content date (`atom-source-date.ts`: the stored `effective_date`, else `computeEffectiveDate`), read with the input before the model call, so classic import and managed publication store that `effective_date` (source `date`) and `extracted_at` stays the processing stamp; transcript-derived atoms carry no `date`. After an unmanaged item completes (`completeUnmanagedItem`), `retireStaleAtoms` soft-deletes that source's atoms from earlier extractions (other or provisional `source_hash`) that this extraction did not re-produce; a cut item neither completes nor retires. Hard `opts.signal` (chat call, pacing, checks before each item and commit) leaves no strike, receipt or rollup; soft `opts.stopSignal` stops before the next item.
+- `src/core/cycle/extract-atoms.ts` — the extract_atoms lens phase: mines eligible pages into `atoms/<date>/<stem>-<hash>` pages. The input cap, the cut and the partial disposition of a cut item are in `extract-atoms-input.ts` (#6503). Transcript input is paste-stripped before the cut (#5812); discovery skips self-captures (#5820) and keeps only the session-corpus files `keepSessionSourceTranscripts` (`synthesize-source-routing.ts`, shared with synthesize) resolves to the run's source, reporting the rest in `details.source_routing` (#7846); the meeting transcripts dir is read only for `default`. Discovery and the backlog count append `connectorAtomExclusionSql` (`connector-atoms.ts`), so connector `email`/`meeting` pages stay out only when `cycle.extract_atoms.connector_pages` is false. The type allowlist is `unionExtractableTypes(packExtractable, packExcluded)`: the legacy floor (meeting, source, article, video, book, original) minus the floor types the active pack declares not extractable (`nonExtractableTypesFromPack`, #5756; a declared type without `extractable: true` is a pack decision, an undeclared type keeps the floor), plus the pack's `extractable: true` types, minus synthesis outputs; gbrain-base is unchanged, gbrain-recommended's `source: false` now holds for atoms as it did for facts. Pinned by `test/pack-extractable-floor-5756.test.ts`. A drain attempt passes `opts.attempt` (`budgetTracker`, `writeWait`) so its batches share one budget and one maintenance publish wait. Eligibility reads source-incarnation/page/full-hash-keyed derived scan state; scan and failure markers do not mutate canonical frontmatter. Managed extraction checks source-wide local authority and owner readiness before provider work, admits outputs plus a completion request together, and replays retained output without another model call. A failed managed batch is retried by the next run instead (#6325, `resumeManagedAtoms` in `persistence/atom-maintenance.ts`); an origin held after `MAX_DETERMINISTIC_FAILURES` (now defined in `extract-atoms-page-state.ts`, re-exported here) is skipped with no model call and counted in `details.held_failed`, never in `failures`. Managed atom files keep provisional source hashes; only the source-incarnation/page/full-hash durable completion receipt excludes an item from discovery and backlog counts. Unmanaged atom completion flips and their cardinality check share a transaction after provenance links persist, so a partial match rolls back and stays retryable. A page-derived atom upserts at its locator-folded slug, else a compatible legacy title-only slug; when both miss, the source page's live bound atoms are loaded once per page per run and exactly one JS-lowercased title match is adopted (a case-only title change never mints a duplicate; zero or several matches mint the new slug). Response parsing lives in `extract-atoms-parse.ts`. Atoms of one item that resolve to one slug (a repeated title) publish once, the first kept (#7514: two items at one key failed the generation, managed `generation_live_conflict`, unmanaged a primary-key violation). A managed item whose atom write did not commit is a write failure (`atomWriteFailure`, `persistence/atom-maintenance.ts`): its `failures` entry carries `request_id`, `code`, and its strike, never a provider classification. An undated source dates its atoms by the source page's creation date (`undated` for transcripts), never the run date. A page-derived atom's frontmatter `date` is the source page's content date (`atom-source-date.ts`: the stored `effective_date`, else `computeEffectiveDate`), read with the input before the model call, so classic import and managed publication store that `effective_date` (source `date`) and `extracted_at` stays the processing stamp; transcript-derived atoms carry no `date`. After an unmanaged item completes (`completeUnmanagedItem`), `retireStaleAtoms` soft-deletes that source's atoms from earlier extractions (other or provisional `source_hash`) that this extraction did not re-produce; a cut item neither completes nor retires. Hard `opts.signal` (chat call, pacing, checks before each item and commit) leaves no strike, receipt or rollup; soft `opts.stopSignal` stops before the next item.
Managed completion admits revision-pinned `managed_atom_delete` children alongside replacement pages, removes canonical files through the page deletion path, and commits its checkpoint only after every publication and retirement commits.
Both paths prevent LLM-chosen titles that drift between extractions from piling up.
Atom identity: page-derived atoms fold the source-page slug into the slug hash (8 chars, NUL-separated) so two same-date pages emitting the same title get DISTINCT slugs; the source CONTENT hash is deliberately NOT folded (a reworded source still upserts rather than duping); transcript atoms keep the legacy title-only 6-char hash. `resolvePageAtomSlug` adopts a legacy-slug atom with a compatible binding in place on re-extraction (upgrade idempotency — no migration, no duplicate), while `assertAtomImportBinding` fail-closed refuses to reuse a slug bound to a DIFFERENT source locator. Both share `isCompatibleAtomBinding(frontmatter, sourcePageSlug)` — the ONE definition of compatible: bound to THIS source page, or carrying no `source_slug`/`source_path` at all (pre-binding-era adoption, not a clobber); a `source_path`-bound legacy transcript atom or a different `source_slug` is a different origin, and a non-atom page squatting on the slug is refused. `source_quote` is verified at extraction time against the exact `truncateUtf8` prompt prefix the model saw (`locateQuote` advances by full code points): located quotes persist original characters + [start,end) offsets + `source_quote_verified`; unlocatable quotes are dropped with the atom kept. Edges write before the completion flip; per-item drain failures stay typed (bounded, sanitized, reconcilable). Transcript items carry the same failure-count/tombstone machinery as pages, in the `extract_atoms_transcript_state` table (keyed `(source_id, file_path, content_hash)`, 16-char hash prefix, so an edit is a new row): `recordItemFailureCount` counts a transcript's malformed output or prompt-level provider block (`providerContentBlockReason`; never feeds the outage halt) toward `MAX_DETERMINISTIC_FAILURES`, a zero-yield transcript tombstones immediately (as pages do), and `tombstonedTranscriptsForHashes` is the batch read-side gate mirroring `atomsExistingForHashes` (fail-open on error). Reported as `tombstoned_transcripts` (paths), separate from the page-slug `tombstoned_for_failures`. Malformed managed output requires a newly approved protected-job retry identity; publication retry reuses retained output and original/current grants.
diff --git a/src/commands/sweep.ts b/src/commands/sweep.ts
index f1cb56fff9..473d42ccab 100644
--- a/src/commands/sweep.ts
+++ b/src/commands/sweep.ts
@@ -51,9 +51,9 @@ files recorded their source (on a brain with more than one source), and
sessions whose source never resolved, are held. --assign-corpus previews
which held sessions would be mapped to <source>; --apply records the
mapping (a session that already names a source is never changed), and the
-next sweep extracts them there. Dream synthesis holds the same files, already
-extracted ones included, until they are mapped (#7805). --session limits it
-to one session.
+next sweep extracts them there. Dream synthesis and extract_atoms hold the
+same files, already extracted ones included, until they are mapped (#7805,
+#7846). --session limits it to one session.
Failed windows (#6785): a corpus window whose extraction fails after the
model call (the fence write was refused, the provider errored) is retried
@@ -240,6 +240,6 @@ async function runAssignCorpus(engine: BrainEngine, args: string[]): Promise<voi
for (const s of sessions) console.log(` ${s}`);
if (sessions.length) console.log(`Apply: gbrain sweep --assign-corpus ${sourceId}${onlySession ? ` --session ${onlySession}` : ''} --apply`);
} else {
- console.log(`Mapped ${assigned.length} held session(s) to source ${sourceId}; the next sweep extracts them (gbrain sweep --once) and dream synthesis writes them there.`);
+ console.log(`Mapped ${assigned.length} held session(s) to source ${sourceId}; the next sweep extracts them (gbrain sweep --once) and dream synthesis and extract_atoms write them there.`);
}
}
diff --git a/src/core/cycle/extract-atoms-source-routing.ts b/src/core/cycle/extract-atoms-source-routing.ts
deleted file mode 100644
index 217b8057eb..0000000000
--- a/src/core/cycle/extract-atoms-source-routing.ts
+++ /dev/null
@@ -1,76 +0,0 @@
-/**
- * #7846: extract_atoms writes a captured session's atoms only to the source
- * #6268 froze for it, never to the source the cycle runs for.
- *
- * Every session-corpus file resolves through `resolveCorpusFileSource`, the
- * resolver the facts sweep and dream synthesis use: a spool file's stamp, or
- * its session's frozen record (an unresolved record with a cwd is frozen first
- * from the brain's resolution chain, as the sweep does), and a top-level legacy
- * file only with durable evidence of its source. A file of the cycle's source
- * is extracted; a file of another source is skipped (`corpus_source_other`)
- * and extracts when the phase runs for that source; a file with no evidence of
- * its source is held with the resolver's reason until
- * `gbrain sweep --assign-corpus <source>` maps its session. Nothing falls back
- * to the cycle's source. Files outside the session corpus (the meeting
- * transcripts dir) are not session captures and pass unchanged.
- */
-import { basename, dirname, isAbsolute, relative, resolve } from 'node:path';
-import type { BrainEngine } from '../engine.ts';
-import type { PhaseResult } from '../cycle.ts';
-import { corpusSpoolDir } from '../context/corpus-segments.ts';
-import { resolveCorpusFileSource, type CorpusSourceHold } from '../context/corpus-source.ts';
-
-export interface AtomSourceRouting {
- /** Corpus files of other sources, counted by the source each belongs to. */
- corpus_source_other: Record<string, number>;
- /** Corpus files with no usable evidence of their source, counted by the resolver's hold reason. */
- held: Record<string, number>;
-}
-
-export const emptyAtomSourceRouting = (): AtomSourceRouting => ({ corpus_source_other: {}, held: {} });
-
-/** The holds an operator mapping releases: the session never recorded a source. */
-const MAPPABLE_HOLDS: ReadonlySet<CorpusSourceHold> = new Set(['corpus_source_unresolved', 'corpus_legacy_unstamped']);
-
-function corpusLocation(filePath: string, corpusDir: string): 'spool' | 'legacy' | null {
- const root = resolve(corpusDir);
- const file = resolve(filePath);
- const rel = relative(root, file);
- if (!rel || rel.startsWith('..') || isAbsolute(rel)) return null;
- return dirname(file) === resolve(corpusSpoolDir(root)) ? 'spool' : 'legacy';
-}
-
-/** The transcripts this run may extract into `sourceId`; every other corpus file is counted in `routing`. */
-export async function keepAtomSourceTranscripts<T extends { filePath: string }>(engine: BrainEngine, transcripts: T[],
- opts: { corpusDir: string; sourceId: string }, routing: AtomSourceRouting): Promise<T[]> {
- const kept: T[] = [];
- for (const t of transcripts) {
- const location = corpusLocation(t.filePath, opts.corpusDir);
- if (!location) { kept.push(t); continue; }
- const owner = await resolveCorpusFileSource(engine, opts.corpusDir, basename(t.filePath), location);
- if (!owner.ok) routing.held[owner.reason] = (routing.held[owner.reason] ?? 0) + 1;
- else if (owner.sourceId !== opts.sourceId) routing.corpus_source_other[owner.sourceId] = (routing.corpus_source_other[owner.sourceId] ?? 0) + 1;
- else kept.push(t);
- }
- return kept;
-}
-
-/** Names every corpus file the phase left out, with the command that extracts or releases it. */
-export function withAtomSourceRouting(result: PhaseResult, routing: AtomSourceRouting, sourceId: string): PhaseResult {
- const others = Object.entries(routing.corpus_source_other).sort(([a], [b]) => a.localeCompare(b));
- const held = Object.entries(routing.held).sort(([a], [b]) => a.localeCompare(b));
- if (!others.length && !held.length) return result;
- const otherCount = others.reduce((n, [, c]) => n + c, 0);
- const heldCount = held.reduce((n, [, c]) => n + c, 0);
- const fix = [
- ...others.map(([id]) => `gbrain dream --source ${id} --phase extract_atoms`),
- ...(held.some(([r]) => MAPPABLE_HOLDS.has(r as CorpusSourceHold)) ? ['gbrain sweep --assign-corpus <source>'] : []),
- ];
- const parts = [
- ...(otherCount ? [`${otherCount} corpus file(s) belong to other sources (${others.map(([id, c]) => `${id}: ${c}`).join(', ')}) and extract only under their own source`] : []),
- ...(heldCount ? [`${heldCount} held with no usable session source (${held.map(([r, c]) => `${r}: ${c}`).join(', ')})`] : []),
- ];
- process.stderr.write(`[extract_atoms] source ${sourceId}: ${parts.join('; ')}.${fix.length ? ` Fix: ${fix.join(' | ')}` : ''}\n`);
- return { ...result, summary: `${result.summary}; ${parts.join('; ')}`,
- details: { ...result.details, source_routing: { source_id: sourceId, ...routing, fix } } };
-}
diff --git a/src/core/cycle/extract-atoms.ts b/src/core/cycle/extract-atoms.ts
index c8e5d2b5a7..4dd38be4e7 100644
--- a/src/core/cycle/extract-atoms.ts
+++ b/src/core/cycle/extract-atoms.ts
@@ -67,7 +67,7 @@ import type { ProgressReporter } from '../progress.ts';
import { chat as gatewayChat, withBudgetTracker } from '../ai/gateway.ts';
import { createGlobalLlmHaltTracker, haltedClassOf, providerContentBlockReason, type GlobalLlmErrorClass } from '../ai/errors.ts';
import { derivedWriteThrough, reportFileHolds } from './derived-write-through.ts';
-import { emptyAtomSourceRouting, keepAtomSourceTranscripts, withAtomSourceRouting } from './extract-atoms-source-routing.ts';
+import { emptySessionSourceRouting, keepSessionSourceTranscripts, withSessionSourceRouting } from './synthesize-source-routing.ts';
import { serializeMarkdown } from '../markdown.ts';
import { DEFAULT_EXTRACT_MAX_INPUT_CHARS, extractAtomsInput, inputTruncatedLine, recordPartialDisposition, releasePartialDispositions, resolveExtractAtomsInputCap, stampTranscriptTombstone, type ExtractAtomsInput, type ExtractAtomsItem } from './extract-atoms-input.ts';
import { commitExtractionPlan, readPageWatermark } from '../extraction-watermark.ts';
@@ -715,7 +715,7 @@ export async function runPhaseExtractAtoms(
// v0.41.2.1: config loader switched to loadConfigWithEngine() so the
// dream.* DB-plane merge from Phase 1 reaches this phase.
let transcripts: Array<{ filePath: string; content: string; contentHash: string; root?: { path: string; label: string } }> = opts._transcripts ?? [];
- const sourceRouting = emptyAtomSourceRouting();
+ const sourceRouting = emptySessionSourceRouting();
// Configured transcript paths are brain-global: every source walks the session
// corpus and keeps only the files #6268 resolves to it (#7846); the meeting
// transcripts dir stays default's.
@@ -737,7 +737,7 @@ export async function runPhaseExtractAtoms(
const corpusDir = dream?.synthesize?.session_corpus_dir;
const meetingDir = dream?.synthesize?.meeting_transcripts_dir;
if (corpusDir !== undefined) {
- const discovered = await keepAtomSourceTranscripts(engine, discoverTranscripts({
+ const discovered = await keepSessionSourceTranscripts(engine, discoverTranscripts({
corpusDir,
meetingTranscriptsDir: sourceId === 'default' ? meetingDir : undefined, selfCaptureSessionIds: claudeCliSelfSessionIds(), // #5820, as synthesize
}), { corpusDir, sourceId }, sourceRouting);
@@ -875,7 +875,7 @@ export async function runPhaseExtractAtoms(
// Phase-level no-op: nothing to extract today.
if (work.length === 0 && transcripts.length === 0 && pages.length === 0) {
- return withAtomSourceRouting(reportFileHolds({
+ return withSessionSourceRouting(reportFileHolds({
phase: 'extract_atoms',
status: 'skipped',
duration_ms: 0,
@@ -895,7 +895,7 @@ export async function runPhaseExtractAtoms(
budget_usd: DEFAULT_BUDGET_USD,
dry_run: opts.dryRun ?? false,
},
- }, atomFiles?.held, 'atom'), sourceRouting, sourceId);
+ }, atomFiles?.held, 'atom'), sourceRouting, sourceId, 'extract_atoms');
}
// 4. Per work-item: extract atoms via the configured extract_atoms model
@@ -1422,7 +1422,7 @@ export async function runPhaseExtractAtoms(
});
}
- return withAtomSourceRouting(reportFileHolds({
+ return withSessionSourceRouting(reportFileHolds({
phase: 'extract_atoms',
// A phase that skipped every work item and produced nothing did not
// succeed, even though skips are not failures and leave failures[] empty.
@@ -1469,7 +1469,7 @@ export async function runPhaseExtractAtoms(
source_id: sourceId,
dry_run: opts.dryRun ?? false,
},
- }, atomFiles?.held, 'atom'), sourceRouting, sourceId);
+ }, atomFiles?.held, 'atom'), sourceRouting, sourceId, 'extract_atoms');
}
/**
diff --git a/src/core/cycle/synthesize-source-routing.ts b/src/core/cycle/synthesize-source-routing.ts
index fda0d81f57..92c21549ef 100644
--- a/src/core/cycle/synthesize-source-routing.ts
+++ b/src/core/cycle/synthesize-source-routing.ts
@@ -13,6 +13,8 @@
* falls back to the cycle's source. Files outside the session corpus (the meeting
* transcripts dir, an `--input` file elsewhere) and imported conversation pages
* (read from the cycle's source) are not session captures and pass unchanged.
+ * extract_atoms routes its session-corpus transcripts through the same two
+ * functions (#7846).
*/
import { basename, dirname, isAbsolute, relative, resolve, sep } from 'node:path';
import type { BrainEngine } from '../engine.ts';
@@ -54,22 +56,23 @@ export async function keepSessionSourceTranscripts(engine: BrainEngine, transcri
return kept;
}
-/** Names every corpus file the phase left out, with the command that synthesizes or releases it. */
-export function withSessionSourceRouting(result: PhaseResult, routing: SessionSourceRouting, sourceId: string): PhaseResult {
+/** Names every corpus file the phase left out, with the command that synthesizes (or extracts) or releases it. */
+export function withSessionSourceRouting(result: PhaseResult, routing: SessionSourceRouting, sourceId: string,
+ phase: 'synthesize' | 'extract_atoms' = 'synthesize'): PhaseResult {
const others = Object.entries(routing.corpus_source_other).sort(([a], [b]) => a.localeCompare(b));
const held = Object.entries(routing.held).sort(([a], [b]) => a.localeCompare(b));
if (!others.length && !held.length) return result;
const otherCount = others.reduce((n, [, c]) => n + c, 0);
const heldCount = held.reduce((n, [, c]) => n + c, 0);
const fix = [
- ...others.map(([id]) => `gbrain dream --source ${id} --phase synthesize`),
+ ...others.map(([id]) => `gbrain dream --source ${id} --phase ${phase}`),
...(heldCount ? ['gbrain sweep --assign-corpus <source>'] : []),
];
const parts = [
- ...(otherCount ? [`${otherCount} corpus file(s) belong to other sources (${others.map(([id, c]) => `${id}: ${c}`).join(', ')}) and synthesize only under their own source`] : []),
+ ...(otherCount ? [`${otherCount} corpus file(s) belong to other sources (${others.map(([id, c]) => `${id}: ${c}`).join(', ')}) and ${phase === 'synthesize' ? 'synthesize' : 'extract'} only under their own source`] : []),
...(heldCount ? [`${heldCount} held with no frozen session source (${held.map(([r, c]) => `${r}: ${c}`).join(', ')})`] : []),
];
- process.stderr.write(`[dream] synthesize for source ${sourceId}: ${parts.join('; ')}. Fix: ${fix.join(' | ')}\n`);
+ process.stderr.write(`[dream] ${phase} for source ${sourceId}: ${parts.join('; ')}. Fix: ${fix.join(' | ')}\n`);
return { ...result, summary: `${result.summary}; ${parts.join('; ')}`,
details: { ...result.details, source_routing: { source_id: sourceId, ...routing, fix } } };
}
diff --git a/test/extract-atoms-session-source-7846.test.ts b/test/extract-atoms-session-source-7846.test.ts
index da33127d28..fc0b092a4c 100644
--- a/test/extract-atoms-session-source-7846.test.ts
+++ b/test/extract-atoms-session-source-7846.test.ts
@@ -132,7 +132,7 @@ describe('extract_atoms honors the session\'s frozen source (#7846)', () => {
fix: [`gbrain dream --source ${b} --phase extract_atoms`, 'gbrain sweep --assign-corpus <source>'],
});
expect(first.result.summary).toContain(`2 corpus file(s) belong to other sources (${b}: 2)`);
- expect(first.result.summary).toContain('3 held with no usable session source');
+ expect(first.result.summary).toContain('3 held with no frozen session source');
const second = await extract(b);
expect(second.read).toEqual([recorded, stamped].sort());