Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix: extract the source window's pages instead of walking the checkout (#8573)
garrytan43121c12d01e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex a12b6ec7e0..853619a8a7 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -12,13 +12,14 @@ identifiers and attribution are available in the pre-removal Git revision## [Unreleased]-**A source whose automatic sync is off no longer re-reads every page on every autopilot cycle.** A per-source cycle that runs no sync for its source (`config.syncEnabled=false`, a claimed source awaiting activation, a gbrain-owned content directory, or a source with no checkout) now reconciles facts and emotional weight only for the source's pages changed since its previous cycle, as connector sources already did. Before, `extract_facts` walked the whole source each time, so a sync-disabled source of about 27,000 pages or more hit the 30-minute job timeout on every cycle and never stamped its freshness. Pages an explicit `gbrain sync --source` imports between cycles land in the next cycle's window, including a write whose transaction was still open while a cycle listed its pages. A source with no recorded cycle walks its pages in chunks of 2,000, one chunk per cycle, and resumes where the last one stopped. Reported by @mlahatte (#8573).+**A source whose automatic sync is off no longer re-reads every page on every autopilot cycle.** A per-source cycle that runs no sync for its source (`config.syncEnabled=false`, a claimed source awaiting activation, a gbrain-owned content directory, or a source with no checkout) now reconciles facts, links and timeline, and emotional weight only for the source's pages changed since its previous cycle, as connector sources already did. Before, `extract_facts` walked the whole source each time and `extract` walked its whole checkout, so a sync-disabled source of about 27,000 pages or more hit the 30-minute job timeout on every cycle and never stamped its freshness. Pages an explicit `gbrain sync --source` imports between cycles land in the next cycle's window, including a write whose transaction was still open while a cycle listed its pages. A source with no recorded cycle walks its pages in chunks of 2,000, one chunk per cycle, and resumes where the last one stopped. Reported by @mlahatte (#8573).### Itemized changes- `src/core/minions/handlers/autopilot-cycle.ts`: `sourceChangedSince` is passed to every per-source cycle whose phases leave out `sync` or that has no checkout, not only to connector sources.+- `src/core/cycle.ts`: the `extract` phase of a cycle with a checkout and a source window extracts the window's pages, instead of walking the checkout. (`lint` still reads every file of the checkout, as it does for every source.)- `src/core/cycle/source-window.ts`: with no recorded cycle the window is one first-walk chunk in slug order (`sourceFirstWalk.pageLimit`, 2,000). The chunk's cursor and the walk's start are stamped in the source config (`last_source_cycle_walk_after`, `last_source_cycle_walk_at`); the last chunk stamps the walk's start as the window, so pages changed during the walk are picked up after it, and a failed chunk is retried. The listing instant is moved back to just before the oldest transaction still open on another connection (at most 10 minutes), because `pages.updated_at` is the writer's transaction start.-- `test/sync-disabled-cycle-window.test.ts` (PGLite and Postgres): an idle cycle of a sync-disabled git source walks no page and an explicit sync's fact page is reconciled by the next cycle; a first walk runs in bounded, resumable chunks; a source with no checkout gets the window; a write committed after a cycle's listing lands in the next window.+- `test/sync-disabled-cycle-window.test.ts` (PGLite and Postgres): an idle cycle of a sync-disabled git source walks no page in `extract_facts` or `extract`, and an explicit sync's fact page is reconciled by the next cycle; a first walk runs in bounded, resumable chunks; a source with no checkout gets the window; a write committed after a cycle's listing lands in the next window.## [0.60.569.0] - 2026-11-04diff --git a/src/core/cycle.ts b/src/core/cycle.tsindex 54f1586e17..324eb300b3 100644--- a/src/core/cycle.ts+++ b/src/core/cycle.ts@@ -2391,9 +2391,10 @@ async function runCycleUnscoped(phaseResults.push({ ...result, duration_ms });} else {// If sync didn't run (phases exclude it) or failed, syncPagesAffected- // is undefined → extract falls back to full walk (safe default).+ // is undefined → extract falls back to full walk (safe default), unless+ // the cycle has a source window (#8573: a checkout whose sync is skipped).progress.start('cycle.extract');- const { result, duration_ms } = await timePhase(() => runPhaseExtract(engine, brainDir, dryRun, syncPagesAffected, cycleSignal, cycleSourceId), 'extract');+ const { result, duration_ms } = await timePhase(() => runPhaseExtract(engine, brainDir, dryRun, sourceWindow?.slugs ?? syncPagesAffected, cycleSignal, cycleSourceId), 'extract');result.duration_ms = duration_ms;phaseResults.push(result);progress.finish();diff --git a/test/sync-disabled-cycle-window.test.ts b/test/sync-disabled-cycle-window.test.tsindex 1b8bcf72aa..44f0e99d5e 100644--- a/test/sync-disabled-cycle-window.test.ts+++ b/test/sync-disabled-cycle-window.test.ts@@ -56,7 +56,7 @@ async function cycle(engine: BrainEngine, sourceId: string, id: number, phases:const weight = out.report.phases.find(p => p.phase === 'recompute_emotional_weight');expect(facts?.status).not.toBe('fail');expect(weight?.status).not.toBe('fail');- return { scanned: Number(facts!.details.pagesScanned), weight: weight!.details };+ return { scanned: Number(facts!.details.pagesScanned), weight: weight!.details, extract: out.report.phases.find(p => p.phase === 'extract')?.details };}async function config(engine: BrainEngine, sourceId: string) {@@ -88,6 +88,7 @@ describe('#8573 sync-disabled source cycle window', () => {const idle = await cycle(engine, id, 2);expect(idle.scanned).toBe(0);expect(idle.weight).toMatchObject({ mode: 'incremental', pages_recomputed: 0 });+ expect(idle.extract).toMatchObject({ incremental: true, pages_processed: 0 });// The external scheduler's explicit sync between cycles imports a page with a facts fence.mkdirSync(join(dir, 'people'));@@ -97,6 +98,7 @@ describe('#8573 sync-disabled source cycle window', () => {const after = await cycle(engine, id, 3);expect(after.scanned).toBe(1);expect(after.weight).toMatchObject({ mode: 'incremental', pages_recomputed: 1 });+ expect(after.extract).toMatchObject({ incremental: true, pages_processed: 1 });expect(await liveFacts(engine, id, 'people/alice-example')).toBe(1);// The fact receipt page cycle 3 wrote after its listing is the next window; then the source is idle again.expect((await cycle(engine, id, 4)).scanned).toBe(1);