Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix: extract the source window's pages instead of walking the checkout (#8573)

garrytan43121c12d01e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a12b6ec7e0..853619a8a7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,13 +12,14 @@ identifiers and attribution are available in the pre-removal Git revision
## [Unreleased]
-**A source whose automatic sync is off no longer re-reads every page on every autopilot cycle.** A per-source cycle that runs no sync for its source (`config.syncEnabled=false`, a claimed source awaiting activation, a gbrain-owned content directory, or a source with no checkout) now reconciles facts and emotional weight only for the source's pages changed since its previous cycle, as connector sources already did. Before, `extract_facts` walked the whole source each time, so a sync-disabled source of about 27,000 pages or more hit the 30-minute job timeout on every cycle and never stamped its freshness. Pages an explicit `gbrain sync --source` imports between cycles land in the next cycle's window, including a write whose transaction was still open while a cycle listed its pages. A source with no recorded cycle walks its pages in chunks of 2,000, one chunk per cycle, and resumes where the last one stopped. Reported by @mlahatte (#8573).
+**A source whose automatic sync is off no longer re-reads every page on every autopilot cycle.** A per-source cycle that runs no sync for its source (`config.syncEnabled=false`, a claimed source awaiting activation, a gbrain-owned content directory, or a source with no checkout) now reconciles facts, links and timeline, and emotional weight only for the source's pages changed since its previous cycle, as connector sources already did. Before, `extract_facts` walked the whole source each time and `extract` walked its whole checkout, so a sync-disabled source of about 27,000 pages or more hit the 30-minute job timeout on every cycle and never stamped its freshness. Pages an explicit `gbrain sync --source` imports between cycles land in the next cycle's window, including a write whose transaction was still open while a cycle listed its pages. A source with no recorded cycle walks its pages in chunks of 2,000, one chunk per cycle, and resumes where the last one stopped. Reported by @mlahatte (#8573).
### Itemized changes
- `src/core/minions/handlers/autopilot-cycle.ts`: `sourceChangedSince` is passed to every per-source cycle whose phases leave out `sync` or that has no checkout, not only to connector sources.
+- `src/core/cycle.ts`: the `extract` phase of a cycle with a checkout and a source window extracts the window's pages, instead of walking the checkout. (`lint` still reads every file of the checkout, as it does for every source.)
- `src/core/cycle/source-window.ts`: with no recorded cycle the window is one first-walk chunk in slug order (`sourceFirstWalk.pageLimit`, 2,000). The chunk's cursor and the walk's start are stamped in the source config (`last_source_cycle_walk_after`, `last_source_cycle_walk_at`); the last chunk stamps the walk's start as the window, so pages changed during the walk are picked up after it, and a failed chunk is retried. The listing instant is moved back to just before the oldest transaction still open on another connection (at most 10 minutes), because `pages.updated_at` is the writer's transaction start.
-- `test/sync-disabled-cycle-window.test.ts` (PGLite and Postgres): an idle cycle of a sync-disabled git source walks no page and an explicit sync's fact page is reconciled by the next cycle; a first walk runs in bounded, resumable chunks; a source with no checkout gets the window; a write committed after a cycle's listing lands in the next window.
+- `test/sync-disabled-cycle-window.test.ts` (PGLite and Postgres): an idle cycle of a sync-disabled git source walks no page in `extract_facts` or `extract`, and an explicit sync's fact page is reconciled by the next cycle; a first walk runs in bounded, resumable chunks; a source with no checkout gets the window; a write committed after a cycle's listing lands in the next window.
## [0.60.569.0] - 2026-11-04
diff --git a/src/core/cycle.ts b/src/core/cycle.ts
index 54f1586e17..324eb300b3 100644
--- a/src/core/cycle.ts
+++ b/src/core/cycle.ts
@@ -2391,9 +2391,10 @@ async function runCycleUnscoped(
phaseResults.push({ ...result, duration_ms });
} else {
// If sync didn't run (phases exclude it) or failed, syncPagesAffected
- // is undefined → extract falls back to full walk (safe default).
+ // is undefined → extract falls back to full walk (safe default), unless
+ // the cycle has a source window (#8573: a checkout whose sync is skipped).
progress.start('cycle.extract');
- const { result, duration_ms } = await timePhase(() => runPhaseExtract(engine, brainDir, dryRun, syncPagesAffected, cycleSignal, cycleSourceId), 'extract');
+ const { result, duration_ms } = await timePhase(() => runPhaseExtract(engine, brainDir, dryRun, sourceWindow?.slugs ?? syncPagesAffected, cycleSignal, cycleSourceId), 'extract');
result.duration_ms = duration_ms;
phaseResults.push(result);
progress.finish();
diff --git a/test/sync-disabled-cycle-window.test.ts b/test/sync-disabled-cycle-window.test.ts
index 1b8bcf72aa..44f0e99d5e 100644
--- a/test/sync-disabled-cycle-window.test.ts
+++ b/test/sync-disabled-cycle-window.test.ts
@@ -56,7 +56,7 @@ async function cycle(engine: BrainEngine, sourceId: string, id: number, phases:
const weight = out.report.phases.find(p => p.phase === 'recompute_emotional_weight');
expect(facts?.status).not.toBe('fail');
expect(weight?.status).not.toBe('fail');
- return { scanned: Number(facts!.details.pagesScanned), weight: weight!.details };
+ return { scanned: Number(facts!.details.pagesScanned), weight: weight!.details, extract: out.report.phases.find(p => p.phase === 'extract')?.details };
}
async function config(engine: BrainEngine, sourceId: string) {
@@ -88,6 +88,7 @@ describe('#8573 sync-disabled source cycle window', () => {
const idle = await cycle(engine, id, 2);
expect(idle.scanned).toBe(0);
expect(idle.weight).toMatchObject({ mode: 'incremental', pages_recomputed: 0 });
+ expect(idle.extract).toMatchObject({ incremental: true, pages_processed: 0 });
// The external scheduler's explicit sync between cycles imports a page with a facts fence.
mkdirSync(join(dir, 'people'));
@@ -97,6 +98,7 @@ describe('#8573 sync-disabled source cycle window', () => {
const after = await cycle(engine, id, 3);
expect(after.scanned).toBe(1);
expect(after.weight).toMatchObject({ mode: 'incremental', pages_recomputed: 1 });
+ expect(after.extract).toMatchObject({ incremental: true, pages_processed: 1 });
expect(await liveFacts(engine, id, 'people/alice-example')).toBe(1);
// The fact receipt page cycle 3 wrote after its listing is the next window; then the source is idle again.
expect((await cycle(engine, id, 4)).scanned).toBe(1);