Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix: window every no-sync source cycle and bound its first walk (#8573)
garrytan63b7ad0e8fda
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 0d325349cd..a12b6ec7e0 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -10,6 +10,16 @@ credits are retained; no result has been reassigned to another provider. Originaidentifiers and attribution are available in the pre-removal Git revision`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).+## [Unreleased]++**A source whose automatic sync is off no longer re-reads every page on every autopilot cycle.** A per-source cycle that runs no sync for its source (`config.syncEnabled=false`, a claimed source awaiting activation, a gbrain-owned content directory, or a source with no checkout) now reconciles facts and emotional weight only for the source's pages changed since its previous cycle, as connector sources already did. Before, `extract_facts` walked the whole source each time, so a sync-disabled source of about 27,000 pages or more hit the 30-minute job timeout on every cycle and never stamped its freshness. Pages an explicit `gbrain sync --source` imports between cycles land in the next cycle's window, including a write whose transaction was still open while a cycle listed its pages. A source with no recorded cycle walks its pages in chunks of 2,000, one chunk per cycle, and resumes where the last one stopped. Reported by @mlahatte (#8573).++### Itemized changes++- `src/core/minions/handlers/autopilot-cycle.ts`: `sourceChangedSince` is passed to every per-source cycle whose phases leave out `sync` or that has no checkout, not only to connector sources.+- `src/core/cycle/source-window.ts`: with no recorded cycle the window is one first-walk chunk in slug order (`sourceFirstWalk.pageLimit`, 2,000). The chunk's cursor and the walk's start are stamped in the source config (`last_source_cycle_walk_after`, `last_source_cycle_walk_at`); the last chunk stamps the walk's start as the window, so pages changed during the walk are picked up after it, and a failed chunk is retried. The listing instant is moved back to just before the oldest transaction still open on another connection (at most 10 minutes), because `pages.updated_at` is the writer's transaction start.+- `test/sync-disabled-cycle-window.test.ts` (PGLite and Postgres): an idle cycle of a sync-disabled git source walks no page and an explicit sync's fact page is reconciled by the next cycle; a first walk runs in bounded, resumable chunks; a source with no checkout gets the window; a write committed after a cycle's listing lands in the next window.+## [0.60.569.0] - 2026-11-04**SkillOpt's iterative loop is deprecated and will be removed on or after 2026-12-03.** The loop tied a single rewrite on gbrain-evals' benchmarks at about 1.8 times the cost, and every edit it kept came from its first step, so it is being retired. `gbrain skillopt --loop`, a `--resume` of a run the loop started, `run_skillopt` with `optimizer_mode: "reflect"` and a queued `skillopt` job in loop mode now print the `skillopt_loop_deprecated` coaching notice once (over MCP, once per session) and otherwise run exactly as before. Nothing replaces the loop, so edit the skill directly or run the default one-shot rewrite, which stays. Nothing is removed and no default changes in this release.diff --git a/src/core/cycle.ts b/src/core/cycle.tsindex e9c66d22a1..54f1586e17 100644--- a/src/core/cycle.ts+++ b/src/core/cycle.ts@@ -586,7 +586,7 @@ export interface CycleOpts {* Validated via `assertValidSourceId` in `cycleLockIdFor` (defense-in-depth).*/sourceId?: string;- /** #6803: a connector source's cycle (no sync phase) works on its pages changed after this (null: all); see cycle/source-window.ts. */+ /** #6803, #8573: a source cycle with no sync works on its pages changed after this (null: a bounded first walk); see cycle/source-window.ts. */sourceChangedSince?: string | null;/*** #4700 — bare `gbrain dream` may opt the brain's default-like sourcediff --git a/src/core/cycle/source-window.ts b/src/core/cycle/source-window.tsindex 97ed6b6203..522418309d 100644--- a/src/core/cycle/source-window.ts+++ b/src/core/cycle/source-window.ts@@ -1,35 +1,80 @@/**- * #6803: the changed-page window of a connector source's cycle. A connector- * cycle has no sync phase, so without a window its extract_facts and+ * The changed-page window of a per-source cycle that runs no sync for its+ * source: a connector source (#6803), a source whose automatic sync is skipped+ * (`syncEnabled=false`, activation pending, a gbrain-owned content directory)+ * or a source with no checkout (#8573). With no sync there is no sync-derived+ * page set, so without a window its extract_facts and* recompute_emotional_weight phases took full-walk mode and read every page of- * the brain on every cycle, past the 30-minute job deadline on a large brain.+ * the source on every cycle, past the 30-minute job deadline on a large one.* The window is the source's pages changed after the previous cycle listed its* pages (`last_source_cycle_window_at`), read before this cycle lists them, so- * a page edited while a cycle runs falls in the next one.+ * a page edited while a cycle runs, or by an explicit `gbrain sync --source`+ * between cycles, falls in the next one.+ *+ * A source with no recorded cycle walks its pages in slug order, at most+ * `sourceFirstWalk.pageLimit` per cycle. The cursor+ * (`last_source_cycle_walk_after`) and the instant the walk began+ * (`last_source_cycle_walk_at`) are stamped so the next cycle resumes; the+ * last chunk stamps that first instant as the window, so every page changed+ * during the walk is in the window after it.*/import type { BrainEngine } from '../engine.ts';-export interface SourceCycleWindow { at: string | undefined; slugs: string[] }+export const sourceFirstWalk = { pageLimit: 2_000 };-/** The source's live pages changed after `since` (every page when null), and when the listing began. */+export interface SourceCycleWindow {+ at: string | undefined;+ slugs: string[];+ /** Set on a first walk: when it began, and the last slug of this chunk when pages remain (null: walk complete). */+ walk?: { startedAt: string | undefined; next: string | null };+}++/**+ * The listing instant, moved back to just before the oldest transaction still+ * open on another connection (at most 10 minutes back): `pages.updated_at` is+ * its writer's transaction start, so a page that commits after the listing+ * still lands after the instant.+ */+const LISTING_INSTANT_SQL = `SELECT GREATEST(LEAST(clock_timestamp(),+ (SELECT min(xact_start) FROM pg_stat_activity WHERE datname = current_database() AND pid <> pg_backend_pid()) - interval '1 millisecond'),+ clock_timestamp() - interval '10 minutes')::text AS at`;++/** The source's live pages changed after `since` (one first-walk chunk when null), and when the listing began. */export async function listSourceChangedPages(engine: BrainEngine, sourceId: string, since: string | null): Promise<SourceCycleWindow> {- const at = (await engine.executeRaw<{ at: string }>('SELECT clock_timestamp()::text AS at'))[0]?.at;+ const at = (await engine.executeRaw<{ at: string }>(LISTING_INSTANT_SQL))[0]?.at;+ if (since !== null) {+ const rows = await engine.executeRaw<{ slug: string }>(+ 'SELECT slug FROM pages WHERE source_id=$1 AND deleted_at IS NULL AND updated_at>$2::timestamptz ORDER BY slug',+ [sourceId, since]);+ return { at, slugs: rows.map(row => row.slug) };+ }+ const [state] = await engine.executeRaw<{ after: string | null; started: string | null }>(+ `SELECT NULLIF(config->>'last_source_cycle_walk_after', '') AS after, NULLIF(config->>'last_source_cycle_walk_at', '') AS started+ FROM sources WHERE id=$1`, [sourceId]);+ const resume = state?.after && state.started && Number.isFinite(Date.parse(state.started)) ? state : null;+ const limit = sourceFirstWalk.pageLimit;const rows = await engine.executeRaw<{ slug: string }>(- 'SELECT slug FROM pages WHERE source_id=$1 AND deleted_at IS NULL AND ($2::timestamptz IS NULL OR updated_at>$2::timestamptz) ORDER BY slug',- [sourceId, since]);- return { at, slugs: rows.map(row => row.slug) };+ 'SELECT slug FROM pages WHERE source_id=$1 AND deleted_at IS NULL AND ($2::text IS NULL OR slug>$2) ORDER BY slug LIMIT $3',+ [sourceId, resume?.after ?? null, limit + 1]);+ const slugs = rows.slice(0, limit).map(row => row.slug);+ return { at, slugs, walk: { startedAt: resume?.started ?? at, next: rows.length > limit ? slugs[slugs.length - 1]! : null } };}/*** The config stamp that starts the next window: this listing's instant, or,* when a phase that consumed the window failed, the same window again- * ('' = every page of the source).+ * ('' = the first walk, resumed from its stamped cursor). A first-walk chunk+ * stamps its cursor; the last one stamps the walk's first instant.*/export function sourceCycleWindowStamp(phases: ReadonlyArray<{ phase: string; status: string }>, since: string | null | undefined,- window: SourceCycleWindow | undefined): { last_source_cycle_window_at?: string } {+ window: SourceCycleWindow | undefined): Record<string, string | null> {if (!window?.at) return {};const failed = phases.some(r => (r.phase === 'extract_facts' || r.phase === 'recompute_emotional_weight') && r.status === 'fail');- return { last_source_cycle_window_at: failed ? (since ?? '') : window.at };+ if (failed) return { last_source_cycle_window_at: since ?? '' };+ if (!window.walk) return { last_source_cycle_window_at: window.at };+ return window.walk.next !== null+ ? { last_source_cycle_window_at: '', last_source_cycle_walk_after: window.walk.next, last_source_cycle_walk_at: window.walk.startedAt ?? window.at }+ : { last_source_cycle_window_at: window.walk.startedAt ?? window.at, last_source_cycle_walk_after: null, last_source_cycle_walk_at: null };}/** The union of the incremental slug sets the cycle collected, or undefined (full walk) when none was collected. */diff --git a/src/core/minions/handlers/autopilot-cycle.ts b/src/core/minions/handlers/autopilot-cycle.tsindex 976675e32e..712feb2ecb 100644--- a/src/core/minions/handlers/autopilot-cycle.ts+++ b/src/core/minions/handlers/autopilot-cycle.ts@@ -163,8 +163,10 @@ export function makeAutopilotCycleHandler(engine: BrainEngine): MinionHandler {deadlineAtMs: job.deadlineAtMs, // #2781: phases budget sub-work from remaining timeprivateQueueOwnerJobId: job.id,...(sourceId ? { sourceId } : {}),- // #6803: a connector cycle has no sync phase; its incremental set is the source's pages changed since its last cycle.- ...(connectorSource ? { sourceChangedSince: lastSourceCycleAt } : {}),+ // #6803, #8573: a cycle that runs no sync for its source (a connector, a skipped automatic sync, no checkout) has no+ // sync-derived page set; its incremental set is the source's pages changed since its last cycle.+ ...(sourceId && (effectiveBrainDir === null || (effectivePhases !== undefined && !effectivePhases.includes('sync')))+ ? { sourceChangedSince: lastSourceCycleAt } : {}),...(effectivePhases !== undefined ? { phases: effectivePhases as any } : {}),// #7241: the phase in progress, for `gbrain jobs get` / get_job_progress while the cycle runs.onPhaseStart: (phase, index, count) => reportJobProgress(job, { phase, phase_index: index, phase_count: count, started_at: new Date().toISOString() }),diff --git a/test/postgres-unit-arms.txt b/test/postgres-unit-arms.txtindex 3d8b84b351..4319a50867 100644--- a/test/postgres-unit-arms.txt+++ b/test/postgres-unit-arms.txt@@ -505,6 +505,7 @@ test/sync-apply-held-deletions-gbra133.test.tstest/sync-capacity-preflight.test.tstest/sync-checkpoint-moved-6834.test.tstest/sync-classic-drift-7199.test.ts+test/sync-disabled-cycle-window.test.tstest/sync-discovery-legacy-origin-7130.test.tstest/sync-drain-claims.test.tstest/sync-drain-effect-recovery-blocked.test.tsdiff --git a/test/sync-disabled-cycle-window.test.ts b/test/sync-disabled-cycle-window.test.tsnew file mode 100644index 0000000000..1b8bcf72aa--- /dev/null+++ b/test/sync-disabled-cycle-window.test.ts@@ -0,0 +1,195 @@+/**+ * #8573: a per-source cycle that runs no sync for its source works on the+ * source's pages changed since its previous cycle. A git source with+ * `config.syncEnabled=false` (an external scheduler runs+ * `gbrain sync --source` itself) got the freshness phases without `sync` and+ * no changed-page window, so extract_facts and recompute_emotional_weight+ * walked every page of the source on every cycle; past about 27k pages each+ * cycle died at the 30-minute job timeout and never stamped. A source with no+ * recorded cycle walks its pages in bounded, resumable chunks.+ *+ * Real handler, cycle and sync on every configured engine. The fixture+ * toggles the brain-wide persistence switch and process env.+ */+import { afterAll, afterEach, beforeAll, describe, expect, test } from 'bun:test';+import postgres from '#postgres';+import { execFileSync } from 'node:child_process';+import { mkdirSync, writeFileSync } from 'node:fs';+import { join } from 'node:path';+import type { BrainEngine } from '../src/core/engine.ts';+import { makeAutopilotCycleHandler } from '../src/core/minions/handlers/autopilot-cycle.ts';+import { SOURCE_FRESHNESS_PHASES } from '../src/core/cycle/phase-scope.ts';+import { sourceFirstWalk } from '../src/core/cycle/source-window.ts';+import { performSync } from '../src/commands/sync.ts';+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';+import { withEnv } from './helpers/with-env.ts';+import { createConnectorFixture } from './helpers/connector-fixture.ts';++const fixture = createConnectorFixture();+const { engines, env, source } = fixture;+beforeAll(fixture.setup, 120_000);+afterAll(fixture.teardown);+const DEFAULT_PAGE_LIMIT = sourceFirstWalk.pageLimit;+afterEach(() => { sourceFirstWalk.pageLimit = DEFAULT_PAGE_LIMIT; });++const NO_SYNC_PHASES = SOURCE_FRESHNESS_PHASES.filter(p => p !== 'sync');+const PAGES = 12;+type Phase = { phase: string; status: string; details: Record<string, unknown> };+type Out = { status: string; report: { phases: Phase[] } };++const git = (root: string, ...args: string[]) => execFileSync('git', ['-C', root, ...args], { stdio: 'pipe' });+const commit = (root: string, message: string) => {+ git(root, 'add', '-A');+ git(root, '-c', 'user.name=Example', '-c', '[email protected]', 'commit', '-qm', message);+};+const note = (i: number) => `---\ntitle: Note ${i}\ntype: note\n---\nSynthetic note ${i}.\n`;+const factPage = `---\ntitle: Alice Example\ntype: person\n---\n# Alice Example\n\nBody.\n\n## Facts\n\n<!--- gbrain:facts:begin -->+| # | claim | kind | confidence | visibility | notability | valid_from | valid_until | source | context |+|---|-------|------|------------|------------|------------|------------|-------------|--------|---------|+| 1 | Founded Acme Example | fact | 1.0 | world | high | 2017-01-01 | | example | |+<!--- gbrain:facts:end -->+`;++async function cycle(engine: BrainEngine, sourceId: string, id: number, phases: readonly string[] | undefined = NO_SYNC_PHASES) {+ const out = await makeAutopilotCycleHandler(engine)({ id, data: { source_id: sourceId, ...(phases ? { phases } : {}) } } as never) as Out;+ const facts = out.report.phases.find(p => p.phase === 'extract_facts');+ const weight = out.report.phases.find(p => p.phase === 'recompute_emotional_weight');+ expect(facts?.status).not.toBe('fail');+ expect(weight?.status).not.toBe('fail');+ return { scanned: Number(facts!.details.pagesScanned), weight: weight!.details };+}++async function config(engine: BrainEngine, sourceId: string) {+ return (await engine.executeRaw<{ config: Record<string, unknown> }>('SELECT config FROM sources WHERE id=$1', [sourceId]))[0]!.config;+}++async function liveFacts(engine: BrainEngine, sourceId: string, slug: string) {+ return Number((await engine.executeRaw<{ n: number }>(+ 'SELECT count(*)::int AS n FROM facts WHERE source_id=$1 AND source_markdown_slug=$2 AND expired_at IS NULL', [sourceId, slug]))[0]!.n);+}++describe('#8573 sync-disabled source cycle window', () => {+ test('a sync-disabled git source reconciles only pages changed since its last cycle; an explicit sync between cycles lands in the next window', async () => withEnv(env, async () => {+ for (const engine of engines) {+ const { id, dir } = await source(engine, { syncEnabled: false });+ await engine.executeRaw('UPDATE persistence_brain SET enabled=false WHERE singleton=1');+ try {+ git(dir, 'init', '-q');+ mkdirSync(join(dir, 'notes'));+ for (let i = 0; i < PAGES; i++) writeFileSync(join(dir, 'notes', `n-${i}.md`), note(i));+ commit(dir, 'seed');+ await performSync(engine, { repoPath: dir, sourceId: id, noPull: true, noEmbed: true, noExtract: true, concurrency: 1 });+ // The source's earlier cycles stamped freshness before its automatic sync was turned off.+ await engine.updateSourceConfig(id, { last_source_cycle_at: new Date(Date.now() - 60_000).toISOString() });++ const first = await cycle(engine, id, 1);+ expect(first.weight).toMatchObject({ mode: 'incremental' });+ // Nothing changed since: no page is walked (pre-fix: every page of the source, every cycle).+ const idle = await cycle(engine, id, 2);+ expect(idle.scanned).toBe(0);+ expect(idle.weight).toMatchObject({ mode: 'incremental', pages_recomputed: 0 });++ // The external scheduler's explicit sync between cycles imports a page with a facts fence.+ mkdirSync(join(dir, 'people'));+ writeFileSync(join(dir, 'people', 'alice-example.md'), factPage);+ commit(dir, 'fact page');+ await performSync(engine, { repoPath: dir, sourceId: id, noPull: true, noEmbed: true, noExtract: true, concurrency: 1 });+ const after = await cycle(engine, id, 3);+ expect(after.scanned).toBe(1);+ expect(after.weight).toMatchObject({ mode: 'incremental', pages_recomputed: 1 });+ expect(await liveFacts(engine, id, 'people/alice-example')).toBe(1);+ // The fact receipt page cycle 3 wrote after its listing is the next window; then the source is idle again.+ expect((await cycle(engine, id, 4)).scanned).toBe(1);+ expect((await cycle(engine, id, 5)).scanned).toBe(0);+ } finally {+ await disposePersistenceConsumer(engine);+ await engine.executeRaw('UPDATE persistence_brain SET enabled=true WHERE singleton=1');+ }+ }+ }), 180_000);++ test('a source with no recorded cycle walks its pages in bounded chunks and resumes; a page changed during the walk lands after it', async () => withEnv(env, async () => {+ for (const engine of engines) {+ const { id } = await source(engine, { syncEnabled: false });+ await engine.executeRaw('UPDATE persistence_brain SET enabled=false WHERE singleton=1');+ try {+ for (let i = 0; i < 5; i++) {+ await engine.putPage(`walk/n-${i}`, { title: `Note ${i}`, type: 'note', compiled_truth: `Body ${i}.` }, { sourceId: id });+ }+ sourceFirstWalk.pageLimit = 2;+ const one = await cycle(engine, id, 1);+ expect(one.scanned).toBe(2);+ expect(one.weight).toMatchObject({ mode: 'incremental', pages_recomputed: 2 });+ expect(await config(engine, id)).toMatchObject({ last_source_cycle_window_at: '', last_source_cycle_walk_after: 'walk/n-1' });+ const startedAt = (await config(engine, id)).last_source_cycle_walk_at as string;+ expect(Number.isFinite(Date.parse(startedAt))).toBe(true);++ // Edited after the first chunk passed it: the walk does not revisit it, the window after the walk does.+ await engine.putPage('walk/n-0', { title: 'Note 0', type: 'note', compiled_truth: 'Body 0, edited.' }, { sourceId: id });+ expect((await cycle(engine, id, 2)).weight).toMatchObject({ pages_recomputed: 2 });+ expect(await config(engine, id)).toMatchObject({ last_source_cycle_walk_after: 'walk/n-3', last_source_cycle_walk_at: startedAt });+ expect((await cycle(engine, id, 3)).weight).toMatchObject({ pages_recomputed: 1 });+ expect(await config(engine, id)).toMatchObject({ last_source_cycle_window_at: startedAt, last_source_cycle_walk_after: null, last_source_cycle_walk_at: null });++ const next = await cycle(engine, id, 4);+ expect(next.weight).toMatchObject({ mode: 'incremental', pages_recomputed: 1 });+ expect((await cycle(engine, id, 5)).weight).toMatchObject({ pages_recomputed: 0 });+ } finally {+ await disposePersistenceConsumer(engine);+ await engine.executeRaw('UPDATE persistence_brain SET enabled=true WHERE singleton=1');+ }+ }+ }), 180_000);++ test('a source with no checkout (repo-less) gets the window too: writes between cycles land, an idle cycle walks nothing', async () => withEnv(env, async () => {+ for (const engine of engines) {+ const { id } = await source(engine, {});+ await engine.executeRaw('UPDATE persistence_brain SET enabled=false WHERE singleton=1');+ try {+ await engine.executeRaw('UPDATE sources SET local_path=NULL WHERE id=$1', [id]);+ for (let i = 0; i < PAGES; i++) {+ await engine.putPage(`db-only/n-${i}`, { title: `Note ${i}`, type: 'note', compiled_truth: `Body ${i}.` }, { sourceId: id });+ }+ await cycle(engine, id, 1, SOURCE_FRESHNESS_PHASES);+ const idle = await cycle(engine, id, 2, SOURCE_FRESHNESS_PHASES);+ expect(idle.scanned).toBe(0);+ expect(idle.weight).toMatchObject({ mode: 'incremental', pages_recomputed: 0 });+ await engine.putPage('db-only/n-3', { title: 'Note 3', type: 'note', compiled_truth: 'Body 3, edited.' }, { sourceId: id });+ expect((await cycle(engine, id, 3, SOURCE_FRESHNESS_PHASES)).weight).toMatchObject({ mode: 'incremental', pages_recomputed: 1 });+ } finally {+ await disposePersistenceConsumer(engine);+ await engine.executeRaw('UPDATE persistence_brain SET enabled=true WHERE singleton=1');+ }+ }+ }), 180_000);+ test('a write whose transaction began before a cycle listed its pages and committed after lands in the next window (Postgres)', async () => withEnv(env, async () => {+ const engine = engines.find(e => e.kind === 'postgres');+ if (!engine) return;+ const { id } = await source(engine, { syncEnabled: false });+ await engine.executeRaw('UPDATE persistence_brain SET enabled=false WHERE singleton=1');+ const [{ name }] = await engine.executeRaw<{ name: string }>('SELECT current_database() AS name');+ const url = new URL(process.env.DATABASE_URL!);+ url.pathname = `/${name}`;+ const writer = postgres(url.toString(), { max: 1, prepare: false });+ try {+ for (let i = 0; i < 3; i++) {+ await engine.putPage(`race/n-${i}`, { title: `Note ${i}`, type: 'note', compiled_truth: `Body ${i}.` }, { sourceId: id });+ }+ await cycle(engine, id, 1);+ expect((await cycle(engine, id, 2)).weight).toMatchObject({ pages_recomputed: 0 });+ // pages.updated_at is the writer's transaction start, which is before the next cycle's listing.+ const reserved = await writer.reserve();+ await reserved.unsafe('BEGIN');+ await reserved.unsafe("UPDATE pages SET compiled_truth='Body 1, edited.', updated_at=now() WHERE source_id=$1 AND slug='race/n-1'", [id]);+ await Bun.sleep(20);+ expect((await cycle(engine, id, 3)).weight).toMatchObject({ pages_recomputed: 0 });+ await reserved.unsafe('COMMIT');+ reserved.release();+ expect((await cycle(engine, id, 4)).weight).toMatchObject({ mode: 'incremental', pages_recomputed: 1 });+ } finally {+ await writer.end();+ await disposePersistenceConsumer(engine);+ await engine.executeRaw('UPDATE persistence_brain SET enabled=true WHERE singleton=1');+ }+ }), 180_000);+});