Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix: bound every source window, advance it only when its phases ran (#8573)
garrytan2f2a3454455e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e0875a9fff..5a0cd8b083 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -12,14 +12,14 @@ identifiers and attribution are available in the pre-removal Git revision## [Unreleased]-**A source whose automatic sync is off no longer re-reads every page on every autopilot cycle.** A per-source cycle that runs no sync for its source (`config.syncEnabled=false`, a claimed source awaiting activation, a gbrain-owned content directory, or a source with no checkout) now reconciles facts, links and timeline, and emotional weight only for the source's pages changed since its previous cycle, as connector sources already did. Before, `extract_facts` walked the whole source each time and `extract` walked its whole checkout, so a sync-disabled source of about 27,000 pages or more hit the 30-minute job timeout on every cycle and never stamped its freshness. Pages an explicit `gbrain sync --source` imports between cycles land in the next cycle's window, including a write whose transaction was still open while a cycle listed its pages. A source with no recorded cycle walks its pages in chunks of 2,000, one chunk per cycle, and resumes where the last one stopped. Reported by @mlahatte (#8573).+**A source whose automatic sync is off no longer extracts and reconciles every page on every autopilot cycle.** A per-source cycle that runs no sync for its source (`config.syncEnabled=false`, a claimed source awaiting activation, a gbrain-owned content directory, or a source with no checkout) now extracts links and timeline, reconciles facts and recomputes emotional weight only for the source's pages changed since its previous cycle, as connector sources already did, at most 2,000 pages per cycle. Before, `extract_facts` walked the whole source each time and `extract` walked its whole checkout, so a sync-disabled source of about 27,000 pages or more hit the 30-minute job timeout on every cycle and never stamped its freshness. Pages an explicit `gbrain sync --source` imports between cycles land in the next window, and a large import is worked through over several cycles instead of timing out. A source with no recorded cycle starts from its oldest page and walks forward the same way. Reported by @mlahatte (#8573).### Itemized changes- `src/core/minions/handlers/autopilot-cycle.ts`: `sourceChangedSince` is passed to every per-source cycle whose phases leave out `sync` or that has no checkout, not only to connector sources.-- `src/core/cycle.ts`: the `extract` phase of a cycle with a checkout and a source window extracts the window's pages, instead of walking the checkout. (`lint` still reads every file of the checkout, as it does for every source.)-- `src/core/cycle/source-window.ts`: with no recorded cycle the window is one first-walk chunk in slug order (`sourceFirstWalk.pageLimit`, 2,000). The chunk's cursor and the walk's start are stamped in the source config (`last_source_cycle_walk_after`, `last_source_cycle_walk_at`); the last chunk stamps the walk's start as the window, so pages changed during the walk are picked up after it. A cycle whose `extract`, `extract_facts` or `recompute_emotional_weight` phase failed keeps its window, so the same pages (or the same chunk) are retried. The listing instant is moved back to just before the oldest transaction still open on another connection (at most 10 minutes), because `pages.updated_at` is the writer's transaction start.-- `test/sync-disabled-cycle-window.test.ts` (PGLite and Postgres): an idle cycle of a sync-disabled git source walks no page in `extract_facts` or `extract`, and an explicit sync's fact page is reconciled by the next cycle; a first walk runs in bounded, resumable chunks; a source with no checkout gets the window; a write committed after a cycle's listing lands in the next window.+- `src/core/cycle/source-window.ts`: the window lists the source's pages after a cursor in (`updated_at`, slug) order, `sourceWindow.pageLimit` (2,000) at a time. A cycle that reaches the end of the window stamps its listing instant (`last_source_cycle_window_at`); one that stops at the limit stamps its last page (`last_source_cycle_window_slug` breaks ties), so the next cycle resumes there. The window advances only when `extract`, `extract_facts` and `recompute_emotional_weight` all ran without failing, so a cycle that failed one of them, or ran without them, gets the same window next time. The listing instant is moved back to just before the oldest transaction still open on another connection, because `pages.updated_at` is the writer's transaction start: a write still open while a cycle listed its pages lands in the next window. That holds for writers whose transactions the cycle's database role can see in `pg_stat_activity` (the same role, or one with `pg_read_all_stats`) and that stay open less than 10 minutes. Cursors bind as text, so Postgres keeps their microseconds.+- `src/core/cycle.ts`: the `extract` phase of a cycle with a checkout and a source window extracts the window's pages instead of walking the checkout. A file edited on disk without a sync is no longer picked up by the cycle's extract on such a source, as on any source whose sync runs; `lint` still reads every file of the checkout, as it does for every source.+- `test/sync-disabled-cycle-window.test.ts` (PGLite and Postgres): an idle cycle of a sync-disabled git source walks no page in `extract_facts` or `extract`, and a page with a facts fence that an explicit sync imports is reconciled by the next cycle; a first walk and a large re-import are worked through in bounded windows that resume; a cycle without the window phases, or with one failed, keeps the window; a source with no checkout gets the window; a write committed after a cycle's listing lands in the next window.## [0.60.569.0] - 2026-11-04diff --git a/src/core/cycle.ts b/src/core/cycle.tsindex 324eb300b3..74bdfe5a30 100644--- a/src/core/cycle.ts+++ b/src/core/cycle.ts@@ -586,7 +586,7 @@ export interface CycleOpts {* Validated via `assertValidSourceId` in `cycleLockIdFor` (defense-in-depth).*/sourceId?: string;- /** #6803, #8573: a source cycle with no sync works on its pages changed after this (null: a bounded first walk); see cycle/source-window.ts. */+ /** #6803, #8573: a source cycle with no sync works on a bounded window of its pages changed after this (null: from its oldest page); see cycle/source-window.ts. */sourceChangedSince?: string | null;/*** #4700 — bare `gbrain dream` may opt the brain's default-like source@@ -3183,7 +3183,7 @@ async function runCycleUnscoped(await engine.updateSourceConfig(opts.sourceId, {last_source_cycle_at: nowIso,last_full_cycle_at: nowIso,- ...sourceCycleWindowStamp(phaseResults, opts.sourceChangedSince, sourceWindow),+ ...sourceCycleWindowStamp(phaseResults, sourceWindow),});} catch (e) {const message = e instanceof Error ? e.message : String(e);diff --git a/src/core/cycle/source-window.ts b/src/core/cycle/source-window.tsindex cbc70913d2..aad57169f4 100644--- a/src/core/cycle/source-window.ts+++ b/src/core/cycle/source-window.ts@@ -3,79 +3,76 @@* source: a connector source (#6803), a source whose automatic sync is skipped* (`syncEnabled=false`, activation pending, a gbrain-owned content directory)* or a source with no checkout (#8573). With no sync there is no sync-derived- * page set, so without a window its extract_facts and+ * page set, so without a window its extract, extract_facts and* recompute_emotional_weight phases took full-walk mode and read every page of* the source on every cycle, past the 30-minute job deadline on a large one.- * The window is the source's pages changed after the previous cycle listed its- * pages (`last_source_cycle_window_at`), read before this cycle lists them, so- * a page edited while a cycle runs, or by an explicit `gbrain sync --source`- * between cycles, falls in the next one.*- * A source with no recorded cycle walks its pages in slug order, at most- * `sourceFirstWalk.pageLimit` per cycle. The cursor- * (`last_source_cycle_walk_after`) and the instant the walk began- * (`last_source_cycle_walk_at`) are stamped so the next cycle resumes; the- * last chunk stamps that first instant as the window, so every page changed- * during the walk is in the window after it.+ * The window is the source's pages after the cursor the previous cycle stamped+ * (`last_source_cycle_window_at`, with `last_source_cycle_window_slug` as the+ * tie-break), in (`updated_at`, slug) order, at most `sourceWindow.pageLimit`+ * per cycle. A cycle that lists the rest of the window stamps its listing+ * instant; one that stops at the limit stamps its last page, so the next cycle+ * resumes there. A source with no recorded cycle starts from its oldest page.+ * A page edited while a cycle runs, or by an explicit `gbrain sync --source`+ * between cycles, moves past the cursor and falls in a later window.*/import type { BrainEngine } from '../engine.ts';-export const sourceFirstWalk = { pageLimit: 2_000 };+export const sourceWindow = { pageLimit: 2_000 };++/** The phases that work on the window's pages; the window advances only when every one of them ran without failing. */+const WINDOW_CONSUMERS = ['extract', 'extract_facts', 'recompute_emotional_weight'];export interface SourceCycleWindow {- at: string | undefined;+ /** The cursor the next window starts after (`next`), and the one this window started after (`since`). */+ next: { at: string; slug: string } | undefined;+ since: { at: string | null; slug: string };slugs: string[];- /** Set on a first walk: when it began, and the last slug of this chunk when pages remain (null: walk complete). */- walk?: { startedAt: string | undefined; next: string | null };}/*** The listing instant, moved back to just before the oldest transaction still- * open on another connection (at most 10 minutes back): `pages.updated_at` is- * its writer's transaction start, so a page that commits after the listing- * still lands after the instant.+ * open on another connection the cycle's role can see (at most 10 minutes+ * back): `pages.updated_at` is its writer's transaction start, so a page that+ * commits after the listing still lands after the cursor.*/const LISTING_INSTANT_SQL = `SELECT GREATEST(LEAST(clock_timestamp(),(SELECT min(xact_start) FROM pg_stat_activity WHERE datname = current_database() AND pid <> pg_backend_pid()) - interval '1 millisecond'),clock_timestamp() - interval '10 minutes')::text AS at`;-/** The source's live pages changed after `since` (one first-walk chunk when null), and when the listing began. */+/**+ * One window of the source's live pages after `since` (every page when null), and the cursor the next one starts after.+ * Cursors bind as text: a driver that reads a timestamp parameter as a JS date keeps only milliseconds.+ */export async function listSourceChangedPages(engine: BrainEngine, sourceId: string, since: string | null): Promise<SourceCycleWindow> {- const at = (await engine.executeRaw<{ at: string }>(LISTING_INSTANT_SQL))[0]?.at;- if (since !== null) {- const rows = await engine.executeRaw<{ slug: string }>(- 'SELECT slug FROM pages WHERE source_id=$1 AND deleted_at IS NULL AND updated_at>$2::timestamptz ORDER BY slug',- [sourceId, since]);- return { at, slugs: rows.map(row => row.slug) };- }- const [state] = await engine.executeRaw<{ after: string | null; started: string | null }>(- `SELECT NULLIF(config->>'last_source_cycle_walk_after', '') AS after, NULLIF(config->>'last_source_cycle_walk_at', '') AS started- FROM sources WHERE id=$1`, [sourceId]);- const resume = state?.after && state.started && Number.isFinite(Date.parse(state.started)) ? state : null;- const limit = sourceFirstWalk.pageLimit;- const rows = await engine.executeRaw<{ slug: string }>(- 'SELECT slug FROM pages WHERE source_id=$1 AND deleted_at IS NULL AND ($2::text IS NULL OR slug>$2) ORDER BY slug LIMIT $3',- [sourceId, resume?.after ?? null, limit + 1]);- const slugs = rows.slice(0, limit).map(row => row.slug);- return { at, slugs, walk: { startedAt: resume?.started ?? at, next: rows.length > limit ? slugs[slugs.length - 1]! : null } };+ const [{ at } = { at: undefined }] = await engine.executeRaw<{ at: string }>(LISTING_INSTANT_SQL);+ const [state] = await engine.executeRaw<{ stamped: string | null; slug: string | null }>(+ `SELECT config->>'last_source_cycle_window_at' AS stamped, config->>'last_source_cycle_window_slug' AS slug FROM sources WHERE id=$1`, [sourceId]);+ const slug = since !== null && state?.stamped === since ? state.slug ?? '' : '';+ const limit = sourceWindow.pageLimit;+ const rows = await engine.executeRaw<{ slug: string; updated_at: string }>(+ `SELECT slug, updated_at::text AS updated_at FROM pages+ WHERE source_id=$1 AND deleted_at IS NULL AND ($2::text IS NULL OR (updated_at, slug) > ($2::text::timestamptz, $3::text))+ ORDER BY updated_at, slug LIMIT $4`,+ [sourceId, since, slug, limit + 1]);+ const window = { since: { at: since, slug }, slugs: rows.slice(0, limit).map(row => row.slug) };+ if (!at) return { ...window, next: undefined };+ if (rows.length <= limit) return { ...window, next: { at, slug: '' } };+ const last = rows[limit - 1]!;+ const [{ before }] = await engine.executeRaw<{ before: boolean }>('SELECT $1::text::timestamptz < $2::text::timestamptz AS before', [at, last.updated_at]);+ return { ...window, next: before ? { at, slug: '' } : { at: last.updated_at, slug: last.slug } };}/**- * The config stamp that starts the next window: this listing's instant, or,- * when a phase that consumed the window (extract, extract_facts,- * recompute_emotional_weight) failed, the same window again- * ('' = the first walk, resumed from its stamped cursor). A first-walk chunk- * stamps its cursor; the last one stamps the walk's first instant.+ * The config stamp that starts the next window: its cursor, or, when a phase+ * that works on the window failed or did not run, the same window again ('' =+ * from the source's oldest page).*/-export function sourceCycleWindowStamp(phases: ReadonlyArray<{ phase: string; status: string }>, since: string | null | undefined,- window: SourceCycleWindow | undefined): Record<string, string | null> {- if (!window?.at) return {};- const failed = phases.some(r => (r.phase === 'extract' || r.phase === 'extract_facts' || r.phase === 'recompute_emotional_weight') && r.status === 'fail');- if (failed) return { last_source_cycle_window_at: since ?? '' };- if (!window.walk) return { last_source_cycle_window_at: window.at };- return window.walk.next !== null- ? { last_source_cycle_window_at: '', last_source_cycle_walk_after: window.walk.next, last_source_cycle_walk_at: window.walk.startedAt ?? window.at }- : { last_source_cycle_window_at: window.walk.startedAt ?? window.at, last_source_cycle_walk_after: null, last_source_cycle_walk_at: null };+export function sourceCycleWindowStamp(phases: ReadonlyArray<{ phase: string; status: string }>, window: SourceCycleWindow | undefined): Record<string, string> {+ if (!window?.next) return {};+ const consumed = WINDOW_CONSUMERS.every(name => phases.some(r => r.phase === name && r.status !== 'fail'));+ const cursor = consumed ? window.next : { at: window.since.at ?? '', slug: window.since.slug };+ return { last_source_cycle_window_at: cursor.at, last_source_cycle_window_slug: cursor.slug };}/** The union of the incremental slug sets the cycle collected, or undefined (full walk) when none was collected. */diff --git a/test/sync-disabled-cycle-window.test.ts b/test/sync-disabled-cycle-window.test.tsindex b789efded1..5994e5e453 100644--- a/test/sync-disabled-cycle-window.test.ts+++ b/test/sync-disabled-cycle-window.test.ts@@ -19,7 +19,7 @@ import { join } from 'node:path';import type { BrainEngine } from '../src/core/engine.ts';import { makeAutopilotCycleHandler } from '../src/core/minions/handlers/autopilot-cycle.ts';import { SOURCE_FRESHNESS_PHASES } from '../src/core/cycle/phase-scope.ts';-import { sourceCycleWindowStamp, sourceFirstWalk } from '../src/core/cycle/source-window.ts';+import { sourceCycleWindowStamp, sourceWindow } from '../src/core/cycle/source-window.ts';import { performSync } from '../src/commands/sync.ts';import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';import { withEnv } from './helpers/with-env.ts';@@ -29,8 +29,8 @@ const fixture = createConnectorFixture();const { engines, env, source } = fixture;beforeAll(fixture.setup, 120_000);afterAll(fixture.teardown);-const DEFAULT_PAGE_LIMIT = sourceFirstWalk.pageLimit;-afterEach(() => { sourceFirstWalk.pageLimit = DEFAULT_PAGE_LIMIT; });+const DEFAULT_PAGE_LIMIT = sourceWindow.pageLimit;+afterEach(() => { sourceWindow.pageLimit = DEFAULT_PAGE_LIMIT; });const NO_SYNC_PHASES = SOURCE_FRESHNESS_PHASES.filter(p => p !== 'sync');const PAGES = 12;@@ -69,13 +69,17 @@ async function liveFacts(engine: BrainEngine, sourceId: string, slug: string) {}describe('#8573 sync-disabled source cycle window', () => {- test('a failed phase that read the window keeps it, extract included; a failed first-walk chunk keeps its cursor', () => {- const window = { at: '2026-01-02T00:00:00Z', slugs: ['a'] };+ test('the window advances only when every phase that works on it ran without failing', () => {+ const window = { since: { at: '2026-01-01T00:00:00Z', slug: 'b' }, next: { at: '2026-01-02T00:00:00Z', slug: '' }, slugs: ['a'] };+ const ran = (fail?: string) => ['extract', 'extract_facts', 'recompute_emotional_weight'].map(phase => ({ phase, status: phase === fail ? 'fail' : 'ok' }));+ expect(sourceCycleWindowStamp(ran(), window)).toEqual({ last_source_cycle_window_at: '2026-01-02T00:00:00Z', last_source_cycle_window_slug: '' });for (const phase of ['extract', 'extract_facts', 'recompute_emotional_weight']) {- expect(sourceCycleWindowStamp([{ phase, status: 'fail' }], '2026-01-01T00:00:00Z', window)).toEqual({ last_source_cycle_window_at: '2026-01-01T00:00:00Z' });- expect(sourceCycleWindowStamp([{ phase, status: 'fail' }], null, { ...window, walk: { startedAt: window.at, next: 'a' } })).toEqual({ last_source_cycle_window_at: '' });+ expect(sourceCycleWindowStamp(ran(phase), window)).toEqual({ last_source_cycle_window_at: '2026-01-01T00:00:00Z', last_source_cycle_window_slug: 'b' });+ expect(sourceCycleWindowStamp(ran().filter(r => r.phase !== phase), window)).toEqual({ last_source_cycle_window_at: '2026-01-01T00:00:00Z', last_source_cycle_window_slug: 'b' });}- expect(sourceCycleWindowStamp([{ phase: 'lint', status: 'fail' }], '2026-01-01T00:00:00Z', window)).toEqual({ last_source_cycle_window_at: window.at });+ expect(sourceCycleWindowStamp(ran(), { ...window, since: { at: null, slug: '' } })).toMatchObject({ last_source_cycle_window_at: '2026-01-02T00:00:00Z' });+ expect(sourceCycleWindowStamp(ran('extract'), { ...window, since: { at: null, slug: '' } })).toEqual({ last_source_cycle_window_at: '', last_source_cycle_window_slug: '' });+ expect(sourceCycleWindowStamp([...ran(), { phase: 'lint', status: 'fail' }], window)).toMatchObject({ last_source_cycle_window_at: '2026-01-02T00:00:00Z' });});test('a sync-disabled git source reconciles only pages changed since its last cycle; an explicit sync between cycles lands in the next window', async () => withEnv(env, async () => {@@ -119,7 +123,7 @@ describe('#8573 sync-disabled source cycle window', () => {}}), 180_000);- test('a source with no recorded cycle walks its pages in bounded chunks and resumes; a page changed during the walk lands after it', async () => withEnv(env, async () => {+ test('a source with no recorded cycle walks its pages in bounded windows and resumes; a page changed during the walk lands in it', async () => withEnv(env, async () => {for (const engine of engines) {const { id } = await source(engine, { syncEnabled: false });await engine.executeRaw('UPDATE persistence_brain SET enabled=false WHERE singleton=1');@@ -127,24 +131,47 @@ describe('#8573 sync-disabled source cycle window', () => {for (let i = 0; i < 5; i++) {await engine.putPage(`walk/n-${i}`, { title: `Note ${i}`, type: 'note', compiled_truth: `Body ${i}.` }, { sourceId: id });}- sourceFirstWalk.pageLimit = 2;+ sourceWindow.pageLimit = 2;const one = await cycle(engine, id, 1);expect(one.scanned).toBe(2);expect(one.weight).toMatchObject({ mode: 'incremental', pages_recomputed: 2 });- expect(await config(engine, id)).toMatchObject({ last_source_cycle_window_at: '', last_source_cycle_walk_after: 'walk/n-1' });- const startedAt = (await config(engine, id)).last_source_cycle_walk_at as string;- expect(Number.isFinite(Date.parse(startedAt))).toBe(true);+ expect(await config(engine, id)).toMatchObject({ last_source_cycle_window_slug: 'walk/n-1' });- // Edited after the first chunk passed it: the walk does not revisit it, the window after the walk does.+ // Edited after the first window passed it: it moves to the end of the walk.await engine.putPage('walk/n-0', { title: 'Note 0', type: 'note', compiled_truth: 'Body 0, edited.' }, { sourceId: id });expect((await cycle(engine, id, 2)).weight).toMatchObject({ pages_recomputed: 2 });- expect(await config(engine, id)).toMatchObject({ last_source_cycle_walk_after: 'walk/n-3', last_source_cycle_walk_at: startedAt });- expect((await cycle(engine, id, 3)).weight).toMatchObject({ pages_recomputed: 1 });- expect(await config(engine, id)).toMatchObject({ last_source_cycle_window_at: startedAt, last_source_cycle_walk_after: null, last_source_cycle_walk_at: null });+ expect(await config(engine, id)).toMatchObject({ last_source_cycle_window_slug: 'walk/n-3' });+ expect((await cycle(engine, id, 3)).weight).toMatchObject({ pages_recomputed: 2 });+ expect(await config(engine, id)).toMatchObject({ last_source_cycle_window_slug: '' });+ expect((await cycle(engine, id, 4)).weight).toMatchObject({ mode: 'incremental', pages_recomputed: 0 });+ } finally {+ await disposePersistenceConsumer(engine);+ await engine.executeRaw('UPDATE persistence_brain SET enabled=true WHERE singleton=1');+ }+ }+ }), 180_000);- const next = await cycle(engine, id, 4);- expect(next.weight).toMatchObject({ mode: 'incremental', pages_recomputed: 1 });- expect((await cycle(engine, id, 5)).weight).toMatchObject({ pages_recomputed: 0 });+ test('a large change between cycles (a bulk explicit sync) is worked through in bounded windows; a cycle without the window phases does not advance it', async () => withEnv(env, async () => {+ for (const engine of engines) {+ sourceWindow.pageLimit = DEFAULT_PAGE_LIMIT;+ const { id } = await source(engine, { syncEnabled: false });+ await engine.executeRaw('UPDATE persistence_brain SET enabled=false WHERE singleton=1');+ try {+ for (let i = 0; i < 5; i++) {+ await engine.putPage(`bulk/n-${i}`, { title: `Note ${i}`, type: 'note', compiled_truth: `Body ${i}.` }, { sourceId: id });+ }+ await cycle(engine, id, 1);+ expect((await cycle(engine, id, 2)).weight).toMatchObject({ pages_recomputed: 0 });+ for (let i = 0; i < 5; i++) {+ await engine.putPage(`bulk/n-${i}`, { title: `Note ${i}`, type: 'note', compiled_truth: `Body ${i}, re-imported.` }, { sourceId: id });+ }+ sourceWindow.pageLimit = 2;+ const before = await config(engine, id);+ await makeAutopilotCycleHandler(engine)({ id: 3, data: { source_id: id, phases: ['lint', 'backlinks'] } } as never);+ expect(await config(engine, id)).toMatchObject({ last_source_cycle_window_at: before.last_source_cycle_window_at, last_source_cycle_window_slug: '' });+ const counts: number[] = [];+ for (let n = 4; n < 8; n++) counts.push(Number((await cycle(engine, id, n)).weight.pages_recomputed));+ expect(counts).toEqual([2, 2, 1, 0]);} finally {await disposePersistenceConsumer(engine);await engine.executeRaw('UPDATE persistence_brain SET enabled=true WHERE singleton=1');