Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(connectors): a connector item stored in segments imports and re-renders, and one the sweep cannot read is held instead of stopping the source (#9062)
garrytan2203597d784b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ca92548655..3c67adca1d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -10,6 +10,16 @@ credits are retained; no result has been reassigned to another provider. Originaidentifiers and attribution are available in the pre-removal Git revision`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).+## [Unreleased]++**A GitHub (or other API) connector source no longer stops for good when one of its items is a page stored in segments.** A long issue (a body plus hundreds of comments) grows past 1 MiB and is stored in segments. Before, that item could never import, and once such a page existed, every later sweep of a bound source stopped on `page_body_segmented` with no item named, so nothing after it imported and the cursor never moved. Now an item up to 24 MiB imports and re-renders like any other, and one the sweep cannot read whole is held on its own (`gbrain sources status`) while the rest of the run imports and the cursor moves. Reported and first diagnosed in #9062.++### Itemized changes++- `src/core/persistence/connector-sync.ts`: the connector publication reads a segmented page back as strings (up to 24 MiB) before it carries its timeline and fences into the re-render, seals a segmented body through its import (never the inline text seal, which read the whole timeline), and runs the import's after-commit step, so a segmented page left unsealed is chunked. The freshness read (`page()`) reads a segmented page back the same way.+- `src/core/github-source.ts`: a stored page the sweep cannot read fails that item (a content hold after three runs), not the run.+- Tests: `test/connector-segmented-page-9062.test.ts` (PGLite) and its PostgreSQL arm `test/e2e/connector-segmented-page-9062.test.ts`.+## [0.60.641.0] - 2026-11-10**Facts, timeline entries, tags and ontology observations now follow the page they are about, so a remote caller who cannot read a page learns nothing about it from fact reads, and a deleted page's facts, timeline and tags stay hidden from everyone until it is restored.** Before this, a fact saved on an open page about a page with `readable_by` or `visibility: private` still reached remote callers through recall, ontology_get, volunteer_chronicle, delta, find_trajectory, hot memory and the entity card, and a soft-deleted page's facts, timeline rows and tags kept answering every reader. Recorded-time reads (get_page and get_versions with known_as_of), plain get_versions with include_body, and delta's belief changes now also keep a quarantined page's bodies and claims from remote callers, including a page quarantined only by a later write and a version history that spans several sources. Nothing is deleted or expired, so restore_page brings every row back as it was. A new page written at a deleted page's slug still shows the deleted page's facts, and a purged page's facts still come back, as before; both are follow-up work.diff --git a/src/core/github-source.ts b/src/core/github-source.tsindex 32905b4644..2472f4e196 100644--- a/src/core/github-source.ts+++ b/src/core/github-source.ts@@ -1239,7 +1239,15 @@ async function runGitHubSyncInner(engine: BrainEngine, sourceId: string, cfg: Gi// so they are re-fetched every sweep. Cost is bounded by the number// of open PRs, which is small in practice.const isOpenPr = item.kind === 'pr' && item.state === 'open';- const { fresh, hasDetail } = await storedPage(deps, filePath, item.updated_at);+ // #9062: a stored page this read cannot load (one stored in segments over the single-string ceiling) fails this item, not the run.+ let stored: Awaited<ReturnType<typeof storedPage>>;+ try {+ stored = await storedPage(deps, filePath, item.updated_at);+ } catch (err) {+ await itemFailed(repo, item, err, ' stored page read');+ continue;+ }+ const { fresh, hasDetail } = stored;if (!opts.full && !isOpenPr && fresh && hasDetail) {// Cursor accounting: a fresh skip is a success too. Without this,// a repo whose newest item is always fresh stays re-listed ondiff --git a/src/core/persistence/connector-sync.ts b/src/core/persistence/connector-sync.tsindex 6d274cdb26..2b084acb21 100644--- a/src/core/persistence/connector-sync.ts+++ b/src/core/persistence/connector-sync.ts@@ -28,6 +28,7 @@ import type { PreparedContentImport } from './prepared-import.ts';import { persistenceFileHash, type PreparedMutation } from './coordinator.ts';import { isTerminal, type WriteRequest } from './model.ts';import { databaseOnlyPublication, prepareFileTarget, preparePrivateTransition } from './page-prepare.ts';+import { materializedSnapshot } from './segmented-apply.ts';import { sourceCanonicalGit } from './git-door.ts';import { isPrivacyPrivatePage } from '../search/private-visibility.ts';import { assertPhysicalRoot } from './physical-root.ts';@@ -628,7 +629,7 @@ export class ManagedConnectorSync {async page(slug: string) {slug = slugifyPath(`${slug}.md`);await this.recover(slug);- const snapshot = await this.engine.readPageSnapshot(slug, { sourceId: this.sourceId });+ const snapshot = await materializedSnapshot(this.engine, await this.engine.readPageSnapshot(slug, { sourceId: this.sourceId }));if (snapshot && this.binding) await prepareFileTarget(this.engine,{ source_id: this.sourceId, worktree_id: this.binding.worktree_id, slug }, snapshot, serializePageToMarkdown(snapshot.page, snapshot.tags));return snapshot?.page ?? null;@@ -1147,7 +1148,8 @@ export async function prepareConnectorMutation(engine: BrainEngine, row: WriteRe[row.source_id, row.source_incarnation, p.newestContentAt ?? null]);return { status: 'checkpointed', source_id: row.source_id };} };- const snapshot = await engine.readPageSnapshot(row.slug, { sourceId: row.source_id, includeDeleted: true });+ // #9062: a page stored in segments is read back as strings (up to PAGE_MAX_BYTES), so its re-render carries its timeline and fences like an inline page's.+ const snapshot = await materializedSnapshot(engine, await engine.readPageSnapshot(row.slug, { sourceId: row.source_id, includeDeleted: true }));if ((snapshot?.revision ?? null) !== p.expected_revision || (snapshot?.page.id ?? null) !== row.page_id ||snapshot?.page.source_path != null && snapshot.page.source_path !== p.sourcePath) {throw connectorPublicationRefusal('revision_conflict', 'The connector page changed after admission.', row,@@ -1222,9 +1224,11 @@ export async function prepareConnectorMutation(engine: BrainEngine, row: WriteRe`The canonical file of ${row.slug} in ${row.source_id} changed on disk while this connector write was being prepared.`);return { observedRevision: ready.observedRevision, sourceExclusive: true,validate: async tx => { await validate(tx); await ready.validate(tx); },- file, ...databaseOnlyPublication(row, file), noop: ready.noop, deferEmbedding: p.noEmbed, apply: async tx => {- await ready.apply(tx);- if (!ready.noop) { await project(tx); await sealPageTextProjection(tx, row.slug, row.source_id); }+ file, ...databaseOnlyPublication(row, file), noop: ready.noop, deferEmbedding: p.noEmbed,+ ...(ready.afterCommit ? { afterCommit: (target: BrainEngine) => ready.afterCommit!(target) } : {}), apply: async tx => {+ const installed = await ready.apply(tx);+ // A body stored in segments was sealed by its import's apply when complete (sealSegmentedPage); the inline text seal reads the whole timeline.+ if (!ready.noop) { await project(tx); if (!installed?.segmented) await sealPageTextProjection(tx, row.slug, row.source_id); }return { status: ready.noop ? 'skipped' : snapshot ? 'updated' : 'created', slug: row.slug, source_id: row.source_id,chunks: result.chunks, noop: ready.noop, imported_file: true, connector_database: !row.worktree_id };} };diff --git a/test/connector-segmented-page-9062.test.ts b/test/connector-segmented-page-9062.test.tsnew file mode 100644index 0000000000..6dfb6e48fc--- /dev/null+++ b/test/connector-segmented-page-9062.test.ts@@ -0,0 +1,119 @@+/**+ * #9062: a GitHub (API) connector source keeps moving when an item's page is stored in segments.+ *+ * A long issue (a body plus hundreds of comments) runs past the inline ceiling, so its page is stored in segments.+ * Before the fix the connector's publication sealed the page's text projection by reading its timeline whole, so the+ * item never imported; and once such a page existed, every later sweep of a bound source read it whole while checking+ * whether the item was fresh, before any item's failure handling ran, so the whole source stopped on+ * `page_body_segmented` with no slug, run after run. Now a segmented page up to the single-string ceiling imports and+ * re-renders like any other, and one over it is held on its own while the rest of the run imports.+ *+ * PGLite here; PostgreSQL through test/e2e/connector-segmented-page-9062.test.ts.+ */+import { afterAll, beforeAll, expect, spyOn, test } from 'bun:test';+import type { BrainEngine } from '../src/core/engine.ts';+import { parseGitHubSourceConfig, runGitHubSync } from '../src/core/github-source.ts';+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';+import { readManagedConnectorState } from '../src/core/persistence/connector-state.ts';+import { readAllSourceHolds } from '../src/core/connectors/item-holds-store.ts';+import { PAGE_INLINE_MAX_BYTES, PAGE_MAX_BYTES } from '../src/core/page-size-policy.ts';+import { createConnectorFixture, options, settleConnectorWrites } from './helpers/connector-fixture.ts';+import { fakeGitHub, githubHoldsFetch } from './helpers/connector-holds-fixture.ts';+import { withEnv } from './helpers/with-env.ts';++const { engines, env, source, boundSource, setup, teardown } = createConnectorFixture();+beforeAll(setup, 120_000);+afterAll(teardown);++const githubConfig = { kind: 'github', gh_scope: 'repos', gh_repos: 'acme-example/app', gh_token_env: 'CONNECTOR_TEST_TOKEN' };+/** A synthetic issue thread past the inline ceiling, so its page is stored in segments. */+const longThread = (tag: string) => {+ const line = (i: number) => `Synthetic comment ${i} (${tag}): ` + 'lorem ipsum '.repeat(8);+ const lines: string[] = [];+ for (let i = 0, bytes = 0; bytes < PAGE_INLINE_MAX_BYTES + 64 * 1024; i++) { lines.push(line(i)); bytes += line(i).length + 2; }+ return lines.join('\n\n');+};++async function state(engine: BrainEngine, id: string) {+ const [row] = await engine.executeRaw<{ incarnation: string }>('SELECT incarnation::text FROM sources WHERE id=$1', [id]);+ return readManagedConnectorState(engine, id, row.incarnation);+}+const storage = async (engine: BrainEngine, id: string, slug: string) =>+ (await engine.executeRaw<{ body_storage: string }>('SELECT body_storage FROM pages WHERE source_id=$1 AND slug=$2 AND deleted_at IS NULL', [id, slug]))[0]?.body_storage ?? null;+const bodyHas = async (engine: BrainEngine, id: string, slug: string, text: string) =>+ (await engine.executeRaw<{ n: number }>(`SELECT count(*)::int AS n FROM content_chunks c JOIN pages p ON p.id=c.page_id+ WHERE p.source_id=$1 AND p.slug=$2 AND strpos(c.chunk_text, $3) > 0`, [id, slug, text]))[0].n > 0;++for (const bound of [false, true]) {+ test(`#9062 (${bound ? 'bound' : 'unbound'} managed source): a long issue stored in segments imports and re-renders, and the rest of the run keeps importing`, async () => withEnv(env, async () => {+ for (const engine of engines) {+ const f = bound ? await boundSource(engine, githubConfig) : await source(engine, githubConfig);+ const cfg = parseGitHubSourceConfig(githubConfig, f.dir);+ const fx = fakeGitHub();+ const run = () => runGitHubSync(engine, f.id, cfg, options, githubHoldsFetch(fx));+ const errors = spyOn(console, 'error').mockImplementation(() => {});+ try {+ fx.issues = [{ number: 1, title: 'Synthetic long issue', body: longThread('first'), updated_at: '2026-02-01T00:00:00Z' }];+ expect((await run()).status).not.toBe('partial');+ await settleConnectorWrites(engine, f.id);+ expect(await storage(engine, f.id, 'gh/acme-example/app/1')).toBe('segmented');+ expect((await state(engine, f.id)).last_failure).toBeNull();+ await disposePersistenceConsumer(engine);++ // A new comment on the long issue and a new issue: both import, and the cursor moves past both.+ fx.issues = [{ number: 1, title: 'Synthetic long issue', body: longThread('second'), updated_at: '2026-02-02T00:00:00Z' },+ { number: 2, title: 'Synthetic short issue', body: 'Short body.', updated_at: '2026-02-03T00:00:00Z' }];+ expect((await run()).status).not.toBe('partial');+ await settleConnectorWrites(engine, f.id);+ expect(await storage(engine, f.id, 'gh/acme-example/app/1')).toBe('segmented');+ expect(await bodyHas(engine, f.id, 'gh/acme-example/app/1', '(second)')).toBe(true);+ expect(await bodyHas(engine, f.id, 'gh/acme-example/app/1', '(first)')).toBe(false);+ expect(await storage(engine, f.id, 'gh/acme-example/app/2')).toBe('inline');+ expect((await state(engine, f.id)).last_failure).toBeNull();+ expect(await readAllSourceHolds(engine, { sourceIds: [f.id] }).then(s => s[0]?.held ?? [])).toEqual([]);+ await disposePersistenceConsumer(engine);+ fx.since.length = 0;+ expect((await run()).status).not.toBe('partial');+ expect(fx.since).toEqual(['2026-02-03T00:00:00Z']);+ await settleConnectorWrites(engine, f.id);+ await disposePersistenceConsumer(engine);+ } finally { errors.mockRestore(); }+ }+ }), 300_000);+}++test('#9062 (bound managed source): a segmented page over the single-string ceiling is held on its own; the rest of the source imports and the cursor moves', async () => withEnv(env, async () => {+ for (const engine of engines) {+ const f = await boundSource(engine, githubConfig);+ const cfg = parseGitHubSourceConfig(githubConfig, f.dir);+ const fx = fakeGitHub();+ const run = () => runGitHubSync(engine, f.id, cfg, options, githubHoldsFetch(fx));+ const errors = spyOn(console, 'error').mockImplementation(() => {});+ const stderr = spyOn(process.stderr, 'write').mockImplementation(() => true);+ try {+ fx.issues = [{ number: 1, title: 'Synthetic long issue', body: longThread('first'), updated_at: '2026-02-01T00:00:00Z' }];+ await run();+ await settleConnectorWrites(engine, f.id);+ await disposePersistenceConsumer(engine);+ // The stored body now reads as over the ceiling a whole-body read takes (a page this size is costly to build in a test).+ await engine.executeRaw('UPDATE pages SET body_bytes=$3 WHERE source_id=$1 AND slug=$2', [f.id, 'gh/acme-example/app/1', PAGE_MAX_BYTES + 1]);+ fx.issues = [{ number: 1, title: 'Synthetic long issue', body: longThread('second'), updated_at: '2026-02-02T00:00:00Z' },+ { number: 2, title: 'Synthetic short issue', body: 'Short body.', updated_at: '2026-02-03T00:00:00Z' }];+ for (let i = 0; i < 3; i++) { await run().catch(() => undefined); await settleConnectorWrites(engine, f.id).catch(() => undefined); await disposePersistenceConsumer(engine); }+ const held = (await readAllSourceHolds(engine, { sourceIds: [f.id] }))[0]?.held ?? [];+ expect(held.map(h => ({ key: h.key, code: h.code, class: h.class, slug: h.slug }))).toEqual([+ { key: 'acme-example/app#1', code: 'page_body_segmented', class: 'content', slug: 'gh/acme-example/app/1' }]);+ const next = await run();+ expect(next.status).not.toBe('partial');+ await settleConnectorWrites(engine, f.id);+ expect(await storage(engine, f.id, 'gh/acme-example/app/2')).toBe('inline');+ expect((await state(engine, f.id)).last_failure).toBeNull();+ await disposePersistenceConsumer(engine);+ fx.since.length = 0;+ await run();+ expect(fx.since).toEqual(['2026-02-03T00:00:00Z']);+ await settleConnectorWrites(engine, f.id);+ await disposePersistenceConsumer(engine);+ } finally { errors.mockRestore(); stderr.mockRestore(); }+ }+}), 300_000);diff --git a/test/e2e/connector-segmented-page-9062.test.ts b/test/e2e/connector-segmented-page-9062.test.tsnew file mode 100644index 0000000000..0f6fa15301--- /dev/null+++ b/test/e2e/connector-segmented-page-9062.test.ts@@ -0,0 +1,4 @@+import { registerPostgresTests } from '../helpers/test-backends.ts';++// #9062: the PostgreSQL arm of the connector segmented-page suite.+await registerPostgresTests(() => import('../connector-segmented-page-9062.test.ts'));