Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(connectors): a connector item stored in segments imports and re-renders, and one the sweep cannot read is held instead of stopping the source (#9062)

garrytan2203597d784b
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ca92548655..3c67adca1d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,16 @@ credits are retained; no result has been reassigned to another provider. Origina
identifiers and attribution are available in the pre-removal Git revision
`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).
+## [Unreleased]
+
+**A GitHub (or other API) connector source no longer stops for good when one of its items is a page stored in segments.** A long issue (a body plus hundreds of comments) grows past 1 MiB and is stored in segments. Before, that item could never import, and once such a page existed, every later sweep of a bound source stopped on `page_body_segmented` with no item named, so nothing after it imported and the cursor never moved. Now an item up to 24 MiB imports and re-renders like any other, and one the sweep cannot read whole is held on its own (`gbrain sources status`) while the rest of the run imports and the cursor moves. Reported and first diagnosed in #9062.
+
+### Itemized changes
+
+- `src/core/persistence/connector-sync.ts`: the connector publication reads a segmented page back as strings (up to 24 MiB) before it carries its timeline and fences into the re-render, seals a segmented body through its import (never the inline text seal, which read the whole timeline), and runs the import's after-commit step, so a segmented page left unsealed is chunked. The freshness read (`page()`) reads a segmented page back the same way.
+- `src/core/github-source.ts`: a stored page the sweep cannot read fails that item (a content hold after three runs), not the run.
+- Tests: `test/connector-segmented-page-9062.test.ts` (PGLite) and its PostgreSQL arm `test/e2e/connector-segmented-page-9062.test.ts`.
+
## [0.60.641.0] - 2026-11-10
**Facts, timeline entries, tags and ontology observations now follow the page they are about, so a remote caller who cannot read a page learns nothing about it from fact reads, and a deleted page's facts, timeline and tags stay hidden from everyone until it is restored.** Before this, a fact saved on an open page about a page with `readable_by` or `visibility: private` still reached remote callers through recall, ontology_get, volunteer_chronicle, delta, find_trajectory, hot memory and the entity card, and a soft-deleted page's facts, timeline rows and tags kept answering every reader. Recorded-time reads (get_page and get_versions with known_as_of), plain get_versions with include_body, and delta's belief changes now also keep a quarantined page's bodies and claims from remote callers, including a page quarantined only by a later write and a version history that spans several sources. Nothing is deleted or expired, so restore_page brings every row back as it was. A new page written at a deleted page's slug still shows the deleted page's facts, and a purged page's facts still come back, as before; both are follow-up work.
diff --git a/src/core/github-source.ts b/src/core/github-source.ts
index 32905b4644..2472f4e196 100644
--- a/src/core/github-source.ts
+++ b/src/core/github-source.ts
@@ -1239,7 +1239,15 @@ async function runGitHubSyncInner(engine: BrainEngine, sourceId: string, cfg: Gi
// so they are re-fetched every sweep. Cost is bounded by the number
// of open PRs, which is small in practice.
const isOpenPr = item.kind === 'pr' && item.state === 'open';
- const { fresh, hasDetail } = await storedPage(deps, filePath, item.updated_at);
+ // #9062: a stored page this read cannot load (one stored in segments over the single-string ceiling) fails this item, not the run.
+ let stored: Awaited<ReturnType<typeof storedPage>>;
+ try {
+ stored = await storedPage(deps, filePath, item.updated_at);
+ } catch (err) {
+ await itemFailed(repo, item, err, ' stored page read');
+ continue;
+ }
+ const { fresh, hasDetail } = stored;
if (!opts.full && !isOpenPr && fresh && hasDetail) {
// Cursor accounting: a fresh skip is a success too. Without this,
// a repo whose newest item is always fresh stays re-listed on
diff --git a/src/core/persistence/connector-sync.ts b/src/core/persistence/connector-sync.ts
index 6d274cdb26..2b084acb21 100644
--- a/src/core/persistence/connector-sync.ts
+++ b/src/core/persistence/connector-sync.ts
@@ -28,6 +28,7 @@ import type { PreparedContentImport } from './prepared-import.ts';
import { persistenceFileHash, type PreparedMutation } from './coordinator.ts';
import { isTerminal, type WriteRequest } from './model.ts';
import { databaseOnlyPublication, prepareFileTarget, preparePrivateTransition } from './page-prepare.ts';
+import { materializedSnapshot } from './segmented-apply.ts';
import { sourceCanonicalGit } from './git-door.ts';
import { isPrivacyPrivatePage } from '../search/private-visibility.ts';
import { assertPhysicalRoot } from './physical-root.ts';
@@ -628,7 +629,7 @@ export class ManagedConnectorSync {
async page(slug: string) {
slug = slugifyPath(`${slug}.md`);
await this.recover(slug);
- const snapshot = await this.engine.readPageSnapshot(slug, { sourceId: this.sourceId });
+ const snapshot = await materializedSnapshot(this.engine, await this.engine.readPageSnapshot(slug, { sourceId: this.sourceId }));
if (snapshot && this.binding) await prepareFileTarget(this.engine,
{ source_id: this.sourceId, worktree_id: this.binding.worktree_id, slug }, snapshot, serializePageToMarkdown(snapshot.page, snapshot.tags));
return snapshot?.page ?? null;
@@ -1147,7 +1148,8 @@ export async function prepareConnectorMutation(engine: BrainEngine, row: WriteRe
[row.source_id, row.source_incarnation, p.newestContentAt ?? null]);
return { status: 'checkpointed', source_id: row.source_id };
} };
- const snapshot = await engine.readPageSnapshot(row.slug, { sourceId: row.source_id, includeDeleted: true });
+ // #9062: a page stored in segments is read back as strings (up to PAGE_MAX_BYTES), so its re-render carries its timeline and fences like an inline page's.
+ const snapshot = await materializedSnapshot(engine, await engine.readPageSnapshot(row.slug, { sourceId: row.source_id, includeDeleted: true }));
if ((snapshot?.revision ?? null) !== p.expected_revision || (snapshot?.page.id ?? null) !== row.page_id ||
snapshot?.page.source_path != null && snapshot.page.source_path !== p.sourcePath) {
throw connectorPublicationRefusal('revision_conflict', 'The connector page changed after admission.', row,
@@ -1222,9 +1224,11 @@ export async function prepareConnectorMutation(engine: BrainEngine, row: WriteRe
`The canonical file of ${row.slug} in ${row.source_id} changed on disk while this connector write was being prepared.`);
return { observedRevision: ready.observedRevision, sourceExclusive: true,
validate: async tx => { await validate(tx); await ready.validate(tx); },
- file, ...databaseOnlyPublication(row, file), noop: ready.noop, deferEmbedding: p.noEmbed, apply: async tx => {
- await ready.apply(tx);
- if (!ready.noop) { await project(tx); await sealPageTextProjection(tx, row.slug, row.source_id); }
+ file, ...databaseOnlyPublication(row, file), noop: ready.noop, deferEmbedding: p.noEmbed,
+ ...(ready.afterCommit ? { afterCommit: (target: BrainEngine) => ready.afterCommit!(target) } : {}), apply: async tx => {
+ const installed = await ready.apply(tx);
+ // A body stored in segments was sealed by its import's apply when complete (sealSegmentedPage); the inline text seal reads the whole timeline.
+ if (!ready.noop) { await project(tx); if (!installed?.segmented) await sealPageTextProjection(tx, row.slug, row.source_id); }
return { status: ready.noop ? 'skipped' : snapshot ? 'updated' : 'created', slug: row.slug, source_id: row.source_id,
chunks: result.chunks, noop: ready.noop, imported_file: true, connector_database: !row.worktree_id };
} };
diff --git a/test/connector-segmented-page-9062.test.ts b/test/connector-segmented-page-9062.test.ts
new file mode 100644
index 0000000000..6dfb6e48fc
--- /dev/null
+++ b/test/connector-segmented-page-9062.test.ts
@@ -0,0 +1,119 @@
+/**
+ * #9062: a GitHub (API) connector source keeps moving when an item's page is stored in segments.
+ *
+ * A long issue (a body plus hundreds of comments) runs past the inline ceiling, so its page is stored in segments.
+ * Before the fix the connector's publication sealed the page's text projection by reading its timeline whole, so the
+ * item never imported; and once such a page existed, every later sweep of a bound source read it whole while checking
+ * whether the item was fresh, before any item's failure handling ran, so the whole source stopped on
+ * `page_body_segmented` with no slug, run after run. Now a segmented page up to the single-string ceiling imports and
+ * re-renders like any other, and one over it is held on its own while the rest of the run imports.
+ *
+ * PGLite here; PostgreSQL through test/e2e/connector-segmented-page-9062.test.ts.
+ */
+import { afterAll, beforeAll, expect, spyOn, test } from 'bun:test';
+import type { BrainEngine } from '../src/core/engine.ts';
+import { parseGitHubSourceConfig, runGitHubSync } from '../src/core/github-source.ts';
+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
+import { readManagedConnectorState } from '../src/core/persistence/connector-state.ts';
+import { readAllSourceHolds } from '../src/core/connectors/item-holds-store.ts';
+import { PAGE_INLINE_MAX_BYTES, PAGE_MAX_BYTES } from '../src/core/page-size-policy.ts';
+import { createConnectorFixture, options, settleConnectorWrites } from './helpers/connector-fixture.ts';
+import { fakeGitHub, githubHoldsFetch } from './helpers/connector-holds-fixture.ts';
+import { withEnv } from './helpers/with-env.ts';
+
+const { engines, env, source, boundSource, setup, teardown } = createConnectorFixture();
+beforeAll(setup, 120_000);
+afterAll(teardown);
+
+const githubConfig = { kind: 'github', gh_scope: 'repos', gh_repos: 'acme-example/app', gh_token_env: 'CONNECTOR_TEST_TOKEN' };
+/** A synthetic issue thread past the inline ceiling, so its page is stored in segments. */
+const longThread = (tag: string) => {
+ const line = (i: number) => `Synthetic comment ${i} (${tag}): ` + 'lorem ipsum '.repeat(8);
+ const lines: string[] = [];
+ for (let i = 0, bytes = 0; bytes < PAGE_INLINE_MAX_BYTES + 64 * 1024; i++) { lines.push(line(i)); bytes += line(i).length + 2; }
+ return lines.join('\n\n');
+};
+
+async function state(engine: BrainEngine, id: string) {
+ const [row] = await engine.executeRaw<{ incarnation: string }>('SELECT incarnation::text FROM sources WHERE id=$1', [id]);
+ return readManagedConnectorState(engine, id, row.incarnation);
+}
+const storage = async (engine: BrainEngine, id: string, slug: string) =>
+ (await engine.executeRaw<{ body_storage: string }>('SELECT body_storage FROM pages WHERE source_id=$1 AND slug=$2 AND deleted_at IS NULL', [id, slug]))[0]?.body_storage ?? null;
+const bodyHas = async (engine: BrainEngine, id: string, slug: string, text: string) =>
+ (await engine.executeRaw<{ n: number }>(`SELECT count(*)::int AS n FROM content_chunks c JOIN pages p ON p.id=c.page_id
+ WHERE p.source_id=$1 AND p.slug=$2 AND strpos(c.chunk_text, $3) > 0`, [id, slug, text]))[0].n > 0;
+
+for (const bound of [false, true]) {
+ test(`#9062 (${bound ? 'bound' : 'unbound'} managed source): a long issue stored in segments imports and re-renders, and the rest of the run keeps importing`, async () => withEnv(env, async () => {
+ for (const engine of engines) {
+ const f = bound ? await boundSource(engine, githubConfig) : await source(engine, githubConfig);
+ const cfg = parseGitHubSourceConfig(githubConfig, f.dir);
+ const fx = fakeGitHub();
+ const run = () => runGitHubSync(engine, f.id, cfg, options, githubHoldsFetch(fx));
+ const errors = spyOn(console, 'error').mockImplementation(() => {});
+ try {
+ fx.issues = [{ number: 1, title: 'Synthetic long issue', body: longThread('first'), updated_at: '2026-02-01T00:00:00Z' }];
+ expect((await run()).status).not.toBe('partial');
+ await settleConnectorWrites(engine, f.id);
+ expect(await storage(engine, f.id, 'gh/acme-example/app/1')).toBe('segmented');
+ expect((await state(engine, f.id)).last_failure).toBeNull();
+ await disposePersistenceConsumer(engine);
+
+ // A new comment on the long issue and a new issue: both import, and the cursor moves past both.
+ fx.issues = [{ number: 1, title: 'Synthetic long issue', body: longThread('second'), updated_at: '2026-02-02T00:00:00Z' },
+ { number: 2, title: 'Synthetic short issue', body: 'Short body.', updated_at: '2026-02-03T00:00:00Z' }];
+ expect((await run()).status).not.toBe('partial');
+ await settleConnectorWrites(engine, f.id);
+ expect(await storage(engine, f.id, 'gh/acme-example/app/1')).toBe('segmented');
+ expect(await bodyHas(engine, f.id, 'gh/acme-example/app/1', '(second)')).toBe(true);
+ expect(await bodyHas(engine, f.id, 'gh/acme-example/app/1', '(first)')).toBe(false);
+ expect(await storage(engine, f.id, 'gh/acme-example/app/2')).toBe('inline');
+ expect((await state(engine, f.id)).last_failure).toBeNull();
+ expect(await readAllSourceHolds(engine, { sourceIds: [f.id] }).then(s => s[0]?.held ?? [])).toEqual([]);
+ await disposePersistenceConsumer(engine);
+ fx.since.length = 0;
+ expect((await run()).status).not.toBe('partial');
+ expect(fx.since).toEqual(['2026-02-03T00:00:00Z']);
+ await settleConnectorWrites(engine, f.id);
+ await disposePersistenceConsumer(engine);
+ } finally { errors.mockRestore(); }
+ }
+ }), 300_000);
+}
+
+test('#9062 (bound managed source): a segmented page over the single-string ceiling is held on its own; the rest of the source imports and the cursor moves', async () => withEnv(env, async () => {
+ for (const engine of engines) {
+ const f = await boundSource(engine, githubConfig);
+ const cfg = parseGitHubSourceConfig(githubConfig, f.dir);
+ const fx = fakeGitHub();
+ const run = () => runGitHubSync(engine, f.id, cfg, options, githubHoldsFetch(fx));
+ const errors = spyOn(console, 'error').mockImplementation(() => {});
+ const stderr = spyOn(process.stderr, 'write').mockImplementation(() => true);
+ try {
+ fx.issues = [{ number: 1, title: 'Synthetic long issue', body: longThread('first'), updated_at: '2026-02-01T00:00:00Z' }];
+ await run();
+ await settleConnectorWrites(engine, f.id);
+ await disposePersistenceConsumer(engine);
+ // The stored body now reads as over the ceiling a whole-body read takes (a page this size is costly to build in a test).
+ await engine.executeRaw('UPDATE pages SET body_bytes=$3 WHERE source_id=$1 AND slug=$2', [f.id, 'gh/acme-example/app/1', PAGE_MAX_BYTES + 1]);
+ fx.issues = [{ number: 1, title: 'Synthetic long issue', body: longThread('second'), updated_at: '2026-02-02T00:00:00Z' },
+ { number: 2, title: 'Synthetic short issue', body: 'Short body.', updated_at: '2026-02-03T00:00:00Z' }];
+ for (let i = 0; i < 3; i++) { await run().catch(() => undefined); await settleConnectorWrites(engine, f.id).catch(() => undefined); await disposePersistenceConsumer(engine); }
+ const held = (await readAllSourceHolds(engine, { sourceIds: [f.id] }))[0]?.held ?? [];
+ expect(held.map(h => ({ key: h.key, code: h.code, class: h.class, slug: h.slug }))).toEqual([
+ { key: 'acme-example/app#1', code: 'page_body_segmented', class: 'content', slug: 'gh/acme-example/app/1' }]);
+ const next = await run();
+ expect(next.status).not.toBe('partial');
+ await settleConnectorWrites(engine, f.id);
+ expect(await storage(engine, f.id, 'gh/acme-example/app/2')).toBe('inline');
+ expect((await state(engine, f.id)).last_failure).toBeNull();
+ await disposePersistenceConsumer(engine);
+ fx.since.length = 0;
+ await run();
+ expect(fx.since).toEqual(['2026-02-03T00:00:00Z']);
+ await settleConnectorWrites(engine, f.id);
+ await disposePersistenceConsumer(engine);
+ } finally { errors.mockRestore(); stderr.mockRestore(); }
+ }
+}), 300_000);
diff --git a/test/e2e/connector-segmented-page-9062.test.ts b/test/e2e/connector-segmented-page-9062.test.ts
new file mode 100644
index 0000000000..0f6fa15301
--- /dev/null
+++ b/test/e2e/connector-segmented-page-9062.test.ts
@@ -0,0 +1,4 @@
+import { registerPostgresTests } from '../helpers/test-backends.ts';
+
+// #9062: the PostgreSQL arm of the connector segmented-page suite.
+await registerPostgresTests(() => import('../connector-segmented-page-9062.test.ts'));