Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(connectors): an unreadable segmented page fails only its item on a bound source, from the first sweep (#9062)
garrytan210279e46b50
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 6ef267e3ea..e7e2a8fe35 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -12,12 +12,13 @@ identifiers and attribution are available in the pre-removal Git revision## [Unreleased]-**A GitHub (or other API) connector source no longer stops for good when one of its items is a page stored in segments.** A long issue (a body plus hundreds of comments) grows past 1 MiB and is stored in segments. Before, that item could never import, and once such a page existed, every later sweep of a bound source stopped on `page_body_segmented` with no item named, so nothing after it imported and the cursor never moved. Now an item up to 24 MiB imports and re-renders like any other, and one the sweep cannot read whole is held on its own (`gbrain sources status`) while the rest of the run imports and the cursor moves. Reported and first diagnosed in #9062.+**A GitHub (or other API) connector source no longer stops for good when one of its items is a page stored in segments.** A long issue (a body plus hundreds of comments) grows past 1 MiB and is stored in segments. Before, that item could never import, and once such a page existed, every later sweep of a bound source stopped on `page_body_segmented` with no item named, so nothing after it imported and the cursor never moved. Now an item up to 24 MiB imports and re-renders like any other, and one the sweep cannot read whole (over 24 MiB) fails on its own from the first sweep, and is held after three (`gbrain sources status`), while the rest of the run imports and the cursor moves; a new comment on it re-tries only that item. Reported and first diagnosed in #9062.### Itemized changes- `src/core/persistence/connector-sync.ts`: the connector publication reads a segmented page back as strings (up to 24 MiB) before it carries its timeline and fences into the re-render, seals a segmented body through its import (never the inline text seal, which read the whole timeline), and runs the import's after-commit step, so a segmented page left unsealed is chunked. The freshness read (`page()`) reads a segmented page back the same way.- `src/core/github-source.ts`: a stored page the sweep cannot read fails that item (a content hold after three runs), not the run.+- `src/core/connectors/connector-hold-session.ts`: on a bound source, `page_body_segmented` raised before any write request is counted against the item instead of aborting the run, as invalid connector text already was.- Tests: `test/connector-segmented-page-9062.test.ts` (PGLite) and its PostgreSQL arm `test/e2e/connector-segmented-page-9062.test.ts`.## [0.60.644.0] - 2026-11-10diff --git a/src/core/connectors/connector-hold-session.ts b/src/core/connectors/connector-hold-session.tsindex 5078be21ab..83f62d8f64 100644--- a/src/core/connectors/connector-hold-session.ts+++ b/src/core/connectors/connector-hold-session.ts@@ -19,11 +19,15 @@ import type { ManagedConnectorSync } from '../persistence/connector-sync.ts';import { clearHoldRetryKeys, readHoldRetryKeys, writeHeldRetryPointer } from './item-holds-store.ts';import { holdsExhaustedError, ItemHoldsRun, type ClassifiedConnectorError, type ItemHoldMeta, type ItemHoldsFinish, type ItemHoldsState } from './item-holds.ts';-/** Errors a managed connector must not absorb (the run aborts). Refusals raised before submission are counted instead. */+/**+ * Errors a managed connector must not absorb (the run aborts). Refusals raised before submission are counted instead:+ * invalid connector text, and (#9062) a stored page a read cannot load whole (`page_body_segmented`, a body over the+ * single-string ceiling), which fails that item while the rest of the run imports.+ */export function abortsManagedRun(error: unknown): boolean {if (error instanceof LockStolenError || (error as { name?: string } | null)?.name === 'ConnectorWaitBudgetStop') return true;if (!(error instanceof OperationError)) return false;- return !(error.code === 'invalid_connector_text' && !error.writeRequest);+ return !((error.code === 'invalid_connector_text' || error.code === 'page_body_segmented') && !error.writeRequest);}export interface HoldRunSummary { held: number; newlyHeld: number; skipped: number; outage: boolean }diff --git a/test/connector-segmented-page-9062.test.ts b/test/connector-segmented-page-9062.test.tsindex 6dfb6e48fc..1b8a75c03e 100644--- a/test/connector-segmented-page-9062.test.ts+++ b/test/connector-segmented-page-9062.test.ts@@ -99,7 +99,13 @@ test('#9062 (bound managed source): a segmented page over the single-string ceilawait engine.executeRaw('UPDATE pages SET body_bytes=$3 WHERE source_id=$1 AND slug=$2', [f.id, 'gh/acme-example/app/1', PAGE_MAX_BYTES + 1]);fx.issues = [{ number: 1, title: 'Synthetic long issue', body: longThread('second'), updated_at: '2026-02-02T00:00:00Z' },{ number: 2, title: 'Synthetic short issue', body: 'Short body.', updated_at: '2026-02-03T00:00:00Z' }];- for (let i = 0; i < 3; i++) { await run().catch(() => undefined); await settleConnectorWrites(engine, f.id).catch(() => undefined); await disposePersistenceConsumer(engine); }+ // Every run before the hold fails only that item: the run finishes partial and the rest of the source imports.+ for (let i = 0; i < 3; i++) {+ expect((await run()).status).toBe('partial');+ await settleConnectorWrites(engine, f.id);+ await disposePersistenceConsumer(engine);+ expect(await storage(engine, f.id, 'gh/acme-example/app/2')).toBe('inline');+ }const held = (await readAllSourceHolds(engine, { sourceIds: [f.id] }))[0]?.held ?? [];expect(held.map(h => ({ key: h.key, code: h.code, class: h.class, slug: h.slug }))).toEqual([{ key: 'acme-example/app#1', code: 'page_body_segmented', class: 'content', slug: 'gh/acme-example/app/1' }]);@@ -114,6 +120,14 @@ test('#9062 (bound managed source): a segmented page over the single-string ceilexpect(fx.since).toEqual(['2026-02-03T00:00:00Z']);await settleConnectorWrites(engine, f.id);await disposePersistenceConsumer(engine);+ // A new comment re-admits the held item; it fails again on its own and a newer item still imports.+ fx.issues = [{ number: 1, title: 'Synthetic long issue', body: longThread('third'), updated_at: '2026-02-04T00:00:00Z' },+ { number: 3, title: 'Synthetic newer issue', body: 'Newer body.', updated_at: '2026-02-05T00:00:00Z' }];+ await run();+ await settleConnectorWrites(engine, f.id);+ await disposePersistenceConsumer(engine);+ expect(await storage(engine, f.id, 'gh/acme-example/app/3')).toBe('inline');+ expect((await state(engine, f.id)).last_failure).toBeNull();} finally { errors.mockRestore(); stderr.mockRestore(); }}}), 300_000);