Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(connectors): an unreadable segmented page fails only its item on a bound source, from the first sweep (#9062)

garrytan210279e46b50
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6ef267e3ea..e7e2a8fe35 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,12 +12,13 @@ identifiers and attribution are available in the pre-removal Git revision
## [Unreleased]
-**A GitHub (or other API) connector source no longer stops for good when one of its items is a page stored in segments.** A long issue (a body plus hundreds of comments) grows past 1 MiB and is stored in segments. Before, that item could never import, and once such a page existed, every later sweep of a bound source stopped on `page_body_segmented` with no item named, so nothing after it imported and the cursor never moved. Now an item up to 24 MiB imports and re-renders like any other, and one the sweep cannot read whole is held on its own (`gbrain sources status`) while the rest of the run imports and the cursor moves. Reported and first diagnosed in #9062.
+**A GitHub (or other API) connector source no longer stops for good when one of its items is a page stored in segments.** A long issue (a body plus hundreds of comments) grows past 1 MiB and is stored in segments. Before, that item could never import, and once such a page existed, every later sweep of a bound source stopped on `page_body_segmented` with no item named, so nothing after it imported and the cursor never moved. Now an item up to 24 MiB imports and re-renders like any other, and one the sweep cannot read whole (over 24 MiB) fails on its own from the first sweep, and is held after three (`gbrain sources status`), while the rest of the run imports and the cursor moves; a new comment on it re-tries only that item. Reported and first diagnosed in #9062.
### Itemized changes
- `src/core/persistence/connector-sync.ts`: the connector publication reads a segmented page back as strings (up to 24 MiB) before it carries its timeline and fences into the re-render, seals a segmented body through its import (never the inline text seal, which read the whole timeline), and runs the import's after-commit step, so a segmented page left unsealed is chunked. The freshness read (`page()`) reads a segmented page back the same way.
- `src/core/github-source.ts`: a stored page the sweep cannot read fails that item (a content hold after three runs), not the run.
+- `src/core/connectors/connector-hold-session.ts`: on a bound source, `page_body_segmented` raised before any write request is counted against the item instead of aborting the run, as invalid connector text already was.
- Tests: `test/connector-segmented-page-9062.test.ts` (PGLite) and its PostgreSQL arm `test/e2e/connector-segmented-page-9062.test.ts`.
## [0.60.644.0] - 2026-11-10
diff --git a/src/core/connectors/connector-hold-session.ts b/src/core/connectors/connector-hold-session.ts
index 5078be21ab..83f62d8f64 100644
--- a/src/core/connectors/connector-hold-session.ts
+++ b/src/core/connectors/connector-hold-session.ts
@@ -19,11 +19,15 @@ import type { ManagedConnectorSync } from '../persistence/connector-sync.ts';
import { clearHoldRetryKeys, readHoldRetryKeys, writeHeldRetryPointer } from './item-holds-store.ts';
import { holdsExhaustedError, ItemHoldsRun, type ClassifiedConnectorError, type ItemHoldMeta, type ItemHoldsFinish, type ItemHoldsState } from './item-holds.ts';
-/** Errors a managed connector must not absorb (the run aborts). Refusals raised before submission are counted instead. */
+/**
+ * Errors a managed connector must not absorb (the run aborts). Refusals raised before submission are counted instead:
+ * invalid connector text, and (#9062) a stored page a read cannot load whole (`page_body_segmented`, a body over the
+ * single-string ceiling), which fails that item while the rest of the run imports.
+ */
export function abortsManagedRun(error: unknown): boolean {
if (error instanceof LockStolenError || (error as { name?: string } | null)?.name === 'ConnectorWaitBudgetStop') return true;
if (!(error instanceof OperationError)) return false;
- return !(error.code === 'invalid_connector_text' && !error.writeRequest);
+ return !((error.code === 'invalid_connector_text' || error.code === 'page_body_segmented') && !error.writeRequest);
}
export interface HoldRunSummary { held: number; newlyHeld: number; skipped: number; outage: boolean }
diff --git a/test/connector-segmented-page-9062.test.ts b/test/connector-segmented-page-9062.test.ts
index 6dfb6e48fc..1b8a75c03e 100644
--- a/test/connector-segmented-page-9062.test.ts
+++ b/test/connector-segmented-page-9062.test.ts
@@ -99,7 +99,13 @@ test('#9062 (bound managed source): a segmented page over the single-string ceil
await engine.executeRaw('UPDATE pages SET body_bytes=$3 WHERE source_id=$1 AND slug=$2', [f.id, 'gh/acme-example/app/1', PAGE_MAX_BYTES + 1]);
fx.issues = [{ number: 1, title: 'Synthetic long issue', body: longThread('second'), updated_at: '2026-02-02T00:00:00Z' },
{ number: 2, title: 'Synthetic short issue', body: 'Short body.', updated_at: '2026-02-03T00:00:00Z' }];
- for (let i = 0; i < 3; i++) { await run().catch(() => undefined); await settleConnectorWrites(engine, f.id).catch(() => undefined); await disposePersistenceConsumer(engine); }
+ // Every run before the hold fails only that item: the run finishes partial and the rest of the source imports.
+ for (let i = 0; i < 3; i++) {
+ expect((await run()).status).toBe('partial');
+ await settleConnectorWrites(engine, f.id);
+ await disposePersistenceConsumer(engine);
+ expect(await storage(engine, f.id, 'gh/acme-example/app/2')).toBe('inline');
+ }
const held = (await readAllSourceHolds(engine, { sourceIds: [f.id] }))[0]?.held ?? [];
expect(held.map(h => ({ key: h.key, code: h.code, class: h.class, slug: h.slug }))).toEqual([
{ key: 'acme-example/app#1', code: 'page_body_segmented', class: 'content', slug: 'gh/acme-example/app/1' }]);
@@ -114,6 +120,14 @@ test('#9062 (bound managed source): a segmented page over the single-string ceil
expect(fx.since).toEqual(['2026-02-03T00:00:00Z']);
await settleConnectorWrites(engine, f.id);
await disposePersistenceConsumer(engine);
+ // A new comment re-admits the held item; it fails again on its own and a newer item still imports.
+ fx.issues = [{ number: 1, title: 'Synthetic long issue', body: longThread('third'), updated_at: '2026-02-04T00:00:00Z' },
+ { number: 3, title: 'Synthetic newer issue', body: 'Newer body.', updated_at: '2026-02-05T00:00:00Z' }];
+ await run();
+ await settleConnectorWrites(engine, f.id);
+ await disposePersistenceConsumer(engine);
+ expect(await storage(engine, f.id, 'gh/acme-example/app/3')).toBe('inline');
+ expect((await state(engine, f.id)).last_failure).toBeNull();
} finally { errors.mockRestore(); stderr.mockRestore(); }
}
}), 300_000);