Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Train C merge fixes: sorted Postgres arm list, graduation carry count 163 (master and #9145 each added a carried table), core-services key-files page back under its size cap

garrytanea74719b9540
diff --git a/docs/architecture/key-files/core-services-1-continued.md b/docs/architecture/key-files/core-services-1-continued.md
index 0d45cc1bd5..50fa9d6759 100644
--- a/docs/architecture/key-files/core-services-1-continued.md
+++ b/docs/architecture/key-files/core-services-1-continued.md
@@ -2,7 +2,9 @@
[Subsystem index](../KEY_FILES.md) · [Previous entries](core-services-1.md). Read only the entries relevant to your change.
Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.
-The content-repair judgment tier under `src/core/content-repair/`, then the Retrieval Reflex and session-context helpers under `src/core/context/`.
+The chat-history connector layer under `src/core/connectors/`, then the content-repair judgment tier under `src/core/content-repair/`, then the Retrieval Reflex and session-context helpers under `src/core/context/`.
+
+- `src/core/connectors/` (directory) — the LIVE chat-history connector layer (front-end to the transcripts pipeline). `types.ts`: the `ChatHistoryProvider` contract (mirror of `TranscriptAdapter` — leaf module per provider; `spoolFormat` names the adapter that parses its output; dated `HostSpecTarget`). `credentials.ts`: file-plane store at `~/.gbrain/connectors/<provider>.json` (0600, dir 0700, atomic tmp+rename), env-above-file `resolveCredential` returning `{cred, source}`; credentials NEVER touch the DB / `sources.config` / config planes / op payloads. `classify.ts`: the pure §2A/§2B response classifier — distinguishes a Cloudflare-fingerprint 403 (`forbidden_fingerprint`, dead-end) from an auth-expired 403 (`auth_required`, re-auth) and flags a parses-but-no-`items` body as `drift`. `client.ts`: `ConnectorClient` modeled on `GitHubClient` (plain fetch against a hardcoded ORIGIN + off-origin guard so the cookie/bearer never leaks, `redirect: 'manual'` with any 3xx a terminal `auth_required`, browser UA, refresh-once-on-401 that is handed the client itself and never runs on a 403 (terminal, #7515), Retry-After backoff, a per-run ceiling `MAX_CONNECTOR_REQUESTS_PER_RUN` (10,000, lowered by `maxRequests`) and a `deadlineAt` checked before each request and retry wait (`ConnectorRunLimitError`), and `probeFailure`, the one error-to-`ProbeResult` table; `fetchImpl`/`sleep`/`now` injectable — NOT `fetchWithSSRFGuard`, which breaks SNI on Cloudflare hosts). `providers/chatgpt.ts` + `providers/claude.ts` (registry in `registry.ts`; Perplexity deliberately absent — no adapter): offset/updated pagination with a per-pass 500-page cap and a `stopBefore` watermark break, epoch/ISO normalization, ChatGPT's `is_archived` second pass, Claude's `chat_conversations` paged with `limit`/`offset` across every chat-capable org (conversations fetched from the org that listed them; timestamps normalized with ChatGPT's `toIso`), a ChatGPT walk that steps back when `total` shrinks mid-walk (a removal shifts later items to lower offsets) and yields each id once across both passes, content-block text assembly for Claude, list pages truncated to the requested size (a truncated page that also ignores offset is drift), Claude's `CLAUDE_MAX_ORGS` (10; more makes the listing incomplete) and `CLAUDE_MAX_REQUESTS_PER_ORG` (5,000) caps, and a `refreshAccessToken` (ChatGPT re-mints from the cookie via `/api/auth/session` through the shared client and throws its terminal errors; Claude's cookie is terminal). `spool.ts`: writes fetched conversations in native-export shape (0600, batched, pruned in the orchestrator `finally`). `sync.ts`: `runConnectorSync` — resolve credential → probe → read the **per-(provider, source) config-scalar watermark** `connectors.<p>.source.<id>.watermark_iso` (NOT op_checkpoint, whose 7-day GC would wipe it → full re-fetch) → list to `watermark − windowDays` → skip stubs whose `updatedAt` the source's `synced` ledger already holds and stubs quarantined in its `failed` ledger (after `QUARANTINE_ATTEMPTS` = 3 fetch OR ingest failures at one `updatedAt`; ingest outcomes are attributed per conversation by session id, a spool-file-level failure fails the batch; an edit or `--full` retries) BEFORE the `--limit` cap, so capped runs progress → fetch → spool → `runTranscriptsIngest` → record each successfully ingested conversation as synced → advance the watermark ONLY on a clean run (quarantined ids no longer block it; `synced` entries older than the next list window are pruned, `failed` entries never are: #6387) → `logIngest` receipt → stamp per-source `last_sync_at` → engine-branched embed kickoff (`maybeKickoffEmbed`: Postgres `submitEmbedBackfill`, PGLite `runEmbedCore` inline — NEVER `submitEmbedBackfill`, which refuses `no_worker_surface`). `failed-ledger.ts` (#6387): the `failed` ledger entry (`attempts`, `updatedAt`, `lastAttemptAt`, `nextRetryAt`, Claude `orgId`) kept until the conversation imports, and the pure `planConnectorFetch` — listed work plus due retained failures retried by id (`RETRY_BUDGET` 10 per run, oldest-due first, slots reserved under `--limit`, quarantine backoff from `item-holds.ts` `HOLD_BACKOFF_MS`/`HOLD_DAILY_MS`); on a `routesByOrg` provider an entry without `orgId` is `held` for `--full`. A sync with any entry left reports `partial` + `unresolved`; aborts and auth stops are never strikes; a complete `--full` listing drops entries it no longer contains (`gone`); `connectors status --json` lists `unresolved` and the doctor check warns "archive incomplete" (ask-first `--full` fix for held entries). `config-keys.ts`: the `connectors.` key builders (sync progress keyed by `(provider, source)`; `auth_error_at` stays per provider because it describes the shared credential), `readConnectorState`/`writeConnectorState` (a legacy per-provider `watermark_iso`/`last_sync_at` is honored only for the scheduled `connectors.source_id` source until that source writes its own key, which retires it), and the pure `isConnectorSyncStale(last, now, floor)` dispatch gate. Status, doctor and the autopilot gate read the scheduled source's state. `oauth-pkce.ts`: dependency-free S256 PKCE loopback (`Bun.serve`, timing-safe state; `portOverride: 0` plus `onListen` binds an OS-assigned port for tests while the authorize and token URLs keep `redirectPort`) — best-effort `--try-oauth` only. Surfaces: the `connectors_status`/`connector_sync` ops (`src/core/ops/connectors.ts`, both `localOnly`), the `gbrain connectors` command family (`src/commands/connectors/` peeled dir; `connectors sync` usage errors, a missing `--source` value or a missing/unknown provider, are `invalid_params` exit 2 with the envelope under `--json` and a `fix` whose `<SOURCE_ID>`/`<PROVIDER>` input it names, and `connectors sync --help` prints the usage; `test/connectors-sync-cli.test.ts`), the `connector-sync` minion handler (`src/core/minions/handlers/connector-sync.ts`, single-flight lock; rechecks `auto_sync` and returns `skipped` unless the job is `explicit`, #7515), `connectors sync --all` (only `auto_sync` providers; `--max-requests`/`--deadline` bound the invocation), the autopilot `maybeDispatchConnectorSyncs` gate, and the `connectors` doctor check. Pinned by `test/connectors-*.test.ts` + `test/e2e/connectors-sync-checkpoints-pglite.test.ts` (per-source watermark, `--limit` progress, quarantine, retitle, #6387 retained retries and the `--full` upgrade-state recovery, #6388 long-message tails) + `test/e2e/connectors-sync-pglite.test.ts` (full pipeline against a `Bun.serve` fixture backend, keyless) + `test/e2e/connector-sync-handler-pglite.test.ts` + `test/e2e/doctor-connectors-pglite.test.ts`.
- `src/core/context/title-matches.ts` — `preferEntityTitleMatches`, the volunteer title arm's one-pointer-per-name-per-source rule: exact-title hits that share a candidate norm within one source collapse to the page the `entity` card would pick (`src/core/entities/exact-match-preference.ts`, shared with `verbs/entity-card.ts`: explicit exact slug, then a linkable type of that source's pack, then any other page), ties broken by slug. Same-title pages in different sources are different entities and all stay. It runs after the privacy and activation gate (a private or withheld entity cannot suppress an eligible fallback) and leaves a norm an alias already resolved alone; the mention policy and packs are read only on a collision. Test: `test/volunteer-title-collision.test.ts`.
diff --git a/docs/architecture/key-files/core-services-1.md b/docs/architecture/key-files/core-services-1.md
index 3dc0e65feb..62285c3f47 100644
--- a/docs/architecture/key-files/core-services-1.md
+++ b/docs/architecture/key-files/core-services-1.md
@@ -48,5 +48,3 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/chunkers/symbol-resolver.ts` — resolves and stamps each bounded chunk batch in one transaction. It takes ordered source/page guards before edge reads, revalidates current chunk/page/slug identity under those guards, and commits edge metadata with its backfill watermark. Concurrent projection replacement cannot certify new edges using an old resolver read; rollback leaves both metadata and watermark unchanged. Pinned by `test/symbol-resolver-projection-race.test.ts` with PGLite serialization and a real PostgreSQL lock-wait receipt. A `calls` edge the extractor flagged `memberCall` (an untyped receiver, `obj.m()` emitted as bare `m`; persisted as `edge_metadata.member_call`) is never resolved within the file, since a member call cannot invoke a same-file top-level function (gbrain-evals N13-8, `test/code-intel/n13-edge-resolution.test.ts`); existing code pages pick this up when they re-import (`gbrain reindex-code`).
- `src/core/code-intel/recursive-walk.ts` (`code_blast` / `code_flow`) + the `chunkId` arm of `getCallersOf` / `getCalleesOf` in `src/core/engine-sql/code-edges.ts` (#5439 item 3, wave 15 P7.8) — the BFS keys its frontier and visited set by source-scoped chunk identity where the hop that reached a node was resolved (a `code_edges_chunk` row's `to_chunk_id`, or `readEdgeResolution(edge_metadata).kind === 'resolved'`), and by qualified name only for unresolved edges; the start is always a name. The next hop from a chunk node reads edges by that chunk (`opts.chunkId`: callees = both tables' `from_chunk_id`; callers = `code_edges_chunk.to_chunk_id` plus `code_edges_symbol` rows whose `edge_metadata.resolved_chunk_id` matches), under the same source filter and current-projection read policy, so two files whose top-level `helper` share the qualified name no longer fan into each other's callers or callees two hops out. `WalkNode.chunk_id` is directional: the caller's chunk on a callers hop, the resolved callee's chunk on a callees hop, absent for an unresolved callee (it used to be the edge's `from_chunk_id` on every hop). SQL lives once in the code-edges domain (SQL-text golden `sql-text/code-edges.json` `chunkId` variants). Pinned on both backends by `test/code-intel/recursive-walk-resolved-ids-5439.test.ts` (`test/e2e/code-intel-walk-resolved-ids-postgres.test.ts`).
-
-- `src/core/connectors/` (directory) — the LIVE chat-history connector layer (front-end to the transcripts pipeline). `types.ts`: the `ChatHistoryProvider` contract (mirror of `TranscriptAdapter` — leaf module per provider; `spoolFormat` names the adapter that parses its output; dated `HostSpecTarget`). `credentials.ts`: file-plane store at `~/.gbrain/connectors/<provider>.json` (0600, dir 0700, atomic tmp+rename), env-above-file `resolveCredential` returning `{cred, source}`; credentials NEVER touch the DB / `sources.config` / config planes / op payloads. `classify.ts`: the pure §2A/§2B response classifier — distinguishes a Cloudflare-fingerprint 403 (`forbidden_fingerprint`, dead-end) from an auth-expired 403 (`auth_required`, re-auth) and flags a parses-but-no-`items` body as `drift`. `client.ts`: `ConnectorClient` modeled on `GitHubClient` (plain fetch against a hardcoded ORIGIN + off-origin guard so the cookie/bearer never leaks, `redirect: 'manual'` with any 3xx a terminal `auth_required`, browser UA, refresh-once-on-401 that is handed the client itself and never runs on a 403 (terminal, #7515), Retry-After backoff, a per-run ceiling `MAX_CONNECTOR_REQUESTS_PER_RUN` (10,000, lowered by `maxRequests`) and a `deadlineAt` checked before each request and retry wait (`ConnectorRunLimitError`), and `probeFailure`, the one error-to-`ProbeResult` table; `fetchImpl`/`sleep`/`now` injectable — NOT `fetchWithSSRFGuard`, which breaks SNI on Cloudflare hosts). `providers/chatgpt.ts` + `providers/claude.ts` (registry in `registry.ts`; Perplexity deliberately absent — no adapter): offset/updated pagination with a per-pass 500-page cap and a `stopBefore` watermark break, epoch/ISO normalization, ChatGPT's `is_archived` second pass, Claude's `chat_conversations` paged with `limit`/`offset` across every chat-capable org (conversations fetched from the org that listed them; timestamps normalized with ChatGPT's `toIso`), a ChatGPT walk that steps back when `total` shrinks mid-walk (a removal shifts later items to lower offsets) and yields each id once across both passes, content-block text assembly for Claude, list pages truncated to the requested size (a truncated page that also ignores offset is drift), Claude's `CLAUDE_MAX_ORGS` (10; more makes the listing incomplete) and `CLAUDE_MAX_REQUESTS_PER_ORG` (5,000) caps, and a `refreshAccessToken` (ChatGPT re-mints from the cookie via `/api/auth/session` through the shared client and throws its terminal errors; Claude's cookie is terminal). `spool.ts`: writes fetched conversations in native-export shape (0600, batched, pruned in the orchestrator `finally`). `sync.ts`: `runConnectorSync` — resolve credential → probe → read the **per-(provider, source) config-scalar watermark** `connectors.<p>.source.<id>.watermark_iso` (NOT op_checkpoint, whose 7-day GC would wipe it → full re-fetch) → list to `watermark − windowDays` → skip stubs whose `updatedAt` the source's `synced` ledger already holds and stubs quarantined in its `failed` ledger (after `QUARANTINE_ATTEMPTS` = 3 fetch OR ingest failures at one `updatedAt`; ingest outcomes are attributed per conversation by session id, a spool-file-level failure fails the batch; an edit or `--full` retries) BEFORE the `--limit` cap, so capped runs progress → fetch → spool → `runTranscriptsIngest` → record each successfully ingested conversation as synced → advance the watermark ONLY on a clean run (quarantined ids no longer block it; `synced` entries older than the next list window are pruned, `failed` entries never are: #6387) → `logIngest` receipt → stamp per-source `last_sync_at` → engine-branched embed kickoff (`maybeKickoffEmbed`: Postgres `submitEmbedBackfill`, PGLite `runEmbedCore` inline — NEVER `submitEmbedBackfill`, which refuses `no_worker_surface`). `failed-ledger.ts` (#6387): the `failed` ledger entry (`attempts`, `updatedAt`, `lastAttemptAt`, `nextRetryAt`, Claude `orgId`) kept until the conversation imports, and the pure `planConnectorFetch` — listed work plus due retained failures retried by id (`RETRY_BUDGET` 10 per run, oldest-due first, slots reserved under `--limit`, quarantine backoff from `item-holds.ts` `HOLD_BACKOFF_MS`/`HOLD_DAILY_MS`); on a `routesByOrg` provider an entry without `orgId` is `held` for `--full`. A sync with any entry left reports `partial` + `unresolved`; aborts and auth stops are never strikes; a complete `--full` listing drops entries it no longer contains (`gone`); `connectors status --json` lists `unresolved` and the doctor check warns "archive incomplete" (ask-first `--full` fix for held entries). `config-keys.ts`: the `connectors.` key builders (sync progress keyed by `(provider, source)`; `auth_error_at` stays per provider because it describes the shared credential), `readConnectorState`/`writeConnectorState` (a legacy per-provider `watermark_iso`/`last_sync_at` is honored only for the scheduled `connectors.source_id` source until that source writes its own key, which retires it), and the pure `isConnectorSyncStale(last, now, floor)` dispatch gate. Status, doctor and the autopilot gate read the scheduled source's state. `oauth-pkce.ts`: dependency-free S256 PKCE loopback (`Bun.serve`, timing-safe state; `portOverride: 0` plus `onListen` binds an OS-assigned port for tests while the authorize and token URLs keep `redirectPort`) — best-effort `--try-oauth` only. Surfaces: the `connectors_status`/`connector_sync` ops (`src/core/ops/connectors.ts`, both `localOnly`), the `gbrain connectors` command family (`src/commands/connectors/` peeled dir; `connectors sync` usage errors, a missing `--source` value or a missing/unknown provider, are `invalid_params` exit 2 with the envelope under `--json` and a `fix` whose `<SOURCE_ID>`/`<PROVIDER>` input it names, and `connectors sync --help` prints the usage; `test/connectors-sync-cli.test.ts`), the `connector-sync` minion handler (`src/core/minions/handlers/connector-sync.ts`, single-flight lock; rechecks `auto_sync` and returns `skipped` unless the job is `explicit`, #7515), `connectors sync --all` (only `auto_sync` providers; `--max-requests`/`--deadline` bound the invocation), the autopilot `maybeDispatchConnectorSyncs` gate, and the `connectors` doctor check. Pinned by `test/connectors-*.test.ts` + `test/e2e/connectors-sync-checkpoints-pglite.test.ts` (per-source watermark, `--limit` progress, quarantine, retitle, #6387 retained retries and the `--full` upgrade-state recovery, #6388 long-message tails) + `test/e2e/connectors-sync-pglite.test.ts` (full pipeline against a `Bun.serve` fixture backend, keyless) + `test/e2e/connector-sync-handler-pglite.test.ts` + `test/e2e/doctor-connectors-pglite.test.ts`.
diff --git a/test/graduation-inventory.test.ts b/test/graduation-inventory.test.ts
index b5d0f1e8ed..6c6987ef83 100644
--- a/test/graduation-inventory.test.ts
+++ b/test/graduation-inventory.test.ts
@@ -60,7 +60,7 @@ describe('graduation inventory', () => {
expect(new Set(names).size).toBe(names.length);
const counts: Record<string, number> = {};
for (const e of GRADUATION_INVENTORY.entries) counts[e.class] = (counts[e.class] ?? 0) + 1;
- expect(counts).toEqual({ carry: 162, rebind: 2, rebuild: 3, discard: 9, schema_owned: 4 });
+ expect(counts).toEqual({ carry: 163, rebind: 2, rebuild: 3, discard: 9, schema_owned: 4 });
for (const e of GRADUATION_INVENTORY.entries) {
expect({ relation: e.relation, reason: e.reason.length > 5, present: e.engines.pglite || e.engines.postgres })
.toEqual({ relation: e.relation, reason: true, present: true });
diff --git a/test/postgres-unit-arms.txt b/test/postgres-unit-arms.txt
index d2da4e55a0..09578abc65 100644
--- a/test/postgres-unit-arms.txt
+++ b/test/postgres-unit-arms.txt
@@ -498,8 +498,8 @@ test/persistence-write-interrupted.test.ts
test/persistence-write-stall.test.ts
test/persistence-writer-fences.test.ts
test/physical-root-birth-tolerance-6501.test.ts
-test/privacy-label-ignored.test.ts
test/physical-root-moved-8625.test.ts
+test/privacy-label-ignored.test.ts
test/provenance-drift-7891.test.ts
test/provenance-stamps-7873.test.ts
test/purge-expires-page-facts.test.ts
@@ -508,8 +508,8 @@ test/purge-global-marker.test.ts
test/purge-missing-cache-6911.test.ts
test/purge-name-stores-9068.test.ts
test/purge-read-surfaces.test.ts
-test/purge-scrub-followups-9068.test.ts
test/purge-root-metafile-7613.test.ts
+test/purge-scrub-followups-9068.test.ts
test/purge-sync-tombstone-hold-7130.test.ts
test/qm-attest-write-boundary.test.ts
test/qm-memory-propose.test.ts