Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(google): checkpoint the contacts full re-list page by page so an expired People sync token can finish (#8906)

garrytan04c5275fdd27
diff --git a/docs/architecture/key-files/google-and-loops.md b/docs/architecture/key-files/google-and-loops.md
index 0812b0271f..ce2780977d 100644
--- a/docs/architecture/key-files/google-and-loops.md
+++ b/docs/architecture/key-files/google-and-loops.md
@@ -22,13 +22,13 @@ Historical Gmail repair selects explicit nested MIME metadata fields, excluding
- `src/core/creds/redirect.ts` — how the authorization code gets back to us: `RedirectStrategy = 'loopback' | 'paste' | 'hosted-callback'`. Loopback is an ephemeral 127.0.0.1 listener; paste mode needs NO listener (the fixed `PASTE_REDIRECT_URI` fails to load, the user pastes the full address-bar URL back) and is auto-selected by `sniffHeadless` (SSH/WSL/container/no-display). Google's device-code flow is NOT an option — Gmail/Calendar/Contacts scopes are excluded from it; don't re-litigate.
- `src/core/creds/relay-client.ts` — the typed CLIENT half of the gbrain.io zero-retention consent relay (`createSession` → user clicks the consent URL → poll a one-time `claim`; the relay deletes tokens on first successful claim or at session TTL). Inert unless `GBRAIN_OAUTH_RELAY_URL` is set (unset = the BYO flow, which always works). `test/creds-relay-client.test.ts` is the server conformance spec — a server that round-trips those fixtures is compatible.
- `src/core/creds/export.ts` — versioned passphrase-encrypted credential bundles (scrypt N=2^15, r=8, p=1 → AES-256-GCM) for machine moves and hosted-upgrade transfer. A bundle carries selected vault entries PLUS the provider client records they depend on (Google refresh tokens are bound to the client that minted them — moving one without the other produces dead tokens). Format frozen here; hosted's import endpoint conforms to it.
-- `src/core/google/types.ts` — pure data shapes for the google source kind: normalized Gmail/Calendar/People payloads, `GoogleSourceConfig` (account pointer, services, historyDays, managed dir), and the `GoogleSourceState` cursor file persisted at `<managed dir>/.google-source.json` (gmail history id, downward-moving backfill floor, per-service syncTokens). No I/O. Exports `DEFAULT_CALENDAR_ID = 'primary'` (the Calendar API's own alias; every `'primary'` literal in google-clients/google-source/sources/sources-ops resolves through it) and `GoogleSourceConfig.calendarId` (one calendar per source). `GoogleSourceState.calendar_id` records which calendar `calendar_sync_token` was minted for — a token is only valid against its own calendar; absent on legacy state, which is therefore primary's. `GoogleSourceState.calendar_horizon_ms` is the ceiling of the last calendar list that reached it (calendar-window.ts); absent on state saved before it existed. `GoogleSourceState.contacts_person_fields` (#6324) is the People field mask `contacts_sync_token` was minted under, written only together with the token; absent on legacy state, which the sweep therefore lists fully once. `contact_groups` caches the group directory (resource name → display name) and is never authoritative on its own. `ContactData` carries `phones: string[]` and `groups: {id, name|null}[]`. `GoogleSourceConfig.futureDays` (#5442) and `loopsExcludeLabels` (#5445) are optional and absent when their keys are unset; `source-config.ts` parses them (`g_future_days` 1..3650 floored, `g_loops_exclude_labels` through `parseExcludeLabelTokens`) and `googleTuningConfig` is the one writer of those keys plus `g_calendar_id`, each only when set, for `addSource`. Both `sources add` parsers accept `--future-days` and `--loops-exclude-labels`: the connected-source parser validates through the operation contract (`sources-lifecycle-args.ts`), the legacy parser through `commands/sources-google-flags.ts` (`parseGoogleAddFlag`, exit 2 on a bad value), so neither `runAdd` nor `addSource` grows for a new tuning key.
+- `src/core/google/types.ts` — pure data shapes for the google source kind: normalized Gmail/Calendar/People payloads, `GoogleSourceConfig` (account pointer, services, historyDays, managed dir), and the `GoogleSourceState` cursor file persisted at `<managed dir>/.google-source.json` (gmail history id, downward-moving backfill floor, per-service syncTokens). No I/O. Exports `DEFAULT_CALENDAR_ID = 'primary'` (the Calendar API's own alias; every `'primary'` literal in google-clients/google-source/sources/sources-ops resolves through it) and `GoogleSourceConfig.calendarId` (one calendar per source). `GoogleSourceState.calendar_id` records which calendar `calendar_sync_token` was minted for — a token is only valid against its own calendar; absent on legacy state, which is therefore primary's. `GoogleSourceState.calendar_horizon_ms` is the ceiling of the last calendar list that reached it (calendar-window.ts); absent on state saved before it existed. `GoogleSourceState.contacts_person_fields` (#6324) is the field mask `contacts_sync_token` was minted under and `contacts_relist` (#8906) the full re-list's resume page. `contact_groups` caches the group directory (resource name → display name) and is never authoritative on its own. `GoogleSourceConfig.futureDays` (#5442) and `loopsExcludeLabels` (#5445) are optional and absent when their keys are unset; `source-config.ts` parses them (`g_future_days` 1..3650 floored, `g_loops_exclude_labels` through `parseExcludeLabelTokens`) and `googleTuningConfig` is the one writer of those keys plus `g_calendar_id`, each only when set, for `addSource`. Both `sources add` parsers accept `--future-days` and `--loops-exclude-labels`: the connected-source parser validates through the operation contract (`sources-lifecycle-args.ts`), the legacy parser through `commands/sources-google-flags.ts` (`parseGoogleAddFlag`, exit 2 on a bad value), so neither `runAdd` nor `addSource` grows for a new tuning key.
- `src/core/google/access.ts` — the pluggable Google-access seam: `GoogleAccessProvider` (`getAccessToken`/`forceRefresh`) with `CommandAccessProvider` (`--access command`: any CLI that prints a token — gog/gcloud/a gateway's mint command; parsed as a bare token line or JSON `{token|access_token, expiry|expires_in}`, cached until expiry with a 60s margin, 30s exec timeout, failures = `access_command_failed` carrying the stderr tail) and `EnvAccessProvider` (`--access env`: a live token read from a NAMED env var each call, refreshed outside gbrain; missing = `access_env_missing`). The vault flow's `GoogleTokenProvider` satisfies the same interface. The token command executes only in the locally-running sync (google config keys are unreachable over MCP; same trust class as recipe health-check argv).
-- `src/core/google/google-clients.ts` — hand-rolled Gmail/Calendar/People REST clients (house style, no googleapis dependency): auth via any `GoogleAccessProvider` (vault-backed `GoogleTokenProvider` by default), 401 → forceRefresh + single retry, Retry-After honored (delta-seconds AND http-date), 403 disabled API → `GoogleApiDisabledError` and insufficient scope → `scope_missing` ([API access](google-api-access.md)), uniform pageToken pagination with a safety cap, `fetchImpl` injectable for tests. `extractCalendarMethod(part)` walks the MIME tree for a `text/calendar`/`application/ics` part and reads the iCalendar `method=` parameter from the PART's own `Content-Type` header (`headers[]` on nested parts with `format=full` — Gmail's `MessagePart.mimeType` is the BARE media type, so a mimeType-only parse would read '' for every real invite and silently downgrade the structural signal to the subject fallback); a mimeType that still carries params is the fallback parse; a bare `.ics` filename with a non-calendar MIME type claims nothing. `CalendarClient.listEvents({calendarId})` sweeps one calendar (default `DEFAULT_CALENDAR_ID`). A People request with a nonempty `syncToken` maps HTTP 400 to `GoogleCursorExpiredError` only for typed `google.rpc.ErrorInfo` reason `EXPIRED_SYNC_TOKEN` or a message beginning `Sync token [is] expired`; any other 400 on such a request is a `GoogleSyncTokenRejectedError` (a `CredentialError('upstream')` subclass the contacts sweep alone treats as cursor-invalid, #6324); initial-list requests and other APIs remain plain upstream failures. `CONTACT_PERSON_FIELDS` (`names,emailAddresses,organizations,phoneNumbers,memberships`) is the one People field mask, repeated on every page of a list; `ContactData` carries deduplicated E.164 `phones` and user-group `groups` (`{id, name: null}`, system groups dropped), and `listContactGroups()` walks `contactGroups` for at most ten 1000-row pages and reports `complete: false` past that. The contacts sweep uses its existing bounded full re-list recovery without restarting Gmail. The Gmail mapper also captures `precedence` (lower-cased), `listId` (present or not) and `replyTo` for the automated-mail loop rule (#5012). A 403 that is not API-disabled, rate-limit or quota is `GoogleForbiddenError` (still `CredentialError` `upstream`); `calendarUnreadableStatus(e)` maps it and a 404 `GoogleCursorExpiredError` to 403/404, and `CalendarClient.probeCalendar(id)` is one `maxResults=1` read. Pinned by injected-response tests in `test/google-clients.test.ts`, `test/contacts-phone-only-groups-6324.test.ts` and `test/google-source-reconcile.test.ts`, not live-provider validation.
+- `src/core/google/google-clients.ts` — hand-rolled Gmail/Calendar/People REST clients (house style, no googleapis dependency): auth via any `GoogleAccessProvider` (vault-backed `GoogleTokenProvider` by default), 401 → forceRefresh + single retry, Retry-After honored (delta-seconds AND http-date), 403 disabled API → `GoogleApiDisabledError` and insufficient scope → `scope_missing` ([API access](google-api-access.md)), uniform pageToken pagination with a safety cap, `fetchImpl` injectable for tests. `extractCalendarMethod(part)` walks the MIME tree for a `text/calendar`/`application/ics` part and reads the iCalendar `method=` parameter from the PART's own `Content-Type` header (`headers[]` on nested parts with `format=full` — Gmail's `MessagePart.mimeType` is the BARE media type, so a mimeType-only parse would read '' for every real invite and silently downgrade the structural signal to the subject fallback); a mimeType that still carries params is the fallback parse; a bare `.ics` filename with a non-calendar MIME type claims nothing. `CalendarClient.listEvents({calendarId})` sweeps one calendar (default `DEFAULT_CALENDAR_ID`). A People request with a nonempty `syncToken` maps HTTP 400 to `GoogleCursorExpiredError` only for typed `google.rpc.ErrorInfo` reason `EXPIRED_SYNC_TOKEN` or a message beginning `Sync token [is] expired`; any other 400 on such a request is a `GoogleSyncTokenRejectedError` (a `CredentialError('upstream')` subclass the contacts sweep alone treats as cursor-invalid, #6324); a connections 400 with a `pageToken` is `GoogleCursorExpiredError` (#8906, `listConnectionsPage` reads one page); other initial-list requests and APIs remain plain upstream failures. `CONTACT_PERSON_FIELDS` (`names,emailAddresses,organizations,phoneNumbers,memberships`) is the one People field mask, repeated on every page of a list; `ContactData` carries deduplicated E.164 `phones` and user-group `groups` (`{id, name: null}`, system groups dropped), and `listContactGroups()` walks `contactGroups` for at most ten 1000-row pages and reports `complete: false` past that. The Gmail mapper also captures `precedence` (lower-cased), `listId` (present or not) and `replyTo` for the automated-mail loop rule (#5012). A 403 that is not API-disabled, rate-limit or quota is `GoogleForbiddenError` (still `CredentialError` `upstream`); `calendarUnreadableStatus(e)` maps it and a 404 `GoogleCursorExpiredError` to 403/404, and `CalendarClient.probeCalendar(id)` is one `maxResults=1` read. Pinned by injected-response tests in `test/google-clients.test.ts`, `test/contacts-phone-only-groups-6324.test.ts` and `test/google-source-reconcile.test.ts`, not live-provider validation.
- `src/core/google/gmail-categories.ts` — `BULK_CATEGORY_LABELS` (`CATEGORY_PROMOTIONS`/`SOCIAL`/`FORUMS`) and `hasBulkCategory`, the one list both open-loop lanes read (#5103): `loopExtractionEligibility` returns `bulk_category` and `detectThreadLoop` withholds an inbound open, in both cases unless the owner wrote a substantive message in the thread; `CATEGORY_UPDATES` is deliberately not in it. `isNoiseSender` additionally matches `GOOGLE_SYSTEM_SENDERS` (the Meet notes digest address) by equality only.
- `src/core/google/google-render.ts` — `isPositivelyAutomated(meta)` (#5012): `Precedence: bulk|list|junk`, a `List-Id`, a noise From or a noise Reply-To. Pure render functions (`{relPath, markdown}` out; no I/O, no engine): thread pages under `emails/YYYY/MM/` (type email), events under `calendar/YYYY/MM/` (type meeting), contacts under `people/` (type person). Gmail deep links are code-generated via the typed `emailCitation` scaffold, never LLM-composed. The noise/signature rules `recipes/email-to-brain.md` specifies as prose for agent-authored collectors are implemented here — keep the recipe and this module in sync. `isCalendarSystemMail({calendarMethod, subject})` gates BOTH loop detectors: the PRIMARY signal is a `calendarMethod` in the `CALENDAR_METHODS` allowlist (RFC 5546 — REQUEST, REPLY, CANCEL, PUBLISH, COUNTER, DECLINECOUNTER, REFRESH, ADD; case-insensitive); an empty or unrecognised method is NOT a stamp and falls to the subject fallback, which is anchored to the start of the subject, refuses any Re:/Fwd: prefix, and matches only Calendar's OWN localised headers (`Invitation:`, `Updated invitation:`, `Accepted:`, `Declined:`, `Tentative:`, `Canceled event:` + localised equivalents) — a generic word like `Notification:` is a human/vendor subject and never matches. `renderThreadPage` stamps `participants:` (every address on the thread) AND `senders:` (sorted unique message AUTHORS only — the list `loops mute sender` gates on, so muting one person cannot silence a whole group thread). `renderPersonPage` (#6324) renders a contact with emails OR phones: `emails:` always, `phones:` and `groups:` (names, or the raw `contactGroups/<id>` when unresolved) only when non-empty, phones never in `aliases`; a contact with neither name nor email lives at `contactIdSlug` = `<dir>/contact-<sha8(resourceName)>`, which `personSlugFromContact` returns for a phone-only nameless contact and the sweep's tombstone path lists as a delete candidate. Pinned by `test/google-render.test.ts` and `test/contacts-phone-only-groups-6324.test.ts`.
- `src/core/google/google-source.ts` — the Google API source sweep. Unmanaged mode materializes Markdown under the source directory and imports through the standard chunks/aliases/links pipeline; managed mode uses journaled connector admission. Sweep order contacts → calendar → gmail: alias rows must exist before the loop detector resolves counterparties. Per-service independent cursors: contacts/calendar syncToken commits only after that service's sweep left no failing item that is not yet held (410 GONE drops the token and re-runs windowed); GBRA-130 A-3: contacts and events go through connector item holds (`contact:`/`event:` keys), so one failing item no longer pins the token or stops Gmail (`test/pathological/a3-google-items.test.ts`); gmail delta rides history.list with an expired-history windowed fallback; the INITIAL backfill drains newest→oldest with a batch-committed floor cursor so a killed 50k-message backfill resumes at the floor instead of restarting, and the historyId anchor is captured BEFORE the backfill so the delta lane takes over with zero gap (overlap re-renders are idempotent). Emits the `sync.google_materialize` progress phase (docs/progress-events.md). Access resolves per source config: the vault (default), a token-printing command (`g_access: command` — gog/gcloud/gateway; identity entry synthesized, scope preflight trusts the configured services, send-as aliases fetched live best-effort), or a named env var (`g_access: env`). Secrets never land in `sources.config` (the command/env NAME is config; tokens are not). Honesty invariants: `last_sync_at` is gated on the GMAIL sweep's success (it feeds `gbrain waiting`'s trust-critical staleness gate, which protects loop freshness — a gmail sweep with real thread failures must not advance it); contacts/calendar failures mark the run `partial` but do not block the stamp, and a source without gmail in its services stamps unconditionally. A thread whose fetch fails on consecutive sweeps lands in the poison ledger and is skipped (steady-state runs honor the ledger; `--full` retries with a fresh one), so one bad thread can't wedge the sync while a poison-skip alone doesn't count as failure. Contact reconcile resolves by contact resourceName (DB lookup by contact id FIRST): deletion tombstones carry only `resourceName + deleted` (no names — slug derivation yields null) and a renamed contact derives a DIFFERENT slug, so both must land on the existing page by id, never by slug. The calendar sweep binds its sync token to its calendar: `sweepCalendar` compares `state.calendar_id` (legacy state without it is treated as primary's) with the configured `calendarId`; on a mismatch it logs `[google] calendar changed (<old> → <new>)`, discards the token and takes the windowed first-sync path (no `syncToken` on the wire), and a freshly banked token stores the calendar id beside it — pairing a re-pointed source with the previous calendar's cursor would import a foreign delta. Pages from the previous calendar are not reconciled on the switch (logged; a filed follow-up). `enqueueLoopsExtraction` enqueues EVERY eligible thread (newest first) under a waiting-depth budget `LOOPS_EXTRACT_ENQUEUE_CEILING − waiting` that counts only THIS source's waiting `loops_extract` jobs (`data->>'sourceId'`), so one account's stalled backlog cannot pin another source's budget at 0; overflow is a logged deferral (a deferred thread re-candidates when it changes) and a failed probe fails open. The budget probe counts PENDING rows — `status IN ('waiting','delayed','active')` — so a retry backlog parked in `delayed` during a flapping provider cannot stack past the ceiling. With no chat provider available (`isAvailable('chat')` false) the enqueue is skipped entirely and one stderr line per sweep names the reason and the count; pages still import and the threads re-candidate on their next touch or a `--full` sweep. Managed Google API sync uses journaled connector checkpoints; an unbound source is explicitly `connector_database`, while a bound source publishes canonical files. Identity-query storage failures do not become successful absence or advance the checkpoint. Unsupported Git/dry-run flags refuse before credentials/network; legacy materialization remains the unmanaged path. FILE MODES (every write, both paths): `writeGoogleState` writes `.google-source.json` through `atomicWriteFileSync(..., { mode: 0o600 })` under `mkdirPrivate(dir)`; a corrupt state file is chmod-ed 0600 and then renamed to `.corrupt`, and if either step fails stderr names the path, the failed step, the exposure and the fix (`chmod 600 <path>` or delete it) while the sync continues with empty cursors. A legacy page goes through `writePrivateTemp`: a stale `<page>.tmp` is removed with `lstat` (never followed; a stale `.tmp` directory fails that item with `Stale temporary path <path> is a directory; remove it and re-run the sync.`), the temp file is created `wx` at 0600 and fchmod-ed past the umask, imported, then renamed over the page; page directories come from `mkdirPrivate(dirname(page), cfg.dir)`. 0600 is reasserted on every rewrite, so a looser mode set by hand is reverted on the next rewrite (unchanged pages are not rewritten). Managed pages get the same modes through `publishMode` (see the persistence coordinator). Mode assertions are skipped on win32. Tests: `test/google-file-modes.test.ts` (child process under umask 0000, `test/helpers/google-file-modes-child.ts`). The loops enqueue runs on every non-aborted sweep in both persistence modes (`--no-extract` gates only the inline link/timeline extract), logs every skip reason and reports `loops_enqueue { enqueued, deferred, skipped_reason }` in the sync result; job payloads carry the page revision (`newestMs`). After the delta and item-hold passes the sweep seeds and settles grace holds, and a managed sweep runs the 30-day catch-up (both in `loop-catchup.ts`); a partial managed run publishes that state with its item holds (`publishHolds` extra), never the Gmail cursor.
-- `src/core/google/calendar-window.ts` — which calendar events a google source keeps pages for: `[now - historyDays, now + futureDays]` (`g_future_days`, 1..3650, default `CALENDAR_HORIZON_DAYS` = 60; the parsed `futureDays` leaf exists only when the key is set, so an unset knob leaves every existing source's connector identity unchanged), checked by the sweep itself on every list because a syncToken delta cannot carry timeMin/timeMax. `CalendarSyncWindow` is built once per sweep: `side(start, end)` follows Google's rule (an event is `before` when it ends at or before the floor, `after` when it starts at or past the ceiling, unparseable dates stay `inside`); `coverageStartMs(horizon, listedWholeWindow)` says where the catch-up list for unchanged instances crossing the leading edge starts (none after a whole-window list, now for state without `calendar_horizon_ms`, the recorded horizon once the ceiling is a day past it); `startsInside` and `startsAfter` are the `--full` reconcile tests. `planCalendarPage` maps one listed event to a page change (a cancellation removes its page anywhere, aged-out history is left as it is, a page past the horizon is removed, otherwise write after removing a copy at an older path). `unlistedCalendarPages` returns sweep pages (with an `event_id`) starting inside the window that a complete `--full` list did not name; `staleCalendarPages` adds the sweep pages starting at or past the ceiling, which no windowed list can name (#5442), and is what the `--full` reconcile deletes through. `people-calendar-sweep.ts` (the contacts and calendar sweeps, peeled from `google-source.ts`) performs the changes with `importRendered`/`deletePageByRelPath`, discards the catch-up list's sync token, advances `calendar_horizon_ms` only after a list reached the ceiling, and sends reconcile deletes through the >200 mass-delete guard (a refusal marks the run partial). Its contacts cursor contract (#6324): `contacts_sync_token` and `contacts_person_fields` (the mask that minted it) commit together after the whole sweep, never on abort or import failure; a recorded mask other than `CONTACT_PERSON_FIELDS` (legacy state has none) drops the token for one full list; an expired or `GoogleSyncTokenRejectedError` token gets exactly one full-list attempt and a second failure propagates with the token nulled, not blessed; a full list that returns no replacement token leaves `contacts_sync_token` null when the mask changed (another full list follows) and keeps the old token when it did not. `resolveContactGroups` reads the directory only when a listed contact carries a user group, replaces the `contact_groups` state cache on a complete read, merges and logs on an incomplete one, and falls back to the cache when the read fails; unresolved groups render as raw ids. Pinned by `test/contacts-phone-only-groups-6324.test.ts`; the migration-176 test in `test/connector-wave3.test.ts` asserts the resumed legacy token is NOT reused (first sweep lists without it, the second reuses the fresh one). Tests: `test/google-calendar-window.test.ts` (pure rules), `test/google-source-reconcile.test.ts` (calendar window suite).
+- `src/core/google/calendar-window.ts` — which calendar events a google source keeps pages for: `[now - historyDays, now + futureDays]` (`g_future_days`, 1..3650, default `CALENDAR_HORIZON_DAYS` = 60; the parsed `futureDays` leaf exists only when the key is set, so an unset knob leaves every existing source's connector identity unchanged), checked by the sweep itself on every list because a syncToken delta cannot carry timeMin/timeMax. `CalendarSyncWindow` is built once per sweep: `side(start, end)` follows Google's rule (an event is `before` when it ends at or before the floor, `after` when it starts at or past the ceiling, unparseable dates stay `inside`); `coverageStartMs(horizon, listedWholeWindow)` says where the catch-up list for unchanged instances crossing the leading edge starts (none after a whole-window list, now for state without `calendar_horizon_ms`, the recorded horizon once the ceiling is a day past it); `startsInside` and `startsAfter` are the `--full` reconcile tests. `planCalendarPage` maps one listed event to a page change (a cancellation removes its page anywhere, aged-out history is left as it is, a page past the horizon is removed, otherwise write after removing a copy at an older path). `unlistedCalendarPages` returns sweep pages (with an `event_id`) starting inside the window that a complete `--full` list did not name; `staleCalendarPages` adds the sweep pages starting at or past the ceiling, which no windowed list can name (#5442), and is what the `--full` reconcile deletes through. `people-calendar-sweep.ts` (the contacts and calendar sweeps, peeled from `google-source.ts`) performs the changes with `importRendered`/`deletePageByRelPath`, discards the catch-up list's sync token, advances `calendar_horizon_ms` only after a list reached the ceiling, and sends reconcile deletes through the >200 mass-delete guard (a refusal marks the run partial). Its contacts cursor contract (#6324): `contacts_sync_token` and `contacts_person_fields` (the mask that minted it) commit together after the whole sweep, never on abort or import failure; a recorded mask other than `CONTACT_PERSON_FIELDS` (legacy state has none) drops the token for one full list; an expired or `GoogleSyncTokenRejectedError` token gets one full list, whose failure propagates with the token nulled; a full list with no replacement token nulls `contacts_sync_token` when the mask changed and keeps it otherwise. A full list (#8906) checkpoints `contacts_relist` per clean page, stops at `CONTACT_RELIST_PAGES_PER_RUN` (10) pages (not partial), resumes under the same mask, restarts once on a stale saved token, clears with the token. `resolveContactGroups` reads the directory once, when a listed contact first has a user group: a complete read replaces the `contact_groups` cache, an incomplete one merges and logs, a failed one falls back to it; unresolved groups render as raw ids. Pinned by `test/contacts-phone-only-groups-6324.test.ts` and `test/contacts-relist-checkpoint{-8906.slow,-managed-8906}.test.ts`; the migration-176 test in `test/connector-wave3.test.ts` asserts the resumed legacy token is not reused. Tests: `test/google-calendar-window.test.ts` (pure rules), `test/google-source-reconcile.test.ts` (calendar window suite).
- `src/core/google/file-modes.ts` — detection and tightening for group/world-readable files gbrain wrote under a Google source directory outside `~/.gbrain` (inside it the 0700 home already protects them). `scanGoogleFileModes(engine, { sourceIds? })` reads non-archived Google sources, skips dirs inside the home (`outsideGbrainHome`, realpath-aware), and inspects only gbrain's own layout: `.google-source.json*` in the root, every recorded page `source_path` under `emails/`, `calendar/`, `people/` (malformed paths ignored), each page's `.tmp`, and the directories between the root and those pages. Each path is `lstat`-ed component by component (a symlink anywhere on the path, another owner, or the wrong file type is counted in `skipped`, never followed); an entry is loose when `mode & 0o077`. `tightenGoogleEntry(root, entry)` re-checks the path, opens the leaf `O_NOFOLLOW` (`O_DIRECTORY` for dirs), re-checks the owner on the fd and `fchmod`s `mode & ~0o077`, so a swap after the scan is refused, not followed; the user-chosen root is never in the candidate set. `googleFileModesPreviewCommand`/`googleFileModesApplyCommand` build the `gbrain repair google-file-modes --source <id> [--apply]` strings every surface prints. win32: scan returns nothing. Tests: `test/google-file-modes-repair.test.ts`.
- `src/commands/google.ts` — `gbrain google connect|status|calendars|disconnect` (+ `setup` and `attachments` dispatch). Agent-first contract: every subcommand supports `--json` emitting `{ ok, status, next_action: { command?, user_message? }, error? }` (`calendars` adds `account` + `calendars[]` — id/summary/primary/accessRole — and carries the `sources add --calendar-id` template in `next_action.command`; it resolves a lone connected account, exits 2 when there are none or several without `--account`, and throws `not_connected` for an unknown `--account`); human copy the harness must relay verbatim is fenced in `[SHOW USER] ... [/SHOW USER]` blocks; secret intake is `--client-json <path|->` (preferred) / env `GOOGLE_CLIENT_ID`+`GOOGLE_CLIENT_SECRET` / TTY prompt, with raw argv flags accepted but documented last-resort; connect is an idempotent state machine (detects what exists, performs only the missing step — re-running is the documented fix for most errors). Credential commands are engine-free; `status` has a best-effort linked-sources listing, while `attachments backfill` requires the explicitly selected local managed brain. Funnel events append to `~/.gbrain/integrations/google/heartbeat.jsonl`; account addresses are hashed to a short non-reversible tag, never raw. The vault entry's `meta.scopes` records what Google ACTUALLY granted (the token response's `scope` — the consent screen lets users uncheck scopes; the requested set is only the fallback), which is what lets downstream preflights report `scope_missing` instead of opaque per-sweep 403s. `status` reports each linked source's calendar readability (`google-calendar-status.ts`, live probe unless `--no-probe`); an unreadable calendar makes the JSON `ok: false` and the exit 1.
- `src/commands/google-calendar-status.ts` — calendar readability per linked Google source for `gbrain google status` (#5012 item 9). `sourceCalendarReadability` returns null for a source that does not sync Calendar; with a probe it makes one live read of the configured calendar, otherwise it reports the `calendar_unreadable` record the last sweep stored in connector state (only when it names the current calendar id). An unreadable result carries `code: calendar_unreadable`, the HTTP status and `calendarUnreadableFix` (`gbrain google calendars --account <a>`, `gbrain sources set-calendar <src> <calendar-id>`). The sweep side is `people-calendar-sweep.ts` `recordCalendarUnreadable`: a 403/404 from `events.list` stores the record, logs both commands and returns without failing the run; a successful list clears it. Tests: `test/google-calendar-unreadable-5012.test.ts`.
diff --git a/docs/guides/google-connect.md b/docs/guides/google-connect.md
index 1829010616..6dcda89aa0 100644
--- a/docs/guides/google-connect.md
+++ b/docs/guides/google-connect.md
@@ -507,9 +507,14 @@ attached. The catalog (also emitted as structured JSON with `--json`):
Cursor expiries (`historyId` older than ~a week, calendar/contacts
`syncToken` 410, or a contacts HTTP 400 explicitly reporting an expired sync
token) use bounded re-lists automatically. Contacts recovery refreshes its
-own cursor without restarting Gmail. A contacts HTTP 400 on a sync-token
-request that is not an expiry (a token minted under other request
-parameters) gets one full-list attempt; every other HTTP 400 remains an
+own cursor without restarting Gmail. A full contacts list (first sync, an
+expired token, `--full`) saves its page cursor after each page and lists at
+most 10 pages (2,000 contacts) per sync, so a large address book finishes
+over several syncs, each resuming where the last stopped, while Calendar and
+Gmail still run every time. A contacts HTTP 400 on a sync-token request that
+is not an expiry (a token minted under other request parameters) gets one
+full-list attempt, and a saved page cursor Google no longer accepts (HTTP 400
+or 410) restarts the list from page 1 once; every other HTTP 400 remains an
upstream failure rather than triggering a re-list.
## Custody, privacy, spend
diff --git a/src/core/google/google-clients.ts b/src/core/google/google-clients.ts
index f1ab4e460f..f48e0df149 100644
--- a/src/core/google/google-clients.ts
+++ b/src/core/google/google-clients.ts
@@ -171,6 +171,11 @@ export class GoogleApiClient {
if (expired) throw new GoogleCursorExpiredError(res.status, url);
throw new GoogleSyncTokenRejectedError(url, body.error?.message ?? 'no detail');
}
+ // #8906: a connections pageToken Google no longer honors (a re-list resumed from its checkpoint).
+ if (res.status === 400 && apiHint === 'people') {
+ const u = new URL(url);
+ if (u.pathname.endsWith('/people/me/connections') && u.searchParams.get('pageToken')) throw new GoogleCursorExpiredError(res.status, url);
+ }
throw new CredentialError('upstream', `: HTTP ${res.status} on ${apiHint} (${body.error?.message ?? 'no detail'})`);
}
throw new CredentialError('upstream', `: unreachable ${apiHint}`);
@@ -769,6 +774,13 @@ function contactData(p: RawPerson): ContactData {
};
}
+function connectionsUrl(syncToken: string | null | undefined, pageToken: string | null): string {
+ const params = new URLSearchParams({ personFields: CONTACT_PERSON_FIELDS, pageSize: '200', requestSyncToken: 'true' });
+ if (syncToken) params.set('syncToken', syncToken);
+ if (pageToken) params.set('pageToken', pageToken);
+ return `${PEOPLE_BASE}/people/me/connections?${params.toString()}`;
+}
+
export class PeopleClient extends GoogleApiClient {
/** Incremental with syncToken; full otherwise. 410 → GoogleCursorExpiredError. */
async listConnections(opts: {
@@ -777,16 +789,7 @@ export class PeopleClient extends GoogleApiClient {
}): Promise<{ contacts: ContactData[]; nextSyncToken: string | null }> {
let nextSyncToken: string | null = null;
const raw = await this.drainPages<RawPerson>(
- (t) => {
- const params = new URLSearchParams({
- personFields: CONTACT_PERSON_FIELDS,
- pageSize: '200',
- requestSyncToken: 'true',
- });
- if (opts.syncToken) params.set('syncToken', opts.syncToken);
- if (t) params.set('pageToken', t);
- return `${PEOPLE_BASE}/people/me/connections?${params.toString()}`;
- },
+ (t) => connectionsUrl(opts.syncToken, t),
(body) => {
if (typeof body.nextSyncToken === 'string') nextSyncToken = body.nextSyncToken;
return {
@@ -800,6 +803,21 @@ export class PeopleClient extends GoogleApiClient {
return { contacts: raw.map(contactData), nextSyncToken };
}
+ /**
+ * #8906: one page of a full list, so a re-list can checkpoint between pages.
+ * `nextSyncToken` arrives only on the last page; a stale `pageToken` (400 or
+ * 410) is a GoogleCursorExpiredError.
+ */
+ async listConnectionsPage(pageToken: string | null, opts: { signal?: AbortSignal } = {}):
+ Promise<{ contacts: ContactData[]; nextPageToken: string | null; nextSyncToken: string | null }> {
+ const body = await this.fetchJSON<Record<string, unknown>>(connectionsUrl(null, pageToken), 'people', opts);
+ return {
+ contacts: ((body.connections as RawPerson[] | undefined) ?? []).map(contactData),
+ nextPageToken: (body.nextPageToken as string | undefined) ?? null,
+ nextSyncToken: typeof body.nextSyncToken === 'string' ? body.nextSyncToken : null,
+ };
+ }
+
/**
* GBRA-130 A-3: one contact by resource name, for re-attempting a held
* contact that no listing of this run returns. Null when it is gone (404/410).
diff --git a/src/core/google/google-source.ts b/src/core/google/google-source.ts
index f8ac5059be..127143c4ee 100644
--- a/src/core/google/google-source.ts
+++ b/src/core/google/google-source.ts
@@ -819,7 +819,7 @@ async function runGoogleSyncInner(engine: BrainEngine, sourceId: string, cfg: Go
if (activeServices.includes('contacts')) {
const stop = startHeartbeat(progress, 'contacts sweep');
try {
- await sweepContacts(deps, people, state, activePack, summary, countedSlugs, holds);
+ await sweepContacts(deps, people, state, activePack, summary, countedSlugs, holds, () => saveGoogleState(deps, state));
reached.add('contacts');
} catch (e) {
if (managed) rethrowConnectorWriteError(e);
diff --git a/src/core/google/people-calendar-sweep.ts b/src/core/google/people-calendar-sweep.ts
index 59837d5dac..dd2254c2b3 100644
--- a/src/core/google/people-calendar-sweep.ts
+++ b/src/core/google/people-calendar-sweep.ts
@@ -2,7 +2,8 @@
* people-calendar-sweep — the Contacts and Calendar sweeps of the google
* source kind (peeled from google-source.ts). Cursor discipline per service:
* the syncToken is committed only after that service's fully-successful
- * sweep; an expired token drops it and re-lists windowed.
+ * sweep; an expired token drops it and re-lists windowed. A full contacts
+ * re-list checkpoints its pageToken page by page and resumes there (#8906).
*
* GBRA-130 A-3: each contact (`contact:<resourceName>`) and each event
* (`event:<calendarId>:<eventId>`) goes through the connector item holds
@@ -31,6 +32,9 @@ import { deletePageByRelPath, importRendered, type ActivePack, type GoogleSyncDe
// ── Contacts sweep ───────────────────────────────────────────────────────────
+/** #8906: pages of a full re-list one run applies before the rest of the sync gets its turn; the next run resumes after them. */
+export const CONTACT_RELIST_PAGES_PER_RUN = 10;
+
export async function sweepContacts(
deps: GoogleSyncDeps,
people: PeopleClient,
@@ -39,6 +43,7 @@ export async function sweepContacts(
summary: GoogleSyncSummary,
countedSlugs: Set<string>,
holds?: ConnectorHoldSession,
+ checkpoint: () => Promise<void> = async () => {},
): Promise<void> {
const signal = deps.opts.signal ? { signal: deps.opts.signal } : {};
// #6324: a syncToken is only valid with the request parameters that minted
@@ -46,28 +51,73 @@ export async function sweepContacts(
// recorded) is unusable, so the sweep lists fully once and then records the
// new token with the mask it was minted under.
const maskChanged = state.contacts_person_fields !== CONTACT_PERSON_FIELDS;
- const syncToken = deps.opts.full || maskChanged ? null : state.contacts_sync_token;
- if (maskChanged && state.contacts_sync_token && !deps.opts.full) deps.log('[google] contacts field mask changed; full re-list');
- let result;
- try {
- result = await people.listConnections({ syncToken, ...signal });
- } catch (e) {
- // One bounded recovery: an expired or rejected token drops to a single
- // full list; a failure of that list propagates and commits nothing.
- if (!syncToken) throw e;
- if (e instanceof GoogleCursorExpiredError) deps.log('[google] contacts syncToken expired; full re-list');
- else if (e instanceof GoogleSyncTokenRejectedError) deps.log('[google] contacts syncToken rejected (HTTP 400); one full re-list');
- else throw e;
- state.contacts_sync_token = null;
- result = await people.listConnections({ syncToken: null, ...signal });
- }
- const groupNames = await resolveContactGroups(deps, people, state, result.contacts);
- const sweep = { activePack, summary, countedSlugs, groupNames };
+ // #8906: a full re-list that a deadline or the page budget stopped resumes at its checkpointed page.
+ const resumeToken = state.contacts_relist?.person_fields === CONTACT_PERSON_FIELDS ? state.contacts_relist.page_token : null;
+ const syncToken = deps.opts.full || maskChanged || resumeToken ? null : state.contacts_sync_token;
+ if (maskChanged && state.contacts_sync_token && !deps.opts.full && !resumeToken) deps.log('[google] contacts field mask changed; full re-list');
+ let groupNames: Record<string, string> | null = null;
+ const sweep = () => ({ activePack, summary, countedSlugs, groupNames: groupNames ?? state.contact_groups ?? {} });
let blocked = false;
- for (const c of result.contacts) {
- if (deps.opts.signal?.aborted) return;
- const item = { key: contactHoldKey(c.resourceName), version: c.etag ?? null, title: c.displayName };
- if (!await throughHolds(deps, summary, holds, item, () => applyContact(deps, c, sweep))) blocked = true;
+ // One listed batch through the holds; false when the run was aborted part-way.
+ const apply = async (contacts: ContactData[]): Promise<boolean> => {
+ groupNames ??= contacts.some((c) => !c.deleted && c.groups.length > 0) ? await resolveContactGroups(deps, people, state) : null;
+ const ctx = sweep();
+ for (const c of contacts) {
+ if (deps.opts.signal?.aborted) return false;
+ const item = { key: contactHoldKey(c.resourceName), version: c.etag ?? null, title: c.displayName };
+ if (!await throughHolds(deps, summary, holds, item, () => applyContact(deps, c, ctx))) blocked = true;
+ }
+ return true;
+ };
+ let delta = null;
+ if (syncToken) {
+ try {
+ delta = await people.listConnections({ syncToken, ...signal });
+ } catch (e) {
+ // One bounded recovery: an expired or rejected token drops to a single
+ // full re-list; a failure of that list propagates and commits nothing.
+ if (e instanceof GoogleCursorExpiredError) deps.log('[google] contacts syncToken expired; full re-list');
+ else if (e instanceof GoogleSyncTokenRejectedError) deps.log('[google] contacts syncToken rejected (HTTP 400); one full re-list');
+ else throw e;
+ state.contacts_sync_token = null;
+ }
+ }
+ let nextSyncToken: string | null = null;
+ let listed = false;
+ if (delta) {
+ if (!await apply(delta.contacts)) return;
+ nextSyncToken = delta.nextSyncToken;
+ listed = true;
+ } else {
+ // #8906: a full re-list goes page by page. Once a page has landed the
+ // next page's token is checkpointed, so a run its job deadline stops keeps
+ // every page before it. A saved token Google no longer honors restarts the
+ // list from page 1, once.
+ let pageToken = resumeToken;
+ if (resumeToken) deps.log('[google] contacts full re-list resuming at its checkpointed page');
+ for (let pages = 0; pages < CONTACT_RELIST_PAGES_PER_RUN && !listed; pages++) {
+ let page;
+ try {
+ page = await people.listConnectionsPage(pageToken, signal);
+ } catch (e) {
+ if (pages > 0 || !resumeToken || !(e instanceof GoogleCursorExpiredError)) throw e;
+ deps.log('[google] contacts re-list page token expired; restarting the full re-list from page 1');
+ state.contacts_relist = null;
+ page = await people.listConnectionsPage(pageToken = null, signal);
+ }
+ if (!await apply(page.contacts)) return;
+ if (!page.nextPageToken) {
+ nextSyncToken = page.nextSyncToken;
+ listed = true;
+ continue;
+ }
+ pageToken = page.nextPageToken;
+ // A failing contact not yet held keeps the cursor before its page; a managed lease cannot write once aborted.
+ if (blocked || deps.opts.signal?.aborted) continue;
+ state.contacts_relist = { page_token: pageToken, person_fields: CONTACT_PERSON_FIELDS };
+ await checkpoint();
+ }
+ if (!listed) deps.log(`[google] contacts full re-list paused after ${CONTACT_RELIST_PAGES_PER_RUN} pages; the next sync resumes it`);
}
const prefix = contactHoldKey('');
for (const key of holds?.dueHeldKeys().filter(k => k.startsWith(prefix)) ?? []) {
@@ -76,17 +126,18 @@ export async function sweepContacts(
const resourceName = key.slice(prefix.length);
const c = await people.getContact(resourceName, signal)
?? { resourceName, displayName: null, emails: [], phones: [], organization: null, title: null, groups: [], deleted: true };
- await applyContact(deps, c, sweep);
+ await applyContact(deps, c, sweep());
});
}
- if (blocked) return;
+ if (blocked || !listed) return;
// Cursor and the mask it was minted under commit together, only after the
// whole sweep succeeded. A full list that returned no replacement token
// leaves no cursor behind (the next sweep lists fully again); a same-mask
// list without one keeps the token it already had.
- if (result.nextSyncToken) state.contacts_sync_token = result.nextSyncToken;
+ if (nextSyncToken) state.contacts_sync_token = nextSyncToken;
else if (maskChanged) state.contacts_sync_token = null;
state.contacts_person_fields = CONTACT_PERSON_FIELDS;
+ state.contacts_relist = null;
}
/**
@@ -204,10 +255,11 @@ async function throughHolds(deps: GoogleSyncDeps, summary: GoogleSyncSummary, ho
}
/**
- * #6324: group names for this sweep's pages. Read the directory only when a
- * listed contact carries a user group; a complete read replaces the cache (a
- * deleted or renamed group is current for every page rendered from now on,
- * an untouched contact's page refreshes when it next changes or on `--full`),
+ * #6324: group names for this sweep's pages. The sweep reads the directory
+ * once, when a listed contact first carries a user group; a complete read
+ * replaces the cache (a deleted or renamed group is current for every page
+ * rendered from now on, an untouched contact's page refreshes when it next
+ * changes or on `--full`),
* an incomplete read is merged over it and said so, an unreadable directory
* falls back to the cache. A group no entry names is rendered as its raw id.
*/
@@ -215,10 +267,8 @@ async function resolveContactGroups(
deps: GoogleSyncDeps,
people: PeopleClient,
state: GoogleSourceState,
- contacts: ContactData[],
): Promise<Record<string, string>> {
const cached = state.contact_groups ?? {};
- if (!contacts.some((c) => !c.deleted && c.groups.length > 0)) return cached;
try {
const { names, complete } = await people.listContactGroups(deps.opts.signal ? { signal: deps.opts.signal } : {});
if (!complete) deps.log('[google] contact groups directory exceeds the bounded read; groups it did not name keep their ids');
diff --git a/src/core/google/types.ts b/src/core/google/types.ts
index 8ecdebde11..93637acac0 100644
--- a/src/core/google/types.ts
+++ b/src/core/google/types.ts
@@ -183,6 +183,13 @@ export interface GoogleSourceState {
* fully. Absent on state saved before the mask grew (legacy = the old mask).
*/
contacts_person_fields?: string | null;
+ /**
+ * #8906: the full contacts re-list in progress: the `pageToken` of its next
+ * page and the `personFields` mask it was started under, checkpointed after
+ * each page lands so a run stopped by its deadline or page budget resumes
+ * there. Cleared when the list completes; one under another mask is ignored.
+ */
+ contacts_relist?: { page_token: string; person_fields: string } | null;
/**
* #6324: contact-group directory cache (resource name → display name) from
* the last successful `contactGroups` read; the fallback when the directory
diff --git a/test/contacts-relist-checkpoint-8906.slow.test.ts b/test/contacts-relist-checkpoint-8906.slow.test.ts
new file mode 100644
index 0000000000..29d1bf67b3
--- /dev/null
+++ b/test/contacts-relist-checkpoint-8906.slow.test.ts
@@ -0,0 +1,200 @@
+/**
+ * #8906: a full contacts re-list (expired syncToken, first sync, changed
+ * mask) checkpoints its People pageToken after each page lands, so a run cut
+ * short by its job deadline resumes at the next page instead of page 1. Runs
+ * on a repo-less PGLite brain: the cursor lives in the source state the sweep
+ * already writes, never in git.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { mkdtempSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { CONTACT_PERSON_FIELDS } from '../src/core/google/google-clients.ts';
+import { googleStateFile, parseGoogleSourceConfig, readGoogleState, runGoogleSync } from '../src/core/google/google-source.ts';
+import type { GoogleSourceState } from '../src/core/google/types.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { withEnv } from './helpers/with-env.ts';
+
+let engine: PGLiteEngine;
+let schemaVersion: string;
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+ schemaVersion = (await engine.getConfig('version')) ?? '7';
+});
+
+afterAll(async () => {
+ await engine.disconnect();
+});
+
+beforeEach(async () => {
+ await resetPgliteState(engine);
+ await engine.setConfig('version', schemaVersion);
+});
+
+const PAGE_SIZE = 200;
+const json = (body: unknown, status = 200) => new Response(JSON.stringify(body), { status, headers: { 'content-type': 'application/json' } });
+type Fetch = (url: string, init?: RequestInit) => Promise<Response>;
+
+/** People connections in pages of 200: page n is asked for with pageToken p<n>; the last page mints ppl-new; a syncToken request is expired (410). */
+function peopleApi(pages: number, o: { stale?: (pageToken: string) => Response; onList?: (pageToken: string | null) => void } = {}) {
+ const served: string[] = [];
+ const requests: Array<{ syncToken: string | null; pageToken: string | null }> = [];
+ let calendarLists = 0;
+ const fetchImpl: Fetch = async (url, init) => {
+ const u = new URL(url);
+ if (u.pathname.includes('/calendars/')) {
+ calendarLists++;
+ return json({ items: [], nextSyncToken: 'cal-1' });
+ }
+ if (!u.pathname.endsWith('/people/me/connections')) return json({ error: { message: 'unexpected fixture route' } }, 400);
+ const syncToken = u.searchParams.get('syncToken');
+ const pageToken = u.searchParams.get('pageToken');
+ requests.push({ syncToken, pageToken });
+ if (syncToken) return json({ error: { code: 410, message: 'Sync token is expired. Clear local cache and retry call without the sync token.' } }, 410);
+ o.onList?.(pageToken);
+ if (init?.signal?.aborted) throw Object.assign(new Error('The operation was aborted.'), { name: 'AbortError' });
+ const n = pageToken === null ? 1 : /^p\d+$/.test(pageToken) ? Number(pageToken.slice(1)) : NaN;
+ if (!(n >= 1 && n <= pages)) return o.stale ? o.stale(pageToken!) : json({ error: { code: 400, message: 'unexpected page token' } }, 400);
+ served.push('p' + n);
+ const connections = Array.from({ length: PAGE_SIZE }, (_, i) => {
+ const id = 'c' + String((n - 1) * PAGE_SIZE + i).padStart(5, '0');
+ return { resourceName: 'people/' + id, etag: 'e-' + id, names: [{ displayName: 'Person ' + id + ' Example' }], emailAddresses: [{ value: id + '@example.invalid' }] };
+ });
+ return json(n < pages ? { connections, nextPageToken: 'p' + (n + 1) } : { connections, nextSyncToken: 'ppl-new' });
+ };
+ return { fetchImpl, served, requests, calendarLists: () => calendarLists };
+}
+
+async function googleSource(services = 'contacts') {
+ const dir = mkdtempSync(join(tmpdir(), 'gsrc-relist-8906-'));
+ const config = { kind: 'google', g_account: '[email protected]', g_services: services, g_history_days: 90, g_dir: dir, g_access: 'env', g_token_env: 'GBRAIN_TEST_8906_TOKEN' };
+ await engine.executeRaw('INSERT INTO sources (id, name, local_path, config) VALUES ($1, $2, $3, $4::text::jsonb)', ['gsrc', 'google', dir, JSON.stringify(config)]);
+ return { dir, cfg: parseGoogleSourceConfig(config, dir) };
+}
+
+function seedState(dir: string, state: Record<string, unknown>): void {
+ writeFileSync(googleStateFile(dir), JSON.stringify({ ...readGoogleState(dir), ...state }));
+}
+
+const relistOf = (dir: string) => (readGoogleState(dir) as GoogleSourceState & { contacts_relist?: { page_token: string; person_fields: string } | null }).contacts_relist ?? null;
+
+const personPages = async () => (await engine.executeRaw<{ n: number }>(
+ "SELECT count(*)::int AS n FROM pages WHERE source_id = 'gsrc' AND deleted_at IS NULL AND frontmatter->>'google_contact_id' IS NOT NULL"))[0]!.n;
+
+const sync = (cfg: ReturnType<typeof parseGoogleSourceConfig>, fetchImpl: Fetch, signal?: AbortSignal) =>
+ withEnv({ GBRAIN_HOME: mkdtempSync(join(tmpdir(), 'gbrain-home-8906-')), GBRAIN_TEST_8906_TOKEN: 'synthetic-fixture-token' }, () =>
+ runGoogleSync(engine, 'gsrc', cfg, { sourceId: 'gsrc', noEmbed: true, noExtract: true, noSchemaPack: true, ...(signal ? { signal } : {}) }, fetchImpl));
+
+describe('contacts full re-list checkpoints (#8906)', () => {
+ test('expired token, 2,000 contacts, a deadline after 3 pages: each run resumes at its next page and the list completes with every page applied once', async () => {
+ const { dir, cfg } = await googleSource();
+ try {
+ seedState(dir, { contacts_sync_token: 'stale-token', contacts_person_fields: CONTACT_PERSON_FIELDS });
+ let controller = new AbortController();
+ let listedThisRun = 0;
+ const onDiskAtDeadline: Array<string | null> = [];
+ const api = peopleApi(10, {
+ onList: () => {
+ // The job deadline fires as the run asks for its 4th page; what is on disk then is what a killed job leaves.
+ if (++listedThisRun !== 4) return;
+ controller.abort();
+ onDiskAtDeadline.push(relistOf(dir)?.page_token ?? null);
+ },
+ });
+ const firstPageTokens: Array<string | null> = [];
+ for (let run = 0; run < 6 && readGoogleState(dir).contacts_sync_token !== 'ppl-new'; run++) {
+ controller = new AbortController();
+ listedThisRun = 0;
+ const before = api.requests.length;
+ await sync(cfg, api.fetchImpl, controller.signal);
+ firstPageTokens.push(api.requests.slice(before).find((r) => r.syncToken === null)?.pageToken ?? null);
+ }
+ expect(firstPageTokens).toEqual([null, 'p4', 'p7', 'p10']);
+ expect(onDiskAtDeadline).toEqual(['p4', 'p7', 'p10']);
+ expect(api.served).toEqual(['p1', 'p2', 'p3', 'p4', 'p5', 'p6', 'p7', 'p8', 'p9', 'p10']);
+ const state = readGoogleState(dir);
+ expect(state.contacts_sync_token).toBe('ppl-new');
+ expect(state.contacts_person_fields).toBe(CONTACT_PERSON_FIELDS);
+ expect(relistOf(dir)).toBeNull();
+ expect(await personPages()).toBe(2_000);
+ const after = api.requests.length;
+ await sync(cfg, api.fetchImpl);
+ expect(api.requests[after]).toEqual({ syncToken: 'ppl-new', pageToken: null });
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ }, 240_000);
+
+ test('a saved page token Google no longer honors (400 or 410) restarts the re-list from page 1 once; a failing restart propagates without looping', async () => {
+ for (const status of [400, 410]) {
+ await resetPgliteState(engine);
+ await engine.setConfig('version', schemaVersion);
+ const { dir, cfg } = await googleSource();
+ try {
+ seedState(dir, { contacts_sync_token: null, contacts_person_fields: CONTACT_PERSON_FIELDS, contacts_relist: { page_token: 'gone-page', person_fields: CONTACT_PERSON_FIELDS } });
+ const api = peopleApi(3, { stale: () => json({ error: { code: status, message: 'Page token is invalid.' } }, status) });
+ expect((await sync(cfg, api.fetchImpl)).status).not.toBe('partial');
+ expect(api.requests.map((r) => r.pageToken)).toEqual(['gone-page', null, 'p2', 'p3']);
+ expect(readGoogleState(dir).contacts_sync_token).toBe('ppl-new');
+ expect(relistOf(dir)).toBeNull();
+ expect(await personPages()).toBe(600);
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ }
+
+ await resetPgliteState(engine);
+ await engine.setConfig('version', schemaVersion);
+ const { dir, cfg } = await googleSource();
+ try {
+ seedState(dir, { contacts_sync_token: null, contacts_person_fields: CONTACT_PERSON_FIELDS, contacts_relist: { page_token: 'gone-page', person_fields: CONTACT_PERSON_FIELDS } });
+ const requests: Array<string | null> = [];
+ const wedged: Fetch = async (url) => {
+ const u = new URL(url);
+ if (u.pathname.endsWith('/people/me/connections')) requests.push(u.searchParams.get('pageToken'));
+ return json({ error: { code: 400, message: 'Page token is invalid.' } }, 400);
+ };
+ expect((await sync(cfg, wedged)).status).toBe('partial');
+ expect(requests).toEqual(['gone-page', null]);
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ }, 120_000);
+
+ test('a page token saved under another field mask is not resumed: the re-list starts at page 1', async () => {
+ const { dir, cfg } = await googleSource();
+ try {
+ seedState(dir, { contacts_sync_token: null, contacts_person_fields: 'names', contacts_relist: { page_token: 'p2', person_fields: 'names' } });
+ const api = peopleApi(2);
+ await sync(cfg, api.fetchImpl);
+ expect(api.requests.map((r) => r.pageToken)).toEqual([null, 'p2']);
+ expect(readGoogleState(dir).contacts_person_fields).toBe(CONTACT_PERSON_FIELDS);
+ expect(relistOf(dir)).toBeNull();
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ }, 60_000);
+
+ test("a re-list longer than one run's page budget stops at the budget and the calendar sweep still runs; the next sync finishes it", async () => {
+ const { dir, cfg } = await googleSource('contacts,calendar');
+ try {
+ const api = peopleApi(12);
+ expect((await sync(cfg, api.fetchImpl)).status).not.toBe('partial');
+ expect(api.served).toHaveLength(10);
+ expect(api.calendarLists()).toBeGreaterThan(0);
+ expect(relistOf(dir)?.page_token).toBe('p11');
+ expect(readGoogleState(dir).contacts_sync_token).toBeNull();
+ await sync(cfg, api.fetchImpl);
+ expect(api.served).toEqual(Array.from({ length: 12 }, (_, i) => 'p' + (i + 1)));
+ expect(readGoogleState(dir).contacts_sync_token).toBe('ppl-new');
+ expect(await personPages()).toBe(2_400);
+ } finally {
+ rmSync(dir, { recursive: true, force: true });
+ }
+ }, 240_000);
+});
diff --git a/test/contacts-relist-checkpoint-managed-8906.test.ts b/test/contacts-relist-checkpoint-managed-8906.test.ts
new file mode 100644
index 0000000000..8e5eccdb88
--- /dev/null
+++ b/test/contacts-relist-checkpoint-managed-8906.test.ts
@@ -0,0 +1,62 @@
+import { afterAll, beforeAll, expect, test } from 'bun:test';
+import { parseGoogleSourceConfig, runGoogleSync } from '../src/core/google/google-source.ts';
+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
+import { withEnv } from './helpers/with-env.ts';
+import { contact, createConnectorFixture, googleConfig, json, options, sourceCheckpoint, withGoogleAccount } from './helpers/connector-fixture.ts';
+
+const { engines, env, source, setup, teardown } = createConnectorFixture();
+beforeAll(setup, 120_000);
+afterAll(teardown);
+
+const PAGES = 4;
+const PER_PAGE = 5;
+
+test('#8906 managed contacts re-list: a deadline mid-list keeps the checkpointed pages in the connector state; the next run resumes there and applies every contact once', async () => withEnv(env, async () => {
+ for (const engine of engines) {
+ const f = await source(engine, googleConfig);
+ const cfg = parseGoogleSourceConfig(googleConfig, f.dir);
+ const pageTokens: Array<string | null> = [];
+ let onList: (() => void) | undefined;
+ const fetcher = async (url: string, init?: RequestInit) => {
+ const u = new URL(url);
+ if (!u.pathname.endsWith('/people/me/connections')) return json({ error: { message: 'unexpected fixture route' } }, 400);
+ const pageToken = u.searchParams.get('pageToken');
+ pageTokens.push(pageToken);
+ onList?.();
+ if (init?.signal?.aborted) throw Object.assign(new Error('The operation was aborted.'), { name: 'AbortError' });
+ const n = pageToken === null ? 1 : Number(pageToken.slice(1));
+ const connections = Array.from({ length: PER_PAGE }, (_, i) => contact('c' + ((n - 1) * PER_PAGE + i), 'Person ' + ((n - 1) * PER_PAGE + i) + ' Example'));
+ return json(n < PAGES ? { connections, nextPageToken: 'p' + (n + 1) } : { connections, nextSyncToken: 'ppl-new' });
+ };
+ const run = (signal?: AbortSignal) => runGoogleSync(engine, f.id, cfg, { ...options, ...(signal ? { signal } : {}) }, withGoogleAccount(fetcher, cfg.account));
+ const imports = async () => (await engine.executeRaw<{ slug: string }>(
+ "SELECT slug FROM persistence_requests WHERE source_id=$1 AND intent->>'kind'='connector_v2_import' ORDER BY slug", [f.id])).map((r) => r.slug);
+ const state = async () => {
+ await disposePersistenceConsumer(engine);
+ const [row] = await sourceCheckpoint(engine, f.id) as { completed_keys: { state?: { contacts_sync_token?: string | null; contacts_relist?: { page_token: string } | null } }[] }[];
+ return row?.completed_keys[0]?.state;
+ };
+
+ // The deadline fires as the run asks for page 3: pages 1 and 2 are banked.
+ const controller = new AbortController();
+ onList = () => { if (pageTokens.length === 3) controller.abort(); };
+ const aborted = await run(controller.signal).then((r) => r.status, (e: Error) => e.name);
+ expect(['partial', 'AbortError']).toContain(aborted);
+ expect(pageTokens).toEqual([null, 'p2', 'p3']);
+ const banked = await state();
+ expect(banked?.contacts_relist?.page_token).toBe('p3');
+ expect(banked?.contacts_sync_token ?? null).toBeNull();
+ expect(await imports()).toHaveLength(2 * PER_PAGE);
+
+ onList = undefined;
+ pageTokens.length = 0;
+ expect((await run()).status).not.toBe('partial');
+ expect(pageTokens).toEqual(['p3', 'p4']);
+ const all = await imports();
+ expect(all).toHaveLength(PAGES * PER_PAGE);
+ expect(new Set(all).size).toBe(PAGES * PER_PAGE);
+ const done = await state();
+ expect(done?.contacts_sync_token).toBe('ppl-new');
+ expect(done?.contacts_relist ?? null).toBeNull();
+ }
+}), 120_000);