Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(google): checkpoint the contacts full re-list page by page so an expired People sync token can finish (#8906)
garrytan04c5275fdd27
diff --git a/docs/architecture/key-files/google-and-loops.md b/docs/architecture/key-files/google-and-loops.mdindex 0812b0271f..ce2780977d 100644--- a/docs/architecture/key-files/google-and-loops.md+++ b/docs/architecture/key-files/google-and-loops.md@@ -22,13 +22,13 @@ Historical Gmail repair selects explicit nested MIME metadata fields, excluding- `src/core/creds/redirect.ts` — how the authorization code gets back to us: `RedirectStrategy = 'loopback' | 'paste' | 'hosted-callback'`. Loopback is an ephemeral 127.0.0.1 listener; paste mode needs NO listener (the fixed `PASTE_REDIRECT_URI` fails to load, the user pastes the full address-bar URL back) and is auto-selected by `sniffHeadless` (SSH/WSL/container/no-display). Google's device-code flow is NOT an option — Gmail/Calendar/Contacts scopes are excluded from it; don't re-litigate.- `src/core/creds/relay-client.ts` — the typed CLIENT half of the gbrain.io zero-retention consent relay (`createSession` → user clicks the consent URL → poll a one-time `claim`; the relay deletes tokens on first successful claim or at session TTL). Inert unless `GBRAIN_OAUTH_RELAY_URL` is set (unset = the BYO flow, which always works). `test/creds-relay-client.test.ts` is the server conformance spec — a server that round-trips those fixtures is compatible.- `src/core/creds/export.ts` — versioned passphrase-encrypted credential bundles (scrypt N=2^15, r=8, p=1 → AES-256-GCM) for machine moves and hosted-upgrade transfer. A bundle carries selected vault entries PLUS the provider client records they depend on (Google refresh tokens are bound to the client that minted them — moving one without the other produces dead tokens). Format frozen here; hosted's import endpoint conforms to it.-- `src/core/google/types.ts` — pure data shapes for the google source kind: normalized Gmail/Calendar/People payloads, `GoogleSourceConfig` (account pointer, services, historyDays, managed dir), and the `GoogleSourceState` cursor file persisted at `<managed dir>/.google-source.json` (gmail history id, downward-moving backfill floor, per-service syncTokens). No I/O. Exports `DEFAULT_CALENDAR_ID = 'primary'` (the Calendar API's own alias; every `'primary'` literal in google-clients/google-source/sources/sources-ops resolves through it) and `GoogleSourceConfig.calendarId` (one calendar per source). `GoogleSourceState.calendar_id` records which calendar `calendar_sync_token` was minted for — a token is only valid against its own calendar; absent on legacy state, which is therefore primary's. `GoogleSourceState.calendar_horizon_ms` is the ceiling of the last calendar list that reached it (calendar-window.ts); absent on state saved before it existed. `GoogleSourceState.contacts_person_fields` (#6324) is the People field mask `contacts_sync_token` was minted under, written only together with the token; absent on legacy state, which the sweep therefore lists fully once. `contact_groups` caches the group directory (resource name → display name) and is never authoritative on its own. `ContactData` carries `phones: string[]` and `groups: {id, name|null}[]`. `GoogleSourceConfig.futureDays` (#5442) and `loopsExcludeLabels` (#5445) are optional and absent when their keys are unset; `source-config.ts` parses them (`g_future_days` 1..3650 floored, `g_loops_exclude_labels` through `parseExcludeLabelTokens`) and `googleTuningConfig` is the one writer of those keys plus `g_calendar_id`, each only when set, for `addSource`. Both `sources add` parsers accept `--future-days` and `--loops-exclude-labels`: the connected-source parser validates through the operation contract (`sources-lifecycle-args.ts`), the legacy parser through `commands/sources-google-flags.ts` (`parseGoogleAddFlag`, exit 2 on a bad value), so neither `runAdd` nor `addSource` grows for a new tuning key.+- `src/core/google/types.ts` — pure data shapes for the google source kind: normalized Gmail/Calendar/People payloads, `GoogleSourceConfig` (account pointer, services, historyDays, managed dir), and the `GoogleSourceState` cursor file persisted at `<managed dir>/.google-source.json` (gmail history id, downward-moving backfill floor, per-service syncTokens). No I/O. Exports `DEFAULT_CALENDAR_ID = 'primary'` (the Calendar API's own alias; every `'primary'` literal in google-clients/google-source/sources/sources-ops resolves through it) and `GoogleSourceConfig.calendarId` (one calendar per source). `GoogleSourceState.calendar_id` records which calendar `calendar_sync_token` was minted for — a token is only valid against its own calendar; absent on legacy state, which is therefore primary's. `GoogleSourceState.calendar_horizon_ms` is the ceiling of the last calendar list that reached it (calendar-window.ts); absent on state saved before it existed. `GoogleSourceState.contacts_person_fields` (#6324) is the field mask `contacts_sync_token` was minted under and `contacts_relist` (#8906) the full re-list's resume page. `contact_groups` caches the group directory (resource name → display name) and is never authoritative on its own. `GoogleSourceConfig.futureDays` (#5442) and `loopsExcludeLabels` (#5445) are optional and absent when their keys are unset; `source-config.ts` parses them (`g_future_days` 1..3650 floored, `g_loops_exclude_labels` through `parseExcludeLabelTokens`) and `googleTuningConfig` is the one writer of those keys plus `g_calendar_id`, each only when set, for `addSource`. Both `sources add` parsers accept `--future-days` and `--loops-exclude-labels`: the connected-source parser validates through the operation contract (`sources-lifecycle-args.ts`), the legacy parser through `commands/sources-google-flags.ts` (`parseGoogleAddFlag`, exit 2 on a bad value), so neither `runAdd` nor `addSource` grows for a new tuning key.- `src/core/google/access.ts` — the pluggable Google-access seam: `GoogleAccessProvider` (`getAccessToken`/`forceRefresh`) with `CommandAccessProvider` (`--access command`: any CLI that prints a token — gog/gcloud/a gateway's mint command; parsed as a bare token line or JSON `{token|access_token, expiry|expires_in}`, cached until expiry with a 60s margin, 30s exec timeout, failures = `access_command_failed` carrying the stderr tail) and `EnvAccessProvider` (`--access env`: a live token read from a NAMED env var each call, refreshed outside gbrain; missing = `access_env_missing`). The vault flow's `GoogleTokenProvider` satisfies the same interface. The token command executes only in the locally-running sync (google config keys are unreachable over MCP; same trust class as recipe health-check argv).-- `src/core/google/google-clients.ts` — hand-rolled Gmail/Calendar/People REST clients (house style, no googleapis dependency): auth via any `GoogleAccessProvider` (vault-backed `GoogleTokenProvider` by default), 401 → forceRefresh + single retry, Retry-After honored (delta-seconds AND http-date), 403 disabled API → `GoogleApiDisabledError` and insufficient scope → `scope_missing` ([API access](google-api-access.md)), uniform pageToken pagination with a safety cap, `fetchImpl` injectable for tests. `extractCalendarMethod(part)` walks the MIME tree for a `text/calendar`/`application/ics` part and reads the iCalendar `method=` parameter from the PART's own `Content-Type` header (`headers[]` on nested parts with `format=full` — Gmail's `MessagePart.mimeType` is the BARE media type, so a mimeType-only parse would read '' for every real invite and silently downgrade the structural signal to the subject fallback); a mimeType that still carries params is the fallback parse; a bare `.ics` filename with a non-calendar MIME type claims nothing. `CalendarClient.listEvents({calendarId})` sweeps one calendar (default `DEFAULT_CALENDAR_ID`). A People request with a nonempty `syncToken` maps HTTP 400 to `GoogleCursorExpiredError` only for typed `google.rpc.ErrorInfo` reason `EXPIRED_SYNC_TOKEN` or a message beginning `Sync token [is] expired`; any other 400 on such a request is a `GoogleSyncTokenRejectedError` (a `CredentialError('upstream')` subclass the contacts sweep alone treats as cursor-invalid, #6324); initial-list requests and other APIs remain plain upstream failures. `CONTACT_PERSON_FIELDS` (`names,emailAddresses,organizations,phoneNumbers,memberships`) is the one People field mask, repeated on every page of a list; `ContactData` carries deduplicated E.164 `phones` and user-group `groups` (`{id, name: null}`, system groups dropped), and `listContactGroups()` walks `contactGroups` for at most ten 1000-row pages and reports `complete: false` past that. The contacts sweep uses its existing bounded full re-list recovery without restarting Gmail. The Gmail mapper also captures `precedence` (lower-cased), `listId` (present or not) and `replyTo` for the automated-mail loop rule (#5012). A 403 that is not API-disabled, rate-limit or quota is `GoogleForbiddenError` (still `CredentialError` `upstream`); `calendarUnreadableStatus(e)` maps it and a 404 `GoogleCursorExpiredError` to 403/404, and `CalendarClient.probeCalendar(id)` is one `maxResults=1` read. Pinned by injected-response tests in `test/google-clients.test.ts`, `test/contacts-phone-only-groups-6324.test.ts` and `test/google-source-reconcile.test.ts`, not live-provider validation.+- `src/core/google/google-clients.ts` — hand-rolled Gmail/Calendar/People REST clients (house style, no googleapis dependency): auth via any `GoogleAccessProvider` (vault-backed `GoogleTokenProvider` by default), 401 → forceRefresh + single retry, Retry-After honored (delta-seconds AND http-date), 403 disabled API → `GoogleApiDisabledError` and insufficient scope → `scope_missing` ([API access](google-api-access.md)), uniform pageToken pagination with a safety cap, `fetchImpl` injectable for tests. `extractCalendarMethod(part)` walks the MIME tree for a `text/calendar`/`application/ics` part and reads the iCalendar `method=` parameter from the PART's own `Content-Type` header (`headers[]` on nested parts with `format=full` — Gmail's `MessagePart.mimeType` is the BARE media type, so a mimeType-only parse would read '' for every real invite and silently downgrade the structural signal to the subject fallback); a mimeType that still carries params is the fallback parse; a bare `.ics` filename with a non-calendar MIME type claims nothing. `CalendarClient.listEvents({calendarId})` sweeps one calendar (default `DEFAULT_CALENDAR_ID`). A People request with a nonempty `syncToken` maps HTTP 400 to `GoogleCursorExpiredError` only for typed `google.rpc.ErrorInfo` reason `EXPIRED_SYNC_TOKEN` or a message beginning `Sync token [is] expired`; any other 400 on such a request is a `GoogleSyncTokenRejectedError` (a `CredentialError('upstream')` subclass the contacts sweep alone treats as cursor-invalid, #6324); a connections 400 with a `pageToken` is `GoogleCursorExpiredError` (#8906, `listConnectionsPage` reads one page); other initial-list requests and APIs remain plain upstream failures. `CONTACT_PERSON_FIELDS` (`names,emailAddresses,organizations,phoneNumbers,memberships`) is the one People field mask, repeated on every page of a list; `ContactData` carries deduplicated E.164 `phones` and user-group `groups` (`{id, name: null}`, system groups dropped), and `listContactGroups()` walks `contactGroups` for at most ten 1000-row pages and reports `complete: false` past that. The Gmail mapper also captures `precedence` (lower-cased), `listId` (present or not) and `replyTo` for the automated-mail loop rule (#5012). A 403 that is not API-disabled, rate-limit or quota is `GoogleForbiddenError` (still `CredentialError` `upstream`); `calendarUnreadableStatus(e)` maps it and a 404 `GoogleCursorExpiredError` to 403/404, and `CalendarClient.probeCalendar(id)` is one `maxResults=1` read. Pinned by injected-response tests in `test/google-clients.test.ts`, `test/contacts-phone-only-groups-6324.test.ts` and `test/google-source-reconcile.test.ts`, not live-provider validation.- `src/core/google/gmail-categories.ts` — `BULK_CATEGORY_LABELS` (`CATEGORY_PROMOTIONS`/`SOCIAL`/`FORUMS`) and `hasBulkCategory`, the one list both open-loop lanes read (#5103): `loopExtractionEligibility` returns `bulk_category` and `detectThreadLoop` withholds an inbound open, in both cases unless the owner wrote a substantive message in the thread; `CATEGORY_UPDATES` is deliberately not in it. `isNoiseSender` additionally matches `GOOGLE_SYSTEM_SENDERS` (the Meet notes digest address) by equality only.- `src/core/google/google-render.ts` — `isPositivelyAutomated(meta)` (#5012): `Precedence: bulk|list|junk`, a `List-Id`, a noise From or a noise Reply-To. Pure render functions (`{relPath, markdown}` out; no I/O, no engine): thread pages under `emails/YYYY/MM/` (type email), events under `calendar/YYYY/MM/` (type meeting), contacts under `people/` (type person). Gmail deep links are code-generated via the typed `emailCitation` scaffold, never LLM-composed. The noise/signature rules `recipes/email-to-brain.md` specifies as prose for agent-authored collectors are implemented here — keep the recipe and this module in sync. `isCalendarSystemMail({calendarMethod, subject})` gates BOTH loop detectors: the PRIMARY signal is a `calendarMethod` in the `CALENDAR_METHODS` allowlist (RFC 5546 — REQUEST, REPLY, CANCEL, PUBLISH, COUNTER, DECLINECOUNTER, REFRESH, ADD; case-insensitive); an empty or unrecognised method is NOT a stamp and falls to the subject fallback, which is anchored to the start of the subject, refuses any Re:/Fwd: prefix, and matches only Calendar's OWN localised headers (`Invitation:`, `Updated invitation:`, `Accepted:`, `Declined:`, `Tentative:`, `Canceled event:` + localised equivalents) — a generic word like `Notification:` is a human/vendor subject and never matches. `renderThreadPage` stamps `participants:` (every address on the thread) AND `senders:` (sorted unique message AUTHORS only — the list `loops mute sender` gates on, so muting one person cannot silence a whole group thread). `renderPersonPage` (#6324) renders a contact with emails OR phones: `emails:` always, `phones:` and `groups:` (names, or the raw `contactGroups/<id>` when unresolved) only when non-empty, phones never in `aliases`; a contact with neither name nor email lives at `contactIdSlug` = `<dir>/contact-<sha8(resourceName)>`, which `personSlugFromContact` returns for a phone-only nameless contact and the sweep's tombstone path lists as a delete candidate. Pinned by `test/google-render.test.ts` and `test/contacts-phone-only-groups-6324.test.ts`.- `src/core/google/google-source.ts` — the Google API source sweep. Unmanaged mode materializes Markdown under the source directory and imports through the standard chunks/aliases/links pipeline; managed mode uses journaled connector admission. Sweep order contacts → calendar → gmail: alias rows must exist before the loop detector resolves counterparties. Per-service independent cursors: contacts/calendar syncToken commits only after that service's sweep left no failing item that is not yet held (410 GONE drops the token and re-runs windowed); GBRA-130 A-3: contacts and events go through connector item holds (`contact:`/`event:` keys), so one failing item no longer pins the token or stops Gmail (`test/pathological/a3-google-items.test.ts`); gmail delta rides history.list with an expired-history windowed fallback; the INITIAL backfill drains newest→oldest with a batch-committed floor cursor so a killed 50k-message backfill resumes at the floor instead of restarting, and the historyId anchor is captured BEFORE the backfill so the delta lane takes over with zero gap (overlap re-renders are idempotent). Emits the `sync.google_materialize` progress phase (docs/progress-events.md). Access resolves per source config: the vault (default), a token-printing command (`g_access: command` — gog/gcloud/gateway; identity entry synthesized, scope preflight trusts the configured services, send-as aliases fetched live best-effort), or a named env var (`g_access: env`). Secrets never land in `sources.config` (the command/env NAME is config; tokens are not). Honesty invariants: `last_sync_at` is gated on the GMAIL sweep's success (it feeds `gbrain waiting`'s trust-critical staleness gate, which protects loop freshness — a gmail sweep with real thread failures must not advance it); contacts/calendar failures mark the run `partial` but do not block the stamp, and a source without gmail in its services stamps unconditionally. A thread whose fetch fails on consecutive sweeps lands in the poison ledger and is skipped (steady-state runs honor the ledger; `--full` retries with a fresh one), so one bad thread can't wedge the sync while a poison-skip alone doesn't count as failure. Contact reconcile resolves by contact resourceName (DB lookup by contact id FIRST): deletion tombstones carry only `resourceName + deleted` (no names — slug derivation yields null) and a renamed contact derives a DIFFERENT slug, so both must land on the existing page by id, never by slug. The calendar sweep binds its sync token to its calendar: `sweepCalendar` compares `state.calendar_id` (legacy state without it is treated as primary's) with the configured `calendarId`; on a mismatch it logs `[google] calendar changed (<old> → <new>)`, discards the token and takes the windowed first-sync path (no `syncToken` on the wire), and a freshly banked token stores the calendar id beside it — pairing a re-pointed source with the previous calendar's cursor would import a foreign delta. Pages from the previous calendar are not reconciled on the switch (logged; a filed follow-up). `enqueueLoopsExtraction` enqueues EVERY eligible thread (newest first) under a waiting-depth budget `LOOPS_EXTRACT_ENQUEUE_CEILING − waiting` that counts only THIS source's waiting `loops_extract` jobs (`data->>'sourceId'`), so one account's stalled backlog cannot pin another source's budget at 0; overflow is a logged deferral (a deferred thread re-candidates when it changes) and a failed probe fails open. The budget probe counts PENDING rows — `status IN ('waiting','delayed','active')` — so a retry backlog parked in `delayed` during a flapping provider cannot stack past the ceiling. With no chat provider available (`isAvailable('chat')` false) the enqueue is skipped entirely and one stderr line per sweep names the reason and the count; pages still import and the threads re-candidate on their next touch or a `--full` sweep. Managed Google API sync uses journaled connector checkpoints; an unbound source is explicitly `connector_database`, while a bound source publishes canonical files. Identity-query storage failures do not become successful absence or advance the checkpoint. Unsupported Git/dry-run flags refuse before credentials/network; legacy materialization remains the unmanaged path. FILE MODES (every write, both paths): `writeGoogleState` writes `.google-source.json` through `atomicWriteFileSync(..., { mode: 0o600 })` under `mkdirPrivate(dir)`; a corrupt state file is chmod-ed 0600 and then renamed to `.corrupt`, and if either step fails stderr names the path, the failed step, the exposure and the fix (`chmod 600 <path>` or delete it) while the sync continues with empty cursors. A legacy page goes through `writePrivateTemp`: a stale `<page>.tmp` is removed with `lstat` (never followed; a stale `.tmp` directory fails that item with `Stale temporary path <path> is a directory; remove it and re-run the sync.`), the temp file is created `wx` at 0600 and fchmod-ed past the umask, imported, then renamed over the page; page directories come from `mkdirPrivate(dirname(page), cfg.dir)`. 0600 is reasserted on every rewrite, so a looser mode set by hand is reverted on the next rewrite (unchanged pages are not rewritten). Managed pages get the same modes through `publishMode` (see the persistence coordinator). Mode assertions are skipped on win32. Tests: `test/google-file-modes.test.ts` (child process under umask 0000, `test/helpers/google-file-modes-child.ts`). The loops enqueue runs on every non-aborted sweep in both persistence modes (`--no-extract` gates only the inline link/timeline extract), logs every skip reason and reports `loops_enqueue { enqueued, deferred, skipped_reason }` in the sync result; job payloads carry the page revision (`newestMs`). After the delta and item-hold passes the sweep seeds and settles grace holds, and a managed sweep runs the 30-day catch-up (both in `loop-catchup.ts`); a partial managed run publishes that state with its item holds (`publishHolds` extra), never the Gmail cursor.-- `src/core/google/calendar-window.ts` — which calendar events a google source keeps pages for: `[now - historyDays, now + futureDays]` (`g_future_days`, 1..3650, default `CALENDAR_HORIZON_DAYS` = 60; the parsed `futureDays` leaf exists only when the key is set, so an unset knob leaves every existing source's connector identity unchanged), checked by the sweep itself on every list because a syncToken delta cannot carry timeMin/timeMax. `CalendarSyncWindow` is built once per sweep: `side(start, end)` follows Google's rule (an event is `before` when it ends at or before the floor, `after` when it starts at or past the ceiling, unparseable dates stay `inside`); `coverageStartMs(horizon, listedWholeWindow)` says where the catch-up list for unchanged instances crossing the leading edge starts (none after a whole-window list, now for state without `calendar_horizon_ms`, the recorded horizon once the ceiling is a day past it); `startsInside` and `startsAfter` are the `--full` reconcile tests. `planCalendarPage` maps one listed event to a page change (a cancellation removes its page anywhere, aged-out history is left as it is, a page past the horizon is removed, otherwise write after removing a copy at an older path). `unlistedCalendarPages` returns sweep pages (with an `event_id`) starting inside the window that a complete `--full` list did not name; `staleCalendarPages` adds the sweep pages starting at or past the ceiling, which no windowed list can name (#5442), and is what the `--full` reconcile deletes through. `people-calendar-sweep.ts` (the contacts and calendar sweeps, peeled from `google-source.ts`) performs the changes with `importRendered`/`deletePageByRelPath`, discards the catch-up list's sync token, advances `calendar_horizon_ms` only after a list reached the ceiling, and sends reconcile deletes through the >200 mass-delete guard (a refusal marks the run partial). Its contacts cursor contract (#6324): `contacts_sync_token` and `contacts_person_fields` (the mask that minted it) commit together after the whole sweep, never on abort or import failure; a recorded mask other than `CONTACT_PERSON_FIELDS` (legacy state has none) drops the token for one full list; an expired or `GoogleSyncTokenRejectedError` token gets exactly one full-list attempt and a second failure propagates with the token nulled, not blessed; a full list that returns no replacement token leaves `contacts_sync_token` null when the mask changed (another full list follows) and keeps the old token when it did not. `resolveContactGroups` reads the directory only when a listed contact carries a user group, replaces the `contact_groups` state cache on a complete read, merges and logs on an incomplete one, and falls back to the cache when the read fails; unresolved groups render as raw ids. Pinned by `test/contacts-phone-only-groups-6324.test.ts`; the migration-176 test in `test/connector-wave3.test.ts` asserts the resumed legacy token is NOT reused (first sweep lists without it, the second reuses the fresh one). Tests: `test/google-calendar-window.test.ts` (pure rules), `test/google-source-reconcile.test.ts` (calendar window suite).+- `src/core/google/calendar-window.ts` — which calendar events a google source keeps pages for: `[now - historyDays, now + futureDays]` (`g_future_days`, 1..3650, default `CALENDAR_HORIZON_DAYS` = 60; the parsed `futureDays` leaf exists only when the key is set, so an unset knob leaves every existing source's connector identity unchanged), checked by the sweep itself on every list because a syncToken delta cannot carry timeMin/timeMax. `CalendarSyncWindow` is built once per sweep: `side(start, end)` follows Google's rule (an event is `before` when it ends at or before the floor, `after` when it starts at or past the ceiling, unparseable dates stay `inside`); `coverageStartMs(horizon, listedWholeWindow)` says where the catch-up list for unchanged instances crossing the leading edge starts (none after a whole-window list, now for state without `calendar_horizon_ms`, the recorded horizon once the ceiling is a day past it); `startsInside` and `startsAfter` are the `--full` reconcile tests. `planCalendarPage` maps one listed event to a page change (a cancellation removes its page anywhere, aged-out history is left as it is, a page past the horizon is removed, otherwise write after removing a copy at an older path). `unlistedCalendarPages` returns sweep pages (with an `event_id`) starting inside the window that a complete `--full` list did not name; `staleCalendarPages` adds the sweep pages starting at or past the ceiling, which no windowed list can name (#5442), and is what the `--full` reconcile deletes through. `people-calendar-sweep.ts` (the contacts and calendar sweeps, peeled from `google-source.ts`) performs the changes with `importRendered`/`deletePageByRelPath`, discards the catch-up list's sync token, advances `calendar_horizon_ms` only after a list reached the ceiling, and sends reconcile deletes through the >200 mass-delete guard (a refusal marks the run partial). Its contacts cursor contract (#6324): `contacts_sync_token` and `contacts_person_fields` (the mask that minted it) commit together after the whole sweep, never on abort or import failure; a recorded mask other than `CONTACT_PERSON_FIELDS` (legacy state has none) drops the token for one full list; an expired or `GoogleSyncTokenRejectedError` token gets one full list, whose failure propagates with the token nulled; a full list with no replacement token nulls `contacts_sync_token` when the mask changed and keeps it otherwise. A full list (#8906) checkpoints `contacts_relist` per clean page, stops at `CONTACT_RELIST_PAGES_PER_RUN` (10) pages (not partial), resumes under the same mask, restarts once on a stale saved token, clears with the token. `resolveContactGroups` reads the directory once, when a listed contact first has a user group: a complete read replaces the `contact_groups` cache, an incomplete one merges and logs, a failed one falls back to it; unresolved groups render as raw ids. Pinned by `test/contacts-phone-only-groups-6324.test.ts` and `test/contacts-relist-checkpoint{-8906.slow,-managed-8906}.test.ts`; the migration-176 test in `test/connector-wave3.test.ts` asserts the resumed legacy token is not reused. Tests: `test/google-calendar-window.test.ts` (pure rules), `test/google-source-reconcile.test.ts` (calendar window suite).- `src/core/google/file-modes.ts` — detection and tightening for group/world-readable files gbrain wrote under a Google source directory outside `~/.gbrain` (inside it the 0700 home already protects them). `scanGoogleFileModes(engine, { sourceIds? })` reads non-archived Google sources, skips dirs inside the home (`outsideGbrainHome`, realpath-aware), and inspects only gbrain's own layout: `.google-source.json*` in the root, every recorded page `source_path` under `emails/`, `calendar/`, `people/` (malformed paths ignored), each page's `.tmp`, and the directories between the root and those pages. Each path is `lstat`-ed component by component (a symlink anywhere on the path, another owner, or the wrong file type is counted in `skipped`, never followed); an entry is loose when `mode & 0o077`. `tightenGoogleEntry(root, entry)` re-checks the path, opens the leaf `O_NOFOLLOW` (`O_DIRECTORY` for dirs), re-checks the owner on the fd and `fchmod`s `mode & ~0o077`, so a swap after the scan is refused, not followed; the user-chosen root is never in the candidate set. `googleFileModesPreviewCommand`/`googleFileModesApplyCommand` build the `gbrain repair google-file-modes --source <id> [--apply]` strings every surface prints. win32: scan returns nothing. Tests: `test/google-file-modes-repair.test.ts`.- `src/commands/google.ts` — `gbrain google connect|status|calendars|disconnect` (+ `setup` and `attachments` dispatch). Agent-first contract: every subcommand supports `--json` emitting `{ ok, status, next_action: { command?, user_message? }, error? }` (`calendars` adds `account` + `calendars[]` — id/summary/primary/accessRole — and carries the `sources add --calendar-id` template in `next_action.command`; it resolves a lone connected account, exits 2 when there are none or several without `--account`, and throws `not_connected` for an unknown `--account`); human copy the harness must relay verbatim is fenced in `[SHOW USER] ... [/SHOW USER]` blocks; secret intake is `--client-json <path|->` (preferred) / env `GOOGLE_CLIENT_ID`+`GOOGLE_CLIENT_SECRET` / TTY prompt, with raw argv flags accepted but documented last-resort; connect is an idempotent state machine (detects what exists, performs only the missing step — re-running is the documented fix for most errors). Credential commands are engine-free; `status` has a best-effort linked-sources listing, while `attachments backfill` requires the explicitly selected local managed brain. Funnel events append to `~/.gbrain/integrations/google/heartbeat.jsonl`; account addresses are hashed to a short non-reversible tag, never raw. The vault entry's `meta.scopes` records what Google ACTUALLY granted (the token response's `scope` — the consent screen lets users uncheck scopes; the requested set is only the fallback), which is what lets downstream preflights report `scope_missing` instead of opaque per-sweep 403s. `status` reports each linked source's calendar readability (`google-calendar-status.ts`, live probe unless `--no-probe`); an unreadable calendar makes the JSON `ok: false` and the exit 1.- `src/commands/google-calendar-status.ts` — calendar readability per linked Google source for `gbrain google status` (#5012 item 9). `sourceCalendarReadability` returns null for a source that does not sync Calendar; with a probe it makes one live read of the configured calendar, otherwise it reports the `calendar_unreadable` record the last sweep stored in connector state (only when it names the current calendar id). An unreadable result carries `code: calendar_unreadable`, the HTTP status and `calendarUnreadableFix` (`gbrain google calendars --account <a>`, `gbrain sources set-calendar <src> <calendar-id>`). The sweep side is `people-calendar-sweep.ts` `recordCalendarUnreadable`: a 403/404 from `events.list` stores the record, logs both commands and returns without failing the run; a successful list clears it. Tests: `test/google-calendar-unreadable-5012.test.ts`.diff --git a/docs/guides/google-connect.md b/docs/guides/google-connect.mdindex 1829010616..6dcda89aa0 100644--- a/docs/guides/google-connect.md+++ b/docs/guides/google-connect.md@@ -507,9 +507,14 @@ attached. The catalog (also emitted as structured JSON with `--json`):Cursor expiries (`historyId` older than ~a week, calendar/contacts`syncToken` 410, or a contacts HTTP 400 explicitly reporting an expired synctoken) use bounded re-lists automatically. Contacts recovery refreshes its-own cursor without restarting Gmail. A contacts HTTP 400 on a sync-token-request that is not an expiry (a token minted under other request-parameters) gets one full-list attempt; every other HTTP 400 remains an+own cursor without restarting Gmail. A full contacts list (first sync, an+expired token, `--full`) saves its page cursor after each page and lists at+most 10 pages (2,000 contacts) per sync, so a large address book finishes+over several syncs, each resuming where the last stopped, while Calendar and+Gmail still run every time. A contacts HTTP 400 on a sync-token request that+is not an expiry (a token minted under other request parameters) gets one+full-list attempt, and a saved page cursor Google no longer accepts (HTTP 400+or 410) restarts the list from page 1 once; every other HTTP 400 remains anupstream failure rather than triggering a re-list.## Custody, privacy, spenddiff --git a/src/core/google/google-clients.ts b/src/core/google/google-clients.tsindex f1ab4e460f..f48e0df149 100644--- a/src/core/google/google-clients.ts+++ b/src/core/google/google-clients.ts@@ -171,6 +171,11 @@ export class GoogleApiClient {if (expired) throw new GoogleCursorExpiredError(res.status, url);throw new GoogleSyncTokenRejectedError(url, body.error?.message ?? 'no detail');}+ // #8906: a connections pageToken Google no longer honors (a re-list resumed from its checkpoint).+ if (res.status === 400 && apiHint === 'people') {+ const u = new URL(url);+ if (u.pathname.endsWith('/people/me/connections') && u.searchParams.get('pageToken')) throw new GoogleCursorExpiredError(res.status, url);+ }throw new CredentialError('upstream', `: HTTP ${res.status} on ${apiHint} (${body.error?.message ?? 'no detail'})`);}throw new CredentialError('upstream', `: unreachable ${apiHint}`);@@ -769,6 +774,13 @@ function contactData(p: RawPerson): ContactData {};}+function connectionsUrl(syncToken: string | null | undefined, pageToken: string | null): string {+ const params = new URLSearchParams({ personFields: CONTACT_PERSON_FIELDS, pageSize: '200', requestSyncToken: 'true' });+ if (syncToken) params.set('syncToken', syncToken);+ if (pageToken) params.set('pageToken', pageToken);+ return `${PEOPLE_BASE}/people/me/connections?${params.toString()}`;+}+export class PeopleClient extends GoogleApiClient {/** Incremental with syncToken; full otherwise. 410 → GoogleCursorExpiredError. */async listConnections(opts: {@@ -777,16 +789,7 @@ export class PeopleClient extends GoogleApiClient {}): Promise<{ contacts: ContactData[]; nextSyncToken: string | null }> {let nextSyncToken: string | null = null;const raw = await this.drainPages<RawPerson>(- (t) => {- const params = new URLSearchParams({- personFields: CONTACT_PERSON_FIELDS,- pageSize: '200',- requestSyncToken: 'true',- });- if (opts.syncToken) params.set('syncToken', opts.syncToken);- if (t) params.set('pageToken', t);- return `${PEOPLE_BASE}/people/me/connections?${params.toString()}`;- },+ (t) => connectionsUrl(opts.syncToken, t),(body) => {if (typeof body.nextSyncToken === 'string') nextSyncToken = body.nextSyncToken;return {@@ -800,6 +803,21 @@ export class PeopleClient extends GoogleApiClient {return { contacts: raw.map(contactData), nextSyncToken };}+ /**+ * #8906: one page of a full list, so a re-list can checkpoint between pages.+ * `nextSyncToken` arrives only on the last page; a stale `pageToken` (400 or+ * 410) is a GoogleCursorExpiredError.+ */+ async listConnectionsPage(pageToken: string | null, opts: { signal?: AbortSignal } = {}):+ Promise<{ contacts: ContactData[]; nextPageToken: string | null; nextSyncToken: string | null }> {+ const body = await this.fetchJSON<Record<string, unknown>>(connectionsUrl(null, pageToken), 'people', opts);+ return {+ contacts: ((body.connections as RawPerson[] | undefined) ?? []).map(contactData),+ nextPageToken: (body.nextPageToken as string | undefined) ?? null,+ nextSyncToken: typeof body.nextSyncToken === 'string' ? body.nextSyncToken : null,+ };+ }+/*** GBRA-130 A-3: one contact by resource name, for re-attempting a held* contact that no listing of this run returns. Null when it is gone (404/410).diff --git a/src/core/google/google-source.ts b/src/core/google/google-source.tsindex f8ac5059be..127143c4ee 100644--- a/src/core/google/google-source.ts+++ b/src/core/google/google-source.ts@@ -819,7 +819,7 @@ async function runGoogleSyncInner(engine: BrainEngine, sourceId: string, cfg: Goif (activeServices.includes('contacts')) {const stop = startHeartbeat(progress, 'contacts sweep');try {- await sweepContacts(deps, people, state, activePack, summary, countedSlugs, holds);+ await sweepContacts(deps, people, state, activePack, summary, countedSlugs, holds, () => saveGoogleState(deps, state));reached.add('contacts');} catch (e) {if (managed) rethrowConnectorWriteError(e);diff --git a/src/core/google/people-calendar-sweep.ts b/src/core/google/people-calendar-sweep.tsindex 59837d5dac..dd2254c2b3 100644--- a/src/core/google/people-calendar-sweep.ts+++ b/src/core/google/people-calendar-sweep.ts@@ -2,7 +2,8 @@* people-calendar-sweep — the Contacts and Calendar sweeps of the google* source kind (peeled from google-source.ts). Cursor discipline per service:* the syncToken is committed only after that service's fully-successful- * sweep; an expired token drops it and re-lists windowed.+ * sweep; an expired token drops it and re-lists windowed. A full contacts+ * re-list checkpoints its pageToken page by page and resumes there (#8906).** GBRA-130 A-3: each contact (`contact:<resourceName>`) and each event* (`event:<calendarId>:<eventId>`) goes through the connector item holds@@ -31,6 +32,9 @@ import { deletePageByRelPath, importRendered, type ActivePack, type GoogleSyncDe// ── Contacts sweep ───────────────────────────────────────────────────────────+/** #8906: pages of a full re-list one run applies before the rest of the sync gets its turn; the next run resumes after them. */+export const CONTACT_RELIST_PAGES_PER_RUN = 10;+export async function sweepContacts(deps: GoogleSyncDeps,people: PeopleClient,@@ -39,6 +43,7 @@ export async function sweepContacts(summary: GoogleSyncSummary,countedSlugs: Set<string>,holds?: ConnectorHoldSession,+ checkpoint: () => Promise<void> = async () => {},): Promise<void> {const signal = deps.opts.signal ? { signal: deps.opts.signal } : {};// #6324: a syncToken is only valid with the request parameters that minted@@ -46,28 +51,73 @@ export async function sweepContacts(// recorded) is unusable, so the sweep lists fully once and then records the// new token with the mask it was minted under.const maskChanged = state.contacts_person_fields !== CONTACT_PERSON_FIELDS;- const syncToken = deps.opts.full || maskChanged ? null : state.contacts_sync_token;- if (maskChanged && state.contacts_sync_token && !deps.opts.full) deps.log('[google] contacts field mask changed; full re-list');- let result;- try {- result = await people.listConnections({ syncToken, ...signal });- } catch (e) {- // One bounded recovery: an expired or rejected token drops to a single- // full list; a failure of that list propagates and commits nothing.- if (!syncToken) throw e;- if (e instanceof GoogleCursorExpiredError) deps.log('[google] contacts syncToken expired; full re-list');- else if (e instanceof GoogleSyncTokenRejectedError) deps.log('[google] contacts syncToken rejected (HTTP 400); one full re-list');- else throw e;- state.contacts_sync_token = null;- result = await people.listConnections({ syncToken: null, ...signal });- }- const groupNames = await resolveContactGroups(deps, people, state, result.contacts);- const sweep = { activePack, summary, countedSlugs, groupNames };+ // #8906: a full re-list that a deadline or the page budget stopped resumes at its checkpointed page.+ const resumeToken = state.contacts_relist?.person_fields === CONTACT_PERSON_FIELDS ? state.contacts_relist.page_token : null;+ const syncToken = deps.opts.full || maskChanged || resumeToken ? null : state.contacts_sync_token;+ if (maskChanged && state.contacts_sync_token && !deps.opts.full && !resumeToken) deps.log('[google] contacts field mask changed; full re-list');+ let groupNames: Record<string, string> | null = null;+ const sweep = () => ({ activePack, summary, countedSlugs, groupNames: groupNames ?? state.contact_groups ?? {} });let blocked = false;- for (const c of result.contacts) {- if (deps.opts.signal?.aborted) return;- const item = { key: contactHoldKey(c.resourceName), version: c.etag ?? null, title: c.displayName };- if (!await throughHolds(deps, summary, holds, item, () => applyContact(deps, c, sweep))) blocked = true;+ // One listed batch through the holds; false when the run was aborted part-way.+ const apply = async (contacts: ContactData[]): Promise<boolean> => {+ groupNames ??= contacts.some((c) => !c.deleted && c.groups.length > 0) ? await resolveContactGroups(deps, people, state) : null;+ const ctx = sweep();+ for (const c of contacts) {+ if (deps.opts.signal?.aborted) return false;+ const item = { key: contactHoldKey(c.resourceName), version: c.etag ?? null, title: c.displayName };+ if (!await throughHolds(deps, summary, holds, item, () => applyContact(deps, c, ctx))) blocked = true;+ }+ return true;+ };+ let delta = null;+ if (syncToken) {+ try {+ delta = await people.listConnections({ syncToken, ...signal });+ } catch (e) {+ // One bounded recovery: an expired or rejected token drops to a single+ // full re-list; a failure of that list propagates and commits nothing.+ if (e instanceof GoogleCursorExpiredError) deps.log('[google] contacts syncToken expired; full re-list');+ else if (e instanceof GoogleSyncTokenRejectedError) deps.log('[google] contacts syncToken rejected (HTTP 400); one full re-list');+ else throw e;+ state.contacts_sync_token = null;+ }+ }+ let nextSyncToken: string | null = null;+ let listed = false;+ if (delta) {+ if (!await apply(delta.contacts)) return;+ nextSyncToken = delta.nextSyncToken;+ listed = true;+ } else {+ // #8906: a full re-list goes page by page. Once a page has landed the+ // next page's token is checkpointed, so a run its job deadline stops keeps+ // every page before it. A saved token Google no longer honors restarts the+ // list from page 1, once.+ let pageToken = resumeToken;+ if (resumeToken) deps.log('[google] contacts full re-list resuming at its checkpointed page');+ for (let pages = 0; pages < CONTACT_RELIST_PAGES_PER_RUN && !listed; pages++) {+ let page;+ try {+ page = await people.listConnectionsPage(pageToken, signal);+ } catch (e) {+ if (pages > 0 || !resumeToken || !(e instanceof GoogleCursorExpiredError)) throw e;+ deps.log('[google] contacts re-list page token expired; restarting the full re-list from page 1');+ state.contacts_relist = null;+ page = await people.listConnectionsPage(pageToken = null, signal);+ }+ if (!await apply(page.contacts)) return;+ if (!page.nextPageToken) {+ nextSyncToken = page.nextSyncToken;+ listed = true;+ continue;+ }+ pageToken = page.nextPageToken;+ // A failing contact not yet held keeps the cursor before its page; a managed lease cannot write once aborted.+ if (blocked || deps.opts.signal?.aborted) continue;+ state.contacts_relist = { page_token: pageToken, person_fields: CONTACT_PERSON_FIELDS };+ await checkpoint();+ }+ if (!listed) deps.log(`[google] contacts full re-list paused after ${CONTACT_RELIST_PAGES_PER_RUN} pages; the next sync resumes it`);}const prefix = contactHoldKey('');for (const key of holds?.dueHeldKeys().filter(k => k.startsWith(prefix)) ?? []) {@@ -76,17 +126,18 @@ export async function sweepContacts(const resourceName = key.slice(prefix.length);const c = await people.getContact(resourceName, signal)?? { resourceName, displayName: null, emails: [], phones: [], organization: null, title: null, groups: [], deleted: true };- await applyContact(deps, c, sweep);+ await applyContact(deps, c, sweep());});}- if (blocked) return;+ if (blocked || !listed) return;// Cursor and the mask it was minted under commit together, only after the// whole sweep succeeded. A full list that returned no replacement token// leaves no cursor behind (the next sweep lists fully again); a same-mask// list without one keeps the token it already had.- if (result.nextSyncToken) state.contacts_sync_token = result.nextSyncToken;+ if (nextSyncToken) state.contacts_sync_token = nextSyncToken;else if (maskChanged) state.contacts_sync_token = null;state.contacts_person_fields = CONTACT_PERSON_FIELDS;+ state.contacts_relist = null;}/**@@ -204,10 +255,11 @@ async function throughHolds(deps: GoogleSyncDeps, summary: GoogleSyncSummary, ho}/**- * #6324: group names for this sweep's pages. Read the directory only when a- * listed contact carries a user group; a complete read replaces the cache (a- * deleted or renamed group is current for every page rendered from now on,- * an untouched contact's page refreshes when it next changes or on `--full`),+ * #6324: group names for this sweep's pages. The sweep reads the directory+ * once, when a listed contact first carries a user group; a complete read+ * replaces the cache (a deleted or renamed group is current for every page+ * rendered from now on, an untouched contact's page refreshes when it next+ * changes or on `--full`),* an incomplete read is merged over it and said so, an unreadable directory* falls back to the cache. A group no entry names is rendered as its raw id.*/@@ -215,10 +267,8 @@ async function resolveContactGroups(deps: GoogleSyncDeps,people: PeopleClient,state: GoogleSourceState,- contacts: ContactData[],): Promise<Record<string, string>> {const cached = state.contact_groups ?? {};- if (!contacts.some((c) => !c.deleted && c.groups.length > 0)) return cached;try {const { names, complete } = await people.listContactGroups(deps.opts.signal ? { signal: deps.opts.signal } : {});if (!complete) deps.log('[google] contact groups directory exceeds the bounded read; groups it did not name keep their ids');diff --git a/src/core/google/types.ts b/src/core/google/types.tsindex 8ecdebde11..93637acac0 100644--- a/src/core/google/types.ts+++ b/src/core/google/types.ts@@ -183,6 +183,13 @@ export interface GoogleSourceState {* fully. Absent on state saved before the mask grew (legacy = the old mask).*/contacts_person_fields?: string | null;+ /**+ * #8906: the full contacts re-list in progress: the `pageToken` of its next+ * page and the `personFields` mask it was started under, checkpointed after+ * each page lands so a run stopped by its deadline or page budget resumes+ * there. Cleared when the list completes; one under another mask is ignored.+ */+ contacts_relist?: { page_token: string; person_fields: string } | null;/*** #6324: contact-group directory cache (resource name → display name) from* the last successful `contactGroups` read; the fallback when the directorydiff --git a/test/contacts-relist-checkpoint-8906.slow.test.ts b/test/contacts-relist-checkpoint-8906.slow.test.tsnew file mode 100644index 0000000000..29d1bf67b3--- /dev/null+++ b/test/contacts-relist-checkpoint-8906.slow.test.ts@@ -0,0 +1,200 @@+/**+ * #8906: a full contacts re-list (expired syncToken, first sync, changed+ * mask) checkpoints its People pageToken after each page lands, so a run cut+ * short by its job deadline resumes at the next page instead of page 1. Runs+ * on a repo-less PGLite brain: the cursor lives in the source state the sweep+ * already writes, never in git.+ */+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';+import { mkdtempSync, rmSync, writeFileSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';++import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { CONTACT_PERSON_FIELDS } from '../src/core/google/google-clients.ts';+import { googleStateFile, parseGoogleSourceConfig, readGoogleState, runGoogleSync } from '../src/core/google/google-source.ts';+import type { GoogleSourceState } from '../src/core/google/types.ts';+import { resetPgliteState } from './helpers/reset-pglite.ts';+import { withEnv } from './helpers/with-env.ts';++let engine: PGLiteEngine;+let schemaVersion: string;++beforeAll(async () => {+ engine = new PGLiteEngine();+ await engine.connect({});+ await engine.initSchema();+ schemaVersion = (await engine.getConfig('version')) ?? '7';+});++afterAll(async () => {+ await engine.disconnect();+});++beforeEach(async () => {+ await resetPgliteState(engine);+ await engine.setConfig('version', schemaVersion);+});++const PAGE_SIZE = 200;+const json = (body: unknown, status = 200) => new Response(JSON.stringify(body), { status, headers: { 'content-type': 'application/json' } });+type Fetch = (url: string, init?: RequestInit) => Promise<Response>;++/** People connections in pages of 200: page n is asked for with pageToken p<n>; the last page mints ppl-new; a syncToken request is expired (410). */+function peopleApi(pages: number, o: { stale?: (pageToken: string) => Response; onList?: (pageToken: string | null) => void } = {}) {+ const served: string[] = [];+ const requests: Array<{ syncToken: string | null; pageToken: string | null }> = [];+ let calendarLists = 0;+ const fetchImpl: Fetch = async (url, init) => {+ const u = new URL(url);+ if (u.pathname.includes('/calendars/')) {+ calendarLists++;+ return json({ items: [], nextSyncToken: 'cal-1' });+ }+ if (!u.pathname.endsWith('/people/me/connections')) return json({ error: { message: 'unexpected fixture route' } }, 400);+ const syncToken = u.searchParams.get('syncToken');+ const pageToken = u.searchParams.get('pageToken');+ requests.push({ syncToken, pageToken });+ if (syncToken) return json({ error: { code: 410, message: 'Sync token is expired. Clear local cache and retry call without the sync token.' } }, 410);+ o.onList?.(pageToken);+ if (init?.signal?.aborted) throw Object.assign(new Error('The operation was aborted.'), { name: 'AbortError' });+ const n = pageToken === null ? 1 : /^p\d+$/.test(pageToken) ? Number(pageToken.slice(1)) : NaN;+ if (!(n >= 1 && n <= pages)) return o.stale ? o.stale(pageToken!) : json({ error: { code: 400, message: 'unexpected page token' } }, 400);+ served.push('p' + n);+ const connections = Array.from({ length: PAGE_SIZE }, (_, i) => {+ const id = 'c' + String((n - 1) * PAGE_SIZE + i).padStart(5, '0');+ return { resourceName: 'people/' + id, etag: 'e-' + id, names: [{ displayName: 'Person ' + id + ' Example' }], emailAddresses: [{ value: id + '@example.invalid' }] };+ });+ return json(n < pages ? { connections, nextPageToken: 'p' + (n + 1) } : { connections, nextSyncToken: 'ppl-new' });+ };+ return { fetchImpl, served, requests, calendarLists: () => calendarLists };+}++async function googleSource(services = 'contacts') {+ const dir = mkdtempSync(join(tmpdir(), 'gsrc-relist-8906-'));+ const config = { kind: 'google', g_account: '[email protected]', g_services: services, g_history_days: 90, g_dir: dir, g_access: 'env', g_token_env: 'GBRAIN_TEST_8906_TOKEN' };+ await engine.executeRaw('INSERT INTO sources (id, name, local_path, config) VALUES ($1, $2, $3, $4::text::jsonb)', ['gsrc', 'google', dir, JSON.stringify(config)]);+ return { dir, cfg: parseGoogleSourceConfig(config, dir) };+}++function seedState(dir: string, state: Record<string, unknown>): void {+ writeFileSync(googleStateFile(dir), JSON.stringify({ ...readGoogleState(dir), ...state }));+}++const relistOf = (dir: string) => (readGoogleState(dir) as GoogleSourceState & { contacts_relist?: { page_token: string; person_fields: string } | null }).contacts_relist ?? null;++const personPages = async () => (await engine.executeRaw<{ n: number }>(+ "SELECT count(*)::int AS n FROM pages WHERE source_id = 'gsrc' AND deleted_at IS NULL AND frontmatter->>'google_contact_id' IS NOT NULL"))[0]!.n;++const sync = (cfg: ReturnType<typeof parseGoogleSourceConfig>, fetchImpl: Fetch, signal?: AbortSignal) =>+ withEnv({ GBRAIN_HOME: mkdtempSync(join(tmpdir(), 'gbrain-home-8906-')), GBRAIN_TEST_8906_TOKEN: 'synthetic-fixture-token' }, () =>+ runGoogleSync(engine, 'gsrc', cfg, { sourceId: 'gsrc', noEmbed: true, noExtract: true, noSchemaPack: true, ...(signal ? { signal } : {}) }, fetchImpl));++describe('contacts full re-list checkpoints (#8906)', () => {+ test('expired token, 2,000 contacts, a deadline after 3 pages: each run resumes at its next page and the list completes with every page applied once', async () => {+ const { dir, cfg } = await googleSource();+ try {+ seedState(dir, { contacts_sync_token: 'stale-token', contacts_person_fields: CONTACT_PERSON_FIELDS });+ let controller = new AbortController();+ let listedThisRun = 0;+ const onDiskAtDeadline: Array<string | null> = [];+ const api = peopleApi(10, {+ onList: () => {+ // The job deadline fires as the run asks for its 4th page; what is on disk then is what a killed job leaves.+ if (++listedThisRun !== 4) return;+ controller.abort();+ onDiskAtDeadline.push(relistOf(dir)?.page_token ?? null);+ },+ });+ const firstPageTokens: Array<string | null> = [];+ for (let run = 0; run < 6 && readGoogleState(dir).contacts_sync_token !== 'ppl-new'; run++) {+ controller = new AbortController();+ listedThisRun = 0;+ const before = api.requests.length;+ await sync(cfg, api.fetchImpl, controller.signal);+ firstPageTokens.push(api.requests.slice(before).find((r) => r.syncToken === null)?.pageToken ?? null);+ }+ expect(firstPageTokens).toEqual([null, 'p4', 'p7', 'p10']);+ expect(onDiskAtDeadline).toEqual(['p4', 'p7', 'p10']);+ expect(api.served).toEqual(['p1', 'p2', 'p3', 'p4', 'p5', 'p6', 'p7', 'p8', 'p9', 'p10']);+ const state = readGoogleState(dir);+ expect(state.contacts_sync_token).toBe('ppl-new');+ expect(state.contacts_person_fields).toBe(CONTACT_PERSON_FIELDS);+ expect(relistOf(dir)).toBeNull();+ expect(await personPages()).toBe(2_000);+ const after = api.requests.length;+ await sync(cfg, api.fetchImpl);+ expect(api.requests[after]).toEqual({ syncToken: 'ppl-new', pageToken: null });+ } finally {+ rmSync(dir, { recursive: true, force: true });+ }+ }, 240_000);++ test('a saved page token Google no longer honors (400 or 410) restarts the re-list from page 1 once; a failing restart propagates without looping', async () => {+ for (const status of [400, 410]) {+ await resetPgliteState(engine);+ await engine.setConfig('version', schemaVersion);+ const { dir, cfg } = await googleSource();+ try {+ seedState(dir, { contacts_sync_token: null, contacts_person_fields: CONTACT_PERSON_FIELDS, contacts_relist: { page_token: 'gone-page', person_fields: CONTACT_PERSON_FIELDS } });+ const api = peopleApi(3, { stale: () => json({ error: { code: status, message: 'Page token is invalid.' } }, status) });+ expect((await sync(cfg, api.fetchImpl)).status).not.toBe('partial');+ expect(api.requests.map((r) => r.pageToken)).toEqual(['gone-page', null, 'p2', 'p3']);+ expect(readGoogleState(dir).contacts_sync_token).toBe('ppl-new');+ expect(relistOf(dir)).toBeNull();+ expect(await personPages()).toBe(600);+ } finally {+ rmSync(dir, { recursive: true, force: true });+ }+ }++ await resetPgliteState(engine);+ await engine.setConfig('version', schemaVersion);+ const { dir, cfg } = await googleSource();+ try {+ seedState(dir, { contacts_sync_token: null, contacts_person_fields: CONTACT_PERSON_FIELDS, contacts_relist: { page_token: 'gone-page', person_fields: CONTACT_PERSON_FIELDS } });+ const requests: Array<string | null> = [];+ const wedged: Fetch = async (url) => {+ const u = new URL(url);+ if (u.pathname.endsWith('/people/me/connections')) requests.push(u.searchParams.get('pageToken'));+ return json({ error: { code: 400, message: 'Page token is invalid.' } }, 400);+ };+ expect((await sync(cfg, wedged)).status).toBe('partial');+ expect(requests).toEqual(['gone-page', null]);+ } finally {+ rmSync(dir, { recursive: true, force: true });+ }+ }, 120_000);++ test('a page token saved under another field mask is not resumed: the re-list starts at page 1', async () => {+ const { dir, cfg } = await googleSource();+ try {+ seedState(dir, { contacts_sync_token: null, contacts_person_fields: 'names', contacts_relist: { page_token: 'p2', person_fields: 'names' } });+ const api = peopleApi(2);+ await sync(cfg, api.fetchImpl);+ expect(api.requests.map((r) => r.pageToken)).toEqual([null, 'p2']);+ expect(readGoogleState(dir).contacts_person_fields).toBe(CONTACT_PERSON_FIELDS);+ expect(relistOf(dir)).toBeNull();+ } finally {+ rmSync(dir, { recursive: true, force: true });+ }+ }, 60_000);++ test("a re-list longer than one run's page budget stops at the budget and the calendar sweep still runs; the next sync finishes it", async () => {+ const { dir, cfg } = await googleSource('contacts,calendar');+ try {+ const api = peopleApi(12);+ expect((await sync(cfg, api.fetchImpl)).status).not.toBe('partial');+ expect(api.served).toHaveLength(10);+ expect(api.calendarLists()).toBeGreaterThan(0);+ expect(relistOf(dir)?.page_token).toBe('p11');+ expect(readGoogleState(dir).contacts_sync_token).toBeNull();+ await sync(cfg, api.fetchImpl);+ expect(api.served).toEqual(Array.from({ length: 12 }, (_, i) => 'p' + (i + 1)));+ expect(readGoogleState(dir).contacts_sync_token).toBe('ppl-new');+ expect(await personPages()).toBe(2_400);+ } finally {+ rmSync(dir, { recursive: true, force: true });+ }+ }, 240_000);+});diff --git a/test/contacts-relist-checkpoint-managed-8906.test.ts b/test/contacts-relist-checkpoint-managed-8906.test.tsnew file mode 100644index 0000000000..8e5eccdb88--- /dev/null+++ b/test/contacts-relist-checkpoint-managed-8906.test.ts@@ -0,0 +1,62 @@+import { afterAll, beforeAll, expect, test } from 'bun:test';+import { parseGoogleSourceConfig, runGoogleSync } from '../src/core/google/google-source.ts';+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';+import { withEnv } from './helpers/with-env.ts';+import { contact, createConnectorFixture, googleConfig, json, options, sourceCheckpoint, withGoogleAccount } from './helpers/connector-fixture.ts';++const { engines, env, source, setup, teardown } = createConnectorFixture();+beforeAll(setup, 120_000);+afterAll(teardown);++const PAGES = 4;+const PER_PAGE = 5;++test('#8906 managed contacts re-list: a deadline mid-list keeps the checkpointed pages in the connector state; the next run resumes there and applies every contact once', async () => withEnv(env, async () => {+ for (const engine of engines) {+ const f = await source(engine, googleConfig);+ const cfg = parseGoogleSourceConfig(googleConfig, f.dir);+ const pageTokens: Array<string | null> = [];+ let onList: (() => void) | undefined;+ const fetcher = async (url: string, init?: RequestInit) => {+ const u = new URL(url);+ if (!u.pathname.endsWith('/people/me/connections')) return json({ error: { message: 'unexpected fixture route' } }, 400);+ const pageToken = u.searchParams.get('pageToken');+ pageTokens.push(pageToken);+ onList?.();+ if (init?.signal?.aborted) throw Object.assign(new Error('The operation was aborted.'), { name: 'AbortError' });+ const n = pageToken === null ? 1 : Number(pageToken.slice(1));+ const connections = Array.from({ length: PER_PAGE }, (_, i) => contact('c' + ((n - 1) * PER_PAGE + i), 'Person ' + ((n - 1) * PER_PAGE + i) + ' Example'));+ return json(n < PAGES ? { connections, nextPageToken: 'p' + (n + 1) } : { connections, nextSyncToken: 'ppl-new' });+ };+ const run = (signal?: AbortSignal) => runGoogleSync(engine, f.id, cfg, { ...options, ...(signal ? { signal } : {}) }, withGoogleAccount(fetcher, cfg.account));+ const imports = async () => (await engine.executeRaw<{ slug: string }>(+ "SELECT slug FROM persistence_requests WHERE source_id=$1 AND intent->>'kind'='connector_v2_import' ORDER BY slug", [f.id])).map((r) => r.slug);+ const state = async () => {+ await disposePersistenceConsumer(engine);+ const [row] = await sourceCheckpoint(engine, f.id) as { completed_keys: { state?: { contacts_sync_token?: string | null; contacts_relist?: { page_token: string } | null } }[] }[];+ return row?.completed_keys[0]?.state;+ };++ // The deadline fires as the run asks for page 3: pages 1 and 2 are banked.+ const controller = new AbortController();+ onList = () => { if (pageTokens.length === 3) controller.abort(); };+ const aborted = await run(controller.signal).then((r) => r.status, (e: Error) => e.name);+ expect(['partial', 'AbortError']).toContain(aborted);+ expect(pageTokens).toEqual([null, 'p2', 'p3']);+ const banked = await state();+ expect(banked?.contacts_relist?.page_token).toBe('p3');+ expect(banked?.contacts_sync_token ?? null).toBeNull();+ expect(await imports()).toHaveLength(2 * PER_PAGE);++ onList = undefined;+ pageTokens.length = 0;+ expect((await run()).status).not.toBe('partial');+ expect(pageTokens).toEqual(['p3', 'p4']);+ const all = await imports();+ expect(all).toHaveLength(PAGES * PER_PAGE);+ expect(new Set(all).size).toBe(PAGES * PER_PAGE);+ const done = await state();+ expect(done?.contacts_sync_token).toBe('ppl-new');+ expect(done?.contacts_relist ?? null).toBeNull();+ }+}), 120_000);