Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(google): a purged Gmail thread no longer rebuilds its open loops on a managed source
garrytan658ef69351a3
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 5003dd0623..9162cc0c05 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -18,7 +18,7 @@ Gmail's change feed reports a thread whenever anything on it changes, a label orOn a GitHub source with a webhook, a refresh waiting for one issue swallowed the event for any other issue of the same source, and deleting an issue right after editing it refetched the page instead of removing it. Each item event now gets its own refresh or delete.-Two unmanaged Google cases ended as held items that only a person could clear: a page whose checkout file was missing only gbrain's Facts table after one of its facts was forgotten, and an item whose page the owner purged while Google kept listing it. Both now sync cleanly; the purged item is skipped and stays purged.+Two unmanaged Google cases ended as held items that only a person could clear: a page whose checkout file was missing only gbrain's Facts table after one of its facts was forgotten, and an item whose page the owner purged while Google kept listing it. Both now sync cleanly; the purged item is skipped and stays purged. On managed and unmanaged sources alike, a purged Gmail thread that the change feed delivers again no longer writes its subject and quoted text back into open loops.### Things to watch@@ -33,6 +33,7 @@ Two unmanaged Google cases ended as held items that only a person could clear: a- `src/commands/serve-http-webhooks.ts`: the item idempotency key carries `delete` or `refresh`, the per-source waiting cap is 50 (was 1), and a cap hit answers 429 `webhook_backlog_full` with `Retry-After: 30` unless the waiting job it returns covers the event (the same item and intent, or an explicit source-wide sync for an edit; an automatic freshness sync, which can skip, covers nothing). In `src/core/minions/handlers/sync.ts` an item job that finds another sync of its source running is deferred 30 s at a time without burning an attempt, instead of completing as a source-wide skip, so its refresh or deletion is not lost (#8799). Reported and first fixed by @javieraldape.- `src/core/facts/file-body-guard.ts`, `src/core/google/sweep-shared.ts`: `replaceableFile`'s `alsoHeldAs` receives each stored state, so the connector puts back that state's own Facts fence (the live row without the withdrawal overlay, versions with it) (#8645). Reported and first fixed by @Masashi-Ono0611.- `src/core/google/sweep-shared.ts`: the unmanaged `importRendered` screens the render against page purge tombstones (`intentCarriesPurgedContent`) before any drift check or temporary write and skips a match with one log line naming `gbrain pages unpurge`; it returns no slug, so the Gmail sweep runs no loop detection or extraction on that thread and drops its grace hold (#8644). Reported and first fixed by @javieraldape.+- `src/core/persistence/connector-sync.ts`, `src/core/google/sweep-shared.ts`: on a managed source, an import the connector session skips as purged content returns `purged: true` on its `skipped` result, and `importRendered` returns no slug for it, so the same Gmail early return applies there (#8938).- Docs: `docs/guides/github-source.md` (backlog refusal, busy-source retry), `docs/architecture/key-files/google-and-loops.md`, new `docs/architecture/key-files/google-sweeps.md`.- Tests: `test/google-gmail-delta-window-8482.test.ts`, `test/google-render-forward-8719.test.ts`, `test/serve-http-github-items-8799.test.ts`, `test/google-withdrawn-fence-carry-8645.test.ts`, `test/google-classic-purged-item-8644.test.ts`, `test/sync-handler-github-item-busy-8799.test.ts`.diff --git a/docs/architecture/key-files/google-sweeps.md b/docs/architecture/key-files/google-sweeps.mdindex 6a3f46b95b..a6f7a4b7d9 100644--- a/docs/architecture/key-files/google-sweeps.md+++ b/docs/architecture/key-files/google-sweeps.md@@ -2,4 +2,4 @@[Subsystem index](../KEY_FILES.md) · [Google connector + open-loop engine](google-and-loops.md) · [Account identity and API access](google-api-access.md).-- `src/core/google/sweep-shared.ts` — the per-run dependency bag (`GoogleSyncDeps`, `GoogleSyncSummary`) and the page write and delete primitives the contacts, calendar and Gmail sweeps share. `importRendered` sends a managed source's render to `ManagedConnectorSync.importMarkdown`. On an unmanaged source it carries the stored fences into the render, then screens it against the source's page purge tombstones (`intentCarriesPurgedContent`, the content hash the import guard compares) and skips a match with one log line naming `gbrain pages unpurge`, before any drift check or file write, as the managed connector skips it. Otherwise `replaceableFile` (`facts/file-body-guard.ts`) decides whether the checkout file may be replaced; its `alsoHeldAs` callback gets each stored state and puts that state's own `## Facts` fence back into a file that lacks one (the live row stores the fence without the withdrawal overlay a page read adds, versions store the overlaid view), so a file missing only gbrain's fence is replaceable and any other file edit refuses `file_database_drift`. The render is written to a private temporary file, imported, checked against the compared bytes and renamed over the page. `deletePageByRelPath` deletes by recorded source path inside the source directory. Tests: `test/google-classic-purged-item-8644.test.ts`, `test/google-withdrawn-fence-carry-8645.test.ts`, `test/google-connector-facts-fence-7292.test.ts`.+- `src/core/google/sweep-shared.ts` — the per-run dependency bag (`GoogleSyncDeps`, `GoogleSyncSummary`) and the page write and delete primitives the contacts, calendar and Gmail sweeps share. `importRendered` sends a managed source's render to `ManagedConnectorSync.importMarkdown` and returns null when the session skipped it as purged content (`purged: true`, #8938), so the Gmail sweep derives no loops from it. On an unmanaged source it carries the stored fences into the render, then screens it against the source's page purge tombstones (`intentCarriesPurgedContent`, the content hash the import guard compares) and skips a match with one log line naming `gbrain pages unpurge`, before any drift check or file write, as the managed connector skips it. Otherwise `replaceableFile` (`facts/file-body-guard.ts`) decides whether the checkout file may be replaced; its `alsoHeldAs` callback gets each stored state and puts that state's own `## Facts` fence back into a file that lacks one (the live row stores the fence without the withdrawal overlay a page read adds, versions store the overlaid view), so a file missing only gbrain's fence is replaceable and any other file edit refuses `file_database_drift`. The render is written to a private temporary file, imported, checked against the compared bytes and renamed over the page. `deletePageByRelPath` deletes by recorded source path inside the source directory. Tests: `test/google-classic-purged-item-8644.test.ts`, `test/google-withdrawn-fence-carry-8645.test.ts`, `test/google-connector-facts-fence-7292.test.ts`.diff --git a/src/core/google/google-source.ts b/src/core/google/google-source.tsindex cad4348a4c..c70641df43 100644--- a/src/core/google/google-source.ts+++ b/src/core/google/google-source.ts@@ -194,8 +194,8 @@ async function processThread(if (!rendered) return thread;const slug = await importRendered(deps, rendered.relPath, rendered.markdown, activePack, summary, countedSlugs,{ thread_id: thread.threadId, message_ids: thread.messages.map((m) => m.id) });- // #8644: an unmanaged purged thread is neither detected nor extracted, and its grace hold is dropped, so its text does- // not come back through open loops.+ // #8644/#8938: a purged thread is neither detected nor extracted, and its grace hold is dropped, so its text does not+ // come back through open loops.if (slug === null) {delete deps.loopState?.loop_grace_holds?.[thread.threadId];return thread;diff --git a/src/core/google/sweep-shared.ts b/src/core/google/sweep-shared.tsindex d58ce70253..f07052f5e2 100644--- a/src/core/google/sweep-shared.ts+++ b/src/core/google/sweep-shared.ts@@ -148,6 +148,7 @@ export async function importRendered(if (labels && isPrivatePage(labels)) markdown = withPrivateVisibility(markdown);if (deps.managed) {const result = await deps.managed.importMarkdown(relPath, markdown);+ if (result.purged) return null;if (result.status === 'imported') {summary.pagesAffected.push(result.slug);summary.chunksCreated += result.chunks;diff --git a/src/core/persistence/connector-sync.ts b/src/core/persistence/connector-sync.tsindex 6d274cdb26..0372b2ec75 100644--- a/src/core/persistence/connector-sync.ts+++ b/src/core/persistence/connector-sync.ts@@ -633,10 +633,11 @@ export class ManagedConnectorSync {{ source_id: this.sourceId, worktree_id: this.binding.worktree_id, slug }, snapshot, serializePageToMarkdown(snapshot.page, snapshot.tags));return snapshot?.page ?? null;}- async importMarkdown(sourcePath: string, content: string): Promise<{ slug: string; chunks: number; status: 'imported' | 'skipped'; created: boolean }> {+ /** `purged` marks a skip because the content matches a page purge tombstone (#8938), so the caller derives nothing from it. */+ async importMarkdown(sourcePath: string, content: string): Promise<{ slug: string; chunks: number; status: 'imported' | 'skipped'; created: boolean; purged?: true }> {const slug = slugifyPath(sourcePath);const done = await this.submit('connector_v2_import', slug, sourcePath, { content });- if (!done.row) return { slug, chunks: 0, status: 'skipped', created: false };+ if (!done.row) return { slug, chunks: 0, status: 'skipped', created: false, ...(done.purged ? { purged: true as const } : {}) };if (done.pending) return { slug, chunks: 0, status: 'imported', created: done.created };return { slug, chunks: Number(done.row.outcome?.chunks ?? 0), status: done.row.outcome?.noop ? 'skipped' : 'imported', created: done.row.outcome?.status === 'created' };}@@ -920,7 +921,7 @@ export class ManagedConnectorSync {}}private async submit(kind: ConnectorIntentKind, slug: string, sourcePath: string | null, extra: Partial<ConnectorIntent>):- Promise<{ row: WriteRequest | null; pending: boolean; created: boolean }> {+ Promise<{ row: WriteRequest | null; pending: boolean; created: boolean; purged?: true }> {await this.awaitPagePending(slug);await this.awaitPageMirror(slug);await this.recover(slug);@@ -980,7 +981,7 @@ export class ManagedConnectorSync {// #6831: a replayed purged refusal stays skipped (even under --retry-failed) while its tombstone stands; once the// owner cleared it, the item gets one automatic retry through the guarded retry path below.if (kind === 'connector_v2_import' && row && purgedRefusal(row) && row.request_id === (this.retryApprovals.get(baseRequestId) ?? baseRequestId)) {- if (await this.purgedNow(slug, extra.content)) { this.skipPurged(slug, row); return { row: null, pending: false, created: false }; }+ if (await this.purgedNow(slug, extra.content)) { this.skipPurged(slug, row); return { row: null, pending: false, created: false, purged: true }; }this.autoRetry.add(baseRequestId);}const automatic = this.autoRetry.has(baseRequestId);@@ -1043,7 +1044,7 @@ export class ManagedConnectorSync {}if (kind === 'connector_v2_import' && await this.carriesPurgedPage(slug, extra.content)) {this.skipPurged(slug, null);- return { row: null, pending: false, created: false };+ return { row: null, pending: false, created: false, purged: true };}row = await this.engine.transaction(async tx => {await this.assertLease(tx);@@ -1076,7 +1077,7 @@ export class ManagedConnectorSync {return { row, pending: false, created: false };}if (isTerminal(row)) {- if (page && purgedRefusal(row)) { this.skipPurged(slug, row); return { row: null, pending: false, created: false }; }+ if (page && purgedRefusal(row)) { this.skipPurged(slug, row); return { row: null, pending: false, created: false, purged: true }; }if (row.state !== 'committed') this.lastFailedItem = { slug, requestId: row.request_id };writeResponse(row);this.receipts.push(row.id);diff --git a/test/google-classic-purged-item-8644.test.ts b/test/google-classic-purged-item-8644.test.tsindex 4709d05ff6..a13fbb5dfe 100644--- a/test/google-classic-purged-item-8644.test.ts+++ b/test/google-classic-purged-item-8644.test.ts@@ -5,9 +5,10 @@* once the file was removed, and held it after three runs. It is now skipped before any file is written, the run stays* clean, and the page stays purged, as on a managed brain (#6831). A skipped Gmail thread is not run through loop* detection or extraction either, so the open loops the purge removed do not come back with the thread's text.+ * #8938: the same holds on a managed source, where the connector session skips the purged render and reports it.* Real PGLite and Postgres (when DATABASE_URL is set), the real Google importer and purge, a fake People API.*/-import { describe, expect, test } from 'bun:test';+import { afterAll, beforeAll, describe, expect, spyOn, test } from 'bun:test';import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';import { tmpdir } from 'node:os';import { join } from 'node:path';@@ -16,7 +17,10 @@ import { parseGoogleSourceConfig, readGoogleState, runGoogleSync } from '../src/import { submitPageMutation } from '../src/core/persistence/page-mutations.ts';import { unpurgePage } from '../src/core/persistence/page-purge.ts';import { _resetWriteThroughCacheForTest } from '../src/core/write-through.ts';-import { contact, json, options, withGoogleAccount } from './helpers/connector-fixture.ts';+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';+import { readManagedConnectorState } from '../src/core/persistence/connector-state.ts';+import { ManagedConnectorSync } from '../src/core/persistence/connector-sync.ts';+import { contact, createConnectorFixture, json, options, settleConnectorWrites, withGoogleAccount } from './helpers/connector-fixture.ts';import { isolatedSharedSkillsEngine } from './helpers/shared-skills-engine.ts';import { requirePostgresTestDatabase, testBackends } from './helpers/test-backends.ts';import { withEnv } from './helpers/with-env.ts';@@ -24,7 +28,7 @@ import { withEnv } from './helpers/with-env.ts';const SOURCE = 'source-a';const SLUG = 'people/alice-example';const CONFIG = { kind: 'google', g_account: '[email protected]', g_services: 'contacts', g_access: 'env', g_token_env: 'CONNECTOR_TEST_TOKEN' };-const ctx = (engine: BrainEngine) => ({ engine, sourceId: SOURCE, config: { engine: engine.kind }, remote: false, dryRun: false,+const ctx = (engine: BrainEngine, sourceId = SOURCE) => ({ engine, sourceId, config: { engine: engine.kind }, remote: false, dryRun: false,logger: { info() {}, warn() {}, error() {} } });interface Brain { engine: BrainEngine; root: string; path: string; render: string; sync(): ReturnType<typeof runGoogleSync> }@@ -157,3 +161,78 @@ for (const backend of testBackends()) describe(`[${backend}] #8644: a purged Gma} finally { rmSync(root, { recursive: true, force: true }); }}, 180_000);});++const managed = createConnectorFixture();+describe('#8938: a purged Gmail thread on a managed Google source', () => {+ beforeAll(managed.setup, 120_000);+ afterAll(managed.teardown);++ test('re-delivering the unchanged thread writes no page and no open loop or quote, and skips cleanly', async () => withEnv(managed.env, async () => {+ for (const engine of managed.engines) {+ const f = await managed.source(engine, GMAIL_CONFIG);+ const cfg = parseGoogleSourceConfig(GMAIL_CONFIG, f.dir);+ const sentAt = Date.now() - 3 * 86_400_000;+ let historyId = '5000';+ let changed: string[] = [];+ const message = { id: '18ff00000000b001', threadId: THREAD, labelIds: ['INBOX'], internalDate: String(sentAt),+ payload: { mimeType: 'text/plain', headers: [{ name: 'From', value: 'Alice Example <[email protected]>' },+ { name: 'To', value: GMAIL_CONFIG.g_account }, { name: 'Subject', value: 'Contract question' }],+ body: { data: b64url('Could you confirm the acme-example contract terms?') } } };+ const fetch = withGoogleAccount(async (url: string) => {+ const u = new URL(url);+ if (u.pathname.endsWith('/settings/sendAs')) return json({ sendAs: [] });+ if (u.pathname.endsWith('/users/me/profile')) return json({ emailAddress: GMAIL_CONFIG.g_account, historyId: '5000' });+ if (u.pathname.endsWith('/users/me/messages')) return json({ messages: Number(/after:(\d+)/.exec(u.searchParams.get('q') ?? '')?.[1] ?? 0) * 1000 <= sentAt+ && sentAt < Number(/before:(\d+)/.exec(u.searchParams.get('q') ?? '')?.[1] ?? Infinity) * 1000 ? [{ id: message.id, threadId: THREAD }] : [] });+ if (u.pathname.endsWith('/users/me/history')) return json({ historyId, history: u.searchParams.get('startHistoryId') === historyId ? []+ : [{ id: historyId, messages: changed.map(threadId => ({ id: message.id, threadId })) }] });+ if (u.pathname.endsWith(`/users/me/threads/${THREAD}`)) return json({ id: THREAD, messages: [message] });+ throw new Error(`Unexpected Google fixture route: ${url}`);+ }, GMAIL_CONFIG.g_account);+ const run = () => runGoogleSync(engine, f.id, cfg, options, withGoogleAccount(fetch, GMAIL_CONFIG.g_account));+ const loops = async () => (await engine.executeRaw<{ n: number }>('SELECT count(*)::int AS n FROM open_loops WHERE source_id=$1 AND thread_id=$2', [f.id, THREAD]))[0]!.n;+ const quoted = async () => (await engine.executeRaw<{ n: number }>(+ "SELECT count(*)::int AS n FROM open_loops WHERE source_id=$1 AND (summary ILIKE '%contract%' OR evidence::text ILIKE '%acme-example contract%')", [f.id]))[0]!.n;+ const livePages = async () => (await engine.executeRaw<{ slug: string }>('SELECT slug FROM pages WHERE source_id=$1 AND deleted_at IS NULL', [f.id])).map(r => r.slug);++ expect((await run()).status).not.toBe('partial');+ await settleConnectorWrites(engine, f.id);+ await disposePersistenceConsumer(engine);+ const [slug] = await livePages();+ expect(slug).toBeDefined();+ expect(await loops()).toBe(1);+ const snapshot = (await engine.readPageSnapshot(slug!, { sourceId: f.id }))!;+ await submitPageMutation(ctx(engine, f.id), { operation: 'delete_page', params: { slug: slug!, purge: true, expected_revision: snapshot.revision } });+ await settleConnectorWrites(engine, f.id);+ expect(await loops()).toBe(0);++ historyId = '5010';+ changed = [THREAD];+ const stderr = spyOn(process.stderr, 'write').mockImplementation(() => true);+ let replay;+ try { replay = await run(); } finally { stderr.mockRestore(); }+ expect(replay.status).not.toBe('partial');+ await settleConnectorWrites(engine, f.id).catch(() => {});+ expect(await livePages()).toEqual([]);+ expect(await loops()).toBe(0);+ expect(await quoted()).toBe(0);+ const [row] = await engine.executeRaw<{ incarnation: string }>('SELECT incarnation::text FROM sources WHERE id=$1', [f.id]);+ const state = await readManagedConnectorState(engine, f.id, row!.incarnation);+ expect(state.last_failure).toBeNull();+ expect(state.last_run).toMatchObject({ purged_skipped: 1 });+ await disposePersistenceConsumer(engine);++ // A render the session's screen misses is admitted and refused by the pages guard at publication; that skip is+ // reported as purged too, so the thread still yields no loop.+ historyId = '5020';+ const screen = spyOn(ManagedConnectorSync.prototype as unknown as { carriesPurgedPage: () => Promise<boolean> }, 'carriesPurgedPage').mockResolvedValue(false);+ const quiet = spyOn(process.stderr, 'write').mockImplementation(() => true);+ try { await run(); } finally { quiet.mockRestore(); screen.mockRestore(); }+ await settleConnectorWrites(engine, f.id).catch(() => {});+ expect(await livePages()).toEqual([]);+ expect(await loops()).toBe(0);+ expect(await quoted()).toBe(0);+ await disposePersistenceConsumer(engine);+ }+ }), 240_000);+});