Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix: record captures of completed jobs at upgrade and resend a managed capture under a fresh request_id (GST-1856 review)
garrytan2cccf7aa45d0
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex d5d6313072..ad30fbfb47 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -12,14 +12,14 @@ identifiers and attribution are available in the pre-removal Git revision## [Unreleased]-**A `POST /ingest` capture that runs again after its write landed no longer reverts a newer edit to the page.** A worker that died or lost its lock after the write committed, but before the job was marked complete, handed the job to another attempt, which wrote the captured body over any edit made in between. A client replaying the same POST after `gbrain jobs prune` removed the finished job got a fresh job that did the same. Each capture is now recorded under its job's idempotency key in the same transaction as the page write, and any later run of that key writes nothing and returns the first run's result. The record lasts 90 days, even after the job row is pruned. Separate captures to one page still land in order, because each has its own key.+**A `POST /ingest` capture that runs again after its write landed no longer reverts a newer edit to the page.** A worker that died or lost its lock after the write committed, but before the job was marked complete, handed the job to another attempt, which wrote the captured body over any edit made in between. A client replaying the same POST after `gbrain jobs prune` removed the finished job got a fresh job that did the same. Each capture is now recorded under its job's idempotency key in the same transaction as the page write, and any later run of that key writes nothing and returns the first run's result. The record lasts 90 days, even after the job row is pruned. The upgrade records the captures whose completed jobs are still in the queue, so replaying one of those after its job row is pruned writes nothing either. Separate captures to one page still land in order, because each has its own key.### Itemized changes-- `src/core/minions/handlers/ingest-capture.ts` + `src/core/ingestion/applied-captures.ts`: a keyed capture checks `ingest_applied_captures` (new migration v296) before writing. On an unmanaged brain the row is inserted inside the page write's transaction, and a run that finds the row already there rolls its write back. On a managed brain each attempt sends the door a `request_id` derived from the key and the attempt number, the row is inserted once the door reports the write committed, and a committed request found without its row counts as applied and fills the row in. Tombstones and jobs queued without a key behave as before.+- `src/core/minions/handlers/ingest-capture.ts` + `src/core/ingestion/applied-captures.ts`: a keyed capture checks `ingest_applied_captures` (new migration v296, which also records the completed `ingest_capture` jobs already in `minion_jobs`) before writing. On an unmanaged brain the row is inserted inside the page write's transaction, and a run that finds the row already there rolls its write back. On a managed brain each attempt sends the door a `request_id` derived from the key and the attempt number, the row is inserted once the door reports the write committed, and a committed request found without its row counts as applied and fills the row in. A run whose `request_id` already names an earlier write of the capture that never committed (a stalled run keeps the attempt number, and `gbrain jobs retry` resets it) sends its write under a fresh `request_id` instead of failing the attempt. Tombstones and jobs queued without a key behave as before.- `src/core/minions/types.ts`: handlers see the job's `idempotency_key` on their context; jobs already waiting when you upgrade carry it from their row.- `src/core/cycle.ts`: the dream cycle's purge phase drops applied-capture records older than 90 days.-- Tests: `test/ingest-capture-applied-once.test.ts` (PGLite) and `test/e2e/ingest-capture-applied-once-postgres.test.ts` (Postgres): a re-run after a newer edit, a replay after the job row is pruned (through the real route), two ordered captures to one slug, and the re-run on a managed brain.+- Tests: `test/ingest-capture-applied-once.test.ts` (PGLite) and `test/e2e/ingest-capture-applied-once-postgres.test.ts` (Postgres): a re-run after a newer edit, a replay after the job row is pruned (through the real route), two ordered captures to one slug, a replay of a capture completed before the upgrade, the re-run on a managed brain, and a retried managed capture whose attempt id already names a refused write.## [0.60.610.0] - 2026-11-08diff --git a/src/core/minions/handlers/ingest-capture.ts b/src/core/minions/handlers/ingest-capture.tsindex ca6c600327..3f885489a4 100644--- a/src/core/minions/handlers/ingest-capture.ts+++ b/src/core/minions/handlers/ingest-capture.ts@@ -32,6 +32,7 @@* pages to canonical homes later.*/+import { randomUUID } from 'node:crypto';import type { MinionJobContext } from '../types.ts';import type { BrainEngine } from '../../engine.ts';import type { IngestionEvent } from '../../ingestion/types.ts';@@ -266,8 +267,8 @@ export function makeIngestCaptureHandler(engine: BrainEngine) {// nothing, so it never reverts a newer edit. Tombstones and jobs queued without a key keep their old behavior.const appliedKey = !isTombstone && job.idempotency_key ? job.idempotency_key : null;const managedOutcome = (outcome: { noop?: boolean; chunks?: number }) => captured(outcome.noop ? 'skipped' : 'imported', Number(outcome.chunks ?? 0));- const applied = appliedKey ? await readAppliedCapture<IngestCaptureResult>(engine, appliedKey,- managed ? { sourceIds: [...new Set([sourceId ?? 'default', 'default'])], slug: slug.toLowerCase(), result: managedOutcome } : undefined) : null;+ const journal = managed ? { sourceIds: [...new Set([sourceId ?? 'default', 'default'])], slug: slug.toLowerCase(), result: managedOutcome } : undefined;+ const applied = appliedKey ? await readAppliedCapture<IngestCaptureResult>(engine, appliedKey, journal) : null;if (applied) return applied;const webhookWrite = untrustedPayload && event.source_kind === 'webhook' && !isTombstone;@@ -336,23 +337,32 @@ export function makeIngestCaptureHandler(engine: BrainEngine) {// #2044/#4548: as `remote: untrustedPayload` does unmanaged, an untrusted body keeps the non-world facts rows and takes rows its sender could not see....(untrustedPayload ? { keep_hidden_rows: true } : {}) });};+ // A stall re-run keeps attempts_made and `gbrain jobs retry` resets it, so this run's id can name an earlier write of+ // this capture that never committed (idempotency_conflict). Unless a run of the key committed since the check above,+ // this run's write goes out under a fresh id with the same prefix.+ let appliedSince = null as IngestCaptureResult | null;+ const submit = async (sid: string, request: string | undefined) => {+ try {+ return await put(sid, request);+ } catch (err) {+ if (!request || !(err instanceof OperationError) || err.code !== 'idempotency_conflict') throw err;+ appliedSince = await readAppliedCapture<IngestCaptureResult>(engine, appliedKey!, journal);+ return appliedSince ? {} : put(sid, `${request}:${randomUUID()}`);+ }+ };let response: Record<string, unknown>;try {- response = await put(sourceId ?? 'default', requestId);+ response = await submit(sourceId ?? 'default', requestId);} catch (err) {- // A re-run of the same attempt whose earlier request did not commit and read another revision: retry it as a new- // attempt (a new request_id) instead of letting the deterministic id end the job as a refusal.- if (requestId && err instanceof OperationError && err.code === 'idempotency_conflict') {- throw new Error(`ingest_capture: attempt ${job.attempts_made} of this capture was already submitted for an older revision; the next attempt retries it`);- }// The door refuses a source that vanished or was archived after the pre-check (the managed form of the FK retry).const vanished = sourceId !== undefined && err instanceof OperationError && err.code === 'source_changed'? await engine.executeRaw<{ archived: boolean | null }>('SELECT archived FROM sources WHERE id = $1', [sourceId]) : null;if (!vanished || (vanished.length > 0 && vanished[0]?.archived !== true)) throw err;sourceFallback = { requested: sourceId!, effective: 'default', reason: vanished.length ? 'archived' : 'fk_violation' };await checkWriter('default');- response = await put('default', requestId && `${requestId}:default`);+ response = await submit('default', requestId && `${requestId}:default`);}+ if (appliedSince) return appliedSince;if (sourceFallback) {console.error(`[WARN] ingest_capture: requested source '${sourceFallback.requested}' unavailable ` +diff --git a/src/core/schema-migrations/v296-ingest-applied-captures.ts b/src/core/schema-migrations/v296-ingest-applied-captures.tsindex 0144f22251..c9ebd16cc9 100644--- a/src/core/schema-migrations/v296-ingest-applied-captures.ts+++ b/src/core/schema-migrations/v296-ingest-applied-captures.ts@@ -10,8 +10,11 @@ import type { Migration } from './types.ts';// body again, so a stalled attempt re-run or a POST /ingest replayed after// `gbrain jobs prune` no longer reverts a newer edit. The dream cycle's purge// phase drops rows past 90 days (purgeStaleAppliedCaptures). Operational-// state, not brain content: a repo-less brain gets it unchanged. Created empty;-// nothing is rewritten on upgrade. No schema.sql mirror: like v282-v290, it+// state, not brain content: a repo-less brain gets it unchanged. The new table+// is seeded from the completed ingest_capture jobs still in minion_jobs, so a+// capture that landed before the upgrade is not applied again when its POST is+// replayed after the job row is pruned; no existing row is rewritten. No+// schema.sql mirror: like v282-v290, it// exists only through migrations. The number is a placeholder restamped at the// merge slot.export const v296: Migration = {@@ -27,6 +30,16 @@ CREATE TABLE IF NOT EXISTS ingest_applied_captures (applied_at TIMESTAMPTZ NOT NULL DEFAULT now());CREATE INDEX IF NOT EXISTS ingest_applied_captures_applied_at_idx ON ingest_applied_captures (applied_at);+INSERT INTO ingest_applied_captures (idempotency_key, source_id, slug, result, applied_at)+SELECT DISTINCT ON (idempotency_key) idempotency_key,+ COALESCE(result->'source_fallback'->>'effective', data->>'sourceId', 'default'),+ COALESCE(result->>'slug', data->>'slug', ''), result, COALESCE(finished_at, now())+ FROM minion_jobs+ WHERE name = 'ingest_capture' AND status = 'completed' AND idempotency_key IS NOT NULL+ AND jsonb_typeof(result) = 'object' AND result->>'status' IS DISTINCT FROM 'error'+ AND data->'event'->>'kind' IS DISTINCT FROM 'tombstone'+ ORDER BY idempotency_key, finished_at DESC NULLS LAST+ON CONFLICT (idempotency_key) DO NOTHING;DO $rls$ BEGINIF EXISTS (SELECT 1 FROM pg_roles pr WHERE pg_has_role(current_user, pr.oid, 'USAGE') AND (pr.rolbypassrls OR pr.rolsuper)) THENALTER TABLE ingest_applied_captures ENABLE ROW LEVEL SECURITY;diff --git a/test/fixtures/goldens/migrations/records.json b/test/fixtures/goldens/migrations/records.jsonindex f249a6baa3..5f3094c3a8 100644--- a/test/fixtures/goldens/migrations/records.json+++ b/test/fixtures/goldens/migrations/records.json@@ -4819,8 +4819,8 @@"idempotent": true,"name": "ingest_applied_captures","sql": {- "length": 556,- "sha256": "685b94ce31a5399b6fa9e456a3280a52a33b5d2f87d60b958f946df5b255144d"+ "length": 1228,+ "sha256": "71295828e31d9b61202fc5e8b05d78f25041b7509c65bef1aeed7e046e761e4c"},"sqlFor": "absent","transaction": "absent",diff --git a/test/ingest-capture-applied-once.test.ts b/test/ingest-capture-applied-once.test.tsindex b42e4c8554..8fca528871 100644--- a/test/ingest-capture-applied-once.test.ts+++ b/test/ingest-capture-applied-once.test.ts@@ -24,7 +24,8 @@ import { GBrainOAuthProvider } from '../src/core/oauth-provider.ts';import { sqlQueryForEngine } from '../src/core/sql-query.ts';import { importFromContent } from '../src/core/import-file.ts';import { computeContentHash } from '../src/core/ingestion/types.ts';-import { purgeStaleAppliedCaptures } from '../src/core/ingestion/applied-captures.ts';+import { appliedCaptureRequestPrefix, purgeStaleAppliedCaptures } from '../src/core/ingestion/applied-captures.ts';+import { MIGRATIONS, runMigrationSQL } from '../src/core/migrate.ts';import { makeIngestCaptureHandler } from '../src/core/minions/handlers/ingest-capture.ts';import type { MinionJobContext } from '../src/core/minions/types.ts';import { submitPageMutation } from '../src/core/persistence/page-mutations.ts';@@ -135,6 +136,25 @@ for (const backend of backends) {expect(await pageText(slug)).not.toContain('Body from the integration.');});+ test('a capture completed before the upgrade keeps the newer edit when its POST is replayed after the job row is pruned', async () => {+ const slug = `notes/pre-upgrade-${backend}`;+ const body = note('Pre-upgrade capture', 'Body captured before the upgrade.');+ const first = await ingest(body, slug);+ const firstResult = await runJob(first);+ await brain().engine.executeRaw("UPDATE minion_jobs SET status = 'completed', finished_at = now(), result = $2::text::jsonb WHERE id = $1", [first, JSON.stringify(firstResult)]);+ // Before the upgrade nothing recorded the capture: the migration records it from the completed job.+ await brain().engine.executeRaw('DELETE FROM ingest_applied_captures WHERE slug = $1', [slug]);+ const migration = MIGRATIONS.find(m => m.name === 'ingest_applied_captures')!;+ await runMigrationSQL(brain().engine, migration, migration.sql!);+ await importFromContent(brain().engine, slug, note('Pre-upgrade capture', 'Newer edit after the upgrade.'), { sourceId: SRC, noEmbed: true });+ await brain().engine.executeRaw('DELETE FROM minion_jobs WHERE id = $1', [first]);++ const replay = await ingest(body, slug);+ expect(await runJob(replay)).toEqual(firstResult);+ expect(await pageText(slug)).toContain('Newer edit after the upgrade.');+ expect(await pageText(slug)).not.toContain('Body captured before the upgrade.');+ });+test('two captures with different content to one slug land in order, and a re-run of the first keeps the second', async () => {const slug = `notes/ordered-${backend}`;const one = await ingest(note('Ordered capture', 'First capture body.'), slug);@@ -256,5 +276,24 @@ for (const backend of backends) {expect(await pageText()).toContain('Managed newer edit.');expect(await pageText()).not.toContain('Managed captured body.');}, { databaseUrl }), 120_000);++ test('a capture whose attempt id already names a refused write of it still lands (jobs retry resets attempts_made)', () => managedBrain(async ({ engine, ctx }) => {+ const slug = 'notes/managed-retried';+ const data = captureData(note('Managed retried', 'Managed retried body.'), slug, 'default');+ const key = 'ingest:webhook:client-example:default:managed-retried';+ const pageText = async () => (await engine.readPageSnapshot(slug, { sourceId: 'default' }))?.page.compiled_truth ?? '';+ await submitPageMutation(ctx, { operation: 'put_page', params: { slug, content: note('Managed retried', 'Page before the capture.') } });+ const stale = await engine.readPageSnapshot(slug, { sourceId: 'default' });+ await submitPageMutation(ctx, { operation: 'put_page', params: { slug, content: note('Managed retried', 'Edit before the capture.'), expected_revision: stale!.revision } });+ // Attempt 0 reached the door for the revision before that edit and was refused; `gbrain jobs retry` reset attempts_made to 0.+ await expect(submitPageMutation(ctx, { operation: 'put_page', params: { slug, content: data.event.content, content_origin: 'tool_output',+ expected_revision: stale!.revision, request_id: `${appliedCaptureRequestPrefix(key)}0` } })).rejects.toThrow();++ const run = makeIngestCaptureHandler(engine);+ const result = await run(jobCtx(5, data, key));+ expect(result.status).toBe('imported');+ expect(await pageText()).toContain('Managed retried body.');+ expect(await run(jobCtx(5, data, key))).toEqual(result);+ }, { databaseUrl }), 120_000);});}