Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix: send each managed /ingest capture run under a new request_id and count only the handler's own journal requests (GST-1856 review)

garrytan7c05e679bf1b
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ad30fbfb47..f09a510378 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -16,10 +16,10 @@ identifiers and attribution are available in the pre-removal Git revision
### Itemized changes
-- `src/core/minions/handlers/ingest-capture.ts` + `src/core/ingestion/applied-captures.ts`: a keyed capture checks `ingest_applied_captures` (new migration v296, which also records the completed `ingest_capture` jobs already in `minion_jobs`) before writing. On an unmanaged brain the row is inserted inside the page write's transaction, and a run that finds the row already there rolls its write back. On a managed brain each attempt sends the door a `request_id` derived from the key and the attempt number, the row is inserted once the door reports the write committed, and a committed request found without its row counts as applied and fills the row in. A run whose `request_id` already names an earlier write of the capture that never committed (a stalled run keeps the attempt number, and `gbrain jobs retry` resets it) sends its write under a fresh `request_id` instead of failing the attempt. Tombstones and jobs queued without a key behave as before.
+- `src/core/minions/handlers/ingest-capture.ts` + `src/core/ingestion/applied-captures.ts`: a keyed capture checks `ingest_applied_captures` (new migration v296, which also records the completed `ingest_capture` jobs already in `minion_jobs`) before writing. On an unmanaged brain the row is inserted inside the page write's transaction, and a run that finds the row already there rolls its write back. On a managed brain each run sends the door a new `request_id` under a prefix derived from the key, the row is inserted once the door reports the write committed, and a committed request the handler sent as the local writer, found without its row, counts as applied and fills the row in; a remote writer's request under the same prefix never counts. Because the id is new on every run, a stalled run or a `gbrain jobs retry` (which resets the attempt count) never replays an earlier write of the capture that failed. Tombstones and jobs queued without a key behave as before.
- `src/core/minions/types.ts`: handlers see the job's `idempotency_key` on their context; jobs already waiting when you upgrade carry it from their row.
- `src/core/cycle.ts`: the dream cycle's purge phase drops applied-capture records older than 90 days.
-- Tests: `test/ingest-capture-applied-once.test.ts` (PGLite) and `test/e2e/ingest-capture-applied-once-postgres.test.ts` (Postgres): a re-run after a newer edit, a replay after the job row is pruned (through the real route), two ordered captures to one slug, a replay of a capture completed before the upgrade, the re-run on a managed brain, and a retried managed capture whose attempt id already names a refused write.
+- Tests: `test/ingest-capture-applied-once.test.ts` (PGLite) and `test/e2e/ingest-capture-applied-once-postgres.test.ts` (Postgres): a re-run after a newer edit, a replay after the job row is pruned (through the real route), two ordered captures to one slug, a replay of a capture completed before the upgrade, the re-run on a managed brain, a retried managed capture whose attempt id already names a refused or failed write, and a remote writer's request forged under the capture's prefix.
## [0.60.610.0] - 2026-11-08
diff --git a/src/core/ingestion/applied-captures.ts b/src/core/ingestion/applied-captures.ts
index 9eef442599..4ecbf7bdc8 100644
--- a/src/core/ingestion/applied-captures.ts
+++ b/src/core/ingestion/applied-captures.ts
@@ -6,8 +6,9 @@
* stalled attempt re-run after its write landed, a POST /ingest replayed after `gbrain jobs prune` removed the job
* row) returns the stored result and writes nothing, so it cannot revert a newer edit. Each capture has its own key, so
* ordered captures to one slug all apply. A managed write's door request carries a client request_id under
- * `appliedCaptureRequestPrefix(key)`, so a committed request found without its row (the worker died between the commit
- * and the insert) counts as applied and backfills the row. Rows live 90 days, at least as long as the job rows the
+ * `appliedCaptureRequestPrefix(key)`, so a committed request the handler sent as the local writer, found without its row
+ * (the worker died between the commit and the insert), counts as applied and backfills the row; a remote writer's request
+ * under the same prefix never counts. Rows live 90 days, at least as long as the job rows the
* queue-level dedupe relied on (`gbrain jobs prune` defaults to 30 days).
*/
@@ -36,7 +37,7 @@ export async function readAppliedCapture<T>(engine: Exec, key: string,
if (!journal) return null;
const [request] = await engine.executeRaw<{ source_id: string; outcome: unknown }>(
`SELECT source_id, outcome FROM persistence_requests
- WHERE source_id = ANY($1::text[]) AND slug = $2 AND state = 'committed' AND client_request_id LIKE $3
+ WHERE source_id = ANY($1::text[]) AND slug = $2 AND state = 'committed' AND principal_kind = 'local_cli' AND client_request_id LIKE $3
ORDER BY sequence LIMIT 1`,
[journal.sourceIds, journal.slug, `${appliedCaptureRequestPrefix(key)}%`]);
if (!request) return null;
diff --git a/src/core/minions/handlers/ingest-capture.ts b/src/core/minions/handlers/ingest-capture.ts
index 3f885489a4..f9f8f454e4 100644
--- a/src/core/minions/handlers/ingest-capture.ts
+++ b/src/core/minions/handlers/ingest-capture.ts
@@ -267,8 +267,8 @@ export function makeIngestCaptureHandler(engine: BrainEngine) {
// nothing, so it never reverts a newer edit. Tombstones and jobs queued without a key keep their old behavior.
const appliedKey = !isTombstone && job.idempotency_key ? job.idempotency_key : null;
const managedOutcome = (outcome: { noop?: boolean; chunks?: number }) => captured(outcome.noop ? 'skipped' : 'imported', Number(outcome.chunks ?? 0));
- const journal = managed ? { sourceIds: [...new Set([sourceId ?? 'default', 'default'])], slug: slug.toLowerCase(), result: managedOutcome } : undefined;
- const applied = appliedKey ? await readAppliedCapture<IngestCaptureResult>(engine, appliedKey, journal) : null;
+ const applied = appliedKey ? await readAppliedCapture<IngestCaptureResult>(engine, appliedKey,
+ managed ? { sourceIds: [...new Set([sourceId ?? 'default', 'default'])], slug: slug.toLowerCase(), result: managedOutcome } : undefined) : null;
if (applied) return applied;
const webhookWrite = untrustedPayload && event.source_kind === 'webhook' && !isTombstone;
@@ -327,9 +327,11 @@ export function makeIngestCaptureHandler(engine: BrainEngine) {
// tool_output stores and gates the capture external_untrusted (#5575 A3), as importOpts does unmanaged.
// A capture replaces the page as it stands, so it binds to the revision read just before (the door requires one to
// replace an existing page); a write landing in between refuses revision_conflict and the job retries.
- // GST-1856: each attempt of a keyed capture sends its own request_id under the capture's prefix, so the applied check
- // above finds a committed attempt in the door's journal even when the worker died before recording it.
- const requestId = appliedKey ? `${appliedCaptureRequestPrefix(appliedKey)}${job.attempts_made}` : undefined;
+ // GST-1856: each run of a keyed capture sends a new request_id under the capture's prefix, so the applied check above
+ // finds a committed run in the door's journal even when the worker died before recording it. The id is new per run
+ // because a stall re-run keeps attempts_made and `gbrain jobs retry` resets it: an id reused for an earlier request
+ // that failed would replay that failure instead of writing.
+ const requestId = appliedKey ? `${appliedCaptureRequestPrefix(appliedKey)}${job.attempts_made}:${randomUUID()}` : undefined;
const put = async (sid: string, request?: string) => {
const snapshot = await engine.readPageSnapshot(slug, { sourceId: sid, includeDeleted: true });
return door('put_page', sid, { content: event.content, content_origin: 'tool_output', source_kind: event.source_kind, source_uri: event.source_uri,
@@ -337,22 +339,9 @@ export function makeIngestCaptureHandler(engine: BrainEngine) {
// #2044/#4548: as `remote: untrustedPayload` does unmanaged, an untrusted body keeps the non-world facts rows and takes rows its sender could not see.
...(untrustedPayload ? { keep_hidden_rows: true } : {}) });
};
- // A stall re-run keeps attempts_made and `gbrain jobs retry` resets it, so this run's id can name an earlier write of
- // this capture that never committed (idempotency_conflict). Unless a run of the key committed since the check above,
- // this run's write goes out under a fresh id with the same prefix.
- let appliedSince = null as IngestCaptureResult | null;
- const submit = async (sid: string, request: string | undefined) => {
- try {
- return await put(sid, request);
- } catch (err) {
- if (!request || !(err instanceof OperationError) || err.code !== 'idempotency_conflict') throw err;
- appliedSince = await readAppliedCapture<IngestCaptureResult>(engine, appliedKey!, journal);
- return appliedSince ? {} : put(sid, `${request}:${randomUUID()}`);
- }
- };
let response: Record<string, unknown>;
try {
- response = await submit(sourceId ?? 'default', requestId);
+ response = await put(sourceId ?? 'default', requestId);
} catch (err) {
// The door refuses a source that vanished or was archived after the pre-check (the managed form of the FK retry).
const vanished = sourceId !== undefined && err instanceof OperationError && err.code === 'source_changed'
@@ -360,9 +349,8 @@ export function makeIngestCaptureHandler(engine: BrainEngine) {
if (!vanished || (vanished.length > 0 && vanished[0]?.archived !== true)) throw err;
sourceFallback = { requested: sourceId!, effective: 'default', reason: vanished.length ? 'archived' : 'fk_violation' };
await checkWriter('default');
- response = await submit('default', requestId && `${requestId}:default`);
+ response = await put('default', requestId && `${requestId}:default`);
}
- if (appliedSince) return appliedSince;
if (sourceFallback) {
console.error(
`[WARN] ingest_capture: requested source '${sourceFallback.requested}' unavailable ` +
diff --git a/test/ingest-capture-applied-once.test.ts b/test/ingest-capture-applied-once.test.ts
index 8fca528871..a8fdaa4f55 100644
--- a/test/ingest-capture-applied-once.test.ts
+++ b/test/ingest-capture-applied-once.test.ts
@@ -29,6 +29,7 @@ import { MIGRATIONS, runMigrationSQL } from '../src/core/migrate.ts';
import { makeIngestCaptureHandler } from '../src/core/minions/handlers/ingest-capture.ts';
import type { MinionJobContext } from '../src/core/minions/types.ts';
import { submitPageMutation } from '../src/core/persistence/page-mutations.ts';
+import { acquireWorktree, getWorktreeBinding } from '../src/core/persistence/ownership.ts';
import { managedBrain } from './helpers/managed-brain.ts';
import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';
import { requirePostgresTestDatabase, testBackends } from './helpers/test-backends.ts';
@@ -75,6 +76,11 @@ afterAll(async () => { for (const brain of brains.values()) await brain.close();
const jobCtx = (id: number, data: Record<string, unknown>, key: string | null, attempts = 0): MinionJobContext => ({ id, name: 'ingest_capture', data,
attempts_made: attempts, idempotency_key: key, signal: new AbortController().signal, deadlineAtMs: null, shutdownSignal: new AbortController().signal,
updateProgress: async () => {}, updateTokens: async () => {}, log: async () => {}, isActive: async () => true, readInbox: async () => [] });
+/** Runs `sql` as a protocol-2 writer, which the writer fence requires of direct writes to journal and page rows. */
+const asWriter = (engine: BrainEngine, sql: string, params: unknown[]) => engine.transaction(async tx => {
+ await tx.executeRaw("SELECT set_config('gbrain.persistence_protocol', '2', true)");
+ await tx.executeRaw(sql, params);
+});
const jsonData = (data: unknown) => (typeof data === 'string' ? JSON.parse(data) : data) as Record<string, unknown>;
const note = (title: string, line: string) => `---\ntype: note\ntitle: ${title}\n---\n# ${title}\n\n${line}\n`;
/** The data a webhook capture job carries for `content` sent to `slug` in `sourceId`. */
@@ -295,5 +301,42 @@ for (const backend of backends) {
expect(await pageText()).toContain('Managed retried body.');
expect(await run(jobCtx(5, data, key))).toEqual(result);
}, { databaseUrl }), 120_000);
+
+ test('a retried capture whose attempt id names a failed write of the same request still lands', () => managedBrain(async ({ engine, ctx }) => {
+ const slug = 'notes/managed-failed';
+ const data = captureData(note('Managed failed', 'Managed body after a failed write.'), slug, 'default');
+ const key = 'ingest:webhook:client-example:default:managed-failed';
+ const pageText = async () => (await engine.readPageSnapshot(slug, { sourceId: 'default' }))?.page.compiled_truth ?? '';
+ // Attempt 0 sent the capture's request (the page did not exist yet) and the write ended failed without landing.
+ const lock = (await acquireWorktree((await getWorktreeBinding(engine, 'default'))!))!;
+ try {
+ await expect(submitPageMutation({ ...ctx, writeWaitMs: 200 }, { operation: 'put_page', params: { slug, source_id: 'default', content: data.event.content,
+ content_origin: 'tool_output', source_kind: 'webhook', source_uri: data.event.source_uri, ingested_via: 'ingest_capture',
+ request_id: `${appliedCaptureRequestPrefix(key)}0`, keep_hidden_rows: true } })).rejects.toThrow(/pending/);
+ await asWriter(engine, "UPDATE persistence_requests SET state = 'failed', error_code = 'storage_error' WHERE client_request_id = $1", [`${appliedCaptureRequestPrefix(key)}0`]);
+ } finally { await lock.release(); }
+ expect(await pageText()).toBe('');
+
+ // `gbrain jobs retry` reset attempts_made to 0, so the run sends the same request again.
+ const result = await makeIngestCaptureHandler(engine)(jobCtx(6, data, key));
+ expect(result.status).toBe('imported');
+ expect(await pageText()).toContain('Managed body after a failed write.');
+ }, { databaseUrl }), 120_000);
+
+ test('a committed request another principal sent under the capture prefix does not count as applied', () => managedBrain(async ({ engine, ctx }) => {
+ const slug = 'notes/managed-forged';
+ const data = captureData(note('Managed forged', 'Managed real capture.'), slug, 'default');
+ const key = 'ingest:webhook:client-example:default:managed-forged';
+ const prefix = appliedCaptureRequestPrefix(key);
+ const pageText = async () => (await engine.readPageSnapshot(slug, { sourceId: 'default' }))?.page.compiled_truth ?? '';
+ // A remote writer that predicted the key committed its own write under the capture's request_id prefix.
+ await submitPageMutation(ctx, { operation: 'put_page', params: { slug, content: note('Managed forged', 'Forged write.'), request_id: `${prefix}0:forged` } });
+ await asWriter(engine, "UPDATE persistence_requests SET principal_kind = 'oauth_client', principal_id = 'client-forger' WHERE client_request_id = $1", [`${prefix}0:forged`]);
+
+ expect((await makeIngestCaptureHandler(engine)(jobCtx(7, data, key))).status).toBe('imported');
+ expect(await pageText()).toContain('Managed real capture.');
+ const own = await engine.executeRaw<{ principal_kind: string }>("SELECT principal_kind FROM persistence_requests WHERE client_request_id LIKE $1 AND client_request_id <> $2", [`${prefix}%`, `${prefix}0:forged`]);
+ expect(own.map(r => r.principal_kind)).toEqual(['local_cli']);
+ }, { databaseUrl }), 120_000);
});
}