Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Stop a retried /ingest capture from reverting a newer edit after upgrading

garrytan338895ef993d
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5017d333fe..8cfed5e71a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,15 @@ credits are retained; no result has been reassigned to another provider. Origina
identifiers and attribution are available in the pre-removal Git revision
`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).
+## [Unreleased]
+
+**A client that retries a `POST /ingest` capture accepted before v0.60.609.0 no longer queues a second job that writes the old body over the page's newer edit.** v0.60.609.0 added the target slug to the capture's queue key, and nothing looked up the old key form, so a retry of a capture that had already landed was taken as a new one. The route now also looks up a capture's pre-v0.60.609.0 key and answers 202 with that job when it was queued for the same target slug, so the retry is the same request again and nothing is queued.
+
+### Itemized changes
+
+- `src/commands/serve-http-webhooks.ts` (`POST /ingest`): before queueing, the route looks up the capture's legacy key (client, write source and content hash, no slug part). A row queued for the same normalized target slug (or with no slug, for a slug-less capture) answers the request; a row for another slug never does, so one body sent to two slugs still queues two captures. A dead or cancelled legacy row frees its key, as under the current key. The 429 at the waiting cap, the 400 for slugs over 1024 bytes and the webhook HMAC are unchanged.
+- Tests: `test/serve-http-ingest-legacy-key-retry.test.ts` (PGLite) and `test/e2e/serve-http-ingest-legacy-key-retry-postgres.test.ts` (Postgres) drive the real route: a retry of a completed legacy capture gets its job back, queues nothing and leaves the newer edit in place.
+
## [0.60.609.0] - 2026-11-07
**`POST /ingest` no longer answers 202 for a capture it drops, and an unsigned GitHub webhook can no longer tell from response time which repositories a brain syncs.** The same body sent to two slugs now queues two captures, where the second caller used to get the first job back and nothing written to its slug. A capture over the 50-capture waiting cap now gets 429 `ingest_backlog_full` with `Retry-After: 30` and nothing queued, where it used to get another capture's job id. A slug over 1024 bytes gets 400 `invalid_slug` before anything is queued, well short of the roughly 2.7 KB where no page can be stored under it. Every webhook request now computes the HMAC over its body, whether the repository is unknown, has no secret or is covered by a source, so a 401 takes the same work either way.
diff --git a/src/commands/serve-http-webhooks.ts b/src/commands/serve-http-webhooks.ts
index 93cb5b103d..18798a715d 100644
--- a/src/commands/serve-http-webhooks.ts
+++ b/src/commands/serve-http-webhooks.ts
@@ -15,6 +15,9 @@ import type { BrainEngine } from '../core/engine.ts';
import type { AuthInfo } from '../core/operations.ts';
import { executeRawJsonb } from '../core/sql-query.ts';
import { MinionQueue } from '../core/minions/queue.ts';
+import { coalesceOnIdempotencyKey } from '../core/minions/idempotency-coalesce.ts';
+import { APPLICATION_AUTHORITY } from '../core/minions/submission-authority.ts';
+import type { MinionJob } from '../core/minions/types.ts';
import { OperationError } from '../core/ops/contract.ts';
import { ERROR_CATALOGUE } from '../core/error-catalogue.ts';
import {
@@ -254,6 +257,30 @@ function ingestTargetSlug(callerSlug: string): string {
return validateSlug(callerSlug);
}
+/**
+ * The job a server before v0.60.609.0 recorded for this capture, or null. Its key had no slug part
+ * (`ingest:webhook:<client>:<write source>:<content hash>`), and a job row keeps its key until `gbrain jobs prune`
+ * removes it, which nothing runs on a schedule, so such a row can answer a retry for as long as the brain keeps it. A
+ * retry that missed it queued a second job, and that job wrote the old body over any edit made since. The row answers
+ * only for the target slug it was queued with: the old key collapsed one body sent to two slugs, which is the dropped
+ * write #8769 fixed. A dead or cancelled row frees its key and answers nothing, as under the current key.
+ */
+async function legacyIngestJob(engine: BrainEngine, clientId: string, writeSourceId: string, contentHash: string, targetSlug: string): Promise<MinionJob | null> {
+ const key = `ingest:webhook:${clientId}:${writeSourceId}:${contentHash}`;
+ return engine.transaction(async (tx) => {
+ const [row] = await tx.executeRaw<{ data: unknown }>('SELECT data FROM minion_jobs WHERE idempotency_key = $1', [key]);
+ if (!row) return null;
+ const data = (typeof row.data === 'string' ? JSON.parse(row.data) : row.data) as { slug?: unknown } | null;
+ let rowSlug = '';
+ try {
+ if (typeof data?.slug === 'string' && data.slug) rowSlug = ingestTargetSlug(data.slug);
+ } catch {
+ return null;
+ }
+ return rowSlug === targetSlug ? coalesceOnIdempotencyKey(tx, key, APPLICATION_AUTHORITY) : null;
+ });
+}
+
/** Stands in for a missing webhook secret; no source holds it and a match against it is still refused. */
const THROWAWAY_WEBHOOK_KEY = randomBytes(32);
@@ -530,7 +557,8 @@ async function handleIngest(
const slugKey = targetSlug ? createHash('sha256').update(targetSlug).digest('hex') : '';
const idempotencyKey = `ingest:webhook:${authInfo.clientId}:${writeSourceId}:${slugKey}:${contentHash}`;
try {
- const job = await ingestQueue.add(
+ const legacyJob = await legacyIngestJob(engine, authInfo.clientId, writeSourceId, contentHash, targetSlug);
+ const job = legacyJob ?? await ingestQueue.add(
'ingest_capture',
{
event,
@@ -554,7 +582,7 @@ async function handleIngest(
);
// At the cap the queue hands back the newest waiting job instead of inserting this capture. Answering 202 with
// that job id would drop this body without a trace, so the caller is told to retry once the backlog drains.
- if (job.idempotency_key !== idempotencyKey) {
+ if (!legacyJob && job.idempotency_key !== idempotencyKey) {
res.setHeader('Retry-After', '30');
res.status(429).json({
error: 'ingest_backlog_full',
diff --git a/test/e2e/serve-http-ingest-legacy-key-retry-postgres.test.ts b/test/e2e/serve-http-ingest-legacy-key-retry-postgres.test.ts
new file mode 100644
index 0000000000..b7394e7aa8
--- /dev/null
+++ b/test/e2e/serve-http-ingest-legacy-key-retry-postgres.test.ts
@@ -0,0 +1,3 @@
+import { registerPostgresTests } from '../helpers/test-backends.ts';
+
+await registerPostgresTests(() => import('../serve-http-ingest-legacy-key-retry.test.ts'));
diff --git a/test/serve-http-ingest-legacy-key-retry.test.ts b/test/serve-http-ingest-legacy-key-retry.test.ts
new file mode 100644
index 0000000000..0f37c1193e
--- /dev/null
+++ b/test/serve-http-ingest-legacy-key-retry.test.ts
@@ -0,0 +1,162 @@
+/**
+ * A retry of a capture recorded under the /ingest key from before v0.60.609.0 must not queue a second job.
+ *
+ * #8798 (v0.60.609.0) added the target slug to the queue idempotency key (#8769). Job rows keep their key until
+ * `gbrain jobs prune` removes them, so a capture that completed under the old key form
+ * (`ingest:webhook:<client>:<write source>:<content hash>`) missed the new key when its client retried after the
+ * upgrade: a second job was queued, and running it wrote the old body over the page's newer edit.
+ * Protects: the retry (to the same slug in any case, or slug-less) answers 202 with the recorded job and queues nothing,
+ * and the page keeps its newer edit after every queued capture runs; a legacy row for another slug or a dead legacy row
+ * never answers for this capture, so #8769's fix and the dead-job retry still hold.
+ * Real token verifier, real Express route, the real ingest_capture handler; PGLite here, Postgres through
+ * test/e2e/serve-http-ingest-legacy-key-retry-postgres.test.ts.
+ */
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import express from 'express';
+import type { Server } from 'node:http';
+import type { AddressInfo } from 'node:net';
+import type { BrainEngine } from '../src/core/engine.ts';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { mintLegacyToken } from '../src/core/token-mint.ts';
+import { mountWebhooks } from '../src/commands/serve-http-webhooks.ts';
+import type { ServeHttpContext } from '../src/commands/serve-http.ts';
+import { GBrainOAuthProvider } from '../src/core/oauth-provider.ts';
+import { sqlQueryForEngine } from '../src/core/sql-query.ts';
+import { importFromContent } from '../src/core/import-file.ts';
+import { makeIngestCaptureHandler } from '../src/core/minions/handlers/ingest-capture.ts';
+import type { MinionJobContext } from '../src/core/minions/types.ts';
+import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';
+import { testBackends } from './helpers/test-backends.ts';
+
+const SRC = 'source-a';
+
+interface Brain { engine: BrainEngine; server: Server; base: string; token: string; close: () => Promise<void> }
+const brains = new Map<string, Brain>();
+
+async function openBrain(engine: BrainEngine, closeEngine: () => Promise<void>): Promise<Brain> {
+ await engine.executeRaw('INSERT INTO sources (id, name) VALUES ($1, $1) ON CONFLICT (id) DO NOTHING', [SRC]);
+ const token = (await mintLegacyToken(engine, { name: 'legacy-retry-writer', scopes: ['read', 'write'], takesHolders: ['world'], sourceGrant: [SRC] })).token;
+ const provider = new GBrainOAuthProvider({ sql: sqlQueryForEngine(engine), transaction: fn => engine.transaction(tx => fn(sqlQueryForEngine(tx))) });
+ const app = express();
+ const pass: express.RequestHandler = (_req, _res, next) => next();
+ mountWebhooks(app, {
+ engine,
+ resourceVerifier: provider,
+ resourceMetadataUrl: 'http://127.0.0.1/.well-known/oauth-protected-resource',
+ ingestRateLimiter: pass,
+ githubWebhookLimiter: pass,
+ broadcastEvent: () => {},
+ } as unknown as ServeHttpContext);
+ const server = await new Promise<Server>(resolve => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });
+ const base = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
+ return { engine, server, base, token, close: async () => { await new Promise<void>(resolve => server.close(() => resolve())); await closeEngine(); } };
+}
+
+const backends = testBackends();
+beforeAll(async () => {
+ if (backends.includes('pglite')) {
+ const pglite = new PGLiteEngine();
+ await pglite.connect({});
+ await pglite.initSchema();
+ brains.set('pglite', await openBrain(pglite, () => pglite.disconnect()));
+ }
+ if (backends.includes('postgres')) {
+ const pg = await isolatedPersistencePostgres(process.env.DATABASE_URL!);
+ brains.set('postgres', await openBrain(pg.engine, pg.close));
+ }
+}, 120_000);
+afterAll(async () => { for (const brain of brains.values()) await brain.close(); }, 60_000);
+
+const jobCtx = (id: number, data: Record<string, unknown>): MinionJobContext => ({ id, name: 'ingest_capture', data, attempts_made: 0,
+ signal: new AbortController().signal, deadlineAtMs: null, shutdownSignal: new AbortController().signal, updateProgress: async () => {},
+ updateTokens: async () => {}, log: async () => {}, isActive: async () => true, readInbox: async () => [] });
+const jsonData = (data: unknown) => (typeof data === 'string' ? JSON.parse(data) : data) as Record<string, unknown>;
+const note = (title: string, line: string) => `---\ntype: note\ntitle: ${title}\n---\n# ${title}\n\n${line}\n`;
+
+for (const backend of backends) {
+ describe(`POST /ingest retry of a capture recorded under the pre-v0.60.609.0 key (${backend})`, () => {
+ const brain = () => brains.get(backend)!;
+ const post = (body: string, slug?: string) => fetch(`${brain().base}/ingest`, {
+ method: 'POST',
+ headers: { authorization: `Bearer ${brain().token}`, 'content-type': 'text/markdown', ...(slug ? { 'x-gbrain-slug': slug } : {}) },
+ body,
+ });
+ const ingest = async (body: string, slug?: string) => {
+ const res = await post(body, slug);
+ expect(res.status).toBe(202);
+ return (await res.json() as { job_id: number }).job_id;
+ };
+ const jobData = async (id: number) =>
+ jsonData((await brain().engine.executeRaw<{ data: unknown }>('SELECT data FROM minion_jobs WHERE id = $1', [id]))[0]!.data);
+ const runJob = async (id: number) => makeIngestCaptureHandler(brain().engine)(jobCtx(id, await jobData(id)));
+ /** Runs every queued capture, as a worker would after the retry. */
+ const drainQueue = async () => {
+ const waiting = await brain().engine.executeRaw<{ id: number }>(
+ "SELECT id FROM minion_jobs WHERE name = 'ingest_capture' AND status = 'waiting' ORDER BY id");
+ for (const { id } of waiting) {
+ await runJob(Number(id));
+ await brain().engine.executeRaw("UPDATE minion_jobs SET status = 'completed', finished_at = now() WHERE id = $1", [id]);
+ }
+ };
+ /** Rewrites a job to how a pre-v0.60.609.0 server recorded it: the key with no slug part, in the given status. */
+ const recordUnderLegacyKey = async (id: number, status: string) => {
+ const data = await jobData(id);
+ const event = data.event as { content_hash: string; metadata: { client_id: string } };
+ const legacyKey = `ingest:webhook:${event.metadata.client_id}:${data.sourceId}:${event.content_hash}`;
+ await brain().engine.executeRaw('UPDATE minion_jobs SET idempotency_key = $2, status = $3, finished_at = now() WHERE id = $1', [id, legacyKey, status]);
+ };
+ const captureJobs = async (marker: string) =>
+ (await brain().engine.executeRaw<{ n: number }>(
+ "SELECT count(*)::int AS n FROM minion_jobs WHERE name = 'ingest_capture' AND data::text LIKE $1", [`%${marker}%`]))[0]!.n;
+ const pageText = async (slug: string) => (await brain().engine.readPageSnapshot(slug, { sourceId: SRC }))?.page.compiled_truth ?? '';
+
+ test('a retry to the same slug gets the recorded job, queues nothing, and the newer edit survives', async () => {
+ const slug = `notes/legacy-retry-${backend}`;
+ const body = note('Legacy capture', 'First body from the integration.');
+ const first = await ingest(body, slug);
+ expect((await runJob(first)).status).toBe('imported');
+ await recordUnderLegacyKey(first, 'completed');
+
+ await importFromContent(brain().engine, slug, note('Legacy capture', 'Newer edit made after the capture.'), { sourceId: SRC });
+ expect(await pageText(slug)).toContain('Newer edit made after the capture.');
+
+ expect(await ingest(body, slug)).toBe(first);
+ expect(await ingest(body, slug.toUpperCase())).toBe(first);
+ expect(await captureJobs('First body from the integration.')).toBe(1);
+
+ await drainQueue();
+ expect(await pageText(slug)).toContain('Newer edit made after the capture.');
+ expect(await pageText(slug)).not.toContain('First body from the integration.');
+ });
+
+ test('a slug-less retry gets the recorded slug-less job', async () => {
+ const body = note('Legacy inbox capture', `Inbox body ${backend}.`);
+ const first = await ingest(body);
+ await recordUnderLegacyKey(first, 'completed');
+ expect(await ingest(body)).toBe(first);
+ expect(await captureJobs(`Inbox body ${backend}.`)).toBe(1);
+ });
+
+ test('a legacy row for another slug never answers for this one (#8769 stays fixed)', async () => {
+ const body = note('Shared legacy body', `Shared body ${backend}.`);
+ const first = await ingest(body, `notes/shared-one-${backend}`);
+ await recordUnderLegacyKey(first, 'completed');
+ const second = await ingest(body, `notes/shared-two-${backend}`);
+ expect(second).not.toBe(first);
+ expect((await runJob(second)).status).toBe('imported');
+ expect(await pageText(`notes/shared-two-${backend}`)).toContain(`Shared body ${backend}.`);
+ expect(await ingest(body)).not.toBe(first);
+ });
+
+ test('a dead legacy row frees the capture for a fresh job', async () => {
+ const slug = `notes/legacy-dead-${backend}`;
+ const body = note('Dead legacy capture', `Dead body ${backend}.`);
+ const first = await ingest(body, slug);
+ await recordUnderLegacyKey(first, 'dead');
+ const retried = await ingest(body, slug);
+ expect(retried).not.toBe(first);
+ expect((await runJob(retried)).status).toBe('imported');
+ expect(await pageText(slug)).toContain(`Dead body ${backend}.`);
+ });
+ });
+}