Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(serve-http): /ingest never 202s a dropped capture; GitHub webhook HMACs every body
garrytan5de00368244c
diff --git a/docs/architecture/key-files/commands-5.md b/docs/architecture/key-files/commands-5.mdindex 917c34bc35..fd19751dda 100644--- a/docs/architecture/key-files/commands-5.md+++ b/docs/architecture/key-files/commands-5.md@@ -21,8 +21,8 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/commands/serve-http-grants.ts` + `admin/src/components/ClientGrant.tsx` — typed admin grant request validation and shared profile/advanced permission editor. Register/rescope/TTL routes in `serve-http-registration.ts`, `serve-http-grants.ts`, and `serve-http-admin-api.ts` use canonical grant validation and audit; legacy response shapes remain available. Registration and permission-edit failures add safe classifications, stage, outcome and inspection remedies; an unacknowledged mutation stays outcome-unknown, while read/preview failures and known refusals remain failed. New UI creation defaults to memory-writer and requires a preview, while edits carry the displayed revision and require a fresh preview after any field change or conflict. Detail/catalog endpoints expose non-secret grants and the shared harness registry. Operation authority snapshots, source sets, delegated tools/namespaces, spend, concurrency, and future TTL are independently reviewable; profile regrant is explicit. `admin/src/pages/Agents.tsx` previews machine/public PKCE/confidential PKCE registration and current permissions, and exposes versioned lifecycle actions. Explicit downloads select machine `HarnessCredentials` or native `OAuthClientSetup`; mixed clients require a flow choice. Loading errors preserve stale rows with Retry and disable creation when required source metadata is missing. Browser downloads and native connection evidence remain distinct. Rebuild committed `admin/dist` and `src/admin-embedded.ts` after SPA edits.-- `src/commands/serve-http.ts` — Express 5 HTTP MCP server with OAuth 2.1, admin dashboard, and SSE live activity feed. The legacy `POST /ingest` queue refuses operation-snapshot clients; they use approved MCP writes so execution stays inside the shared grant contract. Started via `gbrain serve --http [--port N] [--token-ttl N] [--enable-dcr] [--public-url URL] [--bind HOST] [--log-full-params]`. OAuth discovery's `scopes_supported` (RFC 8414 AS metadata and both RFC 9728 protected-resource documents, one value through the SDK router) comes from `scopesSupportedForDiscovery({ enableDcr })` in `src/core/scope.ts`: with `--enable-dcr` it is exactly the self-registration ceiling (`read write`), so a client that copies discovery into its `/register` request succeeds; without DCR it is every scope except operator-only `agent`. The DCR ceiling still rejects an explicit privileged request, and operator-registered clients are never narrowed by the advertisement. Combines MCP SDK's `mcpAuthRouter` (authorize/token/register/revoke), a custom `client_credentials` handler running BEFORE the router (SDK's token endpoint throws `UnsupportedGrantTypeError` for CC; custom handler falls through for `auth_code` / `refresh_token`), `requireBearerAuth` middleware for `/mcp` with scope enforcement + `localOnly` rejection before op dispatch, and per-IP `express-rate-limit` buckets: the MCP SDK's defaults on `/authorize` (100 / 15 min), `/register` (20 / hour) and `/token` (50 / 15 min) because `authRouterOptions` passes no `rateLimit: false`, the custom `client_credentials` handler's `ccRateLimiter`, and independent owner limits: 60 total authentication requests/min/IP, 10 failed authentications/min/IP (verified authority, independent of response status), and 60 consent requests/min/authenticated session in `serve-http-admin-limits.ts`. Admin no-store, no-referrer and framing headers precede login/link handlers. Explicit pending IDs survive fresh-browser links; invalid or restarted requests require native authorization to restart. Serves the built admin SPA from `admin/dist/` with SPA fallback. `/admin/events` SSE broadcasts every MCP request. `cookie-parser` wired (Express 5 has no built-in). Startup logging prints port, engine, issuer URL (honors `--public-url`), client count, DCR status, and the admin bootstrap token line — but the generated token's raw value only prints when stderr is an interactive TTY (`shouldSuppressBootstrapPrint`): a non-TTY (containerized/piped) start hides it so the secret never lands in centralized log storage, env-sourced tokens (`$GBRAIN_ADMIN_BOOTSTRAP_TOKEN`) are always hidden, `--print-admin-token` forces the raw value on a trusted terminal, and `--suppress-bootstrap-token` hides everything. The `/mcp` request handler's OperationContext literal sets `remote: true` explicitly (without it `submit_job`'s protected-name guard would see a falsy undefined and a `read+write`-scoped OAuth token could submit `shell` jobs). `summarizeMcpParams` from `src/mcp/dispatch.ts` feeds both `mcp_request_log` writes and the SSE feed by default (raw via `--log-full-params`). Cookie `Secure` flag set behind HTTPS or a public-URL proxy; magic-link nonce store LRU-bounded; DCR disable routes through the `GBrainOAuthProvider` `dcrDisabled` constructor option (not a router monkey-patch); `transport.handleRequest` wrapped in try/catch to return a JSON-RPC 500 envelope; OperationError + unexpected exceptions unified through `buildError` / `serializeError` so `/mcp` always returns the same envelope. `/health` is liveness-only via `probeLiveness(engine, engineName, version, timeoutMs)` racing `engine.executeRaw('SELECT 1', undefined, { signal })` against the exported `HEALTH_TIMEOUT_MS = 3000`; when the timeout wins, an `AbortController` cancels the Postgres query (PGLite can only discard its eventual result) before the same tagged `ProbeHealthResult` 503 envelope is returned (single timer-cleanup site); body shape `{status, version, engine}` only. Full stats live at admin-only `/admin/api/full-stats` (gated by `requireAdmin`, calls `probeHealth(engine, ...)`) — keeps `getStats()`'s 6× count(*) off the public route so a saturated pool doesn't trigger orchestrator restart cascades. OAuth/admin/audit SQL outside the abortable liveness probe routes through `sqlQueryForEngine(engine)` from `src/core/sql-query.ts` so it works against PGLite; the four `mcp_request_log.params` INSERT sites (success / auth_failed / scope_denied / server-error) go through `executeRawJsonb(engine, ...)` so the column stores real objects (`params->>'op'` returns `search`, not the quoted string). `--bind HOST` defaults `127.0.0.1` (self-hosters pass `--bind 0.0.0.0`); a stderr WARN fires when `--public-url` is set without `--bind`; the banner prints a `Bind:` line. `AuthInfo.sourceId` + `AuthInfo.allowedSources` + `AuthInfo.takesHoldersAllowList` are the typed source of truth, populated by `oauth-provider.ts:verifyAccessToken` (source scope from the `oauth_clients` row; takes-holders from `access_tokens.permissions.takes_holders` for legacy bearer tokens). The `/mcp` dispatch site reads `authInfo.takesHoldersAllowList ?? ['world']` — absent grants (OAuth-client tokens, pre-v29 brains) fail closed to world-only takes visibility, while an explicit `[]` grant is preserved as deny-all; pinned end-to-end by `test/e2e/serve-http-takes-holders.test.ts`. The HTTP MCP `tools/list` handler at `:837-849` uses `paramDefToSchema(v)` from `src/mcp/tool-defs.ts` so array params keep `items` (strict-mode OAuth clients otherwise reject the whole tool list). `POST /ingest` enforces the slug-prefix write fence at the ROUTE, not the op layer: the route hands its payload to the `ingest_capture` minion handler, which deliberately bypasses `put_page`, so no `OperationContext` exists and `enforceClientSlugFence` never runs — a slug-bound client must therefore supply `X-Gbrain-Slug` and it must satisfy `slugUnderBoundPrefixes`, else 403 (without the check a bound client could overwrite any page inside its granted source). The write source is resolved server-side as `authInfo.sourceId ?? 'default'` and travels on the job as `job.data.sourceId`; the caller-supplied `X-Gbrain-Source-Id` header routes nothing and only names the emitter (`webhook-<clientId>`), which is what the event's `source_id` and the 202's back-compat `source_id` field carry. The 202 additionally reports the routed destination as `write_source_id`, and the resolved write source joins the queue idempotency key so a rescoped client lands a new capture instead of deduping against its old job. confidential revoke: a pre-router `/revoke` handler validates the RFC 7009 body, verifies hash-only secrets for both `client_secret_post` and `client_secret_basic`, rejects mixed authentication, preserves the SDK path for public clients, and separates opaque client-auth failures from retryable/backend failures. OAuth metadata advertises both confidential methods. Pinned by `test/e2e/serve-http-oauth.test.ts`. three admin routes: `/admin/api/calibration/profile`, `/admin/api/calibration/charts/:type` (image/svg+xml; type in {brier-trend, domain-bars, pattern-statements, abandoned-threads}), `/admin/api/calibration/pattern/:id` (drill-down). The manual mint route `/admin/api/register-client` in `serve-http-registration.ts` accepts optional `source` + `federatedRead` bindings mirroring the CLI's `--source`/`--federated-read` (validated via `normalizeSourceInput`/`normalizeFederatedReadInput` from `src/core/source-id.ts`; omitting both preserves the default binding `source_id='default'` / `federated_read=[source_id]`, invalid values return a structured 400 `invalid_source`), so an admin SPA or provisioning proxy can mint a source-confined client over HTTP. The route mirrors the CLI lane end to end: structured 400s for `unknown_source` / `archived_source` (one batched `ANY($1::text[])` existence check), `invalid_token_ttl` (shared `TOKEN_TTL_MIN/MAX_SECONDS` bounds, integer-validated BEFORE the tx), and `brain_too_old` (column pre-flight OUTSIDE the tx — a pre-scoped-clients brain refuses up front instead of aborting mid-transaction); the duplicate-name pre-check + INSERT run in ONE transaction under the SAME name-scoped advisory lock the CLI takes (`registerClientNameLockKey`), returning 409 `duplicate_name` with the existing `client_id`; the INSERT composes `registerScopedClient` (the CLI's registration core) so the two paths cannot drift; any post-commit failure includes the created `client_id` so the operator can revoke (never a false "nothing was created"). Pinned by `test/register-client-source-normalize.test.ts`. The `/mcp` tools/list is the honest catalog: per-request filters (token scope incl. the `agentCallable` carve-out, bound-client fence via `opAllowedForBoundClient`, publish gates via `disabledOpsForPublishGates`) over the surface-filtered op set, schemas via `buildToolDefs` (strict emission when `mcp.strict_params` resolves reject); the tools/list `mcp_request_log` row records the listed size as `params.tool_count`. Call-time op-level denials the list should have prevented — the inline scope deny, publish-gate backstop (`config_key=...`), fence op-level deny (`fence=op`, classified via `isListLevelDenialEnvelope`) — log `status='denied_after_list'` instead of `'error'` (the trend-to-zero honest-catalog metric, see docs/operations/mcp-surface-runbook.md) — every tools/call row's status resolves through `requestLogStatusForResult` (dispatch.ts) on BOTH transports. The admin health-indicators error rate counts `status NOT IN ('success','success_with_warnings')` and excludes `operation='surface_change'` audit rows from numerator AND denominator (audit rows record operator/self actions, not traffic). Per-request surface resolution lives in `resolveEffectiveSurface`: a `verbs` ceiling skips the default-surface config read, and a failed read applies the last successfully read default per process (fail-closed — a transient config outage cannot widen a NULL-surface client). OAuth protected-resource metadata (RFC 9728) names `/mcp` as the resource (`resourceServerUrl`), so the SDK mounts the PRM at `/.well-known/oauth-protected-resource/mcp` and the 401 `resource_metadata` URL is derived from the same value; the bare root path is kept as an alias by rewriting `req.url` onto the SDK's handler (same document, same CORS). Shutdown: `waitForHttpServerLifecycle` tracks accepted sockets weakly (`trackServerSockets`; Bun never emits socket `close`, so a strong set leaked one per request), severs the still-reachable ones right after `server.close()`, and bounds the close wait at `CLOSE_TIMEOUT_MS` (5s) — a collected keep-alive wrapper whose native handle keeps `close()` from returning gets one stderr line and shutdown proceeds instead of hanging. DCR startup copy: the SECURITY WARNING states self-registered clients are limited to read/write (read-only for `client_credentials` under `--enable-dcr-insecure`) and that every authorization_code connection needs owner approval in the admin UI; banner `DCR:` values are `enabled (consent; DCR max read write)` / `enabled (INSECURE: read-only M2M)` / `disabled`.- Layout: `runServeHttp` binds through `listenOrRefuse` (`serve-http-listen.ts`: `serve_port_in_use` on a taken port; with `options.adoptServer` from status-only recovery it hands the app to the already-bound listener instead of binding). `buildServeHttpApp` builds one shared `ServeHttpContext` (sessions, nonce stores, limiters, `requireAdmin`, SSE fan-out, OAuth provider, resource verifier, non-`localOnly` op list) and mounts in registration order `serve-http-oauth.ts` (CORS gates, DCR TTL hook, confidential credentials, SDK router, discovery, owner login links, consent), `serve-http-metrics.ts` (`/health`, `/metrics`, probes), `serve-http-admin-api.ts` (`requireAdmin`, `/admin/login`, `/admin/api/*`, `/admin/events`), `serve-http-spa.ts`, `serve-http-mcp.ts` (`/mcp`; writes `[gbrain-serve] dispatch op=<name> args_sha256=<16 hex>` to stderr before each tool call so a stall's last line names its request, never the arguments) and `serve-http-webhooks.ts` (`/ingest`, which maps a legacy-authority `permission_denied` from submission to 409 `{error, message, hint, docs_url}` (`test/serve-http-ingest-legacy.test.ts`) and every other submission error to 500, `/webhooks/github`); `runServeHttp` keeps listen, banner, resolve-IPC binding and shutdown, and the façade re-exports moved symbols. Order is pinned by `test/serve-http-route-golden.test.ts` (AST) and `test/serve-http-route-runtime-golden.test.ts` (router stack). Size ceiling: `scripts/module-size-limits.tsv`.+- `src/commands/serve-http.ts` — Express 5 HTTP MCP server with OAuth 2.1, admin dashboard, and SSE live activity feed. The legacy `POST /ingest` queue refuses operation-snapshot clients; they use approved MCP writes so execution stays inside the shared grant contract. Started via `gbrain serve --http [--port N] [--token-ttl N] [--enable-dcr] [--public-url URL] [--bind HOST] [--log-full-params]`. OAuth discovery's `scopes_supported` (RFC 8414 AS metadata and both RFC 9728 protected-resource documents, one value through the SDK router) comes from `scopesSupportedForDiscovery({ enableDcr })` in `src/core/scope.ts`: with `--enable-dcr` it is exactly the self-registration ceiling (`read write`), so a client that copies discovery into its `/register` request succeeds; without DCR it is every scope except operator-only `agent`. The DCR ceiling still rejects an explicit privileged request, and operator-registered clients are never narrowed by the advertisement. Combines MCP SDK's `mcpAuthRouter` (authorize/token/register/revoke), a custom `client_credentials` handler running BEFORE the router (SDK's token endpoint throws `UnsupportedGrantTypeError` for CC; custom handler falls through for `auth_code` / `refresh_token`), `requireBearerAuth` middleware for `/mcp` with scope enforcement + `localOnly` rejection before op dispatch, and per-IP `express-rate-limit` buckets: the MCP SDK's defaults on `/authorize` (100 / 15 min), `/register` (20 / hour) and `/token` (50 / 15 min) because `authRouterOptions` passes no `rateLimit: false`, the custom `client_credentials` handler's `ccRateLimiter`, and independent owner limits: 60 total authentication requests/min/IP, 10 failed authentications/min/IP (verified authority, independent of response status), and 60 consent requests/min/authenticated session in `serve-http-admin-limits.ts`. Admin no-store, no-referrer and framing headers precede login/link handlers. Explicit pending IDs survive fresh-browser links; invalid or restarted requests require native authorization to restart. Serves the built admin SPA from `admin/dist/` with SPA fallback. `/admin/events` SSE broadcasts every MCP request. `cookie-parser` wired (Express 5 has no built-in). Startup logging prints port, engine, issuer URL (honors `--public-url`), client count, DCR status, and the admin bootstrap token line — but the generated token's raw value only prints when stderr is an interactive TTY (`shouldSuppressBootstrapPrint`): a non-TTY (containerized/piped) start hides it so the secret never lands in centralized log storage, env-sourced tokens (`$GBRAIN_ADMIN_BOOTSTRAP_TOKEN`) are always hidden, `--print-admin-token` forces the raw value on a trusted terminal, and `--suppress-bootstrap-token` hides everything. The `/mcp` request handler's OperationContext literal sets `remote: true` explicitly (without it `submit_job`'s protected-name guard would see a falsy undefined and a `read+write`-scoped OAuth token could submit `shell` jobs). `summarizeMcpParams` from `src/mcp/dispatch.ts` feeds both `mcp_request_log` writes and the SSE feed by default (raw via `--log-full-params`). Cookie `Secure` flag set behind HTTPS or a public-URL proxy; magic-link nonce store LRU-bounded; DCR disable routes through the `GBrainOAuthProvider` `dcrDisabled` constructor option (not a router monkey-patch); `transport.handleRequest` wrapped in try/catch to return a JSON-RPC 500 envelope; OperationError + unexpected exceptions unified through `buildError` / `serializeError` so `/mcp` always returns the same envelope. `/health` is liveness-only via `probeLiveness(engine, engineName, version, timeoutMs)` racing `engine.executeRaw('SELECT 1', undefined, { signal })` against the exported `HEALTH_TIMEOUT_MS = 3000`; when the timeout wins, an `AbortController` cancels the Postgres query (PGLite can only discard its eventual result) before the same tagged `ProbeHealthResult` 503 envelope is returned (single timer-cleanup site); body shape `{status, version, engine}` only. Full stats live at admin-only `/admin/api/full-stats` (gated by `requireAdmin`, calls `probeHealth(engine, ...)`) — keeps `getStats()`'s 6× count(*) off the public route so a saturated pool doesn't trigger orchestrator restart cascades. OAuth/admin/audit SQL outside the abortable liveness probe routes through `sqlQueryForEngine(engine)` from `src/core/sql-query.ts` so it works against PGLite; the four `mcp_request_log.params` INSERT sites (success / auth_failed / scope_denied / server-error) go through `executeRawJsonb(engine, ...)` so the column stores real objects (`params->>'op'` returns `search`, not the quoted string). `--bind HOST` defaults `127.0.0.1` (self-hosters pass `--bind 0.0.0.0`); a stderr WARN fires when `--public-url` is set without `--bind`; the banner prints a `Bind:` line. `AuthInfo.sourceId` + `AuthInfo.allowedSources` + `AuthInfo.takesHoldersAllowList` are the typed source of truth, populated by `oauth-provider.ts:verifyAccessToken` (source scope from the `oauth_clients` row; takes-holders from `access_tokens.permissions.takes_holders` for legacy bearer tokens). The `/mcp` dispatch site reads `authInfo.takesHoldersAllowList ?? ['world']` — absent grants (OAuth-client tokens, pre-v29 brains) fail closed to world-only takes visibility, while an explicit `[]` grant is preserved as deny-all; pinned end-to-end by `test/e2e/serve-http-takes-holders.test.ts`. The HTTP MCP `tools/list` handler at `:837-849` uses `paramDefToSchema(v)` from `src/mcp/tool-defs.ts` so array params keep `items` (strict-mode OAuth clients otherwise reject the whole tool list). `POST /ingest` enforces the slug-prefix write fence at the ROUTE, not the op layer: the route hands its payload to the `ingest_capture` minion handler, which deliberately bypasses `put_page`, so no `OperationContext` exists and `enforceClientSlugFence` never runs — a slug-bound client must therefore supply `X-Gbrain-Slug` and it must satisfy `slugUnderBoundPrefixes`, else 403 (without the check a bound client could overwrite any page inside its granted source). The write source is resolved server-side as `authInfo.sourceId ?? 'default'` and travels on the job as `job.data.sourceId`; the caller-supplied `X-Gbrain-Source-Id` header routes nothing and only names the emitter (`webhook-<clientId>`), which is what the event's `source_id` and the 202's back-compat `source_id` field carry. The 202 additionally reports the routed destination as `write_source_id`, and the write source and normalized target slug join the queue idempotency key, so a rescoped client or a second slug gets its own capture (`test/serve-http-ingest-write-loss.test.ts`). confidential revoke: a pre-router `/revoke` handler validates the RFC 7009 body, verifies hash-only secrets for both `client_secret_post` and `client_secret_basic`, rejects mixed authentication, preserves the SDK path for public clients, and separates opaque client-auth failures from retryable/backend failures. OAuth metadata advertises both confidential methods. Pinned by `test/e2e/serve-http-oauth.test.ts`. three admin routes: `/admin/api/calibration/profile`, `/admin/api/calibration/charts/:type` (image/svg+xml; type in {brier-trend, domain-bars, pattern-statements, abandoned-threads}), `/admin/api/calibration/pattern/:id` (drill-down). The manual mint route `/admin/api/register-client` in `serve-http-registration.ts` accepts optional `source` + `federatedRead` bindings mirroring the CLI's `--source`/`--federated-read` (validated via `normalizeSourceInput`/`normalizeFederatedReadInput` from `src/core/source-id.ts`; omitting both preserves the default binding `source_id='default'` / `federated_read=[source_id]`, invalid values return a structured 400 `invalid_source`), so an admin SPA or provisioning proxy can mint a source-confined client over HTTP. The route mirrors the CLI lane end to end: structured 400s for `unknown_source` / `archived_source` (one batched `ANY($1::text[])` existence check), `invalid_token_ttl` (shared `TOKEN_TTL_MIN/MAX_SECONDS` bounds, integer-validated BEFORE the tx), and `brain_too_old` (column pre-flight OUTSIDE the tx — a pre-scoped-clients brain refuses up front instead of aborting mid-transaction); the duplicate-name pre-check + INSERT run in ONE transaction under the SAME name-scoped advisory lock the CLI takes (`registerClientNameLockKey`), returning 409 `duplicate_name` with the existing `client_id`; the INSERT composes `registerScopedClient` (the CLI's registration core) so the two paths cannot drift; any post-commit failure includes the created `client_id` so the operator can revoke (never a false "nothing was created"). Pinned by `test/register-client-source-normalize.test.ts`. The `/mcp` tools/list is the honest catalog: per-request filters (token scope incl. the `agentCallable` carve-out, bound-client fence via `opAllowedForBoundClient`, publish gates via `disabledOpsForPublishGates`) over the surface-filtered op set, schemas via `buildToolDefs` (strict emission when `mcp.strict_params` resolves reject); the tools/list `mcp_request_log` row records the listed size as `params.tool_count`. Call-time op-level denials the list should have prevented — the inline scope deny, publish-gate backstop (`config_key=...`), fence op-level deny (`fence=op`, classified via `isListLevelDenialEnvelope`) — log `status='denied_after_list'` instead of `'error'` (the trend-to-zero honest-catalog metric, see docs/operations/mcp-surface-runbook.md) — every tools/call row's status resolves through `requestLogStatusForResult` (dispatch.ts) on BOTH transports. The admin health-indicators error rate counts `status NOT IN ('success','success_with_warnings')` and excludes `operation='surface_change'` audit rows from numerator AND denominator (audit rows record operator/self actions, not traffic). Per-request surface resolution lives in `resolveEffectiveSurface`: a `verbs` ceiling skips the default-surface config read, and a failed read applies the last successfully read default per process (fail-closed — a transient config outage cannot widen a NULL-surface client). OAuth protected-resource metadata (RFC 9728) names `/mcp` as the resource (`resourceServerUrl`), so the SDK mounts the PRM at `/.well-known/oauth-protected-resource/mcp` and the 401 `resource_metadata` URL is derived from the same value; the bare root path is kept as an alias by rewriting `req.url` onto the SDK's handler (same document, same CORS). Shutdown: `waitForHttpServerLifecycle` tracks accepted sockets weakly (`trackServerSockets`; Bun never emits socket `close`, so a strong set leaked one per request), severs the still-reachable ones right after `server.close()`, and bounds the close wait at `CLOSE_TIMEOUT_MS` (5s) — a collected keep-alive wrapper whose native handle keeps `close()` from returning gets one stderr line and shutdown proceeds instead of hanging. DCR startup copy: the SECURITY WARNING states self-registered clients are limited to read/write (read-only for `client_credentials` under `--enable-dcr-insecure`) and that every authorization_code connection needs owner approval in the admin UI; banner `DCR:` values are `enabled (consent; DCR max read write)` / `enabled (INSECURE: read-only M2M)` / `disabled`.+ Layout: `runServeHttp` binds through `listenOrRefuse` (`serve-http-listen.ts`: `serve_port_in_use` on a taken port; with `options.adoptServer` from status-only recovery it hands the app to the already-bound listener instead of binding). `buildServeHttpApp` builds one shared `ServeHttpContext` (sessions, nonce stores, limiters, `requireAdmin`, SSE fan-out, OAuth provider, resource verifier, non-`localOnly` op list) and mounts in registration order `serve-http-oauth.ts` (CORS gates, DCR TTL hook, confidential credentials, SDK router, discovery, owner login links, consent), `serve-http-metrics.ts` (`/health`, `/metrics`, probes), `serve-http-admin-api.ts` (`requireAdmin`, `/admin/login`, `/admin/api/*`, `/admin/events`), `serve-http-spa.ts`, `serve-http-mcp.ts` (`/mcp`; writes `[gbrain-serve] dispatch op=<name> args_sha256=<16 hex>` to stderr before each tool call so a stall's last line names its request, never the arguments) and `serve-http-webhooks.ts` (`/ingest`, which maps a legacy-authority `permission_denied` from submission to 409 `{error, message, hint, docs_url}` (`test/serve-http-ingest-legacy.test.ts`) answers 429 `ingest_backlog_full` when the 50-capture waiting cap hands back another job, and every other submission error to 500; `/webhooks/github`, which HMACs every body, with a throwaway key when there is no source or secret); `runServeHttp` keeps listen, banner, resolve-IPC binding and shutdown, and the façade re-exports moved symbols. Order is pinned by `test/serve-http-route-golden.test.ts` (AST) and `test/serve-http-route-runtime-golden.test.ts` (router stack). Size ceiling: `scripts/module-size-limits.tsv`.- `src/commands/serve.ts` — `gbrain serve` stdio MCP entrypoint with idempotent shutdown across every parent-disconnect signal. Surface: stdio resolves `GBRAIN_SURFACE` > `--surface` > config > full (`resolveStdioSurface`; an invalid env value is ignored with a stderr line and handed to `startMcpServer` as `invalidSurfaceEnv`), `--http` resolves `--surface` > config > full and says on stderr that it ignores `GBRAIN_SURFACE`; both print `surface=<x> (source: …)` at start. `--http --token-ttl` goes through `parseTokenTtl` (`src/commands/auth.ts`) and refuses anything outside 60 to 7,776,000 seconds before the server starts. Stdio EOF, SIGTERM, SIGINT, SIGHUP, and parent-process death (every reparent case — PID 1, launchd subreaper, systemd, tmux, or a parent shell with `PR_SET_CHILD_SUBREAPER`) all funnel into one `cleanup(reason)` that first awaits `shutdownDelegatedSync()` (idempotent shared promise; a running delegated sync is aborted and settles its checkpoint against the live engine, with the cleanup deadline extended by `GBRAIN_SERVE_SYNC_SETTLE_MS` only while a job runs) and then releases the engine and the PGLite write-lock dir within 5 seconds (otherwise the lock is held indefinitely after Claude Desktop / Cursor / launchd-managed gateways disconnect, forcing a 5-minute stale-lock wait on next start). Watchdog reparent check is `getParentPid() !== initialParentPid` (the `=== 1` check missed the subreaper case under launchd/systemd). Bun's `process.ppid` cache is stale across reparenting ([oven-sh/bun#30305](https://github.com/oven-sh/bun/issues/30305)) so `getParentPid()` runs `spawnSync('ps', ['-o', 'ppid=', '-p', PID])` per tick. Startup probe verifies `ps` is on PATH; if not (stripped containers, busybox), the watchdog skips installing AND emits a loud `[gbrain serve] watchdog disabled: ps unavailable ...` stderr line so operators see the degraded mode. The idle maintenance sweep skips ticks while a delegated sync runs and drains deferred embeds afterwards (`maybeDrainDeferredEmbeds`); it targets the source `resolveMcpStdioSourceScope` resolves (GBRAIN_SOURCE > `.gbrain-source` dotfile > local_path > `sources.default` > …) — the SAME ladder as stdio dispatch and the startup sweep, so dotfile/local_path-scoped serves reconcile their in-session MCP writes (pinned by `test/serve-idle-sweep-source.serial.test.ts`). #7599: `lean` (`readOnlyServeSkipsResidentDuties`, a read-only stdio serve on Postgres; PGLite keeps both) turns off the idle sweep and `installResidentTickers` (facts drain, movement watch, on-commit head watcher). Both stdin activity listeners (the default-on idle sweep and the opt-in `--stdio-idle-timeout`) attach only AFTER the MCP SDK transport's own stdin listener — the sweep on its first interval tick, the idle timeout through the activator `installStdioLifecycle` returns, which `runServe` invokes once `startMcpServer` resolves (and which re-arms the countdown so boot time does not eat into the idle window) — so neither can flip stdin into flowing mode and consume a fast client's initialize frame before the SDK sees it. The boot-readiness deadline (`GBRAIN_SERVE_BOOT_TIMEOUT_SECONDS`, default 60, 0 disables) measures time WITHOUT boot progress: every `onBootPhase` change and every `noteForwardProgress()` note (`src/core/forward-progress.ts`; progress ticks and each completed stdio request via `trackStdioRpc` in `src/mcp/server.ts`) restarts the window, so a large brain's slow but advancing boot is never killed; only a full window without progress releases the engine and exits 1 with the `serve_boot_timeout` line (stuck phase, pool pressure, seconds since boot started). Pinned by `test/serve-stdio-lifecycle.test.ts` (boot-readiness block). #6543: after the pool stall watch, `recordServeTransport` (`src/commands/serve-health.ts`) reads the read pool's transport once (live `prepare`/`max_pipeline` with `max_pipeline_reason`, the doctor probe's `SHOW statement_timeout` and its ALTER ROLE fix when a configured timeout reads 0, `application_name` when Supavisor), logs one `[gbrain serve] transport` line and caches it for `/health?deep=1` (never re-read per probe). `GBRAIN_SERVE_RESTART_REQUIRED_EXIT_SECONDS=<n>` (>= 120, else `invalid_params` at start) arms `installRestartRequiredExit`: an in-process `restart_required` held continuously for n seconds (the hold resets when it clears) disconnects under a 5 s deadline and exits `SERVE_UNHEALTHY_EXIT_CODE` (69, `serve_unhealthy_exit`); a durable ledger (`serve-self-exit.json` under the gbrain home) suppresses the exit after 3 within an hour (`serve_self_exit_suppressed`, `self_exit_suppressed` in the deep 503). `probeDeepHealth` (`serve-http-metrics.ts`) answers 503 `persistence_restart_required` before its pooled probe, `pool_inflight_over_budget` or `deep_health_probe_failed` (`reason`), each with `fix`/`docs` from the registry and the cached `transport`, no request ids. Pinned by `test/serve-health-6543.test.ts` (a real `PersistenceConsumer` driven to restart_required).diff --git a/src/commands/serve-http-webhooks.ts b/src/commands/serve-http-webhooks.tsindex 8cda00497b..a2af9754d3 100644--- a/src/commands/serve-http-webhooks.ts+++ b/src/commands/serve-http-webhooks.ts@@ -5,7 +5,7 @@*/import express, { type Express, type Request, type Response } from 'express';import rateLimit from 'express-rate-limit';-import { createHmac } from 'crypto';+import { createHmac, randomBytes } from 'crypto';import { readFileSync } from 'node:fs';import { join } from 'node:path';import { requireBearerAuth } from '@modelcontextprotocol/sdk/server/auth/middleware/bearerAuth.js';@@ -103,14 +103,20 @@ export function selectGitHubItemSources<Row extends { local_path: string | null;): { verified: Row[]; legacyMatched: boolean } {const verified: Row[] = [];let legacyMatched = false;+ let legacyRows = 0;+ // Every row is checked before scope decides anything, and a repo with no legacy row pays for one throwaway check, so+ // the work done for an unsigned request is the same whether a source covers the repository or not (#8768).for (const row of rows) {const cfg = (typeof row.config === 'string' ? JSON.parse(row.config) : (row.config ?? {})) as Record<string, unknown>;- if (cfg.kind === 'github' && githubKindCoversRepo(cfg, row.local_path, repo) && verify(cfg)) {+ const signed = verify(cfg);+ if (!(cfg.kind === 'github' && cfg.webhook_secret != null)) legacyRows++;+ if (cfg.kind === 'github' && githubKindCoversRepo(cfg, row.local_path, repo) && signed) {verified.push(row);continue;}- if (cfg.github_repo === repo && verify(cfg)) legacyMatched = true;+ if (cfg.github_repo === repo && signed) legacyMatched = true;}+ if (legacyRows === 0) verify({});return { verified, legacyMatched };}@@ -171,7 +177,7 @@ async function handleGitHubItemEvent([ref.repo],);const { verified, legacyMatched } = selectGitHubItemSources(rows, ref.repo, (cfg) =>- verifyWebhookSig(cfg, sigHeader, payload),+ webhookSignatureMatches(cfg.webhook_secret, sigHeader, payload),);if (verified.length > 1) {res.status(500).json({@@ -228,14 +234,22 @@ async function handleGitHubItemEvent(}}-function verifyWebhookSig(cfg: Record<string, unknown>, sigHeader: string, payload: Buffer): boolean {- const secret = cfg.webhook_secret;- if (typeof secret !== 'string' || secret === '') return false;- // Strict hex shape first: a malformed 64-char signature would make- // safeHexEqual throw (500 instead of 401).- if (!/^sha256=[0-9a-f]{64}$/.test(sigHeader)) return false;- const computedHex = createHmac('sha256', secret).update(payload).digest('hex');- return safeHexEqual(sigHeader.slice('sha256='.length), computedHex);+/** Stands in for a missing webhook secret; no source holds it and a match against it is still refused. */+const THROWAWAY_WEBHOOK_KEY = randomBytes(32);++/**+ * True when `sigHeader` is GitHub's HMAC-SHA256 of `payload` under `secret`. The HMAC over the body is computed on+ * every call, with a throwaway key when there is no source or no secret, so an unsigned caller cannot time responses+ * to learn which repositories this brain syncs with a live secret (#8768). GitHub sends "sha256=<hex>": the prefix is+ * stripped before safeHexEqual because Buffer.from('sha256=...', 'hex') truncates at the 's' and would make every+ * signature "match", and the strict hex shape is checked first because a malformed 64-char signature makes+ * safeHexEqual throw (500 instead of 401).+ */+function webhookSignatureMatches(secret: unknown, sigHeader: string, payload: Buffer): boolean {+ const keyed = typeof secret === 'string' && secret !== '';+ const computedHex = createHmac('sha256', keyed ? secret : THROWAWAY_WEBHOOK_KEY).update(payload).digest('hex');+ const matches = /^sha256=[0-9a-f]{64}$/.test(sigHeader) && safeHexEqual(sigHeader.slice('sha256='.length), computedHex);+ return matches && keyed;}/** POST /ingest (OAuth write scope) and POST /webhooks/github (per-source HMAC), in registration order. */@@ -473,9 +487,10 @@ async function handleIngest(// The job re-checks this writer at write time; refusing here answers the caller before anything is queued.const writeSourceId = authInfo.sourceId ?? 'default';const writer = authInfo.principal ? { principal: { kind: authInfo.principal.kind, id: authInfo.principal.id } } : undefined;+ let targetSlug = '';if (callerSlug) {try {- validateSlug(callerSlug);+ targetSlug = validateSlug(callerSlug);} catch (err) {res.status(400).json({ error: 'invalid_slug', message: err instanceof Error ? err.message : String(err) });return;@@ -489,6 +504,10 @@ async function handleIngest(}}+ // The same body sent to two slugs is two writes, so the normalized target slug (empty when the job picks one) is part+ // of the key next to the write source; dropping it handed the second caller the first job, a 202 with nothing+ // written to its slug (#8769).+ const idempotencyKey = `ingest:webhook:${authInfo.clientId}:${writeSourceId}:${targetSlug}:${contentHash}`;try {const job = await ingestQueue.add('ingest_capture',@@ -506,12 +525,22 @@ async function handleIngest(// The effective write source is part of the key: a client rescoped// from source X to Y must land a NEW capture in Y rather than being// deduped against its old X-bound job.- idempotency_key: `ingest:webhook:${authInfo.clientId}:${writeSourceId}:${contentHash}`,- // Cap waiting jobs from a single client so a runaway integration+ idempotency_key: idempotencyKey,+ // Cap waiting captures per write source so a runaway integration// can't fill the queue.maxWaiting: 50,},);+ // At the cap the queue hands back the newest waiting job instead of inserting this capture. Answering 202 with+ // that job id would drop this body without a trace, so the caller is told to retry once the backlog drains.+ if (job.idempotency_key !== idempotencyKey) {+ res.setHeader('Retry-After', '30');+ res.status(429).json({+ error: 'ingest_backlog_full',+ message: 'POST /ingest has 50 captures waiting for this source; nothing was queued. Retry after the backlog drains.',+ });+ return;+ }const latency = Date.now() - startTime;try {@@ -689,14 +718,12 @@ function mountGitHubWebhook(app: Express, ctx: ServeHttpContext): void {};// HMAC verify before anything about the repository is answered: an unknown repo, a source with no secret and a- // bad signature all get the same 401, so an unsigned request cannot tell which repositories this brain syncs.- // GitHub sends "sha256=<hex>" — strip the prefix BEFORE safeHexEqual because Buffer.from('sha256=...', 'hex')- // silently truncates at the first non-hex char (the 's'), leaving both operands as 0-byte buffers and making- // every signature "match". Strict hex shape first: a malformed 64-char signature would make safeHexEqual throw.+ // bad signature all get the same 401 after the same HMAC work, so an unsigned request cannot tell which+ // repositories this brain syncs.const secret = cfg.webhook_secret;- const computedHex = typeof secret === 'string' && secret ? createHmac('sha256', secret).update(payload).digest('hex') : null;- if (!source || !computedHex || !/^sha256=[0-9a-f]{64}$/.test(sigHeader) || !safeHexEqual(sigHeader.slice('sha256='.length), computedHex)) {- if (source && !computedHex) console.error(`webhook: push for source ${source.id} refused: no webhook secret is set (run: gbrain sources webhook set ${source.id})`);+ const verified = webhookSignatureMatches(secret, sigHeader, payload);+ if (!source || !verified) {+ if (source && !(typeof secret === 'string' && secret)) console.error(`webhook: push for source ${source.id} refused: no webhook secret is set (run: gbrain sources webhook set ${source.id})`);res.status(401).json({ error: 'signature_mismatch' });return;}diff --git a/test/e2e/serve-http-ingest-webhook.test.ts b/test/e2e/serve-http-ingest-webhook.test.tsindex 643b9c64f9..46951687e1 100644--- a/test/e2e/serve-http-ingest-webhook.test.ts+++ b/test/e2e/serve-http-ingest-webhook.test.ts@@ -22,7 +22,7 @@* 5. Header overrides: X-Gbrain-Slug is forwarded; X-Gbrain-Source-Id* cannot override the OAuth client's trusted write source* 6. Idempotency: same content + same client → job_id returned twice- * should match (queue dedup on (client_id, write_source_id, content_hash))+ * should match (queue dedup on (client_id, write_source_id, slug, content_hash))** Mirrors the spawn + mint pattern from test/e2e/serve-http-oauth.test.ts* exactly so future maintainers see one pattern, not two.@@ -601,7 +601,7 @@ describeE2E('serve-http POST /ingest webhook (v0.38)', () => {const secondBody = (await second.json()) as { job_id?: number | string };// Queue idempotency_key:- // `ingest:webhook:${clientId}:${writeSourceId}:${contentHash}` — same input+ // `ingest:webhook:${clientId}:${writeSourceId}:${slug}:${contentHash}` — same input// and same write source, same key, MinionQueue.add returns the existing job.expect(secondBody.job_id).toBe(firstBody.job_id!);});diff --git a/test/serve-http-github-push-webhook.test.ts b/test/serve-http-github-push-webhook.test.tsindex 14c4fac8b8..532d275606 100644--- a/test/serve-http-github-push-webhook.test.ts+++ b/test/serve-http-github-push-webhook.test.ts@@ -4,9 +4,13 @@* ref_mismatch (naming the tracked branch) and webhook_not_configured (naming the source id) before it checked the* HMAC, so an unauthenticated peer could tell which repositories a brain syncs. Protects: every unverified push gets* one 401 signature_mismatch body; a verified push for another branch still gets 202 ref_mismatch.+ * #8768: every unverified request also computes the HMAC over its body, whether the repository is unknown, has no+ * secret or is in scope of a github-kind source, so response time does not say which repositories this brain syncs.+ * Counted on the HMAC itself rather than timed, so the test cannot flake.*/-import { afterAll, beforeAll, describe, expect, test } from 'bun:test';+import { afterAll, afterEach, beforeAll, describe, expect, spyOn, test } from 'bun:test';import express from 'express';+import * as crypto from 'crypto';import { createHmac } from 'node:crypto';import type { Server } from 'node:http';import type { AddressInfo } from 'node:net';@@ -27,6 +31,8 @@ beforeAll(async () => {['source-signed', JSON.stringify({ github_repo: 'acme-example/signed', tracked_branch: 'main', webhook_secret: SECRET })]);await engine.executeRaw('INSERT INTO sources (id, name, config) VALUES ($1, $1, $2::jsonb)',['source-unset', JSON.stringify({ github_repo: 'acme-example/unset', tracked_branch: 'main' })]);+ await engine.executeRaw('INSERT INTO sources (id, name, config) VALUES ($1, $1, $2::jsonb)',+ ['source-items', JSON.stringify({ kind: 'github', gh_scope: 'repos', gh_repos: 'acme-example/covered', webhook_secret: SECRET })]);const app = express();const pass: express.RequestHandler = (_req, _res, next) => next();mountWebhooks(app, {@@ -84,3 +90,47 @@ describe('unverified GitHub push webhooks', () => {expect(JSON.parse(answer.text)).toMatchObject({ status: 'ignored', reason: 'ref_mismatch' });});});++describe('unverified webhooks compute the HMAC whatever the repository (#8768)', () => {+ let spy: ReturnType<typeof spyOn> | undefined;+ afterEach(() => { spy?.mockRestore(); spy = undefined; });++ async function hmacsFor(event: string, payload: string, signature: string) {+ spy = spyOn(crypto, 'createHmac');+ const res = await fetch(`${base}/webhooks/github`, {+ method: 'POST',+ headers: { 'content-type': 'application/json', 'x-github-event': event, 'x-hub-signature-256': signature },+ body: payload,+ });+ const calls = spy.mock.calls.length;+ spy.mockRestore();+ spy = undefined;+ return { status: res.status, calls };+ }+ const forged = `sha256=${'0'.repeat(64)}`;+ const pushBody = (repo: string) => JSON.stringify({ repository: { full_name: repo }, ref: 'refs/heads/main' });+ const issueBody = (repo: string) => JSON.stringify({ repository: { full_name: repo }, issue: { number: 7 }, action: 'edited' });++ test('a push for an unknown repo, a repo with no secret and a signed repo each compute one HMAC', async () => {+ for (const repo of ['acme-example/unknown', 'acme-example/unset', 'acme-example/signed']) {+ const answer = await hmacsFor('push', pushBody(repo), forged);+ expect(answer).toEqual({ status: 401, calls: 1 });+ }+ });++ test('a malformed signature header still computes the HMAC', async () => {+ for (const repo of ['acme-example/unknown', 'acme-example/signed']) {+ expect(await hmacsFor('push', pushBody(repo), 'sha256=not-hex')).toEqual({ status: 401, calls: 1 });+ }+ });++ test('an item event computes as many HMACs for a covered, a legacy-registered and an unknown repo', async () => {+ const covered = await hmacsFor('issues', issueBody('acme-example/covered'), forged);+ const legacy = await hmacsFor('issues', issueBody('acme-example/signed'), forged);+ const unknown = await hmacsFor('issues', issueBody('acme-example/unknown'), forged);+ expect(covered.calls).toBeGreaterThan(0);+ expect(legacy.calls).toBe(covered.calls);+ expect(unknown.calls).toBe(covered.calls);+ expect(new Set([covered.status, legacy.status, unknown.status])).toEqual(new Set([404]));+ });+});diff --git a/test/serve-http-github-webhook.test.ts b/test/serve-http-github-webhook.test.tsindex 33856e334e..6637236c7c 100644--- a/test/serve-http-github-webhook.test.ts+++ b/test/serve-http-github-webhook.test.ts@@ -95,6 +95,27 @@ describe('selectGitHubItemSources', () => {expect(verified.map((r) => r.id)).toEqual(['gh']);});+ test('every candidate row is signature-checked, in scope or not (#8768)', () => {+ let calls = 0;+ const counting = (cfg: Record<string, unknown>) => { calls++; return verify(cfg); };+ selectGitHubItemSources([ghRow('gh', 'good')], 'other/repo', counting);+ const outOfScope = calls;+ calls = 0;+ selectGitHubItemSources([ghRow('gh', 'good')], 'acme/app', counting);+ expect(outOfScope).toBeGreaterThan(0);+ expect(outOfScope).toBe(calls);+ });++ test('a repo with a legacy push source costs as many signature checks as an unknown repo (#8768)', () => {+ let calls = 0;+ const counting = (cfg: Record<string, unknown>) => { calls++; return verify(cfg); };+ selectGitHubItemSources([ghRow('gh', 'bad')], 'acme/unknown', counting);+ const unknown = calls;+ calls = 0;+ selectGitHubItemSources([ghRow('gh', 'bad'), legacyRow('old', 'bad')], 'acme/app', counting);+ expect(calls).toBe(unknown);+ });+test('out-of-scope github-kind source neither verifies nor flags legacy', () => {const { verified, legacyMatched } = selectGitHubItemSources([ghRow('gh', 'good')], 'other/repo', verify);expect(verified).toEqual([]);diff --git a/test/serve-http-ingest-legacy.test.ts b/test/serve-http-ingest-legacy.test.tsindex 62a7674193..f4e6c1cad3 100644--- a/test/serve-http-ingest-legacy.test.ts+++ b/test/serve-http-ingest-legacy.test.ts@@ -49,7 +49,7 @@ beforeEach(async () => { await engine.executeRaw('DELETE FROM minion_jobs'); });function post(content: string) {return fetch(`${base}/ingest`, { method: 'POST', headers: { authorization: 'Bearer test-only', 'content-type': 'text/markdown' }, body: content });}-const keyFor = (content: string) => `ingest:webhook:${CLIENT}:default:${computeContentHash(content)}`;+const keyFor = (content: string) => `ingest:webhook:${CLIENT}:default::${computeContentHash(content)}`;/** A job holding the capture's exact key, rewritten to a pre-cutover row. */async function legacyRow(content: string, status: string, authority: string | null = null): Promise<number> {diff --git a/test/serve-http-ingest-write-loss.test.ts b/test/serve-http-ingest-write-loss.test.tsnew file mode 100644index 0000000000..60033f0cdf--- /dev/null+++ b/test/serve-http-ingest-write-loss.test.ts@@ -0,0 +1,137 @@+/**+ * POST /ingest never answers 202 for a capture it did not queue.+ *+ * #8769: the queue idempotency key now carries the target slug. Before this fix the key was+ * client + write source + content hash, so one client posting the same body to two slugs while the first job was+ * still queued (or after it completed) got the first job back: a 202 for the second slug and nothing written there.+ * Protects: the same body to two slugs queues two jobs and writes both pages; a retry to the same slug (any case)+ * and a slug-less retry still collapse onto one job; a slug-less post and a slug post stay separate.+ * Waiting cap: at 50 waiting captures for a source the queue hands back the newest waiting job instead of inserting;+ * the route used to answer 202 with that job, dropping the body. Protects: the capture over the cap gets 429 with+ * Retry-After and nothing claimed, and the same capture lands its own job once the backlog drains.+ * Real token verifier, real Express route, repo-less PGLite brain, the real ingest_capture handler.+ */+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';+import express from 'express';+import type { Server } from 'node:http';+import type { AddressInfo } from 'node:net';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { mintLegacyToken } from '../src/core/token-mint.ts';+import { mountWebhooks } from '../src/commands/serve-http-webhooks.ts';+import type { ServeHttpContext } from '../src/commands/serve-http.ts';+import { GBrainOAuthProvider } from '../src/core/oauth-provider.ts';+import { sqlQueryForEngine } from '../src/core/sql-query.ts';+import { makeIngestCaptureHandler } from '../src/core/minions/handlers/ingest-capture.ts';+import type { MinionJobContext } from '../src/core/minions/types.ts';++const SRC = 'source-a';++let engine: PGLiteEngine;+let server: Server;+let base: string;+let token: string;++beforeAll(async () => {+ engine = new PGLiteEngine();+ await engine.connect({});+ await engine.initSchema();+ await engine.executeRaw('INSERT INTO sources (id, name) VALUES ($1, $1) ON CONFLICT (id) DO NOTHING', [SRC]);+ token = (await mintLegacyToken(engine, { name: 'writer', scopes: ['read', 'write'], takesHolders: ['world'], sourceGrant: [SRC] })).token;+ const provider = new GBrainOAuthProvider({ sql: sqlQueryForEngine(engine), transaction: fn => engine.transaction(tx => fn(sqlQueryForEngine(tx))) });+ const app = express();+ const pass: express.RequestHandler = (_req, _res, next) => next();+ mountWebhooks(app, {+ engine,+ resourceVerifier: provider,+ resourceMetadataUrl: 'http://127.0.0.1/.well-known/oauth-protected-resource',+ ingestRateLimiter: pass,+ githubWebhookLimiter: pass,+ broadcastEvent: () => {},+ } as unknown as ServeHttpContext);+ server = await new Promise<Server>(resolve => { const s = app.listen(0, '127.0.0.1', () => resolve(s)); });+ base = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;+}, 60_000);++afterAll(async () => {+ await new Promise<void>(resolve => server?.close(() => resolve()));+ await engine?.disconnect();+}, 60_000);++const post = (body: string, slug?: string) => fetch(`${base}/ingest`, {+ method: 'POST',+ headers: { authorization: `Bearer ${token}`, 'content-type': 'text/markdown', ...(slug ? { 'x-gbrain-slug': slug } : {}) },+ body,+});+const ingest = async (body: string, slug?: string) => {+ const res = await fetch(`${base}/ingest`, {+ method: 'POST',+ headers: { authorization: `Bearer ${token}`, 'content-type': 'text/markdown', ...(slug ? { 'x-gbrain-slug': slug } : {}) },+ body,+ });+ expect(res.status).toBe(202);+ return (await res.json() as { job_id: number }).job_id;+};+const jobCtx = (id: number, data: Record<string, unknown>): MinionJobContext => ({ id, name: 'ingest_capture', data, attempts_made: 0,+ signal: new AbortController().signal, deadlineAtMs: null, shutdownSignal: new AbortController().signal, updateProgress: async () => {},+ updateTokens: async () => {}, log: async () => {}, isActive: async () => true, readInbox: async () => [] });+const runJob = async (id: number) => {+ const [row] = await engine.executeRaw<{ data: unknown }>('SELECT data FROM minion_jobs WHERE id = $1', [id]);+ const data = (typeof row!.data === 'string' ? JSON.parse(row!.data) : row!.data) as Record<string, unknown>;+ return makeIngestCaptureHandler(engine)(jobCtx(id, data));+};+const note = (title: string) => `---\ntype: note\ntitle: ${title}\n---\n# ${title}\n\nSame body for every target.\n`;++describe('POST /ingest idempotency includes the target slug', () => {+ test('the same body to two slugs while the first job is queued: two jobs, both pages written', async () => {+ const body = note('Shared capture');+ const first = await ingest(body, 'notes/target-one');+ const second = await ingest(body, 'notes/target-two');+ expect(second).not.toBe(first);+ expect((await runJob(first)).status).toBe('imported');+ expect((await runJob(second)).status).toBe('imported');+ expect((await engine.readPageSnapshot('notes/target-one', { sourceId: SRC }))?.page.compiled_truth).toContain('Same body for every target.');+ expect((await engine.readPageSnapshot('notes/target-two', { sourceId: SRC }))?.page.compiled_truth).toContain('Same body for every target.');+ });++ test('the same body to a third slug after the first job completed still gets its own job', async () => {+ const body = note('Shared capture');+ const third = await ingest(body, 'notes/target-three');+ expect((await runJob(third)).status).toBe('imported');+ expect(await engine.readPageSnapshot('notes/target-three', { sourceId: SRC })).not.toBeNull();+ });++ test('a retry to the same slug, in any case, collapses onto the first job', async () => {+ const body = note('Retried capture');+ const first = await ingest(body, 'notes/retried');+ expect(await ingest(body, 'notes/retried')).toBe(first);+ expect(await ingest(body, 'Notes/Retried')).toBe(first);+ });++ test('a slug-less post and a slug post of the same body are separate jobs; slug-less retries collapse', async () => {+ const body = note('Inbox capture');+ const inbox = await ingest(body);+ expect(await ingest(body)).toBe(inbox);+ expect(await ingest(body, 'notes/inbox-copy')).not.toBe(inbox);+ });+});++describe('POST /ingest at the waiting cap', () => {+ test('the capture over the cap gets 429 and nothing queued; it lands its own job once the backlog drains', async () => {+ await engine.executeRaw('DELETE FROM minion_jobs');+ const queued = new Set<number>();+ for (let i = 0; i < 50; i++) queued.add(await ingest(note(`Backlog ${i}`)));+ expect(queued.size).toBe(50);++ const over = await post(note('Over the cap'));+ expect(over.status).toBe(429);+ expect(over.headers.get('retry-after')).toBe('30');+ expect((await over.json() as { error: string }).error).toBe('ingest_backlog_full');+ const [held] = await engine.executeRaw<{ n: number }>("SELECT count(*)::int AS n FROM minion_jobs WHERE data::text LIKE '%Over the cap%'");+ expect(held!.n).toBe(0);++ await engine.executeRaw("UPDATE minion_jobs SET status = 'completed' WHERE id = $1", [Math.min(...queued)]);+ const retried = await ingest(note('Over the cap'));+ expect(queued.has(retried)).toBe(false);+ expect((await runJob(retried)).status).toBe('imported');+ });+});