Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Keyword search comes back within a minute after a cold-cache timeout, not ten (#7082) (#9495)

Garry Tanb42565177882
diff --git a/behavior-changes.d/keyword_skip_rerun_7082.md b/behavior-changes.d/keyword_skip_rerun_7082.md
new file mode 100644
index 0000000000..f957a5ec87
--- /dev/null
+++ b/behavior-changes.d/keyword_skip_rerun_7082.md
@@ -0,0 +1,4 @@
+---
+kind: fix
+---
+On Postgres, a hybrid keyword statement that hit the keyword statement timeout is still skipped by later identical searches in the same serving process (`keyword_arm_failed: skipped_after_timeout`), but the skip no longer always lasts the whole `search.keyword_timeout_negative_cache_seconds` (default 600 s, unchanged, now the longest skip). The serve re-runs the statement once in the background 30 s after the timeout, then after 60 s, 120 s and so on while it keeps failing, with the same scope and the same bound, only when the pool has no waiting query and keeps a connection free for other queries, and one at a time per brain. Its rows are thrown away and it is not recorded as a search. The first re-run that answers ends the skip, so a timeout on a cold cache stops costing keyword results within about a minute of the cache warming. One-shot CLI commands are unchanged.
diff --git a/changelog.d/capy-keyword-skip-rerun.md b/changelog.d/capy-keyword-skip-rerun.md
new file mode 100644
index 0000000000..e0b8be0606
--- /dev/null
+++ b/changelog.d/capy-keyword-skip-rerun.md
@@ -0,0 +1,6 @@
+**A keyword search that timed out on a cold Postgres cache comes back within about a minute instead of ten.**
+
+### Fixed
+- After hybrid search's keyword statement hits the keyword statement timeout, `gbrain serve` and MCP sessions still skip that exact statement (`keyword_arm_failed: skipped_after_timeout`), and now also re-run it once in the background 30 seconds later, then after 60, 120 and so on seconds while it keeps failing, never past `search.keyword_timeout_negative_cache_seconds` (default 600, unchanged). The first re-run that answers ends the skip. A re-run uses the same statement, scope and bound, starts only when the pool has no waiting query and keeps a connection free for other queries, runs one at a time per brain, throws its rows away, writes no timeout counter row and is not recorded as a search. Before, one timeout on a cold cache left every identical search keyword-less for ten minutes even after the cache had warmed (#7082).
+
+<!-- behavior-change: keyword_skip_rerun_7082 -->
diff --git a/docs/architecture/key-files/engines-2-continued.md b/docs/architecture/key-files/engines-2-continued.md
index da32608aa2..2745736cd5 100644
--- a/docs/architecture/key-files/engines-2-continued.md
+++ b/docs/architecture/key-files/engines-2-continued.md
@@ -3,7 +3,7 @@
[Subsystem index](../KEY_FILES.md) · [Previous entries](engines-2.md). Read only the entries relevant to your change.
Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.
-- `src/core/postgres-engine/` + `src/core/pglite-engine/` — per-engine module directories for code that is genuinely dialect-specific. Migrated storage domains (the `migrated` rows of `scripts/engine-sql-baseline.tsv`) do not live here: their SQL exists once in `src/core/engine-sql/` (see that entry in engines-2.md). What remains: `pglite-engine/checkpoint-guard.ts`, `pglite-engine/code-edges.ts` (PGLite's `getEdgesByChunk`, whose master SQL behaves differently from Postgres's: one UNION ALL under a shared LIMIT; listed in `docs/designs/refactor-wave-1/w1-inventory.md`), `postgres-engine/cancellation.ts`, `postgres-engine/init-schema-lock.ts`, `postgres-engine/keyword-statement-timeout.ts` (#6495: `KEYWORD_STATEMENT_TIMEOUT_KEY` = `search.keyword_statement_timeout_ms`, default 8000, range 1000-60000; `keywordStatementTimeoutMs` accepts an integer in range, `resolveKeywordStatementTimeoutMs(file, db)` applies file > DB > default, `keywordStatementTimeoutValueProblem` is the `config set` refusal text, and `loadKeywordStatementTimeout(reader, fileValue)` yields the `<ms>ms` setting one connection binds; pinned by `test/keyword-statement-timeout-knob-6495.test.ts` on a recording handle and, with DATABASE_URL, on a real Postgres whose locked tables make each statement wait until the bound cancels it), `postgres-engine/keyword-timeout-cache.ts` (#6495 negative cache: `KeywordTimeoutNegativeCache.run(enabled, db, keyParts, exec)` wraps `searchKeyword`'s strict and relaxed statements when hybrid's keyword arm opts in with `SearchOpts.keywordTimeoutCache`; the key hashes the statement text, its parameters, the RLS source scope and the statement timeout, so only the exact statement is skipped; a genuine statement-timeout 57014, never a user cancel, records an entry for `search.keyword_timeout_negative_cache_seconds` (default 600, 0 = off, read once on the first timeout) and later identical runs throw `KeywordSkippedAfterTimeoutError` (code `keyword_skipped_after_timeout`, arm reason `skipped_after_timeout`) without a statement; at most 256 entries per engine; each timeout writes one bounded `search.keyword_timeout_stats` row that doctor `keyword_plan` reads; pinned by `test/keyword-timeout-negative-cache-6495.test.ts`), `postgres-engine/schema-lock-timeout.ts` (#5227: `resolveSchemaLockTimeoutMs` reads `GBRAIN_SCHEMA_LOCK_TIMEOUT_SECONDS`, default 60 s; `withSchemaLockTimeout` sets a session-level `lock_timeout` on the reserved DDL backend and restores the prior value on release, used by the `schema.sql` replay in `postgres-engine.ts` and the `transaction: false` path of `migrate.ts`, while a transaction-wrapped migration binds the same wait with `SET LOCAL`; `schemaDiagnosticConnection` opens a one-backend connection that samples `pg_blocking_pids()` of the waiting backend while the wait is active so `schemaLockBlockedError` names blockers by pid, application, state and transaction age, never query text; `init-schema-retry.ts` retries `schema_lock_blocked` / SQLSTATE 55P03 with a short backoff; handler migrations and the post-migration `verifySchema` run outside the bound) and `postgres-engine/search-settings.ts` (search transactions run with `jit = off`; `withSearchSettings` applies a read's settings in one statement and restores a caller's values when nested), `postgres-engine/reserved-transactions.ts` (`withReservedConnection` lends its connection a `transaction()`; BEGIN READ WRITE and SAVEPOINT go out with the statements after them, and a COMMIT answered with ROLLBACK throws), `postgres-engine/read-write-transactions.ts` (#7183: a server connection whose `default_transaction_read_only` is on, from a session SET left on a pooled connection or a role or database default, refuses writes with 25006; `beginReadWrite` / `readWriteTransactions` open every pool transaction `BEGIN READ WRITE`, falling back to a plain BEGIN on a hot standby that refuses it, and `retryReadOnlyAutocommit` reruns an autocommit statement refused with 25006 once inside one; explicit `begin('read only')` and `SET TRANSACTION ... READ ONLY` stay read-only; pinned by `test/inherited-read-only-7183.test.ts` and, on direct Postgres and PgBouncer, `test/e2e/pooler-inherited-read-only-7183.test.ts`), `postgres-engine/transaction-abort.ts` (`transaction(fn, { signal })`: an abort calls the begin handle's `cancelAndDiscard()`, which sends the running fenced statement's CancelRequest, keeps successors out until it settles (bounded by `GBRAIN_CANCEL_SETTLE_MS`), then discards the connection, so a statement blocked on a lock stops instead of waiting out its statement timeout (#6598) and the server rolls the transaction back; the caller gets an `AbortError` with the driver's `CONNECTION_CLOSED` as `cause` and, when the cancel could not be confirmed (unfenced statement, transport failure, settle timeout), the `CANCEL_UNCONFIRMED` rejection as `cancelFailure`; an abort after COMMIT keeps the committed result; an already-aborted signal rejects before BEGIN; pinned offline by `test/postgres-transaction-cancel-order.test.ts` and on direct Postgres and PgBouncer by `test/e2e/transaction-cancel-held-lock.test.ts`); the forward-reference bootstrap lives once in `engine-sql/bootstrap.ts` for both engines. Façades import these modules statically (engine-live no-runtime-dynamic-import rule) and keep their full public surface.
+- `src/core/postgres-engine/` + `src/core/pglite-engine/` — per-engine module directories for code that is genuinely dialect-specific. Migrated storage domains (the `migrated` rows of `scripts/engine-sql-baseline.tsv`) do not live here: their SQL exists once in `src/core/engine-sql/` (see that entry in engines-2.md). What remains: `pglite-engine/checkpoint-guard.ts`, `pglite-engine/code-edges.ts` (PGLite's `getEdgesByChunk`, whose master SQL behaves differently from Postgres's: one UNION ALL under a shared LIMIT; listed in `docs/designs/refactor-wave-1/w1-inventory.md`), `postgres-engine/cancellation.ts`, `postgres-engine/init-schema-lock.ts`, `postgres-engine/keyword-statement-timeout.ts` (#6495: `KEYWORD_STATEMENT_TIMEOUT_KEY` = `search.keyword_statement_timeout_ms`, default 8000, range 1000-60000; `keywordStatementTimeoutMs` accepts an integer in range, `resolveKeywordStatementTimeoutMs(file, db)` applies file > DB > default, `keywordStatementTimeoutValueProblem` is the `config set` refusal text, and `loadKeywordStatementTimeout(reader, fileValue)` yields the `<ms>ms` setting one connection binds; pinned by `test/keyword-statement-timeout-knob-6495.test.ts` on a recording handle and, with DATABASE_URL, on a real Postgres whose locked tables make each statement wait until the bound cancels it), `postgres-engine/keyword-timeout-cache.ts` (#6495 negative cache: `KeywordTimeoutNegativeCache.run(enabled, db, keyParts, exec)` wraps `searchKeyword`'s strict and relaxed statements when hybrid's keyword arm opts in with `SearchOpts.keywordTimeoutCache`; the key hashes the statement text, its parameters, the RLS source scope and the statement timeout, so only the exact statement is skipped; a genuine statement-timeout 57014, never a user cancel, records an entry for `search.keyword_timeout_negative_cache_seconds` (default 600, 0 = off, read once on the first timeout) and later identical runs throw `KeywordSkippedAfterTimeoutError` (code `keyword_skipped_after_timeout`, arm reason `skipped_after_timeout`) without a statement; at most 256 entries per engine; each timeout writes one bounded `search.keyword_timeout_stats` row that doctor `keyword_plan` reads; #7082: in a serving process (`setKeywordTimeoutReruns(true)` from `gbrain serve`, `false` when its shutdown begins) each entry is re-run in the background 30 s after the timeout, then 60 s, 120 s and so on, never past its TTL, one at a time per engine and only when `keywordRerunPoolIdle`, reading the pool the engine queries (`enginePoolStats`, so a serve's process pool too), sees no waiting query and a connection left free beside the re-run, its rows thrown away, and the first re-run that answers ends the skip; pinned by `test/keyword-timeout-negative-cache-6495.test.ts` and `test/keyword-timeout-rerun-7082.test.ts`), `postgres-engine/schema-lock-timeout.ts` (#5227: `resolveSchemaLockTimeoutMs` reads `GBRAIN_SCHEMA_LOCK_TIMEOUT_SECONDS`, default 60 s; `withSchemaLockTimeout` sets a session-level `lock_timeout` on the reserved DDL backend and restores the prior value on release, used by the `schema.sql` replay in `postgres-engine.ts` and the `transaction: false` path of `migrate.ts`, while a transaction-wrapped migration binds the same wait with `SET LOCAL`; `schemaDiagnosticConnection` opens a one-backend connection that samples `pg_blocking_pids()` of the waiting backend while the wait is active so `schemaLockBlockedError` names blockers by pid, application, state and transaction age, never query text; `init-schema-retry.ts` retries `schema_lock_blocked` / SQLSTATE 55P03 with a short backoff; handler migrations and the post-migration `verifySchema` run outside the bound) and `postgres-engine/search-settings.ts` (search transactions run with `jit = off`; `withSearchSettings` applies a read's settings in one statement and restores a caller's values when nested), `postgres-engine/reserved-transactions.ts` (`withReservedConnection` lends its connection a `transaction()`; BEGIN READ WRITE and SAVEPOINT go out with the statements after them, and a COMMIT answered with ROLLBACK throws), `postgres-engine/read-write-transactions.ts` (#7183: a server connection whose `default_transaction_read_only` is on, from a session SET left on a pooled connection or a role or database default, refuses writes with 25006; `beginReadWrite` / `readWriteTransactions` open every pool transaction `BEGIN READ WRITE`, falling back to a plain BEGIN on a hot standby that refuses it, and `retryReadOnlyAutocommit` reruns an autocommit statement refused with 25006 once inside one; explicit `begin('read only')` and `SET TRANSACTION ... READ ONLY` stay read-only; pinned by `test/inherited-read-only-7183.test.ts` and, on direct Postgres and PgBouncer, `test/e2e/pooler-inherited-read-only-7183.test.ts`), `postgres-engine/transaction-abort.ts` (`transaction(fn, { signal })`: an abort calls the begin handle's `cancelAndDiscard()`, which sends the running fenced statement's CancelRequest, keeps successors out until it settles (bounded by `GBRAIN_CANCEL_SETTLE_MS`), then discards the connection, so a statement blocked on a lock stops instead of waiting out its statement timeout (#6598) and the server rolls the transaction back; the caller gets an `AbortError` with the driver's `CONNECTION_CLOSED` as `cause` and, when the cancel could not be confirmed (unfenced statement, transport failure, settle timeout), the `CANCEL_UNCONFIRMED` rejection as `cancelFailure`; an abort after COMMIT keeps the committed result; an already-aborted signal rejects before BEGIN; pinned offline by `test/postgres-transaction-cancel-order.test.ts` and on direct Postgres and PgBouncer by `test/e2e/transaction-cancel-held-lock.test.ts`); the forward-reference bootstrap lives once in `engine-sql/bootstrap.ts` for both engines. Façades import these modules statically (engine-live no-runtime-dynamic-import rule) and keep their full public surface.
- `src/core/analyze-lock.ts` — #7083: `lockTableForAnalyze` takes a table's SHARE UPDATE EXCLUSIVE lock with NOWAIT before every ANALYZE gbrain runs on Postgres on its own (the projection and chunk refreshes, `repair planner-stats`, `analyzeAfterIndexBuild` through `analyzeYieldingToVacuum`), and `analyzeDeferred` recognizes its 55P03 refusal. An ANALYZE that waited `deadlock_timeout` on a running autovacuum made Postgres cancel it, so a refresh after every embed drain kept a 10M-chunk content_chunks from ever being vacuumed; a request that never waits never signals a vacuum, and the caller defers its ANALYZE. Doctor `planner_stats_stale` reports a hot table whose autovacuum stays starved (`readVacuumStarvation`). Test: `test/e2e/analyze-yields-to-autovacuum-postgres.test.ts`.
- `src/core/migration-preludes.ts` — #7108 runner-side preludes for published migrations whose SQL rewrites a large table in one transaction (published migration files never change; restamp refuses an edit). `runMigrationPrelude(engine, version)` runs right before that migration's SQL inside the runner's retry wrapper: `batchedUpdate` repeats the migration's own SET and predicate over id-ordered batches (5,000 rows, inlined so a generic prepared plan cannot scan the table per batch; `__setBackfillBatchRowsForTests`), each its own transaction with the migration statement budget and the schema lock timeout, so progress is the data itself and a rerun resumes; a column the migration adds first is added first, in the migration's column order (fresh and upgraded ordinals stay equal). Preludes: v011 legacy `links.link_source` (skipped below PG15, where the migration refuses), v028 chunk keyword vectors, v033 request-log agent names (NULL-token rows stay the migration's own), v116 code-edge `source_id`, v161 (Postgres) the partial pages index built concurrently under the budget on a reserved connection (the migration caps its own build at 30 s; another session's build or an invalid result stops the run), v221 `embedding_pending_since` (v221's active-column choice; the first stamping batch writes `embedding_pending_since_backfilled_at` in its transaction). Pinned by `test/e2e/migration-v221-batched-backfill-7108.test.ts`. Migrations that cannot be moved here are listed in docs/guides/troubleshooting.md#migration-statement-timeout.
diff --git a/docs/guides/error-codes.md b/docs/guides/error-codes.md
index e4ae5b9dbe..f2560407b5 100644
--- a/docs/guides/error-codes.md
+++ b/docs/guides/error-codes.md
@@ -3058,7 +3058,7 @@ More: [docs/guides/multi-language-fts.md#cjk-keyword-budget](../../docs/guides/m
| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable |
|---|---|---|---|---|---|---|
-| Hybrid search's Postgres keyword arm skipped a statement that hit the keyword statement timeout within the negative-cache TTL; the search reports keyword_arm_failed with reason skipped_after_timeout and the vector arm still answers. | Running the same broad keyword statement again would hold a backend for the whole search.keyword_statement_timeout_ms and fail the same way. The skip covers that exact statement and scope only, lasts search.keyword_timeout_negative_cache_seconds (default 600, 0 turns it off) and is per serving process. A skipped keyword arm is not evidence of absence. | Narrow the search (source_id, types, since or until) to run keyword search now, or wait for the entry to expire; the user may raise search.keyword_statement_timeout_ms or change the TTL. Run: gbrain doctor --only keyword_plan --json | agent | `gbrain doctor --only keyword_plan --json` | 1 | no |
+| Hybrid search's Postgres keyword arm skipped a statement that hit the keyword statement timeout within the negative-cache TTL; the search reports keyword_arm_failed with reason skipped_after_timeout and the vector arm still answers. | Running the same broad keyword statement again would hold a backend for the whole search.keyword_statement_timeout_ms and fail the same way. The skip covers that exact statement and scope only, lasts at most search.keyword_timeout_negative_cache_seconds (default 600, 0 turns it off), ends as soon as a background re-run of the statement in the serve answers, and is per serving process. A skipped keyword arm is not evidence of absence. | Narrow the search (source_id, types, since or until) to run keyword search now, or wait for the entry to expire; the user may raise search.keyword_statement_timeout_ms or change the TTL. Run: gbrain doctor --only keyword_plan --json | agent | `gbrain doctor --only keyword_plan --json` | 1 | no |
More: [docs/guides/troubleshooting.md#keyword-search-times-out-on-a-large-postgres-brain](../../docs/guides/troubleshooting.md#keyword-search-times-out-on-a-large-postgres-brain)
diff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.md
index ae44e5065c..302c6e1073 100644
--- a/docs/guides/troubleshooting.md
+++ b/docs/guides/troubleshooting.md
@@ -580,25 +580,34 @@ strict count still read every match of a term that matches most of the corpus,
so prefer narrowing with `--source-id` (MCP `source_id`), `--types` or
`--since`.
-**A timed-out keyword search is skipped for 10 minutes.** After hybrid search's
-keyword statement hits the bound, a serving process (`gbrain serve`, an MCP
-session) skips that exact statement for
+**A timed-out keyword search is skipped for up to 10 minutes.** After hybrid
+search's keyword statement hits the bound, a serving process (`gbrain serve`,
+an MCP session) skips that exact statement for at most
`search.keyword_timeout_negative_cache_seconds` (integer seconds, default 600,
range 0 to 86400, 0 turns it off) instead of holding a backend for the whole
-bound again. The skipped search answers from the vector arm at once and its
+bound again. A timeout on a cold cache (the first searches after a restart or
+an eviction) is often gone a minute later, so the serve re-runs the timed-out
+statement once in the background 30 seconds after the timeout, then after 60,
+120 and so on seconds while it keeps failing, never past that limit. The first
+re-run that answers ends the skip, and the next search runs keyword search
+again. A re-run uses the same statement, scope and bound, starts only when the
+pool has no waiting query and keeps a connection free for other queries
+(otherwise it waits for the next step), runs one at
+a time per brain, throws its rows away and is not recorded as a search. The skipped search answers from the vector arm at once and its
metadata carries `keyword_arm_failed` with reason `skipped_after_timeout`
(error code `keyword_skipped_after_timeout` in `gbrain errors`). The skip
covers the same statement only: the same terms with the same source scope,
types, dates, slug filters, visibility and trust floor of the caller, and the
same keyword bound. A narrower query (another `--source-id`, `--types`,
`--since`) still runs keyword search, another brain is never affected, and
-the skip ends by itself when the entry expires, so a brief contention spike
-recovers without anyone acting. A call with `use_cache: false` runs the
+the skip ends by itself when a re-run answers or the entry expires, so a brief
+contention spike recovers without anyone acting. A call with `use_cache: false` runs the
statement anyway.
Limits: the cache lives in the serving process's memory, so a one-shot CLI
-command (`gbrain search …`) starts with an empty cache and gains nothing from
-it, and a restart clears it. The TTL is read on the first timeout after the
+command (`gbrain search …`) starts with an empty cache, runs no re-runs and
+gains nothing from it, and a restart clears it. A CLI search that the serve
+answers through its warm CLI socket uses the serve's cache. The TTL is read on the first timeout after the
process starts; a changed value takes effect after a restart. Each timeout
writes one counter row (`search.keyword_timeout_stats`: timeouts, skips counted
since the previous timeout, last timeout time), never one per search, and
diff --git a/docs/mcp/TOOL_REFERENCE.md b/docs/mcp/TOOL_REFERENCE.md
index cb660370d5..f507a07c67 100644
--- a/docs/mcp/TOOL_REFERENCE.md
+++ b/docs/mcp/TOOL_REFERENCE.md
@@ -413,7 +413,7 @@ Cheap hybrid search (vector + keyword + RRF) with no LLM expansion. Best for exa
Every call also reports how many pages match the query's keywords (`keyword_total`, `keyword_truncated`; "10000+" past 10,000) in one line after the rows. That count covers keyword (full-text) matches only, not the ranked rows. To list every one of those pages, pass `match: "keyword"`: rows come one per page in keyword-score order with `total`, `truncated` and `next`, a ready-to-send argument object; keep calling with `next` until it is absent. A page whose text did not fit `token_budget` still gets a row, with `evidence_omitted: true`. If the count times out, the line says it is unavailable (degraded stage `keyword_count_unavailable`), never zero.
-On a large Postgres brain a broad keyword statement can hit the keyword time limit; the response's degraded stages then carry `keyword_arm_failed` with reason `timeout`, and for the next 10 minutes (`search.keyword_timeout_negative_cache_seconds`, 0 = off) the same search in the same serving process reports reason `skipped_after_timeout` at once instead of waiting again. Vector hits still come back. A skipped keyword arm is not evidence of absence: narrow the search (`source_id`, `types`, `since` or `until`) to run keyword matching, and `gbrain doctor --only keyword_plan --json` on the brain host reports the bound and the recorded timeouts. The same applies to `query`.
+On a large Postgres brain a broad keyword statement can hit the keyword time limit; the response's degraded stages then carry `keyword_arm_failed` with reason `timeout`, and for up to 10 minutes (`search.keyword_timeout_negative_cache_seconds`, 0 = off) the same search in the same serving process reports reason `skipped_after_timeout` at once instead of waiting again. The serve re-runs the statement in the background from 30 seconds after the timeout, and the skip ends at the first re-run that answers. Vector hits still come back. A skipped keyword arm is not evidence of absence: narrow the search (`source_id`, `types`, `since` or `until`) to run keyword matching, and `gbrain doctor --only keyword_plan --json` on the brain host reports the bound and the recorded timeouts. The same applies to `query`.
| Parameter | Type | Guidance |
| --- | --- | --- |
diff --git a/llms-full.txt b/llms-full.txt
index 94d47d4d6f..26309a23c4 100644
--- a/llms-full.txt
+++ b/llms-full.txt
@@ -9776,25 +9776,34 @@ strict count still read every match of a term that matches most of the corpus,
so prefer narrowing with `--source-id` (MCP `source_id`), `--types` or
`--since`.
-**A timed-out keyword search is skipped for 10 minutes.** After hybrid search's
-keyword statement hits the bound, a serving process (`gbrain serve`, an MCP
-session) skips that exact statement for
+**A timed-out keyword search is skipped for up to 10 minutes.** After hybrid
+search's keyword statement hits the bound, a serving process (`gbrain serve`,
+an MCP session) skips that exact statement for at most
`search.keyword_timeout_negative_cache_seconds` (integer seconds, default 600,
range 0 to 86400, 0 turns it off) instead of holding a backend for the whole
-bound again. The skipped search answers from the vector arm at once and its
+bound again. A timeout on a cold cache (the first searches after a restart or
+an eviction) is often gone a minute later, so the serve re-runs the timed-out
+statement once in the background 30 seconds after the timeout, then after 60,
+120 and so on seconds while it keeps failing, never past that limit. The first
+re-run that answers ends the skip, and the next search runs keyword search
+again. A re-run uses the same statement, scope and bound, starts only when the
+pool has no waiting query and keeps a connection free for other queries
+(otherwise it waits for the next step), runs one at
+a time per brain, throws its rows away and is not recorded as a search. The skipped search answers from the vector arm at once and its
metadata carries `keyword_arm_failed` with reason `skipped_after_timeout`
(error code `keyword_skipped_after_timeout` in `gbrain errors`). The skip
covers the same statement only: the same terms with the same source scope,
types, dates, slug filters, visibility and trust floor of the caller, and the
same keyword bound. A narrower query (another `--source-id`, `--types`,
`--since`) still runs keyword search, another brain is never affected, and
-the skip ends by itself when the entry expires, so a brief contention spike
-recovers without anyone acting. A call with `use_cache: false` runs the
+the skip ends by itself when a re-run answers or the entry expires, so a brief
+contention spike recovers without anyone acting. A call with `use_cache: false` runs the
statement anyway.
Limits: the cache lives in the serving process's memory, so a one-shot CLI
-command (`gbrain search …`) starts with an empty cache and gains nothing from
-it, and a restart clears it. The TTL is read on the first timeout after the
+command (`gbrain search …`) starts with an empty cache, runs no re-runs and
+gains nothing from it, and a restart clears it. A CLI search that the serve
+answers through its warm CLI socket uses the serve's cache. The TTL is read on the first timeout after the
process starts; a changed value takes effect after a restart. Each timeout
writes one counter row (`search.keyword_timeout_stats`: timeouts, skips counted
since the previous timeout, last timeout time), never one per search, and
diff --git a/scripts/bench/retrieval-scale/keyword-broad.ts b/scripts/bench/retrieval-scale/keyword-broad.ts
index 44edc03080..d81ce7a2ff 100644
--- a/scripts/bench/retrieval-scale/keyword-broad.ts
+++ b/scripts/bench/retrieval-scale/keyword-broad.ts
@@ -5,7 +5,7 @@
* Opt-in bench code, never run in CI.
*
* bun scripts/bench/retrieval-scale/keyword-broad.ts --url <direct postgres url> [--terms a,b,c] [--n 10]
- * [--calls arm,arm_phrase,arm_remote,arm_all_sources,arm_small_source,arm_person,pages,pages_remote,count] [--explain <dir>] [--label <name>] [--out results.jsonl]
+ * [--calls arm,arm_phrase,arm_remote,arm_all_sources,arm_small_source,arm_person,pages,pages_remote,count,count_remote,vector,vector_remote,titles,titles_remote] [--explain <dir>] [--label <name>] [--out results.jsonl]
* [--rows rows.jsonl] [--tree <checkout>] [--no-share]
*
* Without --terms it picks one term per band from a 1% sample of chunks
@@ -49,7 +49,9 @@ async function pickTerms(): Promise<Array<{ band: string; term: string }>> {
}
const terms = flag('--terms') ? flag('--terms')!.split(',').map(t => ({ band: 'given', term: t })) : await pickTerms();
-const [{ chunks }] = await sql.unsafe('SELECT count(*)::int AS chunks FROM content_chunks') as any;
+const [{ chunks }] = await sql.unsafe(argv.includes('--no-share') ? "SELECT reltuples::bigint::int AS chunks FROM pg_class WHERE relname = 'content_chunks'" : 'SELECT count(*)::int AS chunks FROM content_chunks') as any;
+let queryVector: Float32Array | undefined;
+const vectorFor = async () => queryVector ??= new Float32Array(JSON.parse((await sql.unsafe("SELECT embedding::text AS e FROM content_chunks WHERE embedding IS NOT NULL ORDER BY id LIMIT 1 OFFSET 4242") as any)[0].e));
const liveSources = (await sql.unsafe('SELECT id FROM sources WHERE NOT archived ORDER BY id') as Array<{ id: string }>).map(r => r.id);
const { createEngine } = await import(join(tree, 'src/core/engine-factory.ts'));
@@ -79,6 +81,11 @@ const calls: Record<string, (term: string) => Promise<unknown[] | number>> = {
pages: term => engine.searchKeywordPages(term, undefined, { limit: 20 }),
pages_remote: term => engine.searchKeywordPages(term, { excludePrivate: true, requireSafeChunks: true }, { limit: 20 }),
count: term => engine.countKeywordPages(term, {}),
+ count_remote: term => engine.countKeywordPages(term, { excludePrivate: true, requireSafeChunks: true, sourceIds: liveSources }),
+ vector: async () => engine.searchVector(await vectorFor(), { limit: 50 }),
+ vector_remote: async () => engine.searchVector(await vectorFor(), { limit: 50, excludePrivate: true, requireSafeChunks: true, sourceIds: liveSources }),
+ titles: term => engine.searchTitles(term, { limit: 50 }),
+ titles_remote: term => engine.searchTitles(term, { limit: 50, excludePrivate: true, requireSafeChunks: true, sourceIds: liveSources }),
};
const only = flag('--calls')?.split(',');
if (only) for (const name of Object.keys(calls)) if (!only.includes(name)) delete calls[name];
diff --git a/src/commands/doctor/checks/keyword-plan.ts b/src/commands/doctor/checks/keyword-plan.ts
index a53d1f08e3..4285335393 100644
--- a/src/commands/doctor/checks/keyword-plan.ts
+++ b/src/commands/doctor/checks/keyword-plan.ts
@@ -119,7 +119,7 @@ export async function keywordPlanCheck(engine: BrainEngine, probe: KeywordPlanPr
const negativeCache = { ttl_seconds: ttlSeconds, timeouts: stats?.timeouts ?? 0, skips: stats?.skips ?? 0, last_timeout_at: stats?.last_timeout_at ?? null };
const recorded = negativeCache.timeouts > 0
? ` Hybrid search's keyword arm hit this bound ${negativeCache.timeouts} time(s) (last ${negativeCache.last_timeout_at ?? 'unknown'}); ${negativeCache.skips} later search(es) skipped the timed-out statement (counted at each timeout write). `
- + (ttlSeconds > 0 ? `A serving process skips the exact statement that timed out for ${ttlSeconds} s (${KEYWORD_TIMEOUT_NEGATIVE_CACHE_KEY}; 0 turns it off), reporting keyword_arm_failed: skipped_after_timeout.` : `The skip is off (${KEYWORD_TIMEOUT_NEGATIVE_CACHE_KEY} = 0).`)
+ + (ttlSeconds > 0 ? `A serving process skips the exact statement that timed out for at most ${ttlSeconds} s (${KEYWORD_TIMEOUT_NEGATIVE_CACHE_KEY}; 0 turns it off), reporting keyword_arm_failed: skipped_after_timeout, and ends the skip once a background re-run of the statement answers.` : `The skip is off (${KEYWORD_TIMEOUT_NEGATIVE_CACHE_KEY} = 0).`)
: '';
const base = { timeout_ms: timeoutMs, canary_timeout_ms: KEYWORD_PLAN_CANARY_TIMEOUT_MS, docs: DOCS, negative_cache: negativeCache };
const chunks = await probe.chunkEstimate();
diff --git a/src/commands/serve.ts b/src/commands/serve.ts
index 7927b4e757..589cc1f778 100644
--- a/src/commands/serve.ts
+++ b/src/commands/serve.ts
@@ -287,6 +287,8 @@ async function serveBrain(
const isHttp = args.includes('--http');
// Wakeshift (docs/guides/wakeshift.md): the 5 s wake drainer for stdio and HTTP alike; a no-op tick while reporting back is off.
(await import('../core/wakeshift/drain.ts')).startWakeDrain(engine);
+ // #7082: a keyword statement that timed out is re-run in the background so its skip ends once the cache is warm.
+ (await import('../core/postgres-engine/keyword-timeout-cache.ts')).setKeywordTimeoutReruns(true);
// MEMORY_VERBS v1: tool-surface mode. stdio: GBRAIN_SURFACE > --surface >
// config `mcp_surface` > 'full'; --http ignores GBRAIN_SURFACE (an HTTP
@@ -695,6 +697,7 @@ function installStdioLifecycle(
const beginShutdown = (reason: string): void => {
if (shuttingDown) return;
shuttingDown = true;
+ void import('../core/postgres-engine/keyword-timeout-cache.ts').then(m => m.setKeywordTimeoutReruns(false));
// Stop the parent-watchdog interval as soon as a shutdown begins so
// it cannot fire a redundant 'parent-died' shutdown while the first
diff --git a/src/core/config.ts b/src/core/config.ts
index bcff8b38f7..3a3db8dd85 100644
--- a/src/core/config.ts
+++ b/src/core/config.ts
@@ -1402,7 +1402,7 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
'search.hnsw_iterative_scan',
'search.cjk_keyword_deadline_ms',
'search.keyword_statement_timeout_ms',
- 'search.keyword_timeout_negative_cache_seconds', // #6495: seconds, default 600, 0 = off (postgres-engine/keyword-timeout-cache.ts)
+ 'search.keyword_timeout_negative_cache_seconds', // #6495: the longest skip in seconds, default 600, 0 = off; #7082: a serve's background re-run ends it sooner (postgres-engine/keyword-timeout-cache.ts)
'search.adaptive_return',
'search.adaptive_return_entity_max',
'search.adaptive_return_other_max',
diff --git a/src/core/error-registry.ts b/src/core/error-registry.ts
index 5df942ee06..9df90d126b 100644
--- a/src/core/error-registry.ts
+++ b/src/core/error-registry.ts
@@ -645,7 +645,7 @@ export const CODES = {
suggestion: 'Rerun the same plain gbrain sync --source <id> --no-pull; it resumes the stopped entry under a fresh request. gbrain sync status --source <id> --json shows what remains.',
fix: { argv: ['gbrain', 'sync', 'status', '--source', '{source_id}', '--json'], consent: [], actor: 'agent', requires_exclusive: false, why: 'Shows the cursor and the next step; read-only. A plain sync of the source resumes the stopped entry.' } },
keyword_candidates_incomplete: { class: 'unavailable', summary: "Hybrid search's keyword arm returned a bounded result on a large corpus (the CJK arm's deadline or candidate cap, or a common-word match ranked from a sample), so some keyword matches can be missing.", why: 'The CJK fallback is an unindexed LIKE scan, and a query whose words match hundreds of thousands of chunks ranks a bounded sample of them (an OR retry also keeps only its rarer words); each bound keeps a large brain from stalling every hybrid search. An incomplete keyword search is not evidence of absence.', docs: 'docs/guides/multi-language-fts.md#cjk-keyword-budget', suggestion: 'Add a rarer word or an exact "phrase", or narrow the search with --source-id (MCP source_id) or --types; for CJK text, ask the user to raise search.cjk_keyword_deadline_ms.', fix: { argv: ['gbrain', 'config', 'get', 'search.cjk_keyword_deadline_ms'], consent: [], actor: 'agent', why: 'Shows the deadline in effect (unset means 3000 ms); read-only.', requires_exclusive: false } },
- keyword_skipped_after_timeout: { class: 'unavailable', summary: "Hybrid search's Postgres keyword arm skipped a statement that hit the keyword statement timeout within the negative-cache TTL; the search reports keyword_arm_failed with reason skipped_after_timeout and the vector arm still answers.", why: 'Running the same broad keyword statement again would hold a backend for the whole search.keyword_statement_timeout_ms and fail the same way. The skip covers that exact statement and scope only, lasts search.keyword_timeout_negative_cache_seconds (default 600, 0 turns it off) and is per serving process. A skipped keyword arm is not evidence of absence.', docs: 'docs/guides/troubleshooting.md#keyword-search-times-out-on-a-large-postgres-brain', suggestion: 'Narrow the search (source_id, types, since or until) to run keyword search now, or wait for the entry to expire; the user may raise search.keyword_statement_timeout_ms or change the TTL.', fix: { argv: ['gbrain', 'doctor', '--only', 'keyword_plan', '--json'], consent: [], actor: 'agent', why: 'Read-only: reports the keyword statement bound, the recorded keyword timeouts and skips, and whether broad terms can be ranked inside the bound.', requires_exclusive: false, verify: { argv: ['gbrain', 'doctor', '--only', 'keyword_plan', '--json'] } } },
+ keyword_skipped_after_timeout: { class: 'unavailable', summary: "Hybrid search's Postgres keyword arm skipped a statement that hit the keyword statement timeout within the negative-cache TTL; the search reports keyword_arm_failed with reason skipped_after_timeout and the vector arm still answers.", why: 'Running the same broad keyword statement again would hold a backend for the whole search.keyword_statement_timeout_ms and fail the same way. The skip covers that exact statement and scope only, lasts at most search.keyword_timeout_negative_cache_seconds (default 600, 0 turns it off), ends as soon as a background re-run of the statement in the serve answers, and is per serving process. A skipped keyword arm is not evidence of absence.', docs: 'docs/guides/troubleshooting.md#keyword-search-times-out-on-a-large-postgres-brain', suggestion: 'Narrow the search (source_id, types, since or until) to run keyword search now, or wait for the entry to expire; the user may raise search.keyword_statement_timeout_ms or change the TTL.', fix: { argv: ['gbrain', 'doctor', '--only', 'keyword_plan', '--json'], consent: [], actor: 'agent', why: 'Read-only: reports the keyword statement bound, the recorded keyword timeouts and skips, and whether broad terms can be ranked inside the bound.', requires_exclusive: false, verify: { argv: ['gbrain', 'doctor', '--only', 'keyword_plan', '--json'] } } },
projection_pending: { class: 'retryable', summary: "The current text projection is not ready." },
proposal_closed: { class: 'caller', summary: "The memory proposal is no longer awaiting the owner (approved, denied, expired, cancelled, failed or in conflict), so it was not approved or denied again.", why: 'A proposal is decided once; the decision and the receipt it left stay as they are.', docs: 'docs/guides/qm.md#proposed-writes', suggestion: 'Read how it ended with gbrain memory-proposals show <request_id>; the proposer sends a new proposal if the change is still wanted.' },
proposal_expired: { class: 'caller', summary: "The memory proposal passed its TTL (proposals.ttl_hours) before the owner decided, so nothing was written.", why: 'A held proposal is only as current as the page QM showed the person; an old one is closed rather than applied.', docs: 'docs/guides/qm.md#proposed-writes', suggestion: 'The proposer sends a new proposal if the change is still wanted; the operator raises proposals.ttl_hours to give owners longer.' },
diff --git a/src/core/interop-notices.ts b/src/core/interop-notices.ts
index c5714c460f..fc32a54959 100644
--- a/src/core/interop-notices.ts
+++ b/src/core/interop-notices.ts
@@ -124,7 +124,7 @@ function rerankFix(reason: RerankFailedReason, retry?: { op: string; args: Recor
}
/** #6495: `keyword_arm_failed` with reason `skipped_after_timeout` (postgres-engine/keyword-timeout-cache.ts). */
-const KEYWORD_SKIPPED_WHY = 'the keyword arm was skipped: this exact keyword search hit the keyword statement timeout (search.keyword_statement_timeout_ms) within the last search.keyword_timeout_negative_cache_seconds (default 600 seconds), so the brain did not wait for the timeout again; a narrower query (source_id, types, since or until) runs keyword search, and the skip ends by itself when the entry expires';
+const KEYWORD_SKIPPED_WHY = 'the keyword arm was skipped: this exact keyword search hit the keyword statement timeout (search.keyword_statement_timeout_ms) within the last search.keyword_timeout_negative_cache_seconds (default 600 seconds), so the brain did not wait for the timeout again; a narrower query (source_id, types, since or until) runs keyword search, and the skip ends by itself when a background re-run in the serve of the statement answers or the entry expires';
const KEYWORD_PLAN_VERIFY = { argv: ['gbrain', 'doctor', '--only', 'keyword_plan', '--json'] };
function keywordPlanFix(): Action {
return {
diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts
index 0be9949dde..436c417ca8 100644
--- a/src/core/postgres-engine.ts
+++ b/src/core/postgres-engine.ts
@@ -54,7 +54,7 @@ import {
} from './retry.ts';
import { isConnectionEndedError } from './retry-matcher.ts';
import { CheckoutGauge, PoisonedDiscardCounter, PoolIncidentCounter, type PoolGaugeSnapshot } from './pool-gauge.ts';
-import { driverPoolStats, type DriverPoolStats } from './postgres-engine/pool-stats.ts';
+import { driverPoolStats, enginePoolStats, type DriverPoolStats } from './postgres-engine/pool-stats.ts';
import {
valueHash,
normalizeDimension,
@@ -163,7 +163,7 @@ import type { BodyFlipInput, BodyFlipResult, BodyRegion, BodyStagingInput, BodyS
import type { SegmentDerivationInput, SegmentDerivationKind, SegmentDerivationRow } from './page-body/derivations.ts';
import { hasCJK } from './cjk.ts';
import { searchKeywordCJK as searchKeywordCJKImpl } from './engine-sql/cjk-search.ts';
-import { KeywordTimeoutNegativeCache } from './postgres-engine/keyword-timeout-cache.ts';
+import { KeywordTimeoutNegativeCache, keywordRerunPoolIdle } from './postgres-engine/keyword-timeout-cache.ts';
import * as titlesImpl from './engine-sql/titles.ts';
import * as keywordPagesImpl from './engine-sql/keyword-pages.ts';
import type { CjkKeywordCtx } from './search/cjk-keyword-sql.ts';
@@ -219,7 +219,7 @@ export class PostgresEngine implements BrainEngine {
readonly [ENGINE_ROOT]: object = this;
private vectorIterativeScan?: Promise<boolean>;
private keywordStatementTimeout?: Promise<string>;
- private keywordTimeouts = new KeywordTimeoutNegativeCache();
+ private keywordTimeouts = new KeywordTimeoutNegativeCache(Date.now, undefined, () => keywordRerunPoolIdle(enginePoolStats(this), this._disconnectPromise)); // #7082
private readonly vectorScope = vectorScopeLoader(() => this.executeRaw<PageSourceStats>(PAGE_SOURCE_STATS_SQL), ids => this.executeRaw<ScopeChunkCount>(SCOPE_CHUNKS_SQL, [ids]));
/** Transaction clones keep chunk invalidation and replacement atomic. */
private _chunkWritesInTransaction = false;
diff --git a/src/core/postgres-engine/keyword-timeout-cache.ts b/src/core/postgres-engine/keyword-timeout-cache.ts
index 65be0054b7..d86270c178 100644
--- a/src/core/postgres-engine/keyword-timeout-cache.ts
+++ b/src/core/postgres-engine/keyword-timeout-cache.ts
@@ -24,6 +24,22 @@
* (MAX_ENTRIES, oldest dropped first). A caller that passes `useCache: false`
* to hybrid search does not consult the cache.
*
+ * Background re-runs (#7082): a timeout on a cold cache is often gone a
+ * minute later, while the TTL kept every search keyword-less for its whole
+ * length. A serving process turns re-runs on (`setKeywordTimeoutReruns`,
+ * from `gbrain serve`); each recorded timeout then re-runs the same statement, scope, settings and bound
+ * in the background KEYWORD_TIMEOUT_RERUN_FIRST_MS after the timeout, doubling
+ * the wait after every failed or skipped re-run and never scheduling one past
+ * the entry's expiry, so the TTL stays the longest skip. The first re-run that
+ * answers deletes the entry. Foreground searches skip exactly as before while
+ * the entry stands. A re-run starts only when the gate reports an open engine
+ * whose pool keeps a connection free beside it and has no waiting query (otherwise it waits for
+ * the next step instead of queueing), at most one runs per engine, its rows
+ * are thrown away, it writes no counter row, and it runs in the serve's
+ * start-up async context, never in the request that recorded the timeout, so
+ * no request signal, connection budget or capture rides along. Shutdown
+ * (`setKeywordTimeoutReruns(false)`) lets no scheduled re-run start.
+ *
* Cost: a search that never times out does nothing extra (no statement, no
* config read). The TTL is read once, on the first timeout after the engine
* is created; each timeout then writes one bounded counter row
@@ -34,7 +50,9 @@
*
* Postgres only: PGLite runs keyword search with no statement timeout.
*/
+import { AsyncResource } from 'node:async_hooks';
import { createHash } from 'node:crypto';
+import type { DriverPoolStats } from './pool-stats.ts';
export const KEYWORD_TIMEOUT_NEGATIVE_CACHE_KEY = 'search.keyword_timeout_negative_cache_seconds';
export const KEYWORD_TIMEOUT_NEGATIVE_CACHE_DEFAULT_S = 600;
@@ -42,6 +60,8 @@ export const KEYWORD_TIMEOUT_NEGATIVE_CACHE_MAX_S = 86_400;
/** The bounded counter row a timeout writes and doctor `keyword_plan` reads (not a `config set` key). */
export const KEYWORD_TIMEOUT_STATS_KEY = 'search.keyword_timeout_stats';
export const KEYWORD_TIMEOUT_NEGATIVE_CACHE_MAX_ENTRIES = 256;
+/** A serve's first background re-run of a timed-out statement starts this long after the timeout; each failed or skipped one doubles it. */
+export const KEYWORD_TIMEOUT_RERUN_FIRST_MS = 30_000;
const COUNTER_CAP = 1_000_000_000;
/** The configured seconds when `raw` is an integer in 0..86400, else undefined. */
@@ -104,13 +124,48 @@ export const KEYWORD_TIMEOUT_STATS_UPSERT_SQL = `INSERT INTO config (key, value)
export interface KeywordTimeoutDb { executeRaw<T>(sql: string, params?: unknown[]): Promise<T[]> }
+/** Starts `fn` after `ms` and returns its cancel; the default never keeps the process alive. */
+export type KeywordRerunScheduler = (fn: () => void, ms: number) => () => void;
+
+const unrefTimer: KeywordRerunScheduler = (fn, ms) => {
+ const timer = setTimeout(fn, ms);
+ timer.unref?.();
+ return () => clearTimeout(timer);
+};
+
+let rerunsOn = false;
+let detached: (fn: () => void) => void = fn => fn();
+
+/**
+ * A serving process turns background re-runs on once, at start-up, and off when it starts shutting down, after which
+ * no scheduled re-run starts. Re-runs start in the async context of the call that turned them on, never in the
+ * search that recorded the timeout, so no request's signal, connection budget or capture rides along.
+ */
+export function setKeywordTimeoutReruns(on: boolean): void {
+ rerunsOn = on;
+ if (on) detached = AsyncResource.bind((fn: () => void) => fn());
+}
+
+/**
+ * The engine's gate: an open engine whose pool (its own, or the process pool a serve opens) has no waiting query and
+ * keeps a connection free for foreground queries while the re-run holds one.
+ */
+export function keywordRerunPoolIdle(pool: DriverPoolStats | null, disconnecting: unknown): boolean {
+ return pool !== null && !disconnecting && pool.checked_out + 1 < pool.max && pool.waiters === 0;
+}
+
+interface Entry { expiresAt: number; rerunAfterMs: number; exec: () => Promise<unknown>; cancel?: () => void }
+
/** One engine's negative cache (see the module comment). */
export class KeywordTimeoutNegativeCache {
- private entries = new Map<string, number>();
+ private entries = new Map<string, Entry>();
private ttlMs?: Promise<number>;
private pendingSkips = 0;
+ private rerunning = false;
- constructor(private readonly now: () => number = Date.now) {}
+ /** `canRerun`: whether this engine may start a background re-run now (keywordRerunPoolIdle for Postgres). */
+ constructor(private readonly now: () => number = Date.now, private readonly schedule: KeywordRerunScheduler = unrefTimer,
+ private readonly canRerun: () => boolean = () => false) {}
/**
* Run one keyword statement. With `enabled` false (no hybrid opt-in, or a
@@ -121,38 +176,76 @@ export class KeywordTimeoutNegativeCache {
async run<T>(enabled: boolean, db: KeywordTimeoutDb, keyParts: unknown[], exec: () => Promise<T>): Promise<T> {
if (!enabled) return exec();
const key = createHash('sha256').update(JSON.stringify(keyParts)).digest('hex');
- const expiresAt = this.entries.get(key);
- if (expiresAt !== undefined) {
- const remaining = expiresAt - this.now();
+ const entry = this.entries.get(key);
+ if (entry !== undefined) {
+ const remaining = entry.expiresAt - this.now();
if (remaining > 0) {
this.pendingSkips++;
throw new KeywordSkippedAfterTimeoutError(Math.ceil(remaining / 1000));
}
- this.entries.delete(key);
+ this.drop(key);
}
try {
return await exec();
} catch (error) {
- if (isGenuineStatementTimeout(error)) await this.recordTimeout(key, db);
+ if (isGenuineStatementTimeout(error)) await this.recordTimeout(key, db, exec);
throw error;
}
}
- private async recordTimeout(key: string, db: KeywordTimeoutDb): Promise<void> {
+ private drop(key: string): void {
+ this.entries.get(key)?.cancel?.();
+ this.entries.delete(key);
+ }
+
+ private async recordTimeout(key: string, db: KeywordTimeoutDb, exec: () => Promise<unknown>): Promise<void> {
const ttlMs = await (this.ttlMs ??= db.executeRaw<{ value: string }>(KEYWORD_TIMEOUT_TTL_CONFIG_SQL)
.then(rows => (keywordTimeoutNegativeCacheSeconds(rows[0]?.value) ?? KEYWORD_TIMEOUT_NEGATIVE_CACHE_DEFAULT_S) * 1000)
.catch(() => KEYWORD_TIMEOUT_NEGATIVE_CACHE_DEFAULT_S * 1000));
if (ttlMs > 0) {
- this.entries.delete(key);
- this.entries.set(key, this.now() + ttlMs);
+ this.drop(key);
+ this.entries.set(key, { expiresAt: this.now() + ttlMs, rerunAfterMs: KEYWORD_TIMEOUT_RERUN_FIRST_MS, exec });
for (const oldest of this.entries.keys()) {
if (this.entries.size <= KEYWORD_TIMEOUT_NEGATIVE_CACHE_MAX_ENTRIES) break;
- this.entries.delete(oldest);
+ this.drop(oldest);
}
+ this.scheduleRerun(key);
}
const skips = this.pendingSkips;
this.pendingSkips = 0;
await db.executeRaw(KEYWORD_TIMEOUT_STATS_UPSERT_SQL, [skips, new Date(this.now()).toISOString()])
.catch(() => { this.pendingSkips += skips; });
}
+
+ /** The next re-run of `key`, unless re-runs are off or it would start after the entry expires anyway. */
+ private scheduleRerun(key: string): void {
+ const entry = this.entries.get(key);
+ if (!entry || !rerunsOn || this.now() + entry.rerunAfterMs >= entry.expiresAt) return;
+ entry.cancel = this.schedule(() => detached(() => void this.rerun(key, entry)), entry.rerunAfterMs);
+ }
+
+ /**
+ * One background re-run: the same statement, scope, settings and bound, its rows thrown away. Success ends the
+ * skip; a failure, or a moment with no free connection or another re-run in flight, doubles the wait instead.
+ */
+ private async rerun(key: string, entry: Entry): Promise<void> {
+ entry.cancel = undefined;
+ if (this.entries.get(key) !== entry || !rerunsOn) return;
+ if (this.rerunning || !this.canRerun()) return this.backOff(key, entry);
+ this.rerunning = true;
+ try {
+ await entry.exec();
+ if (this.entries.get(key) === entry) this.entries.delete(key);
+ } catch {
+ this.backOff(key, entry);
+ } finally {
+ this.rerunning = false;
+ }
+ }
+
+ private backOff(key: string, entry: Entry): void {
+ if (this.entries.get(key) !== entry) return;
+ entry.rerunAfterMs *= 2;
+ this.scheduleRerun(key);
+ }
}
diff --git a/test/keyword-timeout-rerun-7082.test.ts b/test/keyword-timeout-rerun-7082.test.ts
new file mode 100644
index 0000000000..9562934ee9
--- /dev/null
+++ b/test/keyword-timeout-rerun-7082.test.ts
@@ -0,0 +1,257 @@
+/**
+ * #7082: the keyword timeout skip ends once a serve's background re-run of the same statement succeeds.
+ *
+ * Contract protected: after a keyword statement hits the statement timeout, foreground searches still skip it
+ * (`keyword_arm_failed: skipped_after_timeout`). In a serving process only, the cache re-runs that exact
+ * statement in the background 30 s later, then after 60 s, 120 s and so on while each re-run fails, never past
+ * the entry's TTL (`search.keyword_timeout_negative_cache_seconds`, still the longest skip). A re-run whose
+ * statement answers ends the skip, so the next search runs the keyword statement again. At most one re-run is
+ * in flight per engine, a re-run starts only when the gate reports an idle connection (otherwise it waits for
+ * the next step instead of queueing), none starts once shutdown begins, its rows are thrown away, it writes no timeout counter row, and it runs
+ * outside the async context of the search that recorded the timeout. Regression that fails it: master keeps
+ * the skip for the whole TTL even when the statement would now answer, so a cold-cache timeout costs every
+ * search its keyword results for ten minutes.
+ *
+ * Unit arm: the cache over a manual scheduler and clock, and the engine's pool gate. Postgres arm (DATABASE_URL): a held lock makes the
+ * real keyword statement time out; once the lock is released the re-run answers through the real pool gate.
+ */
+import { AsyncLocalStorage } from 'node:async_hooks';
+import { afterAll, afterEach, beforeAll, describe, expect, test } from 'bun:test';
+import postgres from '#postgres';
+import { PostgresEngine } from '../src/core/postgres-engine.ts';
+import {
+ KEYWORD_TIMEOUT_RERUN_FIRST_MS,
+ KEYWORD_TIMEOUT_STATS_KEY,
+ KeywordSkippedAfterTimeoutError,
+ KeywordTimeoutNegativeCache,
+ keywordRerunPoolIdle,
+ setKeywordTimeoutReruns,
+ type KeywordRerunScheduler,
+} from '../src/core/postgres-engine/keyword-timeout-cache.ts';
+import { KEYWORD_STATEMENT_TIMEOUT_KEY } from '../src/core/postgres-engine/keyword-statement-timeout.ts';
+import type { SearchOpts } from '../src/core/types.ts';
+import { installFixtureChunks } from './helpers/page-projection.ts';
+import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';
+import { testBackends } from './helpers/test-backends.ts';
+
+const TIMEOUT = Object.assign(new Error('canceling statement due to statement timeout'), { code: '57014' });
+const CACHE: SearchOpts = { keywordTimeoutCache: true };
+
+/** A scheduler that runs nothing until the test fires it; `pending` lists each scheduled delay. */
+function manualScheduler() {
+ const jobs: Array<{ fn: () => void; ms: number; cancelled: boolean }> = [];
+ const schedule: KeywordRerunScheduler = (fn, ms) => {
+ const job = { fn, ms, cancelled: false };
+ jobs.push(job);
+ return () => { job.cancelled = true; };
+ };
+ const live = () => jobs.filter(j => !j.cancelled);
+ return {
+ schedule,
+ pending: () => live().map(j => j.ms),
+ /** Runs every live job once (removing it) and waits for the re-runs they start to settle. */
+ fire: async () => {
+ const due = live();
+ jobs.splice(0, jobs.length);
+ for (const job of due) job.fn();
+ for (let i = 0; i < 5; i++) await Promise.resolve();
+ await new Promise(resolve => setTimeout(resolve, 0));
+ },
+ };
+}
+
+function setup(opts: { ttlSeconds?: number; canRerun?: () => boolean; serving?: boolean } = {}) {
+ let clock = 1_000_000;
+ const sched = manualScheduler();
+ const cache = new KeywordTimeoutNegativeCache(() => clock, sched.schedule, opts.canRerun ?? (() => true));
+ const writes: string[] = [];
+ const db = { executeRaw: async (sql: string) => { writes.push(sql); return sql.includes('SELECT value FROM config') ? [{ value: String(opts.ttlSeconds ?? 600) }] : []; } };
+ setKeywordTimeoutReruns(opts.serving !== false);
+ const run = <T>(key: string, exec: () => Promise<T>) => cache.run(true, db as never, [key], exec);
+ const outcome = async (key: string, exec: () => Promise<unknown>) => {
+ try { await run(key, exec); return 'ran'; } catch (e) { return e instanceof KeywordSkippedAfterTimeoutError ? 'skipped' : 'failed'; }
+ };
+ return { cache, sched, run, outcome, writes, advance: (ms: number) => { clock += ms; } };
+}
+
+afterEach(() => setKeywordTimeoutReruns(false));
+
+describe('keyword timeout re-run (#7082): serving process', () => {
+ test('a successful background re-run ends the skip, and its rows reach no caller', async () => {
+ const s = setup();
+ let calls = 0;
+ let answers = false;
+ const exec = async () => { calls++; if (!answers) throw TIMEOUT; return ['row from call ' + calls]; };
+ expect(await s.outcome('q', exec)).toBe('failed');
+ expect(await s.outcome('q', exec)).toBe('skipped');
+ expect(s.sched.pending()).toEqual([KEYWORD_TIMEOUT_RERUN_FIRST_MS]);
+ answers = true;
+ s.advance(KEYWORD_TIMEOUT_RERUN_FIRST_MS);
+ await s.sched.fire();
+ expect(calls).toBe(2);
+ expect(s.sched.pending()).toEqual([]);
+ expect(await s.run('q', exec)).toEqual(['row from call 3']);
+ });
+
+ test('a re-run that times out doubles the wait, never schedules past the TTL, and writes no counter row', async () => {
+ const s = setup({ ttlSeconds: 300 });
+ let calls = 0;
+ const exec = async () => { calls++; throw TIMEOUT; };
+ await s.outcome('q', exec);
+ const counterWrites = () => s.writes.filter(sql => sql.includes(KEYWORD_TIMEOUT_STATS_KEY)).length;
+ expect(counterWrites()).toBe(1);
+ const delays: number[] = [];
+ for (let i = 0; i < 6 && s.sched.pending().length; i++) {
+ const [ms] = s.sched.pending();
+ delays.push(ms!);
+ s.advance(ms!);
+ await s.sched.fire();
+ }
+ expect(delays).toEqual([30_000, 60_000, 120_000]);
+ expect(calls).toBe(4);
+ expect(counterWrites()).toBe(1);
+ expect(await s.outcome('q', exec)).toBe('skipped');
+ s.advance(300_000);
+ expect(await s.outcome('q', exec)).toBe('failed');
+ });
+
+ test('no idle connection: the re-run is skipped, not queued, and the next one waits twice as long', async () => {
+ let idle = false;
+ const s = setup({ canRerun: () => idle });
+ let calls = 0;
+ const exec = async () => { calls++; if (calls === 1) throw TIMEOUT; return []; };
+ await s.outcome('q', exec);
+ await s.sched.fire();
+ expect(calls).toBe(1);
+ expect(s.sched.pending()).toEqual([60_000]);
+ idle = true;
+ await s.sched.fire();
+ expect(calls).toBe(2);
+ expect(await s.outcome('q', exec)).toBe('ran');
+ });
+
+ test('at most one re-run is in flight per engine', async () => {
+ const s = setup();
+ let release!: () => void;
+ const held = new Promise<void>(resolve => { release = resolve; });
+ let reruns = 0;
+ let first = true;
+ const slow = async () => { if (first) { first = false; throw TIMEOUT; } reruns++; await held; return []; };
+ let firstB = true;
+ const other = async () => { if (firstB) { firstB = false; throw TIMEOUT; } reruns++; return []; };
+ await s.outcome('a', slow);
+ await s.outcome('b', other);
+ await s.sched.fire();
+ expect(reruns).toBe(1);
+ expect(s.sched.pending()).toEqual([60_000]);
+ release();
+ await new Promise(resolve => setTimeout(resolve, 0));
+ await s.sched.fire();
+ expect(reruns).toBe(2);
+ });
+
+ test('once shutdown begins no scheduled re-run starts and none is scheduled', async () => {
+ const s = setup();
+ let calls = 0;
+ const exec = async () => { calls++; throw TIMEOUT; };
+ await s.outcome('q', exec);
+ expect(s.sched.pending()).toHaveLength(1);
+ setKeywordTimeoutReruns(false);
+ await s.sched.fire();
+ expect(calls).toBe(1);
+ await s.outcome('other', exec);
+ expect(s.sched.pending()).toEqual([]);
+ expect(calls).toBe(2);
+ });
+
+ test('the re-run runs outside the async context of the search that recorded the timeout', async () => {
+ const request = new AsyncLocalStorage<string>();
+ const s = setup();
+ const seen: Array<string | undefined> = [];
+ let calls = 0;
+ const exec = async () => { calls++; seen.push(request.getStore()); if (calls === 1) throw TIMEOUT; return []; };
+ await request.run('user request', () => s.outcome('q', exec));
+ await request.run('later request', () => s.sched.fire());
+ expect(seen).toEqual(['user request', undefined]);
+ });
+});
+
+describe('keyword timeout re-run (#7082): one-shot processes', () => {
+ test('without a serve turning re-runs on, nothing is scheduled and the skip lasts the TTL as before', async () => {
+ const s = setup({ serving: false, ttlSeconds: 60 });
+ await s.outcome('q', async () => { throw TIMEOUT; });
+ expect(s.sched.pending()).toEqual([]);
+ s.advance(59_000);
+ expect(await s.outcome('q', async () => [])).toBe('skipped');
+ s.advance(1_000);
+ expect(await s.outcome('q', async () => [])).toBe('ran');
+ });
+
+ test('the engine gate needs an open pool that keeps a connection free and has no waiting query', () => {
+ const pool = (checked_out: number, waiters: number, max = 3) => ({ checked_out, max, waiters, inflight_oldest_ms: 0, completed: 0 });
+ expect(keywordRerunPoolIdle(null, null)).toBe(false);
+ expect(keywordRerunPoolIdle(pool(0, 0), null)).toBe(true);
+ expect(keywordRerunPoolIdle(pool(1, 0), null)).toBe(true);
+ expect(keywordRerunPoolIdle(pool(0, 0), Promise.resolve())).toBe(false);
+ expect(keywordRerunPoolIdle(pool(2, 0), null)).toBe(false);
+ expect(keywordRerunPoolIdle(pool(0, 0, 1), null)).toBe(false);
+ expect(keywordRerunPoolIdle(pool(1, 2), null)).toBe(false);
+ });
+});
+
+if (testBackends().includes('postgres')) {
+ describe('postgres: a timed-out keyword statement answers again after a background re-run', () => {
+ let engine: PostgresEngine;
+ let databaseUrl: string;
+ let close: () => Promise<void>;
+ let locker: ReturnType<typeof postgres>;
+
+ beforeAll(async () => {
+ ({ engine, databaseUrl, close } = await isolatedPersistencePostgres(process.env.DATABASE_URL!, 'module'));
+ await engine.putPage('notes/alpha', { type: 'note', title: 'Alpha beta', compiled_truth: 'alpha beta gamma' });
+ await installFixtureChunks(engine, 'notes/alpha', [{ chunk_index: 0, chunk_text: 'alpha beta gamma', chunk_source: 'compiled_truth' }]);
+ locker = postgres(databaseUrl, { max: 1, prepare: false });
+ }, 120_000);
+ afterAll(async () => {
+ await locker?.end({ timeout: 5 }).catch(() => {});
+ await close?.();
+ });
+
+ test('skipped while the statement times out, answering after the re-run once the lock is gone', async () => {
+ await engine.setConfig(KEYWORD_STATEMENT_TIMEOUT_KEY, '1000');
+ const serving = engine;
+ const sched = manualScheduler();
+ const priv = serving as unknown as { keywordTimeouts: KeywordTimeoutNegativeCache };
+ priv.keywordTimeouts = new KeywordTimeoutNegativeCache(Date.now, sched.schedule, (priv.keywordTimeouts as unknown as { canRerun: () => boolean }).canRerun);
+ setKeywordTimeoutReruns(true);
+ let release!: () => void;
+ const released = new Promise<void>(resolve => { release = resolve; });
+ let locked!: () => void;
+ const lockedUp = new Promise<void>(resolve => { locked = resolve; });
+ const held = locker.begin(async tx => {
+ await tx.unsafe('LOCK TABLE content_chunks, pages IN ACCESS EXCLUSIVE MODE');
+ locked();
+ await released;
+ }).catch(() => {});
+ try {
+ await lockedUp;
+ await expect(serving.searchKeyword('alpha beta', CACHE)).rejects.toMatchObject({ code: '57014' });
+ await expect(serving.searchKeyword('alpha beta', CACHE)).rejects.toBeInstanceOf(KeywordSkippedAfterTimeoutError);
+ expect(sched.pending()).toEqual([KEYWORD_TIMEOUT_RERUN_FIRST_MS]);
+ release();
+ await held;
+ await sched.fire();
+ for (let i = 0; i < 50 && sched.pending().length === 0; i++) {
+ try { expect((await serving.searchKeyword('alpha beta', CACHE)).map(r => r.slug)).toEqual(['notes/alpha']); return; } catch (e) { if (!(e instanceof KeywordSkippedAfterTimeoutError)) throw e; }
+ await new Promise(resolve => setTimeout(resolve, 100));
+ }
+ throw new Error('the keyword statement was still skipped after its background re-run');
+ } finally {
+ release();
+ await held;
+ await engine.unsetConfig(KEYWORD_STATEMENT_TIMEOUT_KEY);
+ await engine.unsetConfig(KEYWORD_TIMEOUT_STATS_KEY);
+ }
+ }, 120_000);
+ });
+}
diff --git a/test/postgres-unit-arms.txt b/test/postgres-unit-arms.txt
index 20c6825d5f..929fd730e8 100644
--- a/test/postgres-unit-arms.txt
+++ b/test/postgres-unit-arms.txt
@@ -211,6 +211,7 @@ test/ingest-capture-keeps-readers.test.ts
test/init-cold-home-embedding.test.ts
test/keyword-statement-timeout-knob-6495.test.ts
test/keyword-timeout-negative-cache-6495.test.ts
+test/keyword-timeout-rerun-7082.test.ts
test/label-only-repairs.test.ts
test/label-repair-derived-stamps-gst2603.test.ts
test/lane-apply-pipelining.test.ts