Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(core): per-brain caches and search telemetry keyed by engine

garrytanec4485eb209d
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 077bc975d8..0ad00d84a4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,17 @@ credits are retained; no result has been reassigned to another provider. Origina
identifiers and attribution are available in the pre-removal Git revision
`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).
+## [Unreleased]
+
+**Mounted brains stop sharing settings and search telemetry.** In a process that serves several brains, search telemetry for every brain was written into the first brain's database, and a handful of per-brain settings were read once from whichever brain asked first and then applied to all of them. Each of those caches now belongs to its engine, and each engine gets its own telemetry writer, which retires when the engine disconnects. A process with one brain behaves exactly as before. Contributed by @bradgessler (#8454).
+
+### Itemized changes
+
+- `src/core/engine-key.ts`: `engineKey(engine)` resolves a page-transaction clone, a read-through view and a statement-deadline wrapper to the engine behind them. Every cache below keys on it.
+- `src/core/search/telemetry.ts`: one writer per engine (`getTelemetryWriter(engine)`), held weakly, flushing only into its own engine. The disconnect drain bounds every live writer, and a writer retires on its engine's disconnect or collection. `setEngine` is gone.
+- Keyed by engine: `search.track_retrieval` (`last-retrieved.ts`), `persistence.publication_ceiling_ms` and the idle bound derived from it (`publication-deadline.ts`), the minion admission policy (`admission.ts`), the `mcp.strict_params` last-known-good fallback (`validate-params.ts`), the `write_capacity_low` throttle (`capacity-snapshot.ts`), and `search.hnsw_iterative_scan` / `search.vector_legacy_guard`, which are still latched once and now latched per engine.
+- `test/engine-keyed-caches.test.ts`: two engines per setting, the second read after the first warmed the cache. All six behavior cases fail on master.
+
## [0.60.554.0] - 2026-11-02
**The admin UI loads again on installed builds, and setup, doctor and MCP verify stop failing on a set of install and serve shapes.** `/admin`, including the OAuth consent page, was blank on every installed build because `admin/dist/index.html` loaded a bundle the embedded manifest didn't carry. It is rebuilt from source, and `check:admin-embedded` now runs in verify so the two can't drift again. Migration 0.53.0's packaged-skill install no longer fails `idempotency_conflict` when a later release ships a different payload. `gbrain setup` refuses the bootstrap harness lane only when that lane wired the same configuration it would write, so one `GBRAIN_HOME` can serve several HOMEs. Doctor and the turn-rule lane now count a gbrain hook in Claude Code's managed settings. `gbrain mcp verify` saves its fixture under its own folder, so a nearby page no longer makes remember refuse it. On the MCP side, `get_versions` and `known_as_of` stop pointing at a `get_page` read that doesn't exist, schema inspection uses the caller's own source pack, and `open_loops` explains an empty answer when none of a scoped caller's sources is a Google source, without revealing anything outside its grant.
diff --git a/src/core/behavior-change-notice.ts b/src/core/behavior-change-notice.ts
index 727fa1fb23..31648eb1d9 100644
--- a/src/core/behavior-change-notice.ts
+++ b/src/core/behavior-change-notice.ts
@@ -1412,6 +1412,8 @@ export const BEHAVIOR_CHANGES: ReadonlyArray<{ since: string; text: ChangeText;
{ since: '0.60.550.0', id: 'mcp_lane_launcher_8426', text: 'The stdio MCP entry `gbrain setup claude-code|codex` and `gbrain bootstrap hooks` (opencode\'s user-global entry included) register now starts `gbrain serve` through `<gbrain config dir>/hook-launcher/gbrain-hook`, and the hooks from `gbrain bootstrap hooks`, `gbrain bootstrap harness`, the Codex SessionEnd capture and the gbrain plugins start gbrain from the launcher\'s or the plugin\'s own directory, so a session opened in a cloned repository never runs that repository\'s bunfig.toml preload or loads its .env; serve still reads .gbrain-source from the session directory. `gbrain post-upgrade` moves a Claude Code setup MCP entry setup still owns and names the command for every Codex (the turn-rule hook included) and bootstrap entry, doctor `hook_session_cwd` covers them, and `gbrain hook` git probes skip core.fsmonitor and run only in the workspace the install receipt names.' },
// GST-927 (GBRA-280) schema-ruled-out empty arguments; release:restamp stamps since at the merge slot.
{ since: '0.60.553.0', id: 'empty_nonstring_args_gst927', text: 'Read tools now treat an optional argument as not provided when its own schema rules the value out: 0 where the range starts above zero (limit: 0, around_chunk_id: 0, schema_version: 0 and others), [] where the list must not be empty (types, only, traverse_graph hops), {} where the object needs a field (get_page lines and bytes), false on get_versions diff, and get_page context without grep. These calls used to refuse with invalid_params; now they answer as if the argument were left out. get_page known_as_of beside window or view arguments still refuses, and the refusal names the arguments to drop.' },
+ // GBRA-283 Wave A (supersedes #8454): per-engine caches; release:restamp stamps since at the merge slot.
+ { since: '0.60.555.0', id: 'engine_keyed_caches_gbra283', text: 'A process serving several brains (mounts) now keeps each brain\'s search telemetry in that brain\'s own database and reads each brain\'s own search.track_retrieval, admission quotas, mcp.strict_params fallback, publication ceiling, capacity-notice throttle and the two vector-search rollback settings. Before, the first brain to be read decided these for every brain in the process, and every brain\'s search telemetry was written into the first brain\'s database. A process with one brain behaves as before.' },
];
/** The newest disclosed change's release: the notice id moves only when a release adds rows. */
diff --git a/src/core/config.ts b/src/core/config.ts
index b250ed0c00..3e6878e2f6 100644
--- a/src/core/config.ts
+++ b/src/core/config.ts
@@ -1393,7 +1393,7 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
// weak-graded query), and unlike crag_think it is reachable by remote
// callers — attacker-shaped weak queries drive that spend (ship security
// review). See docs/operations/spend-controls.md.
- // #5824 one-release rollback, latched per process (search/vector-legacy-guard.ts).
+ // #5824 one-release rollback, latched per engine (search/vector-legacy-guard.ts).
'search.vector_legacy_guard',
'search.hnsw_iterative_scan',
'search.cjk_keyword_deadline_ms',
diff --git a/src/core/engine-key.ts b/src/core/engine-key.ts
new file mode 100644
index 0000000000..815812645f
--- /dev/null
+++ b/src/core/engine-key.ts
@@ -0,0 +1,18 @@
+import { viewedEngine } from './persistence/switches.ts';
+import { engineCacheKey } from './statement-deadline.ts';
+
+/**
+ * The object a per-engine cache keys on: the engine itself, reached from a page-transaction clone (an object whose
+ * prototype is the engine that opened it), a read-through view (`registerEngineView`) or a statement-deadline wrapper.
+ * A process with several engines (mounted brains) keeps one entry per engine, and a transaction or view shares its
+ * engine's entry, so per-brain settings and buffers never cross brains.
+ */
+export function engineKey(engine: object): object {
+ let key = engine;
+ for (;;) {
+ const transaction = Object.hasOwn(key, '_pageTransaction') && (key as { _pageTransaction?: unknown })._pageTransaction === true;
+ const next = transaction ? Object.getPrototypeOf(key) as object | null : viewedEngine(engineCacheKey(key));
+ if (next === null || next === key) return key;
+ key = next;
+ }
+}
diff --git a/src/core/facts/question-recall.ts b/src/core/facts/question-recall.ts
index c0aba1a4fc..84f692bac3 100644
--- a/src/core/facts/question-recall.ts
+++ b/src/core/facts/question-recall.ts
@@ -211,7 +211,7 @@ export async function rankFactsByQuestion(ctx: OperationContext, input: Question
const [pool, uncomparable, rule] = await Promise.all([
collectFactCandidates(ctx.engine, question, scope, embedding, {
depth: { keyword: QUESTION_POOL_DEPTH, cosine: QUESTION_POOL_DEPTH, entity: QUESTION_POOL_DEPTH }, filters, cosineForAll: true,
- hnswIterativeScan: resolveHnswIterativeScan(ctx.config),
+ hnswIterativeScan: resolveHnswIterativeScan(ctx.engine, ctx.config),
}),
embedding ? countUncomparableFacts(ctx.engine, scope, filters, embedding.length) : [],
admissionRule(ctx.engine),
diff --git a/src/core/last-retrieved.ts b/src/core/last-retrieved.ts
index 936d0f1297..b4bb29f14d 100644
--- a/src/core/last-retrieved.ts
+++ b/src/core/last-retrieved.ts
@@ -48,8 +48,10 @@
import type { BrainEngine } from './engine.ts';
import { isMissingSchemaObject } from './utils.ts';
import { deferUntilResponse, registerBackgroundWorkDrainer } from './background-work.ts';
+import { engineKey } from './engine-key.ts';
-let _trackRetrievalCache: { ts: number; enabled: boolean } | null = null;
+/** Per engine, so each mounted brain honors its own setting. */
+let _trackRetrievalCache = new WeakMap<object, { ts: number; enabled: boolean }>();
const TRACK_RETRIEVAL_CACHE_TTL_MS = 30_000;
/**
@@ -147,14 +149,16 @@ registerBackgroundWorkDrainer({
});
/**
- * Resolve `search.track_retrieval` config with a 30s in-process cache so
- * hot-path callers don't pay a SELECT per search. Default-on: missing
+ * Resolve `search.track_retrieval` config with a 30s in-process cache per
+ * engine so hot-path callers don't pay a SELECT per search. Default-on: missing
* config OR unparseable value → true (D13 default).
*/
async function isTrackingEnabled(engine: BrainEngine): Promise<boolean> {
const now = Date.now();
- if (_trackRetrievalCache && now - _trackRetrievalCache.ts < TRACK_RETRIEVAL_CACHE_TTL_MS) {
- return _trackRetrievalCache.enabled;
+ const key = engineKey(engine);
+ const cached = _trackRetrievalCache.get(key);
+ if (cached && now - cached.ts < TRACK_RETRIEVAL_CACHE_TTL_MS) {
+ return cached.enabled;
}
let enabled = true;
try {
@@ -165,13 +169,13 @@ async function isTrackingEnabled(engine: BrainEngine): Promise<boolean> {
} catch {
// getConfig miss / connection blip → default to enabled (D13 default).
}
- _trackRetrievalCache = { ts: now, enabled };
+ _trackRetrievalCache.set(key, { ts: now, enabled });
return enabled;
}
/** Test seam — drops the cache so subsequent calls re-read config. */
export function _resetTrackRetrievalCacheForTests(): void {
- _trackRetrievalCache = null;
+ _trackRetrievalCache = new WeakMap();
}
/**
diff --git a/src/core/minions/admission.ts b/src/core/minions/admission.ts
index 235beddec1..cfc9c4f5f2 100644
--- a/src/core/minions/admission.ts
+++ b/src/core/minions/admission.ts
@@ -39,6 +39,7 @@
import { createHash } from 'node:crypto';
import type { BrainEngine } from '../engine.ts';
+import { engineKey } from '../engine-key.ts';
/** Names whose parentless submits coalesce on identical params by default. */
export const PARAM_COALESCE_DEFAULT: Readonly<Record<string, boolean>> = {
@@ -197,16 +198,17 @@ function parsePositiveNumber(v: string | null): number | null {
return n;
}
-// ~60s in-process cache: add() runs per submission; a config read per submit
-// would add a query to the hot path for a value that changes at human speed.
+// ~60s in-process cache per engine and job name: add() runs per submission; a
+// config read per submit would add a query to the hot path for a value that
+// changes at human speed.
type CacheEntry = { at: number; policy: AdmissionPolicy };
-const policyCache = new Map<string, CacheEntry>();
+let policyCache = new WeakMap<object, Map<string, CacheEntry>>();
const POLICY_CACHE_MS = 60_000;
let warnedFailOpen = false;
/** Test seam: drop the cache so config changes are visible immediately. */
export function _resetAdmissionCacheForTest(): void {
- policyCache.clear();
+ policyCache = new WeakMap();
warnedFailOpen = false;
}
@@ -219,7 +221,10 @@ export async function resolveAdmissionPolicy(engine: BrainEngine, jobName: strin
if (admissionKilled()) {
return { coalesceParams: false, ttlWaitingHours: null, quotaMaxWaiting: null };
}
- const cached = policyCache.get(jobName);
+ const key = engineKey(engine);
+ let byName = policyCache.get(key);
+ if (!byName) policyCache.set(key, byName = new Map());
+ const cached = byName.get(jobName);
if (cached && Date.now() - cached.at < POLICY_CACHE_MS) return cached.policy;
const policy: AdmissionPolicy = {
@@ -252,7 +257,7 @@ export async function resolveAdmissionPolicy(engine: BrainEngine, jobName: strin
);
}
}
- policyCache.set(jobName, { at: Date.now(), policy });
+ byName.set(jobName, { at: Date.now(), policy });
return policy;
}
diff --git a/src/core/persistence/capacity-snapshot.ts b/src/core/persistence/capacity-snapshot.ts
index 9bf8cb5faa..77dfdca6f7 100644
--- a/src/core/persistence/capacity-snapshot.ts
+++ b/src/core/persistence/capacity-snapshot.ts
@@ -27,6 +27,7 @@ import { BACKGROUND_PRODUCERS } from './admission-origin.ts';
import { readCapacityConfig, readCapacityForecast } from './capacity-forecast.ts';
import { CAPACITY_VERIFY } from './capacity-verify.ts';
import { deadlineBoundEngine, deadlineFailure, engineCacheKey, withStatementDeadline } from '../statement-deadline.ts';
+import { engineKey } from '../engine-key.ts';
export const CAPACITY_SNAPSHOT_TTL_MS = 60_000;
export const CAPACITY_NOTICE_INTERVAL_MS = 60 * 60_000;
@@ -51,8 +52,8 @@ export interface CapacitySnapshot {
}
const snapshots = new WeakMap<object, { at: number; value: Promise<CapacitySnapshot> }>();
-const lowShownAt = new Map<string, number>();
-const pausedShown = new Set<string>();
+/** Per engine: when each scope|resource last showed `write_capacity_low`, and which are inside the pause. */
+let noticeState = new WeakMap<object, { lowShownAt: Map<string, number>; pausedShown: Set<string> }>();
const UNKNOWN: CapacitySnapshot = { rows: [], low: [], paused: [], pause_pct: 0, status: 'unknown' };
@@ -125,7 +126,7 @@ function pausedNotice(rows: CapacityAttention[], pausePct: number): Notice {
/**
* The capacity notices one output carries. `throttle` (write and sync
* receipts) shows `write_capacity_low` at most once an hour per process per
- * scope and resource; unthrottled (sync status, doctor, cycle output) shows it
+ * brain, scope and resource; unthrottled (sync status, doctor, cycle output) shows it
* whenever a cap needs attention. The pause notice fires on each transition
* into the pause, in either mode. `scopes` limits both to the scopes a
* caller may see (a remote client: the brain and its own principal).
@@ -137,6 +138,10 @@ export async function capacityNotices(engine: SqlEngine, opts: { throttle: boole
const snapshot = opts.snapshot ?? (opts.deadlineAt !== undefined ? await capacitySnapshotWithin(engine, opts.deadlineAt, now) : await cachedCapacitySnapshot(engine, now));
if (snapshot.status === 'unknown') return [];
const visible = (row: CapacityAttention) => !opts.scopes || opts.scopes.includes(row.scope);
+ const stateKey = engineKey(engine);
+ let state = noticeState.get(stateKey);
+ if (!state) noticeState.set(stateKey, state = { lowShownAt: new Map(), pausedShown: new Set() });
+ const { lowShownAt, pausedShown } = state;
const notices: Notice[] = [];
const candidates = snapshot.low.filter(visible);
const low = opts.throttle ? candidates.filter(row => now - (lowShownAt.get(key(row)) ?? -Infinity) >= CAPACITY_NOTICE_INTERVAL_MS) : candidates;
@@ -155,8 +160,7 @@ export async function capacityNotices(engine: SqlEngine, opts: { throttle: boole
}
export function __resetCapacityNoticesForTests(): void {
- lowShownAt.clear();
- pausedShown.clear();
+ noticeState = new WeakMap();
}
/** #6696 (A4): a write receipt's capacity notices, throttled hourly per scope; a remote caller sees only the brain and its own principal. Never throws. */
diff --git a/src/core/persistence/publication-deadline.ts b/src/core/persistence/publication-deadline.ts
index 3683c73ced..c47307b864 100644
--- a/src/core/persistence/publication-deadline.ts
+++ b/src/core/persistence/publication-deadline.ts
@@ -22,6 +22,7 @@ import { AsyncLocalStorage } from 'node:async_hooks';
import type { BrainEngine, TransactionOptions } from '../engine.ts';
import { gucMilliseconds, resolveSessionTimeouts } from '../db.ts';
import { declareDurablePersistence } from './protocol.ts';
+import { engineKey } from '../engine-key.ts';
export const PUBLICATION_CEILING_KEY = 'persistence.publication_ceiling_ms';
/** `persistence.publication_ceiling_ms`' default (60000 to 3600000 accepted). */
@@ -36,23 +37,34 @@ export function publicationLeaseMs(ceilingMs: number, marginMs = 30_000): number
}
const CEILING_TTL_MS = 5_000;
-let ceiling = { value: PUBLICATION_CEILING_DEFAULT_MS, at: 0 };
+/** Per engine; a publication transaction reads its engine's entry. */
+let ceilings = new WeakMap<object, { value: number; at: number }>();
const signals = new AsyncLocalStorage<AbortSignal>();
/** Test seam (`_setPublicationTimingForTest`): a ceiling and grace below the accepted range, so a test can overrun the ceiling. */
let testTiming: { ceilingMs: number; graceMs: number } | undefined;
-/** The ceiling (cached for 5 s); anything missing or outside 60000..3600000 reads as the default. */
+/** The engine's ceiling (cached per engine for 5 s); anything missing or outside 60000..3600000 reads as the default. */
export async function readPublicationCeilingMs(engine: Pick<BrainEngine, 'getConfig'>): Promise<number> {
if (testTiming) return testTiming.ceilingMs;
- if (Date.now() - ceiling.at < CEILING_TTL_MS) return ceiling.value;
+ const key = engineKey(engine);
+ const cached = ceilings.get(key);
+ if (cached && Date.now() - cached.at < CEILING_TTL_MS) return cached.value;
const n = Number((await engine.getConfig(PUBLICATION_CEILING_KEY).catch(() => null))?.trim());
- ceiling = { value: Number.isInteger(n) && n >= 60_000 && n <= 3_600_000 ? n : PUBLICATION_CEILING_DEFAULT_MS, at: Date.now() };
- return ceiling.value;
+ const value = Number.isInteger(n) && n >= 60_000 && n <= 3_600_000 ? n : PUBLICATION_CEILING_DEFAULT_MS;
+ ceilings.set(key, { value, at: Date.now() });
+ return value;
+}
+
+/** The ceiling `engine` (or the engine its transaction runs on) last read, without a read; the default before the first. */
+function lastReadCeilingMs(engine: object): number {
+ return ceilings.get(engineKey(engine))?.value ?? PUBLICATION_CEILING_DEFAULT_MS;
}
/** Test seam: the next readPublicationCeilingMs reads the config again. */
-export function _resetPublicationCeilingForTest(): void { ceiling = { ...ceiling, at: 0 }; }
+export function _resetPublicationCeilingForTest(): void {
+ ceilings = new WeakMap();
+}
/** Test seam: every publication's ceiling and grace (undefined restores the configured ceiling and `PUBLICATION_GRACE_MS`). */
export function _setPublicationTimingForTest(timing: { ceilingMs: number; graceMs: number } | undefined): void { testTiming = timing; }
@@ -66,7 +78,7 @@ export function publicationDeadlinePassed(): boolean {
}
/** The idle-in-transaction bound a publish transaction gets: the ceiling, never above the pool's own session value. */
-export function publicationIdleTimeoutMs(ceilingMs = ceiling.value): number {
+export function publicationIdleTimeoutMs(ceilingMs: number): number {
const session = gucMilliseconds(resolveSessionTimeouts().idle_in_transaction_session_timeout);
return session !== null && session > 0 ? Math.min(ceilingMs, session) : ceilingMs;
}
@@ -78,7 +90,7 @@ export function publicationIdleTimeoutMs(ceilingMs = ceiling.value): number {
*/
export async function bindPublicationTimeouts(tx: BrainEngine): Promise<void> {
if (tx.kind !== 'postgres') return;
- await tx.executeRaw("SELECT set_config('idle_in_transaction_session_timeout',$1,true)", [`${publicationIdleTimeoutMs()}ms`]);
+ await tx.executeRaw("SELECT set_config('idle_in_transaction_session_timeout',$1,true)", [`${publicationIdleTimeoutMs(lastReadCeilingMs(tx))}ms`]);
}
/**
@@ -86,7 +98,7 @@ export async function bindPublicationTimeouts(tx: BrainEngine): Promise<void> {
* `declareDurablePersistence` in one round trip. A refused SET fails the publication's transaction as above.
*/
export async function declarePublication(tx: BrainEngine, bound: Pick<BrainEngine, 'kind'> = tx): Promise<string | null> {
- return declareDurablePersistence(tx, undefined, undefined, bound.kind === 'postgres' ? `${publicationIdleTimeoutMs()}ms` : undefined);
+ return declareDurablePersistence(tx, undefined, undefined, bound.kind === 'postgres' ? `${publicationIdleTimeoutMs(lastReadCeilingMs(tx))}ms` : undefined);
}
/**
diff --git a/src/core/search/hnsw-iterative-scan.ts b/src/core/search/hnsw-iterative-scan.ts
index f9a4ae6880..322bcc5923 100644
--- a/src/core/search/hnsw-iterative-scan.ts
+++ b/src/core/search/hnsw-iterative-scan.ts
@@ -16,6 +16,7 @@
* re-reads config. An unreadable value falls back to the default.
*/
import type { GBrainConfig } from '../config.ts';
+import { engineKey } from '../engine-key.ts';
export const HNSW_ITERATIVE_SCAN_ENV = 'GBRAIN_HNSW_ITERATIVE_SCAN';
export const HNSW_ITERATIVE_SCAN_KEY = 'search.hnsw_iterative_scan';
@@ -40,17 +41,21 @@ export function readHnswIterativeScan(
return { mode: HNSW_ITERATIVE_SCAN_DEFAULT, via: null };
}
-let resolved: HnswIterativeScanMode | undefined;
+let resolved = new WeakMap<object, HnswIterativeScanMode>();
+const logged = new Set<HnswIterativeScanMode>();
-/** First call decides for the process; logs once to stderr when an override is active. */
-export function resolveHnswIterativeScan(cfg: Pick<GBrainConfig, 'search'> | null | undefined): HnswIterativeScanMode {
- if (resolved !== undefined) return resolved;
+/** The first call for an engine decides for that engine; logs once per process per override mode. */
+export function resolveHnswIterativeScan(engine: object, cfg: Pick<GBrainConfig, 'search'> | null | undefined): HnswIterativeScanMode {
+ const key = engineKey(engine);
+ const known = resolved.get(key);
+ if (known !== undefined) return known;
const setting = readHnswIterativeScan(cfg);
- resolved = setting.mode;
- if (setting.via && setting.mode !== HNSW_ITERATIVE_SCAN_DEFAULT) {
+ resolved.set(key, setting.mode);
+ if (setting.via && setting.mode !== HNSW_ITERATIVE_SCAN_DEFAULT && !logged.has(setting.mode)) {
+ logged.add(setting.mode);
console.error(`[gbrain] vector search uses hnsw.iterative_scan=${setting.mode} (${setting.via === 'env' ? HNSW_ITERATIVE_SCAN_ENV : HNSW_ITERATIVE_SCAN_KEY}); the default is ${HNSW_ITERATIVE_SCAN_DEFAULT}.`);
}
- return resolved;
+ return setting.mode;
}
/** `config set search.hnsw_iterative_scan <value>` refusal text, or null when the value is valid. */
@@ -60,5 +65,6 @@ export function hnswIterativeScanValueProblem(value: string): string | null {
/** @internal test seam */
export function _resetHnswIterativeScanForTests(): void {
- resolved = undefined;
+ resolved = new WeakMap();
+ logged.clear();
}
diff --git a/src/core/search/hybrid/request.ts b/src/core/search/hybrid/request.ts
index f7981dead9..6cfe19d7f4 100644
--- a/src/core/search/hybrid/request.ts
+++ b/src/core/search/hybrid/request.ts
@@ -280,8 +280,8 @@ export async function resolveHybridRequest(
// via normalizeEngineColumn; the descriptor path is the strict one.
embeddingColumn: resolvedCol,
// #5824 rollback switch, latched once per process from env/config.
- vectorLegacyGuard: resolveVectorLegacyGuard(cfgForColumn),
- hnswIterativeScan: resolveHnswIterativeScan(cfgForColumn),
+ vectorLegacyGuard: resolveVectorLegacyGuard(engine, cfgForColumn),
+ hnswIterativeScan: resolveHnswIterativeScan(engine, cfgForColumn),
// #5989: bound the CJK keyword arm by one deadline; its outcome and wall time ride the meta.
cjkKeyword: {
deadlineMs: resolveCjkKeywordDeadlineMs(cfgForColumn),
diff --git a/src/core/search/telemetry.ts b/src/core/search/telemetry.ts
index 12c89e6452..757f214623 100644
--- a/src/core/search/telemetry.ts
+++ b/src/core/search/telemetry.ts
@@ -40,6 +40,7 @@
import type { BrainEngine } from '../engine.ts';
import type { HybridSearchMeta } from '../types.ts';
import { registerBackgroundWorkDrainer, type BackgroundWorkDrainMode } from '../background-work.ts';
+import { engineKey } from '../engine-key.ts';
interface Bucket {
date: string;
@@ -119,27 +120,30 @@ const FLUSH_INTERVAL_MS = 60_000;
export const FLUSH_THRESHOLD_CALLS = 100;
/**
- * Per-process telemetry singleton. Each gbrain process (CLI, stdio MCP,
- * HTTP MCP) gets one instance. The flush timer is installed lazily by the
- * first `setEngine()` call, not by `record()`, so importing this module has
- * no side effects and a caller can wire an engine without recording. No exit
- * hooks are installed — see `ensureExitHook`.
+ * One engine's telemetry writer (`getTelemetryWriter(engine)`): a process
+ * with one engine (CLI, stdio MCP, HTTP MCP) has one, a process with mounted
+ * brains has one per brain, and each flushes into its own engine's database.
+ * The writer holds its engine weakly. The flush timer is installed when the
+ * writer is created at the first search, not by importing this module. No
+ * exit hooks are installed — see `ensureExitHook`.
*/
class TelemetryWriter {
private buckets = new Map<string, Bucket>();
private pendingCount = 0;
- private engine: BrainEngine | null = null;
+ private readonly engine: WeakRef<BrainEngine>;
private timer: NodeJS.Timeout | null = null;
private exitHookInstalled = false;
private flushInFlight: Promise<void> | null = null;
- /** Wire the engine. Called once per process at search-time. Subsequent calls are a no-op. */
- setEngine(engine: BrainEngine): void {
- if (!this.engine) {
- this.engine = engine;
- this.ensureExitHook();
- this.ensureTimer();
- }
+ constructor(engine: BrainEngine) {
+ this.engine = new WeakRef(engine);
+ this.ensureExitHook();
+ this.ensureTimer();
+ }
+
+ /** The engine, or undefined once it was garbage collected (its buffer can never be written). */
+ liveEngine(): BrainEngine | undefined {
+ return this.engine.deref();
}
/**
@@ -249,7 +253,8 @@ class TelemetryWriter {
*/
async flush(): Promise<void> {
if (this.flushInFlight) return this.flushInFlight;
- if (!this.engine || this.buckets.size === 0) {
+ const engine = this.engine.deref();
+ if (!engine || this.buckets.size === 0) {
this.pendingCount = 0;
return;
}
@@ -259,7 +264,6 @@ class TelemetryWriter {
this.buckets = new Map();
this.pendingCount = 0;
- const engine = this.engine;
this.flushInFlight = (async () => {
try {
for (const b of snapshot.values()) {
@@ -298,9 +302,10 @@ class TelemetryWriter {
}
/**
- * Stop the timer and drop the buffer. Test-only in practice: the sole
- * caller is `_resetTelemetryWriterForTest`. Nothing in production shuts
- * the writer down — processes exit and the buffer goes with them.
+ * Stop the timer and drop the buffer. Called by `_resetTelemetryWriterForTest`
+ * and for a writer whose engine was garbage collected. Nothing in production
+ * shuts down a writer whose engine is alive — processes exit and the buffer
+ * goes with them.
*/
stop(): void {
if (this.timer) {
@@ -336,6 +341,7 @@ class TelemetryWriter {
private ensureTimer(): void {
if (this.timer) return;
this.timer = setInterval(() => {
+ if (!this.liveEngine()) { retireWriter(this); return; }
void this.flush().catch(() => { /* swallow */ });
}, FLUSH_INTERVAL_MS);
// Unref so the timer doesn't keep the process alive on its own.
@@ -375,12 +381,49 @@ class TelemetryWriter {
}
}
-/** Module-level singleton, one per process. */
-let _writer: TelemetryWriter | null = null;
+/**
+ * One writer per engine (a transaction or view shares its engine's); `live` lets the drain reach each one. A writer is
+ * retired when its engine disconnects (after its in-flight flush; residual buckets drop there, as the disconnect drain
+ * already drops them) or is garbage collected, so an engine opened and closed per request leaves no timer behind.
+ */
+let writers = new WeakMap<object, TelemetryWriter>();
+const live = new Set<TelemetryWriter>();
+/** Each engine's disconnect-hook removal, held weakly so the registry never keeps an engine alive. */
+const unregisters = new WeakMap<object, () => void>();
+
+export function getTelemetryWriter(engine: BrainEngine): TelemetryWriter {
+ const key = engineKey(engine) as BrainEngine;
+ let writer = writers.get(key);
+ if (!writer) {
+ const created = new TelemetryWriter(key);
+ writer = created;
+ writers.set(key, created);
+ // Never awaits: the bounded disconnect drain (awaitPendingTelemetryFlush) still sees the in-flight flush.
+ const unregister = key.registerBeforeDisconnect?.(async () => {
+ if (writers.get(key) === created) writers.delete(key);
+ const inFlight = created.pendingFlush();
+ if (inFlight) void inFlight.catch(() => undefined).finally(() => retireWriter(created));
+ else retireWriter(created);
+ });
+ if (unregister) unregisters.set(key, unregister);
+ live.add(created);
+ }
+ return writer;
+}
-export function getTelemetryWriter(): TelemetryWriter {
- if (!_writer) _writer = new TelemetryWriter();
- return _writer;
+/** Test-only — how many engines still hold a writer. */
+export function _telemetryWriterCountForTest(): number {
+ return live.size;
+}
+
+function retireWriter(writer: TelemetryWriter): void {
+ writer.stop();
+ const engine = writer.liveEngine();
+ if (engine) {
+ unregisters.get(engine)?.();
+ unregisters.delete(engine);
+ }
+ live.delete(writer);
}
/**
@@ -401,11 +444,9 @@ export async function awaitPendingTelemetryFlush(
timeoutMs = 2000,
mode: BackgroundWorkDrainMode = 'disconnect',
): Promise<{ unfinished: number }> {
- const w = _writer;
- if (!w) return { unfinished: 0 };
- const inFlight = w.pendingFlush();
- const wantsResidual = mode === 'exit' && w.hasBuffered();
- if (!inFlight && !wantsResidual) return { unfinished: 0 };
+ for (const w of [...live]) if (!w.liveEngine()) retireWriter(w);
+ const pending = [...live].filter(w => w.pendingFlush() || (mode === 'exit' && w.hasBuffered()));
+ if (pending.length === 0) return { unfinished: 0 };
let timer: ReturnType<typeof setTimeout> | undefined;
const timedOut = Symbol('telemetry-drain-timeout');
@@ -413,18 +454,17 @@ export async function awaitPendingTelemetryFlush(
timer = setTimeout(() => resolve(timedOut), timeoutMs);
timer.unref?.();
});
+ const drainOne = async (w: TelemetryWriter): Promise<void> => {
+ await w.pendingFlush()?.catch(() => undefined);
+ if (mode === 'exit' && w.hasBuffered()) await w.flush().catch(() => undefined);
+ };
+ const settled = new Set<TelemetryWriter>();
try {
- if (inFlight) {
- const r = await Promise.race([inFlight.catch(() => undefined), bound]);
- if (r === timedOut) return { unfinished: 1 };
- }
- if (mode === 'exit' && w.hasBuffered()) {
- const r = await Promise.race([w.flush().catch(() => undefined), bound]);
- if (r === timedOut) return { unfinished: 1 };
- }
- return { unfinished: 0 };
+ const all = Promise.all(pending.map(w => drainOne(w).then(() => { settled.add(w); })));
+ const r = await Promise.race([all, bound]);
+ return { unfinished: r === timedOut ? pending.length - settled.size : 0 };
} catch {
- return { unfinished: 1 };
+ return { unfinished: pending.length - settled.size };
} finally {
if (timer) clearTimeout(timer);
}
@@ -450,9 +490,7 @@ export function recordSearchTelemetry(
opts: { results_count: number; tokens_estimate?: number; rank1_score?: number } = { results_count: 0 },
): void {
try {
- const w = getTelemetryWriter();
- w.setEngine(engine);
- w.record(meta, opts);
+ getTelemetryWriter(engine).record(meta, opts);
} catch {
// swallow — telemetry is best-effort.
}
@@ -712,12 +750,10 @@ function nowDate(): string {
return new Date().toISOString().slice(0, 10);
}
-/** Test-only — reset the module-level singleton between test cases. */
+/** Test-only — drop every engine's writer between test cases. */
export function _resetTelemetryWriterForTest(): void {
- if (_writer) {
- _writer.stop();
- _writer = null;
- }
+ for (const w of [...live]) retireWriter(w);
+ writers = new WeakMap();
}
// ---------------------------------------------------------------------------
diff --git a/src/core/search/vector-legacy-guard.ts b/src/core/search/vector-legacy-guard.ts
index d2bb2268ce..96e10c9c24 100644
--- a/src/core/search/vector-legacy-guard.ts
+++ b/src/core/search/vector-legacy-guard.ts
@@ -5,11 +5,12 @@
* wave.
*
* Read from `GBRAIN_VECTOR_LEGACY_GUARD` (wins when set) or the config key
- * `search.vector_legacy_guard`, resolved once per process by the caller that
+ * `search.vector_legacy_guard`, resolved once per engine by the caller that
* builds search options, so the owning service (serve, autopilot, worker)
* picks a change up on restart and the hot path never re-reads config.
*/
import type { GBrainConfig } from '../config.ts';
+import { engineKey } from '../engine-key.ts';
export const VECTOR_LEGACY_GUARD_ENV = 'GBRAIN_VECTOR_LEGACY_GUARD';
export const VECTOR_LEGACY_GUARD_KEY = 'search.vector_legacy_guard';
@@ -27,7 +28,7 @@ function parseFlag(value: string | undefined): boolean | undefined {
return undefined;
}
-/** What the setting says right now (doctor reports it; search uses the per-process latch). */
+/** What the setting says right now (doctor reports it; search uses the per-engine latch). */
export function readVectorLegacyGuard(cfg: Pick<GBrainConfig, 'search'> | null | undefined, env: NodeJS.ProcessEnv = process.env): VectorLegacyGuardSetting {
const fromEnv = parseFlag(env[VECTOR_LEGACY_GUARD_ENV]);
if (fromEnv !== undefined) return { enabled: fromEnv, via: 'env' };
@@ -35,21 +36,26 @@ export function readVectorLegacyGuard(cfg: Pick<GBrainConfig, 'search'> | null |
return { enabled: false, via: null };
}
-let resolved: boolean | undefined;
+let resolved = new WeakMap<object, boolean>();
+let logged = false;
-/** First call decides for the process; logs once to stderr when the guard is active. */
-export function resolveVectorLegacyGuard(cfg: Pick<GBrainConfig, 'search'> | null | undefined): boolean {
- if (resolved !== undefined) return resolved;
+/** The first call for an engine decides for that engine; logs once per process when the guard is active. */
+export function resolveVectorLegacyGuard(engine: object, cfg: Pick<GBrainConfig, 'search'> | null | undefined): boolean {
+ const key = engineKey(engine);
+ const known = resolved.get(key);
+ if (known !== undefined) return known;
const setting = readVectorLegacyGuard(cfg);
- resolved = setting.enabled;
- if (resolved) {
+ resolved.set(key, setting.enabled);
+ if (setting.enabled && !logged) {
+ logged = true;
console.error(`[gbrain] vector legacy guard active (${setting.via === 'env' ? VECTOR_LEGACY_GUARD_ENV : VECTOR_LEGACY_GUARD_KEY}): `
+ 'vector search keeps the pre-#5824 freshness guard inside the HNSW candidate scan; remove it after upgrading.');
}
- return resolved;
+ return setting.enabled;
}
/** @internal test seam */
export function _resetVectorLegacyGuardForTests(): void {
- resolved = undefined;
+ resolved = new WeakMap();
+ logged = false;
}
diff --git a/src/mcp/validate-params.ts b/src/mcp/validate-params.ts
index 504db54980..06d670dbde 100644
--- a/src/mcp/validate-params.ts
+++ b/src/mcp/validate-params.ts
@@ -16,6 +16,7 @@ import type { Operation } from '../core/operations.ts';
import type { BrainEngine } from '../core/engine.ts';
import type { GBrainConfig } from '../core/config.ts';
import { suggestNearest } from '../core/levenshtein.ts';
+import { engineKey } from '../core/engine-key.ts';
import type { OperationError } from '../core/ops/contract.ts';
import { invalidParam } from '../core/ops/op-fix.ts';
import type { Action } from '../core/agent-output.ts';
@@ -256,7 +257,7 @@ export function parseStrictParamsMode(v: unknown): StrictParamsMode | null {
* Dual-plane resolution for `mcp.strict_params`: DB plane (`engine.getConfig`)
* wins, file plane (`config.mcp.strict_params`) is the fallback, absent or
* unparseable on both = 'warn' (the grace-period default). A failed DB read
- * holds the LAST successfully resolved DB mode (per process) before falling
+ * holds the LAST successfully resolved DB mode (per engine) before falling
* to the file plane — a transient config outage on a reject-mode server must
* not silently re-open the warn-mode grace period. Validation posture never
* takes a tool call down. Resolved once per dispatch (amendment 12);
@@ -265,11 +266,11 @@ export function parseStrictParamsMode(v: unknown): StrictParamsMode | null {
* (stdio + the legacy bearer transport resolve once at startup — restart to
* flip there, deliberate).
*/
-let lastKnownDbStrictMode: StrictParamsMode | null = null;
+let lastKnownDbStrictMode = new WeakMap<object, StrictParamsMode | null>();
/** Test seam: clear the last-known-good DB mode between cases. */
export function resetStrictParamsModeCache(): void {
- lastKnownDbStrictMode = null;
+ lastKnownDbStrictMode = new WeakMap();
}
export async function resolveStrictParamsMode(
@@ -278,12 +279,13 @@ export async function resolveStrictParamsMode(
): Promise<StrictParamsMode> {
try {
const dbMode = parseStrictParamsMode(await engine.getConfig('mcp.strict_params'));
- lastKnownDbStrictMode = dbMode;
+ lastKnownDbStrictMode.set(engineKey(engine), dbMode);
if (dbMode) return dbMode;
} catch {
// Engine without a config table / transient error: last-known-good DB
// mode wins; with no prior successful read, the file plane decides.
- if (lastKnownDbStrictMode) return lastKnownDbStrictMode;
+ const lastKnown = lastKnownDbStrictMode.get(engineKey(engine));
+ if (lastKnown) return lastKnown;
}
return parseStrictParamsMode(config?.mcp?.strict_params) ?? 'warn';
}
diff --git a/test/commands-search.test.ts b/test/commands-search.test.ts
index a5270e0c52..7e2344e30b 100644
--- a/test/commands-search.test.ts
+++ b/test/commands-search.test.ts
@@ -165,8 +165,7 @@ describe('gbrain search stats', () => {
});
test('after telemetry writes → hit rate + intent mix surfaced', async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }), { results_count: 5 });
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }), { results_count: 7 });
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'miss' } }), { results_count: 9 });
@@ -206,8 +205,7 @@ describe('gbrain search stats', () => {
});
test('--json includes a coverage disclosure (non-empty table)', async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }), { results_count: 5 });
await w.flush();
@@ -260,8 +258,7 @@ describe('gbrain search stats', () => {
});
test('human output surfaces the exact coverage caveat (non-empty table)', async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }), { results_count: 5 });
await w.flush();
@@ -300,8 +297,7 @@ describe('gbrain search tune (recommendations)', () => {
test('conservative + high budget drop rate → recommends balanced', async () => {
await engine.setConfig('search.mode', 'conservative');
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
// 30 calls, each dropping 5 results — strong signal.
for (let i = 0; i < 30; i++) {
recordSearchTelemetry(engine, makeMeta({
@@ -324,8 +320,7 @@ describe('gbrain search tune (recommendations)', () => {
test('has_recommendations → human output notes the exact coverage caveat', async () => {
await engine.setConfig('search.mode', 'conservative');
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
for (let i = 0; i < 30; i++) {
recordSearchTelemetry(engine, makeMeta({
mode: 'conservative',
@@ -341,8 +336,7 @@ describe('gbrain search tune (recommendations)', () => {
test('tokenmax + Haiku subagent → recommends balanced', async () => {
await engine.setConfig('search.mode', 'tokenmax');
await engine.setConfig('models.tier.subagent', 'anthropic:claude-haiku-4-5');
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
for (let i = 0; i < 25; i++) {
recordSearchTelemetry(engine, makeMeta({ mode: 'tokenmax' }), { results_count: 30 });
}
@@ -359,8 +353,7 @@ describe('gbrain search tune (recommendations)', () => {
test('--apply mutates config', async () => {
await engine.setConfig('search.mode', 'conservative');
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
for (let i = 0; i < 30; i++) {
recordSearchTelemetry(engine, makeMeta({
mode: 'conservative',
diff --git a/test/engine-keyed-caches.test.ts b/test/engine-keyed-caches.test.ts
new file mode 100644
index 0000000000..72c13828aa
--- /dev/null
+++ b/test/engine-keyed-caches.test.ts
@@ -0,0 +1,154 @@
+/**
+ * Per-brain caches keyed by engine (GBRA-283 Wave A). A process with several
+ * engines (mounted brains) must read each brain's own settings and write each
+ * brain's telemetry into its own database. Every case runs the second engine
+ * after the first one warmed the cache, the order that leaked on master.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import type { BrainEngine } from '../src/core/engine.ts';
+import type { HybridSearchMeta } from '../src/core/types.ts';
+import { engineKey } from '../src/core/engine-key.ts';
+import { registerEngineView } from '../src/core/persistence/switches.ts';
+import { deadlineBoundEngine } from '../src/core/statement-deadline.ts';
+import { recordSearchTelemetry, awaitPendingTelemetryFlush, _resetTelemetryWriterForTest, _telemetryWriterCountForTest } from '../src/core/search/telemetry.ts';
+import { bumpLastRetrievedAt, awaitPendingLastRetrievedWrites, _resetTrackRetrievalCacheForTests } from '../src/core/last-retrieved.ts';
+import { resolveAdmissionPolicy, _resetAdmissionCacheForTest } from '../src/core/minions/admission.ts';
+import { resolveStrictParamsMode, resetStrictParamsModeCache } from '../src/mcp/validate-params.ts';
+import { readPublicationCeilingMs, _resetPublicationCeilingForTest } from '../src/core/persistence/publication-deadline.ts';
+import { capacityNotices, __resetCapacityNoticesForTests, type CapacitySnapshot } from '../src/core/persistence/capacity-snapshot.ts';
+
+let brainA: PGLiteEngine;
+let brainB: PGLiteEngine;
+
+beforeAll(async () => {
+ brainA = new PGLiteEngine();
+ await brainA.connect({});
+ await brainA.initSchema();
+ brainB = new PGLiteEngine();
+ await brainB.connect({});
+ await brainB.initSchema();
+});
+
+afterAll(async () => {
+ _resetTelemetryWriterForTest();
+ await brainA.disconnect();
+ await brainB.disconnect();
+});
+
+beforeEach(() => {
+ _resetTelemetryWriterForTest();
+ _resetTrackRetrievalCacheForTests();
+ _resetAdmissionCacheForTest();
+ resetStrictParamsModeCache();
+ _resetPublicationCeilingForTest();
+ __resetCapacityNoticesForTests();
+});
+
+/** A stand-in engine whose config reads answer from \`values\` (a throwing value makes the read fail). */
+function configEngine(values: Record<string, string | Error | null>, calls: string[] = []): BrainEngine {
+ return {
+ async getConfig(key: string) {
+ const v = values[key];
+ if (v instanceof Error) throw v;
+ return v ?? null;
+ },
+ async executeRaw(sql: string) { calls.push(sql); return []; },
+ } as unknown as BrainEngine;
+}
+
+const meta: HybridSearchMeta = { vector_enabled: true, detail_resolved: null, expansion_applied: false, intent: 'general', mode: 'balanced' };
+
+describe('engineKey', () => {
+ test('a page transaction, a read-through view and a deadline wrapper share their engine\'s key', () => {
+ const engine = configEngine({});
+ const tx = Object.create(engine) as object;
+ Object.defineProperty(tx, '_pageTransaction', { value: true });
+ const view = registerEngineView({}, engine);
+ expect(engineKey(engine)).toBe(engine);
+ expect(engineKey(tx)).toBe(engine);
+ expect(engineKey(view)).toBe(engine);
+ expect(engineKey(deadlineBoundEngine(engine))).toBe(engine);
+ expect(engineKey(configEngine({}))).not.toBe(engine);
+ });
+});
+
+describe('search telemetry flushes into the searching brain', () => {
+ test('each engine\'s rows land in its own database, whichever recorded first', async () => {
+ recordSearchTelemetry(brainA, meta, { results_count: 1 });
+ recordSearchTelemetry(brainB, meta, { results_count: 1 });
+ recordSearchTelemetry(brainB, meta, { results_count: 1 });
+ await awaitPendingTelemetryFlush(5000, 'exit');
+ const count = async (e: PGLiteEngine) => (await e.executeRaw<{ n: number }>('SELECT COALESCE(SUM(count),0)::int AS n FROM search_telemetry'))[0]!.n;
+ expect(await count(brainA)).toBe(1);
+ expect(await count(brainB)).toBe(2);
+ await brainA.executeRaw('DELETE FROM search_telemetry');
+ await brainB.executeRaw('DELETE FROM search_telemetry');
+ });
+
+ test('a transaction records into its engine\'s writer, not a writer of its own', () => {
+ const tx = Object.create(brainA) as BrainEngine;
+ Object.defineProperty(tx, '_pageTransaction', { value: true });
+ recordSearchTelemetry(brainA, meta, { results_count: 1 });
+ recordSearchTelemetry(tx, meta, { results_count: 1 });
+ expect(_telemetryWriterCountForTest()).toBe(1);
+ });
+
+ test('an engine\'s disconnect retires its writer, so engines opened and closed per request leave none behind', async () => {
+ const closers: Array<() => Promise<void>> = [];
+ for (let i = 0; i < 20; i++) {
+ const hooks = new Set<() => Promise<void>>();
+ const engine = { ...configEngine({}), registerBeforeDisconnect(stop: () => Promise<void>) { hooks.add(stop); return () => { hooks.delete(stop); }; } } as unknown as BrainEngine;
+ recordSearchTelemetry(engine, meta, { results_count: 1 });
+ closers.push(async () => { for (const stop of [...hooks]) await stop(); expect(hooks.size).toBe(0); });
+ }
+ expect(_telemetryWriterCountForTest()).toBe(20);
+ for (const close of closers) await close();
+ expect(_telemetryWriterCountForTest()).toBe(0);
+ });
+});
+
+describe('per-brain settings are read per engine', () => {
+ test('retrieval tracking honors each brain\'s own search.track_retrieval', async () => {
+ const writesA: string[] = [];
+ const writesB: string[] = [];
+ bumpLastRetrievedAt(configEngine({ 'search.track_retrieval': 'false' }, writesA), [1]);
+ await awaitPendingLastRetrievedWrites(2000);
+ bumpLastRetrievedAt(configEngine({ 'search.track_retrieval': 'true' }, writesB), [1]);
+ await awaitPendingLastRetrievedWrites(2000);
+ expect(writesA).toHaveLength(0);
+ expect(writesB).toHaveLength(1);
+ });
+
+ test('a job name resolves each brain\'s own admission quota', async () => {
+ const a = await resolveAdmissionPolicy(configEngine({ 'minions.quota_max_waiting.probe-job': '5' }), 'probe-job');
+ const b = await resolveAdmissionPolicy(configEngine({ 'minions.quota_max_waiting.probe-job': '7' }), 'probe-job');
+ expect(a.quotaMaxWaiting).toBe(5);
+ expect(b.quotaMaxWaiting).toBe(7);
+ });
+
+ test('a failed strict-params read falls back to its own brain, never another brain\'s last mode', async () => {
+ expect(await resolveStrictParamsMode(configEngine({ 'mcp.strict_params': 'reject' }), null)).toBe('reject');
+ const flaky = configEngine({ 'mcp.strict_params': new Error('config table unavailable') });
+ expect(await resolveStrictParamsMode(flaky, null)).toBe('warn');
+ const recovered = configEngine({ 'mcp.strict_params': 'reject' });
+ expect(await resolveStrictParamsMode(recovered, null)).toBe('reject');
+ (recovered as unknown as { getConfig: () => Promise<never> }).getConfig = async () => { throw new Error('blip'); };
+ expect(await resolveStrictParamsMode(recovered, null)).toBe('reject');
+ });
+
+ test('each brain reads its own publication ceiling inside the cache window', async () => {
+ expect(await readPublicationCeilingMs(configEngine({ 'persistence.publication_ceiling_ms': '120000' }))).toBe(120_000);
+ expect(await readPublicationCeilingMs(configEngine({ 'persistence.publication_ceiling_ms': '180000' }))).toBe(180_000);
+ });
+
+ test('the write_capacity_low throttle is per brain', async () => {
+ const row = { scope: 'brain', resource: 'lifetime_ids' as const, used: 90, limit: 100, pct: 90, days_to_exhaustion: null,
+ config_key: 'persistence.capacity.lifetime_ids', recommended_limit: 200, command: 'gbrain config set persistence.capacity.lifetime_ids 200' };
+ const snapshot: CapacitySnapshot = { rows: [row], low: [row], paused: [], pause_pct: 95, status: 'ok' };
+ const a = configEngine({});
+ expect(await capacityNotices(a, { throttle: true, snapshot })).toHaveLength(1);
+ expect(await capacityNotices(a, { throttle: true, snapshot })).toHaveLength(0);
+ expect(await capacityNotices(configEngine({}), { throttle: true, snapshot })).toHaveLength(1);
+ });
+});
diff --git a/test/search-telemetry-cache-wiring.serial.test.ts b/test/search-telemetry-cache-wiring.serial.test.ts
index 302e972697..d88a5e813d 100644
--- a/test/search-telemetry-cache-wiring.serial.test.ts
+++ b/test/search-telemetry-cache-wiring.serial.test.ts
@@ -59,7 +59,7 @@ interface Counters {
/** Flush the writer and read the summed counters back from the table. */
async function readCounters(): Promise<Counters> {
- await getTelemetryWriter().flush();
+ await getTelemetryWriter(engine).flush();
const rows = await engine.executeRaw<Counters>(
`SELECT COALESCE(SUM(count), 0)::int AS c,
COALESCE(SUM(cache_hit), 0)::int AS hit,
diff --git a/test/search-telemetry-disconnect-hang.test.ts b/test/search-telemetry-disconnect-hang.test.ts
index 7b62e15871..f66b4f62b5 100644
--- a/test/search-telemetry-disconnect-hang.test.ts
+++ b/test/search-telemetry-disconnect-hang.test.ts
@@ -26,8 +26,8 @@ import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
import { hybridSearch } from '../src/core/search/hybrid.ts';
import {
FLUSH_THRESHOLD_CALLS,
- getTelemetryWriter,
_resetTelemetryWriterForTest,
+ _telemetryWriterCountForTest,
} from '../src/core/search/telemetry.ts';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
@@ -74,7 +74,7 @@ describe('telemetry flush vs disconnect (#4143, serial)', () => {
]);
if (timer) clearTimeout(timer);
expect(winner).toBe('disconnected'); // pre-fix: 'hung' (600s CI kill)
- // The writer must not be left holding an in-flight flush.
- expect(getTelemetryWriter().pendingFlush()).toBeNull();
+ // The writer must not be left holding an in-flight flush: disconnect retires the engine's writer.
+ expect(_telemetryWriterCountForTest()).toBe(0);
}, 60_000);
});
diff --git a/test/search-telemetry.test.ts b/test/search-telemetry.test.ts
index 6369bb848d..4e963a49d0 100644
--- a/test/search-telemetry.test.ts
+++ b/test/search-telemetry.test.ts
@@ -47,16 +47,14 @@ const makeMeta = (overrides: Partial<HybridSearchMeta> = {}): HybridSearchMeta =
describe('recordSearchTelemetry — in-memory bucket', () => {
test('first record creates a bucket; record() never blocks the caller', () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
expect(w.bucketCountForTest()).toBe(0);
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
expect(w.bucketCountForTest()).toBe(1);
});
test('same (date, mode, intent) accumulates into one bucket', () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
recordSearchTelemetry(engine, makeMeta(), { results_count: 7 });
recordSearchTelemetry(engine, makeMeta(), { results_count: 3 });
@@ -68,8 +66,7 @@ describe('recordSearchTelemetry — in-memory bucket', () => {
});
test('different modes / intents create distinct buckets', () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, makeMeta({ mode: 'conservative', intent: 'entity' }), { results_count: 2 });
recordSearchTelemetry(engine, makeMeta({ mode: 'tokenmax', intent: 'temporal' }), { results_count: 9 });
recordSearchTelemetry(engine, makeMeta({ mode: 'balanced', intent: 'event' }), { results_count: 4 });
@@ -77,8 +74,7 @@ describe('recordSearchTelemetry — in-memory bucket', () => {
});
test('cache_hit / cache_miss counters fire from meta.cache.status', () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }));
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }));
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'miss' } }));
@@ -91,8 +87,7 @@ describe('recordSearchTelemetry — in-memory bucket', () => {
});
test('sum_budget_dropped accumulates from meta.token_budget.dropped', () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, makeMeta({ token_budget: { budget: 4000, used: 3800, kept: 8, dropped: 12 } }));
recordSearchTelemetry(engine, makeMeta({ token_budget: { budget: 4000, used: 4000, kept: 10, dropped: 7 } }));
const today = new Date().toISOString().slice(0, 10);
@@ -101,8 +96,7 @@ describe('recordSearchTelemetry — in-memory bucket', () => {
});
test('missing mode / intent fall back to "unset" — telemetry is non-blocking', () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, { vector_enabled: true, detail_resolved: null, expansion_applied: false });
const today = new Date().toISOString().slice(0, 10);
expect(w.bucketForTest(today, 'unset', 'unset')).not.toBeNull();
@@ -111,8 +105,7 @@ describe('recordSearchTelemetry — in-memory bucket', () => {
describe('flush() writes to search_telemetry', () => {
test('flush drains the bucket map atomically', async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
recordSearchTelemetry(engine, makeMeta(), { results_count: 7 });
expect(w.bucketCountForTest()).toBe(1);
@@ -128,8 +121,7 @@ describe('flush() writes to search_telemetry', () => {
});
test('ON CONFLICT DO UPDATE adds raw values (concurrent-flush semantics)', async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
// First flush: 3 calls under balanced/general.
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
@@ -151,16 +143,14 @@ describe('flush() writes to search_telemetry', () => {
});
test('flush is no-op when bucket map is empty', async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
await w.flush(); // no records, no rows
const rows = await engine.executeRaw<{ n: number }>('SELECT COUNT(*)::int AS n FROM search_telemetry');
expect(rows[0].n).toBe(0);
});
test('concurrent flush() calls coalesce (flushInFlight reuse)', async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, makeMeta(), { results_count: 1 });
// Two simultaneous flush() awaits → both observe the same underlying drain.
const [a, b] = await Promise.all([w.flush(), w.flush()]);
@@ -180,8 +170,7 @@ describe('readSearchStats — read-time derived averages', () => {
});
test('one bucket flushed → stats derive averages from sums/counts', async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, makeMeta(), { results_count: 10 });
recordSearchTelemetry(engine, makeMeta(), { results_count: 20 });
recordSearchTelemetry(engine, makeMeta(), { results_count: 30 });
@@ -193,8 +182,7 @@ describe('readSearchStats — read-time derived averages', () => {
});
test('cache_hit_rate computed from hits + misses (excludes disabled)', async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }));
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }));
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }));
@@ -209,8 +197,7 @@ describe('readSearchStats — read-time derived averages', () => {
});
test('intent_distribution and mode_distribution surface counts', async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, makeMeta({ mode: 'conservative', intent: 'entity' }));
recordSearchTelemetry(engine, makeMeta({ mode: 'conservative', intent: 'entity' }));
recordSearchTelemetry(engine, makeMeta({ mode: 'tokenmax', intent: 'temporal' }));
@@ -254,8 +241,7 @@ describe('readSearchStats — read-time derived averages', () => {
// diverts them out of the call/intent/mode aggregates.
describe('empty_result bucket keyed by cause (WP2/T3)', () => {
test('vector down + empty response → vector_disabled cause', async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, makeMeta({ vector_enabled: false }), { results_count: 0 });
await w.flush();
const s = await readSearchStats(engine, { days: 7 });
@@ -264,8 +250,7 @@ describe('empty_result bucket keyed by cause (WP2/T3)', () => {
});
test('budget dropped everything → budget_dropped_all cause (beats vector_disabled)', async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(
engine,
makeMeta({
@@ -280,8 +265,7 @@ describe('empty_result bucket keyed by cause (WP2/T3)', () => {
});
test('degraded budget_dropped_all stage classifies the same way', async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(
engine,
makeMeta({ degraded: [{ stage: 'budget_dropped_all' }] }),
@@ -293,8 +277,7 @@ describe('empty_result bucket keyed by cause (WP2/T3)', () => {
});
test('healthy pipeline + zero hits → keyword_zero cause', async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, makeMeta(), { results_count: 0 });
await w.flush();
const s = await readSearchStats(engine, { days: 7 });
@@ -302,8 +285,7 @@ describe('empty_result bucket keyed by cause (WP2/T3)', () => {
});
test('empty cache HIT (offset artifact) is NOT counted as an empty-result cause', async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }), { results_count: 0 });
await w.flush();
const s = await readSearchStats(engine, { days: 7 });
@@ -311,8 +293,7 @@ describe('empty_result bucket keyed by cause (WP2/T3)', () => {
});
test('reserved rows never skew total_calls or the distributions', async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
recordSearchTelemetry(engine, makeMeta({ vector_enabled: false }), { results_count: 0 });
await w.flush();
@@ -334,8 +315,7 @@ describe('awaitPendingTelemetryFlush (#4143 drain)', () => {
});
test("exit mode flushes residual buckets — BOTH the normal and the empty_result bucket land", async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
// One normal record + one zero-result record (the empty_result cause
// bucket #4096 added — the second INSERT that turned the flush into the
// close()-deadlock trigger).
@@ -355,8 +335,7 @@ describe('awaitPendingTelemetryFlush (#4143 drain)', () => {
});
test('disconnect mode awaits only the in-flight flush and DROPS residual buckets (lossy by design)', async () => {
- const w = getTelemetryWriter();
- w.setEngine(engine);
+ const w = getTelemetryWriter(engine);
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
// Kick a flush so it is in flight, then buffer MORE — the post-swap
// records that a disconnect-mode drain must NOT try to write.
@@ -375,7 +354,7 @@ describe('awaitPendingTelemetryFlush (#4143 drain)', () => {
});
test('a drain that exceeds its bound reports unfinished instead of hanging', async () => {
- const w = getTelemetryWriter();
+ const w = getTelemetryWriter(engine);
// A flush that never settles (simulated) — the drain must give up at the bound.
(w as unknown as { flushInFlight: Promise<void> | null }).flushInFlight = new Promise<void>(() => {});
const started = Date.now();
diff --git a/test/search/hnsw-iterative-scan.test.ts b/test/search/hnsw-iterative-scan.test.ts
index fffc5a71e7..a1785ea7c3 100644
--- a/test/search/hnsw-iterative-scan.test.ts
+++ b/test/search/hnsw-iterative-scan.test.ts
@@ -53,9 +53,12 @@ describe('hnsw.iterative_scan mode', () => {
expect(readHnswIterativeScan(null, {})).toEqual({ mode: 'relaxed_order', via: null });
});
- test('resolved once per process', () => {
- expect(resolveHnswIterativeScan({ search: { hnsw_iterative_scan: 'strict_order' } })).toBe('strict_order');
- expect(resolveHnswIterativeScan({ search: { hnsw_iterative_scan: 'off' } })).toBe('strict_order');
+ test('resolved once per engine', () => {
+ const engineA = {};
+ const engineB = {};
+ expect(resolveHnswIterativeScan(engineA, { search: { hnsw_iterative_scan: 'strict_order' } })).toBe('strict_order');
+ expect(resolveHnswIterativeScan(engineA, { search: { hnsw_iterative_scan: 'off' } })).toBe('strict_order');
+ expect(resolveHnswIterativeScan(engineB, { search: { hnsw_iterative_scan: 'off' } })).toBe('off');
});
test('config set refuses an invalid value with the valid modes and an example', () => {
diff --git a/test/search/rerank-gate-telemetry.test.ts b/test/search/rerank-gate-telemetry.test.ts
index 9cb2e127a6..0dbf8cccaa 100644
--- a/test/search/rerank-gate-telemetry.test.ts
+++ b/test/search/rerank-gate-telemetry.test.ts
@@ -39,7 +39,7 @@ const off: Gate = { mode: 'shadow', eligible: false, ineligible_reason: 'reranke
async function flushFrom(engine: BrainEngine, metas: HybridSearchMeta[]): Promise<void> {
_resetTelemetryWriterForTest();
for (const m of metas) recordSearchTelemetry(engine, m, { results_count: 3 });
- await getTelemetryWriter().flush();
+ await getTelemetryWriter(engine).flush();
}
for (const backend of testBackends()) {
diff --git a/test/search/telemetry-rank1.test.ts b/test/search/telemetry-rank1.test.ts
index 2738776361..49c7093011 100644
--- a/test/search/telemetry-rank1.test.ts
+++ b/test/search/telemetry-rank1.test.ts
@@ -29,7 +29,7 @@ describe('rank-1 score telemetry', () => {
recordSearchTelemetry(engine, meta, { results_count: 5, rank1_score: 0.95 }); // high
recordSearchTelemetry(engine, meta, { results_count: 3, rank1_score: 0.70 }); // solid
recordSearchTelemetry(engine, meta, { results_count: 1, rank1_score: 0.40 }); // lt_solid
- await getTelemetryWriter().flush();
+ await getTelemetryWriter(engine).flush();
const stats = await readSearchStats(engine, { days: 1 });
expect(stats.rank1_count).toBe(3);
@@ -40,7 +40,7 @@ describe('rank-1 score telemetry', () => {
test('queries with no result (no rank1_score) do not pollute the average', async () => {
recordSearchTelemetry(engine, meta, { results_count: 0 }); // no rank1
recordSearchTelemetry(engine, meta, { results_count: 2, rank1_score: 0.88 });
- await getTelemetryWriter().flush();
+ await getTelemetryWriter(engine).flush();
const stats = await readSearchStats(engine, { days: 1 });
expect(stats.total_calls).toBe(2);
diff --git a/test/search/vector-statement.test.ts b/test/search/vector-statement.test.ts
index 140e1fe0a2..1b6163b9dd 100644
--- a/test/search/vector-statement.test.ts
+++ b/test/search/vector-statement.test.ts
@@ -372,15 +372,20 @@ describe('vector legacy guard setting', () => {
});
});
- test('resolves once per process and logs activation once to stderr', async () => {
+ test('resolves once per engine and logs activation once to stderr', async () => {
const stderr = spyOn(console, 'error').mockImplementation(() => {});
+ const engineA = {};
+ const engineB = {};
+ const engineC = {};
try {
await withEnv({ GBRAIN_VECTOR_LEGACY_GUARD: 'true' }, async () => {
- expect(resolveVectorLegacyGuard(null)).toBe(true);
- expect(resolveVectorLegacyGuard(null)).toBe(true);
+ expect(resolveVectorLegacyGuard(engineA, null)).toBe(true);
+ expect(resolveVectorLegacyGuard(engineA, null)).toBe(true);
+ expect(resolveVectorLegacyGuard(engineB, null)).toBe(true);
});
await withEnv({ GBRAIN_VECTOR_LEGACY_GUARD: undefined }, async () => {
- expect(resolveVectorLegacyGuard(null)).toBe(true);
+ expect(resolveVectorLegacyGuard(engineA, null)).toBe(true);
+ expect(resolveVectorLegacyGuard(engineC, null)).toBe(false);
});
expect(stderr).toHaveBeenCalledTimes(1);
expect(String(stderr.mock.calls[0][0])).toStartWith('[gbrain] vector legacy guard active');
diff --git a/test/validate-params.test.ts b/test/validate-params.test.ts
index 9b55768cdd..5179402ecf 100644
--- a/test/validate-params.test.ts
+++ b/test/validate-params.test.ts
@@ -323,18 +323,27 @@ describe('resolveStrictParamsMode (DB > file > warn)', () => {
getConfig: async () => { throw new Error('no config table'); },
} as unknown as BrainEngine;
- // The resolver holds the last successfully read DB mode per process
+ // The resolver holds the last successfully read DB mode per engine
// (adversarial fix: a transient outage must not re-open warn mode) —
// isolate each case from its neighbors.
beforeEach(() => resetStrictParamsModeCache());
+ /** One engine whose config read answers `state.value`, or fails while `state.down`. */
+ const flakyEngine = (state: { value: unknown; down: boolean }) =>
+ ({ getConfig: async () => { if (state.down) throw new Error('no config table'); return state.value; } } as unknown as BrainEngine);
+
test('a failed DB read holds the last-known-good DB mode (reject survives an outage)', async () => {
- expect(await resolveStrictParamsMode(engineWith('reject'), null)).toBe('reject');
+ const state = { value: 'reject' as unknown, down: false };
+ const engine = flakyEngine(state);
+ expect(await resolveStrictParamsMode(engine, null)).toBe('reject');
// DB now unreadable + file plane says warn → cached reject still wins.
- expect(await resolveStrictParamsMode(throwingEngine, { engine: 'pglite', mcp: { strict_params: 'warn' } } as never)).toBe('reject');
+ state.down = true;
+ expect(await resolveStrictParamsMode(engine, { engine: 'pglite', mcp: { strict_params: 'warn' } } as never)).toBe('reject');
// A later successful read of "unset" clears the cache → file plane decides.
- expect(await resolveStrictParamsMode(engineWith(null), { engine: 'pglite', mcp: { strict_params: 'warn' } } as never)).toBe('warn');
- expect(await resolveStrictParamsMode(throwingEngine, null)).toBe('warn');
+ state.down = false; state.value = null;
+ expect(await resolveStrictParamsMode(engine, { engine: 'pglite', mcp: { strict_params: 'warn' } } as never)).toBe('warn');
+ state.down = true;
+ expect(await resolveStrictParamsMode(engine, null)).toBe('warn');
});
test('DB plane wins over the file plane', async () => {