Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

search: the readiness probe skips the by_design walk while pages are pending (#8000)

garrytanc0f9d851c506
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 375c769249..b200c2e56e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,19 @@ credits are retained; no result has been reassigned to another provider. Origina
identifiers and attribution are available in the pre-removal Git revision
`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).
+## [Unreleased]
+
+**A search on a brain with many pages still waiting for a projection no longer spends seconds in the readiness check: the probe stops after it finds a pending page instead of also walking every unsealed page for the `by_design` answer it then discards (#8000).**
+
+The readiness probe that runs before every uncached `search` and `query` asked two questions in one statement: is any page in scope pending, and are the only unsealed pages protected ones (`by_design`). The second answer is only used when nothing is pending, but Postgres evaluated both. While pages are pending there is usually no protected page for its `LIMIT 1` to stop at, so it read every unsealed page on every cold call. On a synthetic brain with 235k pending image pages that was most of the `meta_readiness` stage. The statement now evaluates `by_design` only when nothing is pending, still in one round trip, and returns the same `ready`, `projection_pending` and `by_design` answers.
+
+Contributed by @Masashi-Ono0611, whose #8004 proposed the short-circuit.
+
+### Itemized changes
+
+- `src/core/search/projection-readiness.ts`: the probe reads `pending` in a materialized CTE and `by_design` inside `CASE WHEN pending THEN false ELSE … END`, so its subplan never runs while pages are pending.
+- Tests: `test/projection-readiness.test.ts` (PGLite) and `test/e2e/projection-readiness-plan-postgres.test.ts` (Postgres) pin the plan: with pending pages only one scan of `pages` executes.
+
## [0.60.512.0] - 2026-10-30
**After an automatic self-upgrade, the autopilot now restarts the serve your installed service runs when it is still on the old build, so one auto-upgrade no longer leaves two builds running until someone runs `gbrain upgrade --restart-only` by hand.**
diff --git a/docs/architecture/key-files/core-search-1.md b/docs/architecture/key-files/core-search-1.md
index a7c8384cf3..d4d3c90005 100644
--- a/docs/architecture/key-files/core-search-1.md
+++ b/docs/architecture/key-files/core-search-1.md
@@ -67,7 +67,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/search/private-visibility.ts` — shared page-visibility resolver and SQL predicates. Explicit local trust disables page filtering; operator env/config opt-outs keep their semantics; failed config reads enforce privacy. `privatePagesFilterFragment` authorizes concrete page rows and `isPrivatePage` is its row-side twin: explicit `private` hides a page, absent visibility is world for ordinary pages and private for extracted atoms (`type = 'atom'`) and synthesized concepts (`type = 'concept'` with `synthesized_by`), because their origin may be a private page or transcript. The page predicate also follows derived origins: an atom whose `source_slug` page is explicitly private, and a synthesized concept with a private input atom (or an input whose origin is private) over `concept-provenance` edges, are private whatever their own field says, so a later private flip of an origin reaches derived pages before any repair. Chronicle events follow `event.depth` (fail closed). Declared lineage matches `derived_from` slugs with `= ANY(ARRAY(...))`, not `IN (SELECT ... jsonb_array_elements_text(...))`: same rows, but the planner priced the IN form as a 100-row semi-join per page and pushed every read carrying the rule past `jit_above_cost` (`test/e2e/private-visibility-plan-postgres.test.ts`). `privateSnapshotFilterFragment` is the same rule on a row's own frontmatter for `page_versions`. `effectiveVisibility(origin)` (transcript or missing origin private, page origin per the read rule) and `strictestVisibility` are used by both `extract_atoms` paths, concept synthesis and `gbrain repair visibility`. Remote publication rechecks (`persistence/page-visibility.ts`) and the alias-inject recheck in `hybrid.ts` use the same rule. Tests: `test/private-visibility.test.ts`, `test/derived-page-visibility.test.ts` (PGLite and Postgres); link origins and timeline projections use independent ID-based predicates. `privateProvenanceFilterFragment` hides a fact whose provenance page (`source_markdown_slug`) is private; the fact-list reads in `engine-sql/facts.ts` (by entity, since, keyset, session, supersessions, trajectory) apply it behind `FactListOpts.excludePrivate` / `TrajectoryOpts.excludePrivate`, which every remote fact surface resolves with `resolveExcludePrivatePages` (`test/remote-fact-list-private-provenance.test.ts`, both engines). Remote fact writes apply the same predicate behind an `excludePrivate` option: `forget` (`guardRemoteForget`, the row lock in `persistence/memory-mutations.ts`, `recordFactWithdrawal`, `similarActiveAfterForget`) and `remember` (`decideSingleFact`'s exact and cosine arms, `decideReplacement`), so a fact remote reads hide is `not_found` to a remote write and never matched as its duplicate (`test/remote-fact-write-private-provenance.test.ts`, both engines). Legacy slug-set helpers only enumerate names; they never authorize another row's data. `ops/context.ts:readPolicyOpts` combines canonical source precedence, resolved privacy and holder permissions. Both engines filter content/history/graph/identity/analytics before limits or aggregation; current and historical visibility are checked independently while page privacy is enforced. Semantic response caching remains disabled, so visibility-key folding alone never authorizes reuse.
-- `src/core/search/projection-readiness.ts` — one statement probes visible unsealed pages: ready, projection_pending, by_design (protected/archived only) or unknown; LIMIT 1 arms walk the v161 pending index, not pages. Cached per scope by DB generation (page clock, pages filenode, archive flags, no open tx); `GBRAIN_PROJECTION_READINESS_CACHE=0` disables.
+- `src/core/search/projection-readiness.ts` — one statement probes visible unsealed pages: ready, projection_pending, by_design (protected/archived only) or unknown; LIMIT 1 arms walk the v161 pending index, not pages; the by_design arm runs only when nothing is pending (#8000). Cached per scope by DB generation (page clock, pages filenode, archive flags, no open tx); `GBRAIN_PROJECTION_READINESS_CACHE=0` disables.
- `src/core/search/projection-statistics.ts` — current-projection expression statistics, strict migration postconditions and owner-authorized batch refresh. Migration 160 verifies object identity and populated samples before advancing the ledger, distinguishing empty tables and row-security-hidden statistics. Import, sync, reindex and drained recovery refresh outside page locks; failures preserve completed writes and give a maintenance hint. A pass that changed fewer than 50 + 10% of the sampled pages leaves the statistics alone once they exist (the projection statistic and, on Postgres, every `SEARCH_STATISTICS_COLUMNS` column: `missingSearchStatistics`), and PGLite then analyzes only the hot tables its row deltas mark stale. On Postgres it also samples the page columns search filters read (`SEARCH_PAGE_STATISTICS_SQL`) and the chunk `SEARCH_STATISTICS_COLUMNS`; a vacuum holding a table defers it (analyze-lock.ts). `refreshChunkStatistics` runs the chunk step after a drain that changed 50 + 10% of chunks (any sans statistics): without them the vector candidate statement sorts every eligible chunk instead of walking the HNSW index (`test/e2e/vector-chunk-statistics-postgres.test.ts`). Absent search statistics (pg_upgrade, a deleted pg_statistic; 50-60 s searches at 5k pages) get `ensureSearchStatistics`: PostgresEngine runs it once per engine in the background on its first pool search (never awaited; disconnect waits), and doctor `planner_stats_stale` plus `repair planner-stats` report and apply the same refresh (`test/search-statistics-guard.test.ts`).
diff --git a/src/core/search/projection-readiness.ts b/src/core/search/projection-readiness.ts
index 842b363a08..7fd1a32483 100644
--- a/src/core/search/projection-readiness.ts
+++ b/src/core/search/projection-readiness.ts
@@ -86,6 +86,8 @@ async function probeUncached(
const filters = coverageScopeFilters(scope, params);
try {
// One bound statement: unsealed pages that need work, then those unsealed by design (the class fragment doctor counts).
+ // by_design is read only when nothing is pending, and an InitPlan runs only when the CASE reaches it: while pages are
+ // pending its LIMIT 1 finds no protected page to stop at, so evaluating it anyway read every unsealed page (#8000).
// The planner has no statistics for a column-to-column comparison, so it believes most pages are unsealed and an
// EXISTS chose a sequential scan of pages that expected to stop at once; on a sealed brain it read every page
// (~600 ms at 600k pages, on every cold call). Ordering by idx_pages_projection_pending's columns under LIMIT 1
@@ -97,8 +99,8 @@ async function probeUncached(
ORDER BY p.source_id, p.page_kind, p.slug LIMIT 1) IS NOT NULL`;
const cls = projectionClassSql('p', 's');
const rows = await engine.executeRaw<{ pending: boolean; by_design: boolean }>(
- `SELECT ${unsealed(`${cls} NOT IN ('protected','archived')`)} AS pending,
- ${unsealed(`${cls} IN ('protected','archived')`)} AS by_design`,
+ `WITH r AS MATERIALIZED (SELECT ${unsealed(`${cls} NOT IN ('protected','archived')`)} AS pending)
+ SELECT r.pending, CASE WHEN r.pending THEN false ELSE ${unsealed(`${cls} IN ('protected','archived')`)} END AS by_design FROM r`,
params,
);
if (typeof rows[0]?.pending !== 'boolean') throw new Error('Invalid readiness result');
diff --git a/test/e2e/projection-readiness-plan-postgres.test.ts b/test/e2e/projection-readiness-plan-postgres.test.ts
index c88799cfc9..657bed80c1 100644
--- a/test/e2e/projection-readiness-plan-postgres.test.ts
+++ b/test/e2e/projection-readiness-plan-postgres.test.ts
@@ -76,4 +76,18 @@ describePostgres('projection readiness probe plan on Postgres (GBRA-156)', () =>
await engine.executeRaw("UPDATE pages SET text_projection_revision=knowledge_revision WHERE slug='filler-7'");
}
});
+
+ test('a brain with many pending pages answers pending without the by_design walk (#8000)', async () => {
+ await engine.executeRaw("UPDATE pages SET text_projection_revision=NULL WHERE source_id='source-a'");
+ try {
+ await engine.executeRaw('ANALYZE pages');
+ const { result, probe } = await probeStatement({ sourceId: 'source-a' });
+ expect(result.status).toBe('projection_pending');
+ const text = await plan(probe.sql, probe.params);
+ expect(text.split('\n').filter(line => / on pages p(_\d+)? /.test(line) && !line.includes('never executed'))).toHaveLength(1);
+ } finally {
+ await engine.executeRaw("UPDATE pages SET text_projection_revision=knowledge_revision WHERE source_id='source-a'");
+ await engine.executeRaw('ANALYZE pages');
+ }
+ });
});
diff --git a/test/projection-readiness.test.ts b/test/projection-readiness.test.ts
index f1a963e173..c55209207f 100644
--- a/test/projection-readiness.test.ts
+++ b/test/projection-readiness.test.ts
@@ -220,3 +220,17 @@ describe('projection readiness cache', () => {
expect(probes).toBe(2);
});
});
+
+describe('a pending brain skips the by_design half (#8000)', () => {
+ test('while pages need work, only the pending subquery reads pages; by_design never runs', async () => {
+ await engine.executeRaw(`INSERT INTO pages (source_id,slug,title,type,page_kind,compiled_truth,chunker_version,frontmatter)
+ SELECT 'source-a', 'pending-' || i, 'Pending', 'concept', 'markdown', 'body', 999, '{}'::jsonb FROM generate_series(1, 300) AS i`);
+ const captured: Array<{ sql: string; params?: unknown[] }> = [];
+ const recording = { executeRaw: (sql: string, params?: unknown[]) => { captured.push({ sql, params }); return engine.executeRaw(sql, params); } };
+ const result = await withEnv({ GBRAIN_PROJECTION_READINESS_CACHE: '0' }, () => probeProjectionReadiness(recording as never, { sourceId: 'source-a' }));
+ expect(result.status).toBe('projection_pending');
+ const probe = captured.find(call => call.sql.includes('IS DISTINCT FROM'))!;
+ const plan = (await engine.executeRaw<Record<string, string>>(`EXPLAIN (ANALYZE, COSTS OFF) ${probe.sql}`, probe.params)).map(row => Object.values(row)[0]);
+ expect(plan.filter(line => / on pages p(_\d+)? /.test(line) && !line.includes('never executed'))).toHaveLength(1);
+ });
+});