BrainAPI is a knowledge graph–powered AI memory layer that transforms unstructured data into structured knowledge, enabling intelligent search, recommendations, and contextual memory for AI agents and applications.
17 — Search surface on the shared knowledge base
Workstream: make BrainAPI a unique source of truth that already powers memory and recommendations, and can also power search, with shared write-time features and isolated query contracts. Binding: 00-scope-and-constraints.md, ADR-005, ADR-006, ADR-007 (Accepted). Recs isolation: 16-recsys-eval-protocol.md. Memory SOTA: 08, 12, 13. Substrate indexes: 03 §G.
Plan mode. This document does not change src/ or harnesses. Tasks are sized for later implementers.
Skills applied: scientific-brainstorming (ideas labeled, no automatic winner), scientific-critical-thinking (severity, construct validity), planning-and-task-breakdown (vertical slices, gates), paper-lookup (arXiv 2026-08-18).
0. Scope of this session
Focal question. How can one brain serve (a) agent memory, (b) document/entity search, and (c) recommendations as separate products that share indexes and cheap scores, without making LoCoMo / LongMemEval / BEAM / recsys worse?
| Record | |
|---|---|
| Purpose | Architecture + literature + implementable plan |
| Audience | Maintainer; retrieval / substrate / eval owners |
| Decision owner | Maintainer (ADR-007 Accepted 2026-08-18; §15 locked below) |
| Time horizon | Core search API in weeks; plugins and LTR later |
| In scope | Read APIs, lexical+ANN indexes, fusion, cheap rankers, search eval, plugin boundary, cross-features that are indexes or scores |
| Out of scope | Replacing ingest; training a proprietary bi-encoder as a prerequisite; making /retrieve/context the search engine; MS MARCO as a product SLO; dual-use / biosafety (not implicated) |
| Real constraints | ADR-006: no LLM on context path; p50 < 1000 ms target unmet (~1.4 s); judge cannot resolve <5–7 LoCoMo points; 3072-d HNSW skipped; keyword path is ILIKE |
| Assumed | Event-leg supersession bug stays fixed going forward; existing unrepaired brains may still exist |
| Negotiable | Whether context later opts into BM25; search p95 number; Postgres FTS vs external inverted-index plugin |
| Unknown | Keyword vs paraphrase mix of future search tenants; corpus size per brain in production |
| Prohibited | LoCoMo judge as a search metric; wiping memory eval brains; silent default changes on /retrieve/context |
Perspectives represented: retrieval engineering, memory eval, recsys (existing plugin split), IR literature. Missing: a production search-traffic owner (no click log). Do not treat LoCoMo questions as a substitute for that mix.
1. Product model (decision, proposed)
Three read products, one write path. Analogous to recsys (16): BrainAPI is the KB; heavy models are plugins.
POST /ingest* (chunks, graph, vectors, topics)
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
/retrieve/context /retrieve/search /retrieve/recommend
memory assembler ranked hits ranked items
LoCoMo/BEAM benchmarks.search benchmarks.recsys
ADR-006 SLO search SLO recs SLO
What “search” is not. It is not “call context and parse text_context.” Context exists to feed an answering model. Search exists to return addressable hits (chunk / entity / event / structured row) with scores, filters, and snippets.
What “cross-feature” is. A structure computed at write time, or a cheap score at read time, that more than one product can consume without inheriting the other’s loops (PPR over event hubs, cross-encoder, LightGCN train, sufficiency retry).
2. Current state (located evidence)
Anchors are the working tree as of 2026-08-18. Older line numbers in 02 / 07 may have drifted; behaviour below was re-checked in retrieve.py, data.py, vectors.py.
2.1 Surfaces that exist
| Surface | Role today | Search-engine analogue |
|---|---|---|
POST /retrieve/context |
Memory: spaCy variants → dense node/rel seeds → event-centric facts + PPR + bridges + topics; passages via RRF(dense, keyword) | RAG retriever, not a SERP |
GET /retrieve/ |
retrieve_data: text + preferred entity types |
Internal lookup |
GET|POST /retrieve/recommend |
Graph recommend (+ optional LightGCN plugin) | Recs, not lexical search |
GET /retrieve/entity/synergies |
Sibling / synergy scores | Related-entities widget |
MCP search_semantically / traverse_graph / search_memory |
Deep / agentic | Escalation, unbounded |
2.2 The “hybrid” passage channel is not BM25∪dense
def _retrieve_passages(...):
...
vector_hits = vector_search.search_data(...)
...
search_result = data_adapter.search(text, brain_id) # ILIKE
fused = reciprocal_rank_fusion([vector_ids, keyword_ids])
data.py search is WHERE text ILIKE %s with '%' + escaped_query + '%'. No inverted index, no term weights, no k1/b. RRF (fact_filter.py, k=60) fuses ANN order with a substring hit list.
Implication (interpretation): ADR-005’s hybrid design is real as architecture; the sparse leg is not the IR method the SOTA brief and HyperMem call BM25. Keyword/navigational search is the first thing this stack cannot honestly sell.
2.3 Dense ANN is optional / silently off on default pgvector
_vector_index_ddl returns empty string when dimension > 2000. Deploy examples set all stores to 3072. search_vectors still ORDER BY embeddings <=> $1 and sets hnsw.ef_search (no-op without an index). Over-fetch + uuid tie-break (ann_overfetch_k, stable_top_k_vectors) is in place for determinism, not speed. Milvus uses AUTOINDEX + COSINE if that backend is selected.
On LoCoMo-sized brains this is not the latency bottleneck (06: passages p50 829 ms of which 682 ms embed). On BEAM-scale or multi-million-chunk brains it becomes a sequential scan. Two different products, two different bottlenecks.
2.4 Memory path latency and quality (do not mix with search claims)
| Signal | Status | Use for search work |
|---|---|---|
| Context p50 | ~1.3–1.4 s after dossier deletion; ADR-006 target 1 s | Shared embed batching helps both; do not put CE on this path |
| Passage EvR | ~97% conv-26 | LoCoMo cannot show BM25 recall gains in aggregate; need a keyword slice or a non-LoCoMo corpus |
| Graph vs passages-only judge acc | McNemar ns at n=152 | Do not justify search graph-channel by LoCoMo accuracy |
| Graph EvR | C3: 47.3% → 56.0% with reserved bridge slots + ANN seed stability | Graph as a search channel (return events/entities as hits) is still a product idea, not a LoCoMo win |
| SOTA compose arm | conv-26 95.4% judge (13); full-10 still open |
Do not touch default context ranking while that claim is live unless the gate in §8 passes |
| Recsys | Isolated brain demorecsys; plugins for GNN / attribute prefs |
Copy this isolation for search eval |
2.5 Ranking on context (no LTR, no CE)
Distance (min over event nodes) → optional PPR replaces distance → MMR diversify. LLM fact filter is a no-op without a deep-tier adapter. No LambdaMART, no linear blend of BM25+dense+PPR. Historical context still re-runs the same passage search (~91.5% duplicate; cost, not wall-clock). Passage collection still serial over 3–6 queries each calling embed_text though embed_texts exists.
2.6 Recs already show the split we want for search
16: core = train-free graph ranker on /retrieve/recommend; plugins/features-rec writes extra edges; plugins/recsys-gnn is optional LightGCN and does not replace ingest. Search should copy that: core BM25+dense+filters; plugins for SPLADE / ColBERT / CE.
3. Literature (located, 2026-08-18)
Retrieval summary
- Query: canonical IR + memory papers by arXiv ID, plus title search for hybrid/RRF/SPLADE/ANCE
- Scope: targeted lookup (not exhaustive)
- Databases: arXiv Atom API (
export.arxiv.org/api/query) parsed withpaper-lookup/scripts/arxiv_atom.py - Access date: 2026-08-18
- Limits: first page of title-OR search (
total_results=59, retrieved 8); ID lists complete (12 + 4). RRF’s original SIGIR 2009 paper is not on arXiv (Cormack, Clarke, Buettcher) — cited by name only.
Warnings: Absence from this bound is not “never studied.” UniIR (2311.17136) is multimodal instruction retrieval — weak fit for text-only BrainAPI v1. HyDE (2212.10496) uses an LLM at query time — incompatible with ADR-006 on context; plugin/deep only.
3.1 What actually supports a BrainAPI search product
| Paper | arXiv | Status for us | Takeaway (not a BrainAPI result) |
|---|---|---|---|
| BEIR | 2104.08663 | support-located (zero-shot IR) | BM25 is a strong OOD baseline; dense often loses zero-shot; hybrid / CE / late-interaction win on average at higher cost. This is the argument for a real lexical index before a custom bi-encoder. |
| DPR | 2004.04906 | support-located (mechanism) | Dual-encoder + ANN can beat BM25 in-domain with labels. We have hosted embeddings, not DPR training. |
| ANCE | 2007.00808 | support-located (mechanism) | Hard negatives from a live ANN index close the train/test gap. High engineering cost; not a v1 gate. |
| ColBERT / v2 / PLAID | 2004.12832, 2112.01488, 2205.09707 | mixed | Token-level matching approaches CE quality; PLAID claims tens of ms on GPU at 140M passages. Index size and serving complexity → plugin, not core. |
| SPLADE / v2 | 2107.05720, 2109.10086 | mixed | Learned sparse on inverted indexes; v2 reports large BEIR gains. Index growth and inference → plugin after BM25 exists. |
| M3-Embedding (BGE-M3) | 2402.03216 | idea-adjacent | One model, three modes: dense, sparse, multi-vector, long context (8k). Strong cross-feature candidate if we ever change the embedder; not a silent swap (would re-embed every brain). |
| Matryoshka / MRL | 2205.13147 | support-located | Truncatable embeddings; 03 already maps this to making HNSW legal. text-embedding-3-large supports dimension cut natively. |
| Fusion analysis | 2210.11934 | challenge-located vs “just RRF” | Convex combination of lexical+dense scores can beat RRF; RRF is parameter-sensitive. Our hardcoded k=60 is a choice, not a law. Prefer exposing RRF and CC on the search API. |
| HyperMem | 2604.08256 | support-located (memory) | LoCoMo SOTA report uses topic→episode→fact + BM25∪dense RRF. We already have topic coarse-to-fine; we lack BM25. Same hybrid is a memory cross-feature, opt-in on context after the gate. |
| BEAM | 2510.27246 | support-located (memory scale) | Long dialogues; retrieval cost grows with N. HNSW legality matters here more than on conv-26. |
| Filtered ANN | 2602.11443 | support-located (substrate) | Metadata filters + ANN; Milvus recall more stable than pgvector plans in their study. Search filters (type, validity, time) are first-class, not a WHERE after LIMIT. |
| Contrastive embeddings | 2201.10005 | background | Off-the-shelf embeddings can be strong for semantic search. Justifies keeping a general embedder in v1. |
| HyDE | 2212.10496 | veto on context | LLM generates a hypothetical doc then embeds. Deep/search-plugin only. |
Not treated as evidence for BrainAPI quality: MS MARCO MRR tables in the user’s SOTA brief (analytical interpretation, different task, no reproduction here).
3.2 What would confirm / refute transferring these papers
- Confirm BM25 value: keyword-slice Recall@10 on a brain where ILIKE misses exact tokens that BM25 posts; latency not worse than ILIKE seqscan on a 100k-chunk fixture.
- Refute BM25 value: no Recall@k lift on that slice and on LoCoMo passage EvR (already saturated).
- Confirm HNSW: recall@k vs exact
ORDER BY <=>within a declared drop (e.g. ≤2 points at k=24) and p95 vector stage drop on a large fixture. - Refute “search will lift LoCoMo judge”: McNemar ns after BM25-on-context opt-in — expected; do not use that as a search-ship failure.
4. Independent idea register
Stage: independent then clustered. Origin: AI-assisted + literature-inspired (this session). These are proposals, not findings.
| ID | Statement | Origin | Predicted observation | Disconfirm |
|---|---|---|---|---|
| S01 | Add POST /retrieve/search returning scored hits, not a prompt blob |
product | Search clients stop scraping text_context |
Nobody uses it; context remains the only caller |
| S02 | Replace ILIKE with Postgres BM25/tsvector (or pg_trgm for short codes) as core lexical |
BEIR, HyperMem | Keyword Recall@k ↑; long-query ILIKE misses ↓ | No lift on keyword slice |
| S03 | Legalize ANN (halfvec HNSW or MRL truncate ≤2000) with loud skip |
03 G, MRL |
BEAM/large-brain vector p95 ↓; seed stability held via overfetch | Recall@k vs exact collapses |
| S04 | Batch embed_texts once per request; concurrent dense+lexical; single query on context |
ADR-006 leftover | Context p50 toward <1 s | Stage timings still show multi-RTT embed |
| S05 | Expose fusion rrf and cc (convex combination) on search |
2210.11934 | CC wins in-domain if a tiny labelled set exists | RRF ≈ CC on our mix |
| S06 | Cheap linear/GBDT ranker over {dense, lexical, PPR, recency, topic, validity} | ADR-006, LTR brief | Precision@10 on search set ↑; microseconds | Features collinear; no lift |
| S07 | Search channels: passages | entities | events | structured |
product | Entity-name queries don’t require passage hits | Graph channel noisy; users ignore non-passage |
| S08 | Metadata filters (label, validity, time range) applied inside retrieval | 2602.11443 | Filtered search correct; no post-LIMIT drop | Planner ignores filter; recall hole |
| S09 | Opt-in lexical=bm25 on context after search BM25 is proven |
HyperMem | Memory keyword misses ↓ without judge drop | EvR or McNemar regresses |
| S10 | Plugin features-search or search-rerank: CE on K≤10 |
brief, ADR-006 | Search nDCG ↑; context untouched | p95 blows search SLO |
| S11 | Plugin late-interaction (ColBERT/PLAID) or SPLADE | ColBERT family, SPLADE | Hard queries ↑ | Index size / reindex cost dominates per-brain |
| S12 | Optional embedder swap to BGE-M3 for dense+sparse+colbert from one model | 2402.03216 | One write, three retrieval modes | Re-embed cost; LoCoMo seed shift |
| S13 | Recs cold-start: /retrieve/search on item text for new SKUs, then graph recs |
recs×search | Cold items appear before walks exist | Duplicate ranking; worse HitRate |
| S14 | Do not train ANCE/DPR in v1 | negative control | — | If we later get labels and a refresh loop, reopen |
| S15 | HyDE / query LLM rewrite only on deep search | 2212.10496, ADR-006 | Hard paraphrases ↑ at high latency | Context accidentally calls it |
| S16 | Search eval harness on a separate brain; ledger benchmarks.search |
16 analog |
Claims don’t contaminate LoCoMo | People still quote judge % for search |
Minority / preserved: S12 (embedder swap) and S11 (ColBERT core) are attractive and wrong as v1 if they force re-ingest of memory brains. S07 graph-as-hits may look unused until a tenant searches people/events rather than documents — keep the channel even if default k focuses on passages.
What became less obvious after evidence check: “Ship hybrid” is already the shape of context; the gap is the sparse implementation and the missing hit-list API, not the absence of RRF. “Need IVF+PQ / 100M HNSW RAM” is the SOTA brief’s catalog assumption; BrainAPI shards per brain. “Cross-encoder on K=10 for 200 ms p99” fights ADR-006 on memory and is optional on search.
5. Clusters (by mechanism, not by slogan)
| Cluster | IDs | Shared mechanism |
|---|---|---|
| C-API | S01, S07, S08 | New contract: hits, channels, filters |
| C-LEX | S02, S09 | Real inverted/lexical index |
| C-ANN | S03, S04 | Vectors actually indexed; embed RTT collapsed |
| C-FUSE | S05, S06 | How scores combine (RRF/CC/LTR) |
| C-PLUG | S10, S11, S12, S15 | Heavy IR off the core path |
| C-X | S09, S13 | Cross-product: memory opt-in BM25; recs cold-start via search |
| C-EVAL | S16 | Isolated measurement |
| C-NOT | S14 | Explicit non-goals for v1 |
Merges: S02 and “SPLADE in core” stay split (different index ops and failure modes). S04 is not the same as S03 (latency vs scale).
6. Criteria (declared before preference)
Direction: higher is better unless noted. Weights are decision aids, not a computed winner.
| Criterion | Weight | Anchors (1–5) | Evidence needed |
|---|---|---|---|
| Memory non-regression | Gate (noncompensatory) | Fail = LoCoMo/BEAM/recsys worse | Paired McNemar, passage EvR, recs HitRate, p50 |
| Search usefulness | 3 | 1 = still ILIKE+blob; 5 = hit API + BM25 + filters | Recall@k / nDCG on search set |
| Latency / SLO fit | 3 | 1 = extra LLM; 5 = index lookup + one embed | profile_stages |
| Cross-feature leverage | 2 | 1 = search-only fork; 5 = one index, three readers | Same DDL used by context/search/recs |
| Feasibility / reversibility | 2 | 1 = re-embed all brains; 5 = additive index + flag | Rollback to ILIKE/exact scan |
| Information gain if null | 1 | Null still teaches query mix | Pre-registered slices |
Vetoes (not averaged): LLM on /retrieve/context; default context ranker change without §8 gate; search eval writing memory brains; dual source of truth.
7. Adversarial review (short)
| Idea | Wrong if… | Alternative explanation | Hidden dependency | Harm |
|---|---|---|---|---|
| S02 BM25 | LoCoMo-like questions are long paraphrases; BM25 adds posting I/O and zero recall | Dense already covers; ILIKE was enough on tiny N | Tokenization mismatch with embedder tokenizer | Over-rank boilerplate terms |
| S03 HNSW | Approx top-k reshuffles seeds → graph EvR jitter (C3 already fought this) | Latency win is still embed RTT | halfvec ops / pgvector version |
Silent quality flake on memory |
| S01 new API | All “search” users were agents who wanted context anyway | Product confusion; two ways to do one thing | Client SDK + docs | Split brain of integrations |
| S06 LTR | No labels except LoCoMo sessions (biased to memory) | Overfit to session IDs | Click log does not exist | Ranking that looks good offline, worse UX |
| S12 M3 embedder | Different geometry vs current 3072-d OpenAI vectors | “One model three modes” marketing | Full re-embed | SOTA compose arm incomparable |
Anchoring risk: the user’s SOTA brief pushes hybrid→CE→LTR on a 200 ms p99 web SLO. We keep the sequence of primitives (lexical, ANN, optional CE) but reject the SLO and the default CE.
8. Non-regression protocol (how we refuse to make memory worse)
Copy recsys isolation (16, benchmarks/AGENTS.md).
8.1 What may change without a memory A/B
- New routes under
/retrieve/search*that do not alter context defaults. - New tables/indexes that context does not read yet (e.g.
tsvectorcolumn unused by_retrieve_passages). - Plugins that only register search rerank.
8.2 What requires the gate
- Context reading BM25 instead of ILIKE.
- Enabling HNSW / changing embedding dimensionality or metric.
- Changing RRF
k, seedk, passage fanout, PPR default, fact diversify. - Shared embed batching that changes variant set or seed order (stabilize with existing uuid/distance quantization).
8.3 Gate (all must hold)
- Memory champion arm (document the exact
run_idat implementation time; today the conv-26 SOTA pointer islocomo-compose-sota-conv26-v4dinREPORTS.json): paired McNemar not significantly worse; point estimate ≥ baseline. Prefer product profile for default-path changes; do not require SOTA compose (SC/gap-fill) to move. - Passage full evidence-session recall held to the digit on that brain (deterministic). Graph EvR: if ANN/seeds change, identical-config agreement ≥95% (
00). - Context p50: not worse than the then-current post-dossier baseline beyond noise (record both numbers). Sub-second remains ADR-006, not a search ship blocker.
- Recsys:
demorecsysHitRate@K unchanged unless the PR is a recs PR. - Brains: search fixtures use IDs like
searchbench*. Never wipelocomoconv26*,beam*clean,demorecsys.
If BM25-on-context is null on LoCoMo: keep ILIKE as context default; search still ships BM25. That is a successful split, not a failed hybrid.
9. Proposed search API (idea → later spec)
Vertical slice: one call that a UI or engine can use without parsing memory blobs.
POST /retrieve/search
{
"query": "alice counseling license",
"brain_id": "default",
"k": 20,
"channels": ["passages", "entities"],
"lexical": "bm25",
"dense": true,
"fusion": "rrf",
"filters": {
"labels": ["PERSON", "EVENT"],
"currently_valid": true,
"happened_after": null
},
"rerank": "none",
"profile_stages": false
}
{
"hits": [
{
"id": "chunk_…",
"channel": "passages",
"score": 0.031,
"scores": { "bm25": 12.4, "dense": 0.22, "rrf": 0.031 },
"snippet": "…",
"source_session_ids": ["session_3"],
"entity_uuid": null
}
],
"insufficient": false,
"stage_timings": null
}
Optional later: GET /retrieve/search/suggest (prefix / trgm). Out of v1 if it expands scope.
rerank: none | linear | plugin:<name>. Unknown plugin → 400, not a silent fallback that looks like core quality.
10. Cross-feature map
| Feature | Memory | Search | Recs | When to share |
|---|---|---|---|---|
| BM25 / tsvector | Opt-in passage channel (HyperMem-like) | Default lexical | Item-text cold start (S13) | Write once; read per product flag |
| HNSW / halfvec | Seed + passage ANN at scale | Same | Node similarity if used | Shared; gate on seed stability |
| Batched embed | Closes ~400 ms gap (06) |
One RTT per search query | If recs embed queries | Shared helper, no rank change if vectors identical |
| Topic memberships | Coarse-to-fine already on context | Facet / boost | Weak | Already written |
Validity / happened_at |
Current-truth vs history (05) |
Filters | Recency decay (already on recs) | Shared properties |
| PPR / hub bridges | Multi-hop coverage | events channel ranking |
Not default | Do not turn search into a second context assembler |
| Linear LTR features | ADR-006 context ranker | Search precision | Could consume overlap scores | Train per product or freeze a tiny global blend with separate weights |
| Cross-encoder | Deep tier only | Search plugin, K≤10 | No | Plugin |
| LightGCN | No | No | Plugin | Unchanged |
Prediction: the highest cross-leverage items are S02+S03+S04 (indexes + embed batch). They improve search immediately and can improve memory latency/keyword recall without touching judge prompts.
11. Architecture decisions for implementers
- Lexical engine v1: Postgres FTS (
tsvector+ts_rank_cdor a BM25 extension available in the deployed image). Avoid requiring Elasticsearch as system of record (ADR-007). A plugin may project to an external engine later. - Do not change
EMBEDDING_*_DIMENSIONas a drive-by in the search PR. HNSW legality is a dedicated slice with a recall-vs-latency measurement (00,03). - Context keeps ILIKE until S09 gate. Implementation may write tsvector in the same migration.
- RRF stays as default fusion (already in code, HyperMem-like). Expose
fusion=ccwithalphafor search; do not retune contextk=60in the same PR. - No ANCE/DPR training, no ColBERT core, no HyDE on context in this workstream’s v1.
12. Search evaluation protocol
Mirror 16 so search cannot hide inside LoCoMo. Binding write-up: 18-search-eval-protocol.md. Harness: benchmarks/search/ + ./search.sh.
| Item | Rule |
|---|---|
| Brain | searchbench* only |
| Harness | benchmarks/search/ + ./search.sh; HTTP only |
| Ledger | benchmarks.search in REPORTS.json only |
| Metrics | Recall@{5,10,20}, nDCG@10, MRR; p50/p95 of /retrieve/search excluding embed.query; optional keyword/paraphrase slice |
| Query mix | At least: (a) keyword / name, (b) paraphrase / semantic. Filter/time deferred until the search API accepts filters. Gold = chunk IDs via DOCID markers, not LLM judge |
| Cheap start | Reuse a held-out subset of memory questions as retrieval labels (gold session → chunks via provenance) without running the answerer. Report as search-from-locomo-qrels, never as LoCoMo accuracy. Not in harness v1 |
| Forbidden | Ingest into memory eval brains; quoting judge % as search quality |
Until a scored evaluate run is on the benchmarks.search leaderboard, do not claim “search is better than BM25-in-the-brief.” Toy search_toy rows are isolation checks, not BEIR.
13. Task list
Vertical slices. Each leaves the system working. No task is XL.
Phase 0 — Contract and isolation
Task 1: Freeze the three-surface contract
Description: ADR-007 accepted or explicitly deferred by the maintainer; this doc linked from 00. No ranking code.
Acceptance criteria:
- Maintainer records accept / amend / reject on ADR-007
-
00workstream row for17remains accurate
Verification: Human review of ADR-007.
Dependencies: None
Files: docs/decisions/007-three-product-surfaces-one-kb.md, docs/research/00-scope-and-constraints.md
Scope: S
Task 2: Search eval skeleton
Description: Package benchmarks/search/ with smoke: ingest N chunks into searchbenchsmoke, call a stub or interim POST /retrieve/search (may 501 until Task 4), ledger key documented in benchmarks/AGENTS.md.
Acceptance criteria:
-
./search.sh smokefails loudly if it would use a memory brain id -
REPORTS.jsonschema comment or emptybenchmarks.searchleaderboard allowed
Verification: ./search.sh smoke documented; unit test on brain-id guard.
Dependencies: Task 1 (or proceed as Proposed)
Files: benchmarks/search/**, benchmarks/AGENTS.md
Scope: M
Checkpoint A
- Product split is written; search eval cannot wipe LoCoMo/BEAM brains
Phase 1 — Core search first stage (does not change context defaults)
Task 3: Lexical index at write time
Description: Persist tsvector (or equivalent) on data_text_chunks (and entity name field if cheap). GIN index. Backfill existing brains lazily or via explicit job. Context does not query it yet.
Acceptance criteria:
- New chunks get lexical vectors on ingest
- ILIKE path still used by
_retrieve_passages - Unit test: token query matches a chunk ILIKE would miss or vice versa, documented
Verification: pytest on FTS helpers; explain/analyze on a fixture shows index use.
Dependencies: None (can parallel Task 2)
Files: src/lib/postgresql/data.py, ingest write path, tests
Scope: M
Task 4: POST /retrieve/search core hybrid
Description: New controller: embed once, dense search_data + BM25/tsvector, fusion RRF default, return hits. No PPR, no fact assembler, no historical duplicate.
Acceptance criteria:
- OpenAPI/schema: query, k, channels⊇passages, lexical, dense, fusion
- Does not import dossier/synergy retrievers
-
profile_stagesworks like context - Context tests unchanged (no default behavior change)
Verification: pytest API tests; ./search.sh smoke green against live API.
Dependencies: Task 3
Files: src/services/api/routes/retrieve.py, new controller module, requests.py
Scope: M
Task 5: Filters + entity channel (optional second slice)
Description: currently_valid, label filter; optional entities channel via node ANN + name FTS.
Acceptance criteria:
- Filter is applied so that invalid entities are not in top-k when
currently_valid=true - Passage-only default remains if
channelsomitted
Verification: Tests with a superseded edge / deprecated node.
Dependencies: Task 4
Files: search controller, graph/data queries
Scope: M
Checkpoint B
- Search API returns BM25∪dense hits
-
/retrieve/contextbyte-level ranking defaults unchanged (diff + LoCoMo smoke optional)
Phase 2 — Shared speed/scale (gated)
Task 6: Batch embeddings on context and search
Description: One embed_texts per request; concurrent dense+lexical; collapse context passage variant loop toward ADR-006 (single query or batched variants sharing the batch). Preserve seed uuid/distance stabilization.
Acceptance criteria:
- Stage timings: a single
embed.query(or one batched stage) on a profiled request - Passage EvR held on a recorded memory arm if context code changed
- Search p50 excluding network policy: one embed RTT
Verification: profile_stages JSON; paired EvR if context touched.
Dependencies: Task 4 if sharing helper; else can start from context-only
Files: retrieve.py, embeddings adapter call sites
Scope: M
Task 7: Legalize HNSW
Description: Measurement-first: halfvec vs truncate-to-2000 vs Milvus-only. Loud log/metric when index DDL skipped. Keep overfetch+uuid order.
Acceptance criteria:
- Default 3072 install either has an ANN index or fails closed with a visible error at store init
- Recall@k vs exact on a ≥10k vector fixture recorded
- Memory gate §8 if context seeds change
Verification: Fixture script + notes in run dir; no silent seqscan.
Dependencies: None vs search API; gate before flipping production default
Files: src/lib/postgresql/vectors.py, config, tests
Scope: M
Checkpoint C
- Context p50 moved toward 1 s or numbers escalate per ADR-006
- ANN skip cannot be silent
Phase 3 — Fusion / cheap LTR / memory opt-in
Task 8: Search fusion knobs + linear rerank
Description: fusion=rrf|cc with documented alpha; optional rerank=linear using dense, lexical, recency, validity — no LLM.
Acceptance criteria:
- Default remains RRF k=60-equivalent
- Linear weights are config/request, not a trained black box in v1
- Search nDCG measured on
searchbench*vs RRF-only
Verification: ./search.sh evaluate two arms.
Dependencies: Task 2, Task 4
Files: search controller, fact_filter.py or shared fusion
Scope: S–M
Task 9: Opt-in BM25 on context (cross-feature)
Description: GetContextRequestBody.lexical: ilike | bm25 default ilike. Document HyperMem-style union as the reason to try bm25.
Acceptance criteria:
- Default path = today’s ILIKE RRF
-
lexical=bm25uses the Task 3 index - Gate §8 on a product LoCoMo arm before considering default flip
Verification: McNemar + passage EvR; do not use SOTA compose as the only arm.
Dependencies: Task 3, Task 6 recommended
Files: retrieve.py _retrieve_passages, requests.py
Scope: S
Checkpoint D
- Search has fusion/linear knobs
- Memory default still ILIKE unless gate passed and maintainer flips
Phase 4 — Plugins (after core is real)
Task 10: Search rerank plugin hook
Description: Same pattern as recsys-gnn: optional package registers rerank=plugin:cross-encoder (or SPLADE document expansion). Core works if plugin absent.
Acceptance criteria:
- Plugin yaml + route or entrypoint
- Context path cannot call the plugin
- K≤10 documented for CE
Verification: Load/unload plugin; 400 on unknown name.
Dependencies: Task 4
Files: plugins/search-rerank/ or features-search/, plugin loader
Scope: M
Task 11: Recs cold-start via search (optional)
Description: When graph walks return < k items, fill from /retrieve/search on item text / attributes. Flag on recommend request, default off.
Acceptance criteria:
- Default HitRate on
demorecsysunchanged - Flag-on arm recorded separately in
benchmarks.recsys
Verification: ./recsys.sh evaluate both flags.
Dependencies: Task 4
Files: recommend.py, recsys harness
Scope: S–M
Checkpoint E
- Heavy IR is optional
- Recs default unchanged
14. Risks
| Risk | Impact | Mitigation |
|---|---|---|
| HNSW changes memory seeds | High | Overfetch+uuid; EvR agreement gate; exact scan fallback flag |
| BM25 on context looks “free SOTA” | Med | Opt-in; HyperMem is not our judge protocol (13) |
| Search API unused; context still scraped | Med | SDKs/docs; chatbot plugin uses search for “find” and context for “answer” |
| FTS tokenization ≠ embed tokenizer | Med | Document analyzer; don’t expect identical hits |
| Scope bleed into ColBERT/ANCE | High | C-NOT / Task 14; plugins only |
| Judge-noise used to “prove” search | High | Separate metrics and brains |
15. Open questions (maintainer)
Locked 2026-08-18 (see ADR-007 Status):
- Routes. Keep search under
/retrieveasGET|POST /retrieve/search. Do not replaceGET /retrieve/. - Lexical engine. Env-gated. When on, Okapi BM25 over Postgres
tsvector+ GIN onpgvector/pgvector:pg16. Do not requirepg_search/rum. Do not shipILIKEorts_rank_cdas the search lexical ranker. - 3072-d ANN. Keep float32 vectors.
halfvecHNSW (≤4000-d) + exact float32 cosine rerank of overfetch. Do not truncate / MRL-cut dimensions. - Chatbot plugin. Out of scope for v1. No
plugins/chatbot-memoryclient. - SLO. p50 < 200 ms excluding embed RTT.
profile_stagesmust exposeembed.queryseparately from retrieve. - Context vs search ranking. Search default is both channels fused (
SEARCH_FUSION=rrf). Context stays today’s dense ∪ ILIKE unlessSEARCH_ENABLED=trueandCONTEXT_PASSAGE_MODEis left athybrid(or set tobm25/dense).ilikefreezes the memory eval path.
16. Overall assessment
BrainAPI is already a knowledge base with a strong memory assembler and a recs surface that knows how to keep plugins off the default path. It is not yet a search engine: there is no hit-list API, no BM25, and default vectors may be unindexed.
The SOTA search brief is useful as a checklist of primitives (lexical, dense ANN, optional CE, LTR). It is the wrong product recipe if pasted onto /retrieve/context (wrong SLO, wrong response shape, wrong eval).
Highest-leverage path that can help search and memory without betting LoCoMo: write a real lexical index and a legal ANN index, batch embeddings, ship /retrieve/search that reads them, keep context defaults frozen until a paired gate. Cross-encoders, SPLADE, ColBERT, HyDE, and ANCE stay plugins or deep-tier — the same split recsys already uses.
Next action: maintainer accept/amend ADR-007, then Task 2+3 in parallel (eval skeleton + tsvector), then Task 4 (search API). Label: protocol development, not a validated quality claim.
Status note (2026-08-21): ADR-007 accepted; /retrieve/search shipped. Subsequent workstreams: 18–24 (eval, ESCI, WANDS, catalog graph), 25 + ADR-008 (query-gated personalize). §16 above is the 2026-08-18 assessment that motivated the ship. Default omitted channels=["passages"] and the 200 ms SLO are unchanged.