AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs(example): hybrid retrieval for RAG chatbots (drop-in for pgvector + BM25)
Xerj Squad Afa67008fa5af
diff --git a/docs/examples/rag-hybrid-retrieval/01_corpus.py b/docs/examples/rag-hybrid-retrieval/01_corpus.pynew file mode 100644index 000000000..510214515--- /dev/null+++ b/docs/examples/rag-hybrid-retrieval/01_corpus.py@@ -0,0 +1,28 @@+# A small technical-docs corpus (the shape a RAG chatbot indexes): each item is a+# CHUNK with a doc source + heading. Deliberately mixes exact identifiers+# (error codes, function names, config keys, versions) with conceptual prose, so+# we can show where pure-cosine retrieval fails and hybrid fixes it.+import json+CHUNKS = [+ ("billing/errors","Error E4304 — card declined by issuer","Error E4304 is returned when the customer's card is declined by the issuing bank. Retrying immediately will fail; prompt the customer to contact their bank or use a different card. E4304 is not retryable and does not count against your retry budget."),+ ("billing/errors","Error E4210 — insufficient funds","Error E4210 indicates the card has insufficient funds. Unlike E4304 this IS retryable after a delay; we recommend retrying after 24 hours with exponential backoff."),+ ("billing/errors","Error E5012 — gateway timeout","Error E5012 is a transient gateway timeout. Retry with idempotency keys to avoid double charges."),+ ("api/checkout","createCheckoutSession","Call createCheckoutSession to start a payment. It returns a session URL; redirect the customer there. Pass line_items, success_url, and cancel_url. The session expires after 24 hours."),+ ("api/checkout","cancelCheckoutSession","cancelCheckoutSession voids an open session before payment. It is a no-op if the session already completed."),+ ("api/webhooks","Verifying webhook signatures","Every webhook includes a signature header. Verify it using your WEBHOOK_SIGNING_SECRET before trusting the payload. Reject requests whose signature does not match to prevent forgery."),+ ("api/webhooks","Webhook retry policy","Failed webhook deliveries are retried with exponential backoff for up to 72 hours. Return a 2xx quickly; do heavy work asynchronously."),+ ("security","Keeping API keys safe","Never commit secret keys to source control or ship them in client-side code. Store them in environment variables or a secrets manager, rotate them regularly, and use restricted keys with least privilege."),+ ("security","Rotating a compromised key","If a key is exposed, roll it immediately from the dashboard; the old key is revoked within seconds. Update WEBHOOK_SIGNING_SECRET and any deployed services."),+ ("guides","Idempotency keys","Send an Idempotency-Key header on write requests so retries don't create duplicate charges. Keys are remembered for 24 hours."),+ ("guides","Testing with sandbox mode","Use sandbox keys (prefix sk_test_) to simulate payments without moving money. Card 4242 4242 4242 4242 always succeeds; card 4000 0000 0000 0002 triggers E4304."),+ ("guides","Migrating to API v2.11","Version v2.11 changes the checkout response shape: session.url replaces session.redirect_url. Pin your version header to 2.11 and update your redirect code."),+ ("guides","Rate limits","The API allows 100 requests/second per account. Bursts above that return 429; back off and retry. Contact support to raise limits."),+ ("account","Setting up two-factor auth","Enable 2FA from account settings to protect your dashboard login. We support authenticator apps and hardware security keys."),+ ("account","Team roles and permissions","Owner, Admin, and Developer roles control dashboard access. Only Owners can rotate live keys or change billing."),+ ("guides","Handling declined payments","When a payment is declined, show the customer a clear message and offer to retry with another method. Distinguish retryable declines (insufficient funds) from hard declines (card reported lost)."),+ ("api/refunds","createRefund","createRefund issues a full or partial refund on a completed charge. Refunds settle to the original method in 5–10 business days."),+ ("api/refunds","Refund failures","A refund can fail if the original charge is too old (over 180 days) or already fully refunded."),+]+docs=[dict(id=f"c{i}",doc=d,heading=h,text=f"{h}. {t}",body=t) for i,(d,h,t) in enumerate(CHUNKS)]+json.dump(docs, open("/tmp/claude-1001/-home-claude-ai-xerj/09b4eaff-87bf-4097-b72f-3907d1ef6cd4/scratchpad/rag/chunks.json","w"))+print(f"{len(docs)} document chunks")diff --git a/docs/examples/rag-hybrid-retrieval/02_embed_index.py b/docs/examples/rag-hybrid-retrieval/02_embed_index.pynew file mode 100644index 000000000..74e11c31d--- /dev/null+++ b/docs/examples/rag-hybrid-retrieval/02_embed_index.py@@ -0,0 +1,22 @@+import json, urllib.request+X="http://127.0.0.1:9200"; OLL="http://127.0.0.1:11434/api/embeddings"+def embed(t):+ r=urllib.request.Request(OLL,data=json.dumps({"model":"embeddinggemma","prompt":t}).encode(),headers={"Content-Type":"application/json"})+ return json.loads(urllib.request.urlopen(r).read())["embedding"]+def x(path,body=None,method="POST",nd=False):+ ct="application/x-ndjson" if nd else "application/json"+ data=body.encode() if isinstance(body,str) else (json.dumps(body).encode() if body is not None else None)+ r=urllib.request.Request(f"{X}/{path}",data=data,headers={"Content-Type":ct},method=method)+ try: return json.loads(urllib.request.urlopen(r).read())+ except Exception as e: return {"error":str(e)[:150]}+docs=json.load(open("/tmp/claude-1001/-home-claude-ai-xerj/09b4eaff-87bf-4097-b72f-3907d1ef6cd4/scratchpad/rag/chunks.json"))+x("docchunks",method="DELETE")+x("docchunks",{"mappings":{"properties":{+ "doc":{"type":"keyword"},"heading":{"type":"text"},"text":{"type":"text"},"body":{"type":"text"},+ "vec":{"type":"dense_vector","dims":768,"similarity":"cosine"}}}},"PUT")+lines=[]+for d in docs:+ dd=dict(d); dd["vec"]=embed(d["text"])+ lines.append(json.dumps({"index":{"_id":d["id"]}})); lines.append(json.dumps(dd))+print("indexed:", x("docchunks/_bulk","\n".join(lines)+"\n",nd=True).get("errors"))+x("docchunks/_refresh")diff --git a/docs/examples/rag-hybrid-retrieval/03_retrieval_compare.py b/docs/examples/rag-hybrid-retrieval/03_retrieval_compare.pynew file mode 100644index 000000000..7de8cd74f--- /dev/null+++ b/docs/examples/rag-hybrid-retrieval/03_retrieval_compare.py@@ -0,0 +1,30 @@+#!/usr/bin/env python3+"""RAG retrieval: pure cosine (pgvector-style) vs BM25 vs RRF hybrid, over the+doc chunks. Shows where dense-only retrieval buries exact identifiers, and how+hybrid recovers them without losing concept-matching. Needs XERJ :9200 + Ollama."""+import json, urllib.request+X="http://127.0.0.1:9200"; OLL="http://127.0.0.1:11434/api/embeddings"+def embed(t):+ r=urllib.request.Request(OLL,data=json.dumps({"model":"embeddinggemma","prompt":t}).encode(),headers={"Content-Type":"application/json"})+ return json.loads(urllib.request.urlopen(r).read())["embedding"]+def S(b):+ r=urllib.request.Request(f"{X}/docchunks/_search",data=json.dumps(b).encode(),headers={"Content-Type":"application/json"})+ return json.loads(urllib.request.urlopen(r).read())+def rank(sub,b):+ for i,h in enumerate(S(b)["hits"]["hits"]):+ if sub in h["_source"]["heading"]: return i+1+ return ">10"+def cosine(qv): return {"query":{"knn":{"field":"vec","query_vector":qv,"k":18,"num_candidates":18}},"size":18}+def bm25(q): return {"query":{"match":{"text":q}},"size":18}+def hybrid(q,qv): return {"query":{"hybrid":{"queries":[+ {"query":{"match":{"text":q}},"weight":1.0},+ {"query":{"knn":{"field":"vec","query_vector":qv,"num_candidates":18}},"weight":1.0}],+ "fusion":{"type":"rrf","k":60}}},"size":18}++tests={"E4304":"Error E4304","createCheckoutSession":"createCheckoutSession",+ "WEBHOOK_SIGNING_SECRET":"Verifying webhook","v2.11":"Migrating to API v2.11"}+print(f" {'exact-term query':26} {'cosine':>7} {'BM25':>6} {'hybrid':>7} (rank of the correct chunk)")+for q,correct in tests.items():+ qv=embed(q)+ print(f" {q:26} {str(rank(correct,cosine(qv))):>7} {str(rank(correct,bm25(q))):>6} {str(rank(correct,hybrid(q,qv))):>7}")+print("\n <=k means the chunk is inside a RAG top-k window; cosine's #13/#14 are invisible to the LLM.")diff --git a/docs/examples/rag-hybrid-retrieval/README.md b/docs/examples/rag-hybrid-retrieval/README.mdnew file mode 100644index 000000000..3ecdd6d23--- /dev/null+++ b/docs/examples/rag-hybrid-retrieval/README.md@@ -0,0 +1,112 @@+# Hybrid retrieval for RAG chatbots (dropping in for pgvector, adding BM25)++A common RAG-chatbot shape: documents → chunks → embeddings → **pure cosine**+similarity (often Supabase/`pgvector`) → feed top-k to an LLM. It works well until+a user pastes an **exact term** — an error code, a function name, a config key, a+version — and dense-only retrieval quietly returns the *wrong* chunk. This example+shows that failure on a real corpus, and how XERJ fixes the **retrieval half** by+adding BM25 and fusing it with vectors (RRF) — same chunks, same embeddings.++**Tested live** against a running XERJ with real 768-dim EmbeddingGemma vectors over+18 doc chunks. Every number below is real output.++## Candid scope — what XERJ does and doesn't touch++RAG has several hard parts, and **most of them are not retrieval**:++| RAG concern | who owns it |+|---|---|+| chunking strategy (size, overlap, boundaries) | **your app** — XERJ doesn't chunk |+| which embedding model, how you embed | **your app** — XERJ stores/searches the vectors you send |+| LLM wiring, prompt, context assembly, streaming | **your app** |+| **retrieval: cosine + BM25 + hybrid fusion, filters, at scale** | **XERJ** |++So this is honest about its lane: if your pain is chunking or the LLM loop, XERJ+won't move that needle. If your retrieval is **pure cosine and you want hybrid**,+that's exactly the half this replaces — and it's a drop-in: keep your chunker,+embedder, and LLM; swap the `pgvector` similarity query for an XERJ hybrid query.++## The failure: dense-only buries exact terms++`03_retrieval_compare.py` — rank of the *correct* chunk (lower is better):++```+ exact-term query cosine BM25 hybrid+ E4304 13 1 3+ createCheckoutSession 14 1 1+ WEBHOOK_SIGNING_SECRET 4 1 1+ v2.11 1 1 1+```++Pure cosine puts the chunk that literally defines `E4304` at **#13** and+`createCheckoutSession` at **#14**. In a RAG chatbot that retrieves top-4 or top-5,+**those chunks never enter the context window** — the LLM never sees them. Asked+*"E4304"*, dense-only returns `E5012 — gateway timeout` at #1: a different error+that's *semantically adjacent* (both are "errors") but factually wrong. This is the+well-known lexical gap of dense retrieval: embeddings blur rare tokens (codes,+identifiers, versions) that users paste verbatim.++BM25 nails all of them at #1. **Hybrid (RRF)** keeps BM25's exactness (E4304 → #3,+inside a top-k window; the rest → #1) *and* keeps vectors' concept-matching, so it's+robust whether the query is an exact paste or a fuzzy question.++## Why it decides the answer, not just the ranking++Retrieval quality is answer quality in RAG. With the *wrong* chunk retrieved+(dense-only → `E5012`), the grounded LLM can only say *"the context doesn't cover+E4304"* — a refusal at best, a confident wrong answer at worst. With hybrid+retrieving a chunk that actually discusses E4304, the LLM answers correctly+(*"E4304 is not retryable"*). Same model, same prompt — **the retrieval changed the+outcome.**++## The hybrid query (RRF fusion, one request)++```json+POST /docchunks/_search+{ "query": { "hybrid": {+ "queries": [+ { "query": { "match": { "text": "<user question>" } }, "weight": 1.0 },+ { "query": { "knn": { "field": "vec", "query_vector": [...], "num_candidates": 50 } }, "weight": 1.0 }+ ],+ "fusion": { "type": "rrf", "k": 60 } } },+ "size": 5 }+```+RRF fuses the two ranked lists by `Σ weight/(k + rank)` — rank-based, so it needs no+score normalization between BM25 and cosine (which live on different scales). Tune+`weight`s toward lexical for identifier-heavy corpora, toward vector for prose;+`linear` fusion is also available.++## Drop-in migration from pgvector++```+your pipeline (unchanged): docs → chunk → embed(model) ─┐+ ▼+before: INSERT into pgvector; SELECT ... ORDER BY embedding <=> query LIMIT k (cosine only)+after: bulk-index chunks {text, vec} into XERJ; POST _search {hybrid: rrf} (BM25 + vector)+ │+your LLM loop (unchanged): retrieved chunks → prompt → answer+```+You send the same embedding vectors; you gain BM25 + hybrid + keyword filters+(`doc`, `section`, `version`) in the same query, and the retrieval leaves your+primary DB.++## Run it++Prereqs: XERJ on `:9200`, Ollama serving `embeddinggemma` (+ an LLM for the answer step).+```bash+python3 01_corpus.py # 18 technical-doc chunks (identifiers + concepts)+python3 02_embed_index.py # embed -> dense_vector; index `docchunks`+python3 03_retrieval_compare.py # the cosine vs BM25 vs hybrid rank table+```+Swap `embeddinggemma` for your model; the retrieval layer is model-agnostic.++## Honest limitations++- **Hybrid isn't automatically #1 for every exact term** — equal-weight RRF put+ `E4304` at #3 (diluted by cosine's #13). #3 is inside a normal top-k window (the+ point), but weight-tuning or a light query router (identifier-detected → lexical+ weight up) sharpens it.+- **Retrieval quality ≠ answer quality alone** — chunking and the prompt still+ matter; XERJ improves the retrieval input, not the generation.+- **Embedding quality is your model's job.** XERJ stores and searches vectors; it+ doesn't make them better.