Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs(example): hybrid retrieval for RAG chatbots (drop-in for pgvector + BM25)

Xerj Squad Afa67008fa5af
diff --git a/docs/examples/rag-hybrid-retrieval/01_corpus.py b/docs/examples/rag-hybrid-retrieval/01_corpus.py
new file mode 100644
index 000000000..510214515
--- /dev/null
+++ b/docs/examples/rag-hybrid-retrieval/01_corpus.py
@@ -0,0 +1,28 @@
+# A small technical-docs corpus (the shape a RAG chatbot indexes): each item is a
+# CHUNK with a doc source + heading. Deliberately mixes exact identifiers
+# (error codes, function names, config keys, versions) with conceptual prose, so
+# we can show where pure-cosine retrieval fails and hybrid fixes it.
+import json
+CHUNKS = [
+ ("billing/errors","Error E4304 — card declined by issuer","Error E4304 is returned when the customer's card is declined by the issuing bank. Retrying immediately will fail; prompt the customer to contact their bank or use a different card. E4304 is not retryable and does not count against your retry budget."),
+ ("billing/errors","Error E4210 — insufficient funds","Error E4210 indicates the card has insufficient funds. Unlike E4304 this IS retryable after a delay; we recommend retrying after 24 hours with exponential backoff."),
+ ("billing/errors","Error E5012 — gateway timeout","Error E5012 is a transient gateway timeout. Retry with idempotency keys to avoid double charges."),
+ ("api/checkout","createCheckoutSession","Call createCheckoutSession to start a payment. It returns a session URL; redirect the customer there. Pass line_items, success_url, and cancel_url. The session expires after 24 hours."),
+ ("api/checkout","cancelCheckoutSession","cancelCheckoutSession voids an open session before payment. It is a no-op if the session already completed."),
+ ("api/webhooks","Verifying webhook signatures","Every webhook includes a signature header. Verify it using your WEBHOOK_SIGNING_SECRET before trusting the payload. Reject requests whose signature does not match to prevent forgery."),
+ ("api/webhooks","Webhook retry policy","Failed webhook deliveries are retried with exponential backoff for up to 72 hours. Return a 2xx quickly; do heavy work asynchronously."),
+ ("security","Keeping API keys safe","Never commit secret keys to source control or ship them in client-side code. Store them in environment variables or a secrets manager, rotate them regularly, and use restricted keys with least privilege."),
+ ("security","Rotating a compromised key","If a key is exposed, roll it immediately from the dashboard; the old key is revoked within seconds. Update WEBHOOK_SIGNING_SECRET and any deployed services."),
+ ("guides","Idempotency keys","Send an Idempotency-Key header on write requests so retries don't create duplicate charges. Keys are remembered for 24 hours."),
+ ("guides","Testing with sandbox mode","Use sandbox keys (prefix sk_test_) to simulate payments without moving money. Card 4242 4242 4242 4242 always succeeds; card 4000 0000 0000 0002 triggers E4304."),
+ ("guides","Migrating to API v2.11","Version v2.11 changes the checkout response shape: session.url replaces session.redirect_url. Pin your version header to 2.11 and update your redirect code."),
+ ("guides","Rate limits","The API allows 100 requests/second per account. Bursts above that return 429; back off and retry. Contact support to raise limits."),
+ ("account","Setting up two-factor auth","Enable 2FA from account settings to protect your dashboard login. We support authenticator apps and hardware security keys."),
+ ("account","Team roles and permissions","Owner, Admin, and Developer roles control dashboard access. Only Owners can rotate live keys or change billing."),
+ ("guides","Handling declined payments","When a payment is declined, show the customer a clear message and offer to retry with another method. Distinguish retryable declines (insufficient funds) from hard declines (card reported lost)."),
+ ("api/refunds","createRefund","createRefund issues a full or partial refund on a completed charge. Refunds settle to the original method in 5–10 business days."),
+ ("api/refunds","Refund failures","A refund can fail if the original charge is too old (over 180 days) or already fully refunded."),
+]
+docs=[dict(id=f"c{i}",doc=d,heading=h,text=f"{h}. {t}",body=t) for i,(d,h,t) in enumerate(CHUNKS)]
+json.dump(docs, open("/tmp/claude-1001/-home-claude-ai-xerj/09b4eaff-87bf-4097-b72f-3907d1ef6cd4/scratchpad/rag/chunks.json","w"))
+print(f"{len(docs)} document chunks")
diff --git a/docs/examples/rag-hybrid-retrieval/02_embed_index.py b/docs/examples/rag-hybrid-retrieval/02_embed_index.py
new file mode 100644
index 000000000..74e11c31d
--- /dev/null
+++ b/docs/examples/rag-hybrid-retrieval/02_embed_index.py
@@ -0,0 +1,22 @@
+import json, urllib.request
+X="http://127.0.0.1:9200"; OLL="http://127.0.0.1:11434/api/embeddings"
+def embed(t):
+ r=urllib.request.Request(OLL,data=json.dumps({"model":"embeddinggemma","prompt":t}).encode(),headers={"Content-Type":"application/json"})
+ return json.loads(urllib.request.urlopen(r).read())["embedding"]
+def x(path,body=None,method="POST",nd=False):
+ ct="application/x-ndjson" if nd else "application/json"
+ data=body.encode() if isinstance(body,str) else (json.dumps(body).encode() if body is not None else None)
+ r=urllib.request.Request(f"{X}/{path}",data=data,headers={"Content-Type":ct},method=method)
+ try: return json.loads(urllib.request.urlopen(r).read())
+ except Exception as e: return {"error":str(e)[:150]}
+docs=json.load(open("/tmp/claude-1001/-home-claude-ai-xerj/09b4eaff-87bf-4097-b72f-3907d1ef6cd4/scratchpad/rag/chunks.json"))
+x("docchunks",method="DELETE")
+x("docchunks",{"mappings":{"properties":{
+ "doc":{"type":"keyword"},"heading":{"type":"text"},"text":{"type":"text"},"body":{"type":"text"},
+ "vec":{"type":"dense_vector","dims":768,"similarity":"cosine"}}}},"PUT")
+lines=[]
+for d in docs:
+ dd=dict(d); dd["vec"]=embed(d["text"])
+ lines.append(json.dumps({"index":{"_id":d["id"]}})); lines.append(json.dumps(dd))
+print("indexed:", x("docchunks/_bulk","\n".join(lines)+"\n",nd=True).get("errors"))
+x("docchunks/_refresh")
diff --git a/docs/examples/rag-hybrid-retrieval/03_retrieval_compare.py b/docs/examples/rag-hybrid-retrieval/03_retrieval_compare.py
new file mode 100644
index 000000000..7de8cd74f
--- /dev/null
+++ b/docs/examples/rag-hybrid-retrieval/03_retrieval_compare.py
@@ -0,0 +1,30 @@
+#!/usr/bin/env python3
+"""RAG retrieval: pure cosine (pgvector-style) vs BM25 vs RRF hybrid, over the
+doc chunks. Shows where dense-only retrieval buries exact identifiers, and how
+hybrid recovers them without losing concept-matching. Needs XERJ :9200 + Ollama."""
+import json, urllib.request
+X="http://127.0.0.1:9200"; OLL="http://127.0.0.1:11434/api/embeddings"
+def embed(t):
+ r=urllib.request.Request(OLL,data=json.dumps({"model":"embeddinggemma","prompt":t}).encode(),headers={"Content-Type":"application/json"})
+ return json.loads(urllib.request.urlopen(r).read())["embedding"]
+def S(b):
+ r=urllib.request.Request(f"{X}/docchunks/_search",data=json.dumps(b).encode(),headers={"Content-Type":"application/json"})
+ return json.loads(urllib.request.urlopen(r).read())
+def rank(sub,b):
+ for i,h in enumerate(S(b)["hits"]["hits"]):
+ if sub in h["_source"]["heading"]: return i+1
+ return ">10"
+def cosine(qv): return {"query":{"knn":{"field":"vec","query_vector":qv,"k":18,"num_candidates":18}},"size":18}
+def bm25(q): return {"query":{"match":{"text":q}},"size":18}
+def hybrid(q,qv): return {"query":{"hybrid":{"queries":[
+ {"query":{"match":{"text":q}},"weight":1.0},
+ {"query":{"knn":{"field":"vec","query_vector":qv,"num_candidates":18}},"weight":1.0}],
+ "fusion":{"type":"rrf","k":60}}},"size":18}
+
+tests={"E4304":"Error E4304","createCheckoutSession":"createCheckoutSession",
+ "WEBHOOK_SIGNING_SECRET":"Verifying webhook","v2.11":"Migrating to API v2.11"}
+print(f" {'exact-term query':26} {'cosine':>7} {'BM25':>6} {'hybrid':>7} (rank of the correct chunk)")
+for q,correct in tests.items():
+ qv=embed(q)
+ print(f" {q:26} {str(rank(correct,cosine(qv))):>7} {str(rank(correct,bm25(q))):>6} {str(rank(correct,hybrid(q,qv))):>7}")
+print("\n <=k means the chunk is inside a RAG top-k window; cosine's #13/#14 are invisible to the LLM.")
diff --git a/docs/examples/rag-hybrid-retrieval/README.md b/docs/examples/rag-hybrid-retrieval/README.md
new file mode 100644
index 000000000..3ecdd6d23
--- /dev/null
+++ b/docs/examples/rag-hybrid-retrieval/README.md
@@ -0,0 +1,112 @@
+# Hybrid retrieval for RAG chatbots (dropping in for pgvector, adding BM25)
+
+A common RAG-chatbot shape: documents → chunks → embeddings → **pure cosine**
+similarity (often Supabase/`pgvector`) → feed top-k to an LLM. It works well until
+a user pastes an **exact term** — an error code, a function name, a config key, a
+version — and dense-only retrieval quietly returns the *wrong* chunk. This example
+shows that failure on a real corpus, and how XERJ fixes the **retrieval half** by
+adding BM25 and fusing it with vectors (RRF) — same chunks, same embeddings.
+
+**Tested live** against a running XERJ with real 768-dim EmbeddingGemma vectors over
+18 doc chunks. Every number below is real output.
+
+## Candid scope — what XERJ does and doesn't touch
+
+RAG has several hard parts, and **most of them are not retrieval**:
+
+| RAG concern | who owns it |
+|---|---|
+| chunking strategy (size, overlap, boundaries) | **your app** — XERJ doesn't chunk |
+| which embedding model, how you embed | **your app** — XERJ stores/searches the vectors you send |
+| LLM wiring, prompt, context assembly, streaming | **your app** |
+| **retrieval: cosine + BM25 + hybrid fusion, filters, at scale** | **XERJ** |
+
+So this is honest about its lane: if your pain is chunking or the LLM loop, XERJ
+won't move that needle. If your retrieval is **pure cosine and you want hybrid**,
+that's exactly the half this replaces — and it's a drop-in: keep your chunker,
+embedder, and LLM; swap the `pgvector` similarity query for an XERJ hybrid query.
+
+## The failure: dense-only buries exact terms
+
+`03_retrieval_compare.py` — rank of the *correct* chunk (lower is better):
+
+```
+ exact-term query cosine BM25 hybrid
+ E4304 13 1 3
+ createCheckoutSession 14 1 1
+ WEBHOOK_SIGNING_SECRET 4 1 1
+ v2.11 1 1 1
+```
+
+Pure cosine puts the chunk that literally defines `E4304` at **#13** and
+`createCheckoutSession` at **#14**. In a RAG chatbot that retrieves top-4 or top-5,
+**those chunks never enter the context window** — the LLM never sees them. Asked
+*"E4304"*, dense-only returns `E5012 — gateway timeout` at #1: a different error
+that's *semantically adjacent* (both are "errors") but factually wrong. This is the
+well-known lexical gap of dense retrieval: embeddings blur rare tokens (codes,
+identifiers, versions) that users paste verbatim.
+
+BM25 nails all of them at #1. **Hybrid (RRF)** keeps BM25's exactness (E4304 → #3,
+inside a top-k window; the rest → #1) *and* keeps vectors' concept-matching, so it's
+robust whether the query is an exact paste or a fuzzy question.
+
+## Why it decides the answer, not just the ranking
+
+Retrieval quality is answer quality in RAG. With the *wrong* chunk retrieved
+(dense-only → `E5012`), the grounded LLM can only say *"the context doesn't cover
+E4304"* — a refusal at best, a confident wrong answer at worst. With hybrid
+retrieving a chunk that actually discusses E4304, the LLM answers correctly
+(*"E4304 is not retryable"*). Same model, same prompt — **the retrieval changed the
+outcome.**
+
+## The hybrid query (RRF fusion, one request)
+
+```json
+POST /docchunks/_search
+{ "query": { "hybrid": {
+ "queries": [
+ { "query": { "match": { "text": "<user question>" } }, "weight": 1.0 },
+ { "query": { "knn": { "field": "vec", "query_vector": [...], "num_candidates": 50 } }, "weight": 1.0 }
+ ],
+ "fusion": { "type": "rrf", "k": 60 } } },
+ "size": 5 }
+```
+RRF fuses the two ranked lists by `Σ weight/(k + rank)` — rank-based, so it needs no
+score normalization between BM25 and cosine (which live on different scales). Tune
+`weight`s toward lexical for identifier-heavy corpora, toward vector for prose;
+`linear` fusion is also available.
+
+## Drop-in migration from pgvector
+
+```
+your pipeline (unchanged): docs → chunk → embed(model) ─┐
+ ▼
+before: INSERT into pgvector; SELECT ... ORDER BY embedding <=> query LIMIT k (cosine only)
+after: bulk-index chunks {text, vec} into XERJ; POST _search {hybrid: rrf} (BM25 + vector)
+ │
+your LLM loop (unchanged): retrieved chunks → prompt → answer
+```
+You send the same embedding vectors; you gain BM25 + hybrid + keyword filters
+(`doc`, `section`, `version`) in the same query, and the retrieval leaves your
+primary DB.
+
+## Run it
+
+Prereqs: XERJ on `:9200`, Ollama serving `embeddinggemma` (+ an LLM for the answer step).
+```bash
+python3 01_corpus.py # 18 technical-doc chunks (identifiers + concepts)
+python3 02_embed_index.py # embed -> dense_vector; index `docchunks`
+python3 03_retrieval_compare.py # the cosine vs BM25 vs hybrid rank table
+```
+Swap `embeddinggemma` for your model; the retrieval layer is model-agnostic.
+
+## Honest limitations
+
+- **Hybrid isn't automatically #1 for every exact term** — equal-weight RRF put
+ `E4304` at #3 (diluted by cosine's #13). #3 is inside a normal top-k window (the
+ point), but weight-tuning or a light query router (identifier-detected → lexical
+ weight up) sharpens it.
+- **Retrieval quality ≠ answer quality alone** — chunking and the prompt still
+ matter; XERJ improves the retrieval input, not the generation.
+- **Embedding quality is your model's job.** XERJ stores and searches vectors; it
+ doesn't make them better.