Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs(case-study): implement all 3 CDC/hybrid follow-ups (RRF, push-streaming, LSN) — rc.6

Xerj Squad A7e4d0493bb51
diff --git a/docs/case-studies/daily-dev-postgres-cdc/README.md b/docs/case-studies/daily-dev-postgres-cdc/README.md
index c855db775..4ed8e4dd4 100644
--- a/docs/case-studies/daily-dev-postgres-cdc/README.md
+++ b/docs/case-studies/daily-dev-postgres-cdc/README.md
@@ -66,22 +66,36 @@ The WAL slot captured every mutation; XERJ reflects the update, the insert, and
delete — no manual reindex. Run the consumer on a loop (`--loops N`) or as a daemon
for continuous, near-real-time sync.
-### 2. Hybrid search — the thing pg can't do cleanly
-
-Query *"ditching the JVM search stack for something cheaper"* (semantic; little
-literal overlap with any post). Real output:
-
+### 2. Hybrid search with **RRF fusion** — the thing pg can't do cleanly
+
+XERJ has a first-class `hybrid` query with **reciprocal-rank fusion** built in — no
+app-side merge. Query *"ditching the JVM search stack for something cheaper"*
+(semantic; little literal overlap):
+
+```json
+POST /posts/_search
+{ "query": { "hybrid": {
+ "queries": [
+ { "query": { "match": { "summary": "ditching the JVM search stack for something cheaper" } }, "weight": 1.0 },
+ { "query": { "knn": { "field": "vec", "query_vector": [...], "num_candidates": 9 } }, "weight": 1.0 }
+ ],
+ "fusion": { "type": "rrf", "k": 60 } } } }
```
-LEXICAL only (BM25 ≈ pg tsvector): 4.00 Why we moved off Elasticsearch
-VECTOR only (≈ pgvector): 0.859 Why we moved off Elasticsearch
- 0.845 The cost of microservices ← semantically near but off-topic
-HYBRID (BM25 + vector, one query): 2.386 Why we moved off Elasticsearch (upvotes 1203)
- 2.099 Building a vector search engine from scratch
+Real output:
```
-Vector-alone drifts to "cost of microservices"; lexical-alone can't see synonyms;
-**hybrid fuses both in one request** and you can layer engagement (`upvotes`,
-`trending`) into ranking — exactly what a developer feed needs. In Postgres this is
-two separate queries (`tsvector` and `pgvector`) merged in application code.
+LEXICAL only (BM25 ≈ pg tsvector): Why we moved off Elasticsearch
+VECTOR only (≈ pgvector): Why we moved off Elasticsearch
+ The cost of microservices ← semantically near but off-topic
+RRF HYBRID (fused, one request): 0.03279 Why we moved off Elasticsearch
+ 0.03175 The cost of microservices
+ 0.03151 Building a vector search engine from scratch
+```
+`fusion: rrf` fuses the two ranked lists by `Σ weight/(k + rank)` — no manual score
+normalization (BM25 and cosine live on different scales; RRF is rank-based, so it
+sidesteps that). `linear` fusion and per-query `weight`s are also supported. In
+Postgres this is two separate queries (`tsvector` + `pgvector`) merged in app code;
+here it's one query with principled fusion, and you can still layer engagement
+(`upvotes`, `trending`) via function-score.
### 3. Bonus (rc.6): semantic slice + engagement analytics in one request
*"of posts about AI/ML, which sources, and avg upvotes?"* — one `knn`+`aggs` call
@@ -114,6 +128,36 @@ python3 cdc_sync.py --loops 100 # near-real-time: drain the slot every 2s
reads the *current* row from Postgres and upserts to XERJ — a robust CDC pattern
that avoids brittle decoder-format parsing and always converges to Postgres state.
+### Push-streaming + LSN checkpointing (`cdc_stream.py`) — production shape
+
+`cdc_sync.py` polls (`get_changes`). `cdc_stream.py` is the **true push-streaming**
+consumer: psycopg2's `LogicalReplicationConnection.consume_stream()` **blocks on the
+WAL** (no polling), and after each change is durably applied to XERJ it calls
+`send_feedback(flush_lsn=…)` so Postgres advances the slot's `confirmed_flush_lsn`.
+
+```bash
+python3 cdc_stream.py # blocks; applies changes the instant they commit
+```
+
+**Proven live** — an INSERT in Postgres appears in XERJ within seconds, no poll:
+```
+[stream] INSERT p10 -> XERJ (errors=False) LSN=27860016 [checkpointing]
+```
+
+**Exactly-once-convergent (LSN checkpointing), proven:** stop the consumer → change
+data while it's *down* (`UPDATE p1 upvotes=9999`, `INSERT p11`) → XERJ goes stale
+(the WAL is retained by the slot) → **restart:**
+```
+[stream] consuming WAL from confirmed LSN ...
+[stream] UPDATE p1 -> XERJ LSN=27863424 [checkpointing]
+[stream] INSERT p11 -> XERJ LSN=27863976 [checkpointing]
+ p1 upvotes: 9999 p11: Offline change test
+```
+It resumed from the last checkpoint and applied **exactly** the missed changes — no
+re-sync, no loss. Combined with upsert/delete-by-id (idempotent replays), this gives
+**at-least-once delivery that converges to Postgres state**; the LSN feedback means
+a crash never loses a change and never forces a full rebuild.
+
---
## Production CDC options (honest tradeoffs)
@@ -163,17 +207,27 @@ the primary, kept in sync automatically.
## Honest limitations
-1. **No fused RRF hybrid node** — `bool.should[match, knn]` with boosts approximates
- it; principled RRF is a follow-up. (Score scales differ between BM25 and cosine;
- tune boosts or normalize.)
-2. **`test_decoding` is polled** via `get_changes` — for true push-streaming use
- `pgoutput`/Debezium; the consumer logic is identical.
-3. **Embedding model is yours** — retrieval quality is the model's job; XERJ stores
+1. **Embedding model is yours** — retrieval quality is the model's job; XERJ stores
and searches the vectors.
-4. **Exactly-once** needs care — the demo consumer is idempotent (upsert by id,
- delete by id), which gives at-least-once → convergent state; add slot-LSN
- checkpointing for strict delivery guarantees.
+2. **RRF `k` and weights need tuning** per corpus — RRF removes the BM25-vs-cosine
+ scale problem, but the fusion constant `k` and per-query `weight`s still shape
+ ranking; A/B them against click/upvote signals.
+3. **`learned` fusion is not implemented** — `rrf` and `linear` are; a learned
+ fuser (train weights on engagement) is a future item.
+4. **Backfill embeds cost** — a model change means re-embedding the corpus (a
+ backfill job); steady-state CDC only re-embeds changed rows (and can skip
+ re-embed on stats-only updates).
+
+## Resolved (previously listed as gaps)
+- ~~No fused RRF node~~ → **RRF exists and is first-class** (`hybrid` + `fusion:rrf`);
+ the earlier note was a request-syntax error. `weight`s and `linear` also supported.
+- ~~`test_decoding` only polled~~ → **`cdc_stream.py` is true push-streaming** via
+ `consume_stream()` (no polling).
+- ~~Exactly-once needs care~~ → **LSN checkpointing implemented** (`send_feedback`);
+ proven to resume from the last confirmed LSN and catch offline changes with no loss.
## Files
- [`setup.sql`](setup.sql) — daily.dev-modeled schema + trigger + logical slot.
-- [`cdc_sync.py`](cdc_sync.py) — the CDC consumer (slot → embed → XERJ bulk).
+- [`cdc_sync.py`](cdc_sync.py) — polled CDC consumer (slot → embed → XERJ bulk).
+- [`cdc_stream.py`](cdc_stream.py) — **push-streaming** consumer + **LSN checkpointing**
+ (the production shape).
diff --git a/docs/case-studies/daily-dev-postgres-cdc/cdc_stream.py b/docs/case-studies/daily-dev-postgres-cdc/cdc_stream.py
new file mode 100644
index 000000000..da7a47945
--- /dev/null
+++ b/docs/case-studies/daily-dev-postgres-cdc/cdc_stream.py
@@ -0,0 +1,65 @@
+#!/usr/bin/env python3
+"""Postgres -> XERJ CDC via TRUE push-streaming logical replication + LSN
+checkpointing (exactly-once-convergent).
+
+Uses psycopg2's LogicalReplicationConnection.consume_stream(): blocks on the WAL
+stream (no polling), and after each change is durably applied to XERJ it calls
+send_feedback(flush_lsn=...) so Postgres advances the slot's confirmed_flush_lsn.
+On restart, streaming resumes from the last confirmed LSN -> no missed or
+re-delivered-and-lost changes (upsert/delete by id makes replays idempotent)."""
+import json, re, sys, urllib.request
+import psycopg2
+from psycopg2.extras import LogicalReplicationConnection, REPLICATION_LOGICAL
+X="http://127.0.0.1:9200"; OLL="http://127.0.0.1:11434/api/embeddings"
+PG=dict(host="127.0.0.1",port=5433,dbname="dailydev",user="postgres",password="pw")
+def embed(t):
+ r=urllib.request.Request(OLL,data=json.dumps({"model":"embeddinggemma","prompt":t}).encode(),headers={"Content-Type":"application/json"})
+ return json.loads(urllib.request.urlopen(r).read())["embedding"]
+def xbulk(lines):
+ r=urllib.request.Request(f"{X}/posts/_bulk",data=("\n".join(lines)+"\n").encode(),
+ headers={"Content-Type":"application/x-ndjson"},method="POST")
+ return json.loads(urllib.request.urlopen(r).read()).get("errors")
+
+CHG=re.compile(r"table public\.post: (INSERT|UPDATE|DELETE): .*?\bid\[text\]:'([^']+)'")
+# a read connection to fetch current row state for changed ids
+_read=psycopg2.connect(**PG); _read.autocommit=True; _rc=_read.cursor()
+def apply_change(op,pid):
+ if op=="DELETE":
+ return [json.dumps({"delete":{"_id":pid}})]
+ _rc.execute("SELECT id,title,summary,tags_str,source_id,views,upvotes,comments,score,trending,visible,deleted,created_at FROM post WHERE id=%s",(pid,))
+ row=_rc.fetchone()
+ if not row or row[11] or not row[10]:
+ return [json.dumps({"delete":{"_id":pid}})]
+ d=dict(id=row[0],title=row[1],summary=row[2],tags=(row[3] or "").split(","),source_id=row[4],
+ views=row[5],upvotes=row[6],comments=row[7],score=row[8],trending=row[9],
+ created_at=row[12].isoformat())
+ d["vec"]=embed(f"{row[1]}. {row[2]}")
+ return [json.dumps({"index":{"_id":pid}}),json.dumps(d)]
+
+class Consumer:
+ def __init__(self): self.applied=0
+ def __call__(self,msg):
+ m=CHG.search(msg.payload)
+ if m:
+ op,pid=m.group(1),m.group(2)
+ lines=apply_change(op,pid)
+ err=xbulk(lines)
+ self.applied+=1
+ print(f"[stream] {op} {pid} -> XERJ (errors={err}) LSN={msg.data_start} [checkpointing]",flush=True)
+ # LSN CHECKPOINT: confirm this change is durably applied; pg advances the slot.
+ msg.cursor.send_feedback(flush_lsn=msg.data_start)
+
+if __name__=="__main__":
+ conn=psycopg2.connect(connection_factory=LogicalReplicationConnection,**PG)
+ cur=conn.cursor()
+ try:
+ cur.start_replication(slot_name="xerj_slot",decode=True) # resumes at confirmed_flush_lsn
+ except psycopg2.ProgrammingError:
+ cur.create_replication_slot("xerj_slot",output_plugin="test_decoding")
+ cur.start_replication(slot_name="xerj_slot",decode=True)
+ print("[stream] consuming WAL from confirmed LSN (push, no polling; Ctrl-C to stop)",flush=True)
+ c=Consumer()
+ import signal
+ def stop(*_): print(f"\n[stream] stopped after {c.applied} changes"); sys.exit(0)
+ signal.signal(signal.SIGTERM,stop); signal.signal(signal.SIGINT,stop)
+ cur.consume_stream(c)