AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs(case-study): auto-replicate Postgres -> XERJ (CDC) + hybrid search (daily.dev)
Xerj Squad A04259a483d6a
diff --git a/docs/case-studies/daily-dev-postgres-cdc/README.md b/docs/case-studies/daily-dev-postgres-cdc/README.mdnew file mode 100644index 000000000..c855db775--- /dev/null+++ b/docs/case-studies/daily-dev-postgres-cdc/README.md@@ -0,0 +1,179 @@+# Case study: auto-replicate Postgres → XERJ (CDC) + hybrid search — for daily.dev++**Who:** [Ido Shamun](https://github.com/idoshamun), founder of+[daily.dev](https://daily.dev) — the personalized developer news feed (1000++sources, open-source API at+[`dailydotdev/daily-api`](https://github.com/dailydotdev/daily-api)). daily.dev+runs on Postgres and uses `pgvector` for semantic features.++**His ask:** *"Interesting! is it possible to replicate automatically from pg?"* —+i.e. keep a search/analytics copy in XERJ **in sync with Postgres automatically**,+then run **hybrid (keyword + vector) queries** over it, moving the search workload+off `pg tsvector + pgvector`.++**Answer: yes — and it's tested end-to-end below.** Real Postgres 16 + pgvector+0.8.2 with **logical replication**, a CDC consumer that streams every+INSERT/UPDATE/DELETE into XERJ (with EmbeddingGemma vectors), then hybrid search.+Every result here is real output from the scripts in this directory.++---++## Grounded in daily.dev's real schema++Modeled on the actual `Post` entity in `daily-api`+([`src/entity/posts/Post.ts`](https://github.com/dailydotdev/daily-api/blob/main/src/entity/posts/Post.ts)):+`title`, `summary`, `tagsStr`, `sourceId`, engagement (`views`, `upvotes`,+`comments`, `score`, `trending`), a **`tsv` tsvector** (they already do lexical+search in pg), and — crucially — a **`metadataChangedAt`** watermark column+(daily.dev bumps it on every change). We add a `pgvector` `embedding` column, which+is where the "moving off pgvector" question lives: **pg's `tsvector` and `pgvector`+don't fuse into one ranked hybrid result** — you run two queries and merge in app+code. That's the friction this removes.++---++## The pipeline (tested)++```+ Postgres (daily.dev `post` table, pgvector)+ │ logical replication slot (WAL → logical decoding)+ ▼+ cdc_sync.py — reads changed ids, fetches current rows,+ │ embeds title+summary, bulk-upserts / deletes+ ▼+ XERJ `posts` index (text for BM25 + dense_vector for kNN, one store)+ │+ ▼+ hybrid query (BM25 + vector, one request) + rc.6 knn+aggregations+```++### 1. Automatic replication — proven live++Initial load streamed 8 posts. Then three live changes in Postgres:+```sql+UPDATE post SET upvotes=2500, trending=99 WHERE id='p7'; -- engagement update+INSERT INTO post (...) VALUES ('p9','WebAssembly beyond the browser', ...); -- new post+DELETE FROM post WHERE id='p5'; -- removed post+```+One `cdc_sync.py` pass (drains the slot → XERJ), real output:+```+[cdc] synced 2 upserts, 1 deletes to XERJ changed ids: [('p7','UPDATE'),('p9','INSERT'),('p5','DELETE')]+ p7 upvotes now 2500 trending 99 (updated)+ p5 deleted from XERJ ✓+ p9 present: WebAssembly beyond the browser+```+The WAL slot captured every mutation; XERJ reflects the update, the insert, and the+delete — no manual reindex. Run the consumer on a loop (`--loops N`) or as a daemon+for continuous, near-real-time sync.++### 2. Hybrid search — the thing pg can't do cleanly++Query *"ditching the JVM search stack for something cheaper"* (semantic; little+literal overlap with any post). Real output:++```+LEXICAL only (BM25 ≈ pg tsvector): 4.00 Why we moved off Elasticsearch+VECTOR only (≈ pgvector): 0.859 Why we moved off Elasticsearch+ 0.845 The cost of microservices ← semantically near but off-topic+HYBRID (BM25 + vector, one query): 2.386 Why we moved off Elasticsearch (upvotes 1203)+ 2.099 Building a vector search engine from scratch+```+Vector-alone drifts to "cost of microservices"; lexical-alone can't see synonyms;+**hybrid fuses both in one request** and you can layer engagement (`upvotes`,+`trending`) into ranking — exactly what a developer feed needs. In Postgres this is+two separate queries (`tsvector` and `pgvector`) merged in application code.++### 3. Bonus (rc.6): semantic slice + engagement analytics in one request+*"of posts about AI/ML, which sources, and avg upvotes?"* — one `knn`+`aggs` call+returns the source breakdown with average upvotes per source. (See the+[analytics case study](../calltree-analytics/) for that feature.)++---++## Step-by-step (runnable)++Prereqs: Docker, XERJ on `:9200`, Ollama serving `embeddinggemma`, `pip install psycopg2-binary`.++```bash+# 1. Postgres 16 + pgvector, logical replication ON+docker run -d --name pg-daily -e POSTGRES_PASSWORD=pw -e POSTGRES_DB=dailydev -p 5433:5432 \+ pgvector/pgvector:pg16 -c wal_level=logical -c max_replication_slots=4 -c max_wal_senders=4++# 2. schema (daily.dev-modeled) + trigger + logical replication slot+docker cp setup.sql pg-daily:/tmp/ && docker exec pg-daily psql -U postgres -d dailydev -f /tmp/setup.sql+# …then INSERT your posts (see the seed block in the case study history)++# 3. initial sync + continuous CDC+python3 cdc_sync.py --init # create the XERJ `posts` index + first drain+python3 cdc_sync.py --loops 100 # near-real-time: drain the slot every 2s++# 4. hybrid query (see the demo in this README)+```++`cdc_sync.py` uses the slot as a **change signal** (which ids changed + op), then+reads the *current* row from Postgres and upserts to XERJ — a robust CDC pattern+that avoids brittle decoder-format parsing and always converges to Postgres state.++---++## Production CDC options (honest tradeoffs)++| approach | latency | setup | notes |+|---|---|--:|---|+| **`test_decoding` slot** (this demo) | seconds (polled `get_changes`) | built-in | readable output; great for a consumer you own; the pattern shown here |+| **`pgoutput` slot** (built-in) | streaming | built-in | native logical-replication protocol; binary — consume via a client lib |+| **`wal2json` slot** | streaming | plugin | JSON change events; easy to parse; needs the plugin installed |+| **Debezium (Kafka Connect)** | streaming | heavier | industry standard for large fleets; exactly-once, schema history, many sinks |+| **watermark polling** (`metadata_changed_at > last_seen`) | seconds–minutes | trivial | daily.dev's schema already has the column; no slot; simplest, at-least-once |++For daily.dev's scale, **Debezium → the consumer**, or a **`pgoutput` slot** read+by a small service, is the production shape; the `test_decoding` consumer here is+the same logic without the Kafka footprint. All of them feed the identical+transform+embed+bulk step.++## Scaling++- **Embedding is the cost, not the sync.** Only changed rows are re-embedded (the+ slot tells you which). A stats-only update (`upvotes`) can skip re-embedding —+ gate on whether `title`/`summary` changed. Batch embeds for throughput.+- **Backfill once, stream forever.** Initial load is a bulk pass; steady state is+ just the change stream. Re-embedding on a model change is a backfill job, not a+ query-time cost.+- **Object-store-backed XERJ** keeps the search copy cheap; hot segments cache.+- **Multi-tenant / per-source:** filter by `source_id`, or per-tenant indices; the+ vector `similarity`/dims are per-field so different embedding models coexist.+- **Ranking:** fold `upvotes`/`trending`/recency into the hybrid score with+ function-score / boosts — feed ranking and search share one index.++## Comparison with staying on pg/pgvector++| | Postgres + pgvector + tsvector | XERJ (CDC from pg) |+|---|---|---|+| hybrid keyword+vector ranking | two queries merged in app code | **one query, fused ranking** |+| search load vs OLTP | competes with primary DB | **offloaded**; pg stays transactional |+| aggregations over a semantic slice | awkward | `knn`+`aggs` in one request (rc.6) |+| keeping in sync | you build it | **logical-replication CDC**, shown here |+| BM25 relevance | `ts_rank` (coarse) | real BM25 + analyzers |+| operational model | one DB (simple) | +1 system, but search/analytics isolated |++**When to stay on pg:** small corpora, low query volume, simplicity over search+quality. **When XERJ wins for daily.dev:** feed/search at scale where hybrid+ranking + engagement signals + analytics matter and you don't want search load on+the primary, kept in sync automatically.++## Honest limitations++1. **No fused RRF hybrid node** — `bool.should[match, knn]` with boosts approximates+ it; principled RRF is a follow-up. (Score scales differ between BM25 and cosine;+ tune boosts or normalize.)+2. **`test_decoding` is polled** via `get_changes` — for true push-streaming use+ `pgoutput`/Debezium; the consumer logic is identical.+3. **Embedding model is yours** — retrieval quality is the model's job; XERJ stores+ and searches the vectors.+4. **Exactly-once** needs care — the demo consumer is idempotent (upsert by id,+ delete by id), which gives at-least-once → convergent state; add slot-LSN+ checkpointing for strict delivery guarantees.++## Files+- [`setup.sql`](setup.sql) — daily.dev-modeled schema + trigger + logical slot.+- [`cdc_sync.py`](cdc_sync.py) — the CDC consumer (slot → embed → XERJ bulk).diff --git a/docs/case-studies/daily-dev-postgres-cdc/cdc_sync.py b/docs/case-studies/daily-dev-postgres-cdc/cdc_sync.pynew file mode 100644index 000000000..db20dbeb7--- /dev/null+++ b/docs/case-studies/daily-dev-postgres-cdc/cdc_sync.py@@ -0,0 +1,75 @@+#!/usr/bin/env python3+"""Postgres -> XERJ CDC sync via logical replication (test_decoding slot).+The slot streams every INSERT/UPDATE/DELETE on `post`; for each changed id we+read the current row, embed title+summary, and upsert into XERJ. Deletes remove+the doc. Keeps XERJ in sync automatically — no manual reindex."""+import json, re, sys, time, urllib.request+import psycopg2+X="http://127.0.0.1:9200"; OLL="http://127.0.0.1:11434/api/embeddings"+PG=dict(host="127.0.0.1",port=5433,dbname="dailydev",user="postgres",password="pw")+def embed(t):+ r=urllib.request.Request(OLL,data=json.dumps({"model":"embeddinggemma","prompt":t}).encode(),headers={"Content-Type":"application/json"})+ return json.loads(urllib.request.urlopen(r).read())["embedding"]+def xreq(path,body=None,method="POST",ndjson=False):+ ct="application/x-ndjson" if ndjson else "application/json"+ data=body.encode() if isinstance(body,str) else (json.dumps(body).encode() if body is not None else None)+ r=urllib.request.Request(f"{X}/{path}",data=data,headers={"Content-Type":ct},method=method)+ try: return json.loads(urllib.request.urlopen(r).read())+ except Exception as e: return {"error":str(e)[:200]}++def ensure_index():+ xreq("posts",method="DELETE")+ xreq("posts",{"mappings":{"properties":{+ "id":{"type":"keyword"},"title":{"type":"text"},"summary":{"type":"text"},+ "tags":{"type":"keyword"},"source_id":{"type":"keyword"},+ "views":{"type":"integer"},"upvotes":{"type":"integer"},"comments":{"type":"integer"},+ "score":{"type":"integer"},"trending":{"type":"integer"},+ "created_at":{"type":"date"},+ "vec":{"type":"dense_vector","dims":768,"similarity":"cosine"}}}},method="PUT")++# parse test_decoding lines: "table public.post: INSERT: id[text]:'p1' ..."+CHG=re.compile(r"table public\.post: (INSERT|UPDATE|DELETE): .*?\bid\[text\]:'([^']+)'")+def drain_slot(cur):+ cur.execute("SELECT data FROM pg_logical_slot_get_changes('xerj_slot',NULL,NULL);")+ ops={} # id -> last op+ for (data,) in cur.fetchall():+ m=CHG.search(data)+ if m: ops[m.group(2)]=m.group(1)+ return ops++def sync(ids_ops, cur):+ up=[]; delete=[]+ for pid,op in ids_ops.items():+ if op=="DELETE": delete.append(pid); continue+ cur.execute("SELECT id,title,summary,tags_str,source_id,views,upvotes,comments,score,trending,visible,deleted,created_at FROM post WHERE id=%s",(pid,))+ row=cur.fetchone()+ if not row or row[11] or not row[10]: # deleted or not visible -> remove+ delete.append(pid); continue+ d=dict(id=row[0],title=row[1],summary=row[2],tags=(row[3] or "").split(","),+ source_id=row[4],views=row[5],upvotes=row[6],comments=row[7],score=row[8],+ trending=row[9],created_at=row[12].isoformat())+ d["vec"]=embed(f"{row[1]}. {row[2]}")+ up.append(d)+ lines=[]+ for d in up:+ lines.append(json.dumps({"index":{"_id":d["id"]}})); lines.append(json.dumps(d))+ for pid in delete:+ lines.append(json.dumps({"delete":{"_id":pid}}))+ if lines:+ r=xreq("posts/_bulk","\n".join(lines)+"\n",ndjson=True)+ xreq("posts/_refresh",method="POST")+ return len(up),len(delete),r.get("errors")+ return 0,0,False++if __name__=="__main__":+ conn=psycopg2.connect(**PG); conn.autocommit=True; cur=conn.cursor()+ if "--init" in sys.argv: ensure_index()+ loops = int(sys.argv[sys.argv.index("--loops")+1]) if "--loops" in sys.argv else 1+ for i in range(loops):+ ops=drain_slot(cur)+ if ops:+ u,dd,err=sync(ops,cur)+ print(f"[cdc] synced {u} upserts, {dd} deletes to XERJ (errors={err}) changed ids: {list(ops.items())}")+ else:+ print("[cdc] no changes")+ if loops>1: time.sleep(2)diff --git a/docs/case-studies/daily-dev-postgres-cdc/setup.sql b/docs/case-studies/daily-dev-postgres-cdc/setup.sqlnew file mode 100644index 000000000..eca39f3ed--- /dev/null+++ b/docs/case-studies/daily-dev-postgres-cdc/setup.sql@@ -0,0 +1,22 @@+-- daily.dev-modeled `post` table (columns mirror dailydotdev/daily-api Post entity:+-- title, summary, tagsStr, sourceId, views/upvotes/comments/score/trending,+-- metadataChangedAt watermark) + a pgvector column for semantic search.+CREATE EXTENSION IF NOT EXISTS vector;+CREATE TABLE IF NOT EXISTS post (+ id text PRIMARY KEY, title text, summary text, tags_str text, source_id text,+ type text default 'article', views int default 0, upvotes int default 0,+ comments int default 0, score int default 0, trending int,+ visible boolean default true, deleted boolean default false,+ created_at timestamptz default now(), metadata_changed_at timestamptz default now(),+ embedding vector(768)+);+-- mirror daily.dev: bump the change-watermark on any update+CREATE OR REPLACE FUNCTION touch_mc() RETURNS trigger AS $BODY$+BEGIN NEW.metadata_changed_at = now(); RETURN NEW; END; $BODY$ LANGUAGE plpgsql;+DROP TRIGGER IF EXISTS post_touch ON post;+CREATE TRIGGER post_touch BEFORE UPDATE ON post FOR EACH ROW EXECUTE FUNCTION touch_mc();+-- logical replication: capture the full row image for UPDATE/DELETE+ALTER TABLE post REPLICA IDENTITY FULL;+-- the CDC slot (built-in test_decoding plugin; swap for wal2json/pgoutput/Debezium in prod)+SELECT pg_create_logical_replication_slot('xerj_slot','test_decoding')+ WHERE NOT EXISTS (SELECT 1 FROM pg_replication_slots WHERE slot_name='xerj_slot');