AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs: verified example — embed via any OpenAI-compatible API
Xerj Squad Ace509a3d2d44
diff --git a/docs/examples/external-embeddings/README.md b/docs/examples/external-embeddings/README.mdnew file mode 100644index 000000000..b526aecd3--- /dev/null+++ b/docs/examples/external-embeddings/README.md@@ -0,0 +1,74 @@+# External embeddings — use any OpenAI-compatible API++XERJ can embed `semantic_text` fields through an **external OpenAI-compatible+`/v1/embeddings` endpoint** instead of its built-in lexical or neural backends.+The same embedder is used at ingest *and* query time, so vectors stay+comparable — you bring the model, XERJ handles indexing, kNN, and hybrid+fusion.++This directory is a **key-free, reproducible proof** of that path.++## Run it++```bash+cargo build --release -p xerj-server # once, from the repo root+docs/examples/external-embeddings/run.sh+```++Expected: XERJ logs `embedding backend: external proxy`, the mock server logs+one call per document at ingest and one per query, and a semantic search for+*"database outage from too many open connections"* returns the document about+*"the connection pool was exhausted"* — proving the query was embedded by the+same external API and matched by vector similarity.++## How it's wired++`xerj-proxy.toml`:++```toml+[embedding]+mode = "proxy"+default_endpoint = "http://127.0.0.1:8900/v1/embeddings"+default_model = "mock-embed-256"+```++Start XERJ with `--config xerj-proxy.toml` (or set `--embed-mode proxy` and put+the endpoint in your config). The API key is read from the+`XERJ_EMBEDDING_API_KEY` environment variable and sent as+`Authorization: Bearer <key>`.++The wire contract is the standard one:++```+POST {default_endpoint}+{"input": ["text one", "text two"], "model": "<default_model>"}+→ {"data": [{"embedding": [ ... ], "index": 0}, { ... }]}+```++`mock_embed_server.py` implements exactly this in ~40 dependency-free lines, so+the example runs offline with no account.++## Point it at a real provider++Any endpoint that speaks the contract above works — just change+`default_endpoint`, `default_model`, and set `XERJ_EMBEDDING_API_KEY`:++| provider | `default_endpoint` |+|---|---|+| OpenAI | `https://api.openai.com/v1/embeddings` |+| Gemini (OpenAI-compatible mode) | `https://generativelanguage.googleapis.com/v1beta/openai/embeddings` |+| Local (text-embeddings-inference, LM Studio, llama.cpp server, …) | `http://localhost:PORT/v1/embeddings` |++> Verified in this repo against the reproducible mock, which implements the+> exact request/response shape XERJ sends. Real providers that speak the same+> OpenAI `/v1/embeddings` contract are drop-in; confirm your provider's dims+> match your `dense_vector`/`semantic_text` field and set the key. Endpoints+> and model names change over time — check the provider's current docs.++## Why this matters for agents++The most common way to get *wrong* RAG results is an embedder mismatch: you+index with model A and later query with model B, and the vectors are silently+incomparable. XERJ uses the **one configured embedder for both**, so an agent+that indexes a corpus and later searches it gets comparable vectors by+construction — no external inference service to keep in sync, just one endpoint.diff --git a/docs/examples/external-embeddings/mock_embed_server.py b/docs/examples/external-embeddings/mock_embed_server.pynew file mode 100644index 000000000..7b1f000b4--- /dev/null+++ b/docs/examples/external-embeddings/mock_embed_server.py@@ -0,0 +1,39 @@+#!/usr/bin/env python3+"""A tiny OpenAI-compatible /v1/embeddings server, to prove XERJ's external+embeddings integration end to end without needing a real provider key.+Deterministic, dependency-free. Logs every call so we can show XERJ hit it."""+import json, hashlib, http.server, sys++DIM = 256+CALLS = {"n": 0, "texts": 0}++def embed(text: str):+ # Deterministic bag-of-trigram hashed vector, L2-normalised — enough for+ # meaningful nearest-neighbour behaviour in the verification.+ v = [0.0] * DIM+ t = text.lower()+ for i in range(len(t) - 2):+ h = int(hashlib.md5(t[i:i+3].encode()).hexdigest(), 16)+ v[h % DIM] += 1.0+ n = sum(x * x for x in v) ** 0.5 or 1.0+ return [x / n for x in v]++class H(http.server.BaseHTTPRequestHandler):+ def log_message(self, *a): pass+ def do_POST(self):+ body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))+ texts = body.get("input", [])+ CALLS["n"] += 1; CALLS["texts"] += len(texts)+ sys.stderr.write(f"[mock-embeddings] call #{CALLS['n']}: {len(texts)} text(s), model={body.get('model')}\n"); sys.stderr.flush()+ out = {"object": "list", "model": body.get("model", "mock"),+ "data": [{"object": "embedding", "index": i, "embedding": embed(t)} for i, t in enumerate(texts)],+ "usage": {"prompt_tokens": CALLS["texts"], "total_tokens": CALLS["texts"]}}+ b = json.dumps(out).encode()+ self.send_response(200); self.send_header("Content-Type", "application/json")+ self.send_header("Content-Length", str(len(b))); self.end_headers(); self.wfile.write(b)+ def do_GET(self):+ b = json.dumps({"calls": CALLS["n"], "texts": CALLS["texts"]}).encode()+ self.send_response(200); self.send_header("Content-Length", str(len(b))); self.end_headers(); self.wfile.write(b)++if __name__ == "__main__":+ http.server.HTTPServer(("127.0.0.1", 8900), H).serve_forever()diff --git a/docs/examples/external-embeddings/run.sh b/docs/examples/external-embeddings/run.shnew file mode 100755index 000000000..4c02fad8b--- /dev/null+++ b/docs/examples/external-embeddings/run.sh@@ -0,0 +1,39 @@+#!/usr/bin/env bash+# Reproducible, key-free proof that XERJ embeds through an external+# OpenAI-compatible /v1/embeddings API. Run from the repo root after+# `cargo build --release -p xerj-server`.+#+# It starts a tiny mock embeddings server, points XERJ at it in `proxy` mode,+# indexes a few docs, and shows that (a) XERJ called the external API for+# ingest and (b) a semantic query is embedded by the same API. Swap the mock+# for any real OpenAI-compatible endpoint (OpenAI, Gemini's OpenAI-compatible+# embeddings, a local text-embeddings server, …) by editing xerj-proxy.toml.+set -euo pipefail+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+XERJ="${XERJ_BIN:-$HERE/../../../engine/target/release/xerj}"+DATA="$(mktemp -d)"++python3 "$HERE/mock_embed_server.py" & MOCK=$!+trap 'kill $MOCK $XPID 2>/dev/null || true; rm -rf "$DATA"' EXIT+sleep 1++# A real provider expects a real key; the mock ignores it.+XERJ_EMBEDDING_API_KEY="${XERJ_EMBEDDING_API_KEY:-sk-mock}" \+ "$XERJ" --insecure --data-dir "$DATA" --config "$HERE/xerj-proxy.toml" > "$DATA/xerj.log" 2>&1 &+XPID=$!+until curl -s -m2 -o /dev/null localhost:9200/ 2>/dev/null; do sleep 1; done++echo "backend chosen:"; grep -m1 -a "embedding backend" "$DATA/xerj.log" || true++H='Content-Type: application/json'+curl -s -XPUT localhost:9200/notes -H "$H" \+ -d '{"mappings":{"properties":{"body":{"type":"semantic_text"}}}}' >/dev/null+curl -s localhost:9200/_bulk -H "$H" --data-binary $'{"index":{"_index":"notes","_id":"0"}}\n{"body":"the payment service went down when the connection pool was exhausted"}\n{"index":{"_index":"notes","_id":"1"}}\n{"body":"kittens should be vaccinated at eight and twelve weeks of age"}\n' >/dev/null+curl -s -XPOST localhost:9200/notes/_refresh >/dev/null++echo "external-API calls after ingest: $(curl -s localhost:8900/ | python3 -c 'import json,sys;print(json.load(sys.stdin)["calls"])')"+echo "semantic query result:"+curl -s "localhost:9200/notes/_search?filter_path=hits.hits._source.body" -H "$H" \+ -d '{"size":1,"query":{"semantic":{"field":"body","query":"database outage from too many open connections","k":1}}}'+echo+echo "external-API calls after query: $(curl -s localhost:8900/ | python3 -c 'import json,sys;print(json.load(sys.stdin)["calls"])')"diff --git a/docs/examples/external-embeddings/xerj-proxy.toml b/docs/examples/external-embeddings/xerj-proxy.tomlnew file mode 100644index 000000000..3f2ebd5a4--- /dev/null+++ b/docs/examples/external-embeddings/xerj-proxy.toml@@ -0,0 +1,6 @@+[embedding]+mode = "proxy"+default_endpoint = "http://127.0.0.1:8900/v1/embeddings"+default_model = "mock-embed-256"+batch_size = 64+timeout_ms = 5000