Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs: verified example — embed via any OpenAI-compatible API

Xerj Squad Ace509a3d2d44
diff --git a/docs/examples/external-embeddings/README.md b/docs/examples/external-embeddings/README.md
new file mode 100644
index 000000000..b526aecd3
--- /dev/null
+++ b/docs/examples/external-embeddings/README.md
@@ -0,0 +1,74 @@
+# External embeddings — use any OpenAI-compatible API
+
+XERJ can embed `semantic_text` fields through an **external OpenAI-compatible
+`/v1/embeddings` endpoint** instead of its built-in lexical or neural backends.
+The same embedder is used at ingest *and* query time, so vectors stay
+comparable — you bring the model, XERJ handles indexing, kNN, and hybrid
+fusion.
+
+This directory is a **key-free, reproducible proof** of that path.
+
+## Run it
+
+```bash
+cargo build --release -p xerj-server # once, from the repo root
+docs/examples/external-embeddings/run.sh
+```
+
+Expected: XERJ logs `embedding backend: external proxy`, the mock server logs
+one call per document at ingest and one per query, and a semantic search for
+*"database outage from too many open connections"* returns the document about
+*"the connection pool was exhausted"* — proving the query was embedded by the
+same external API and matched by vector similarity.
+
+## How it's wired
+
+`xerj-proxy.toml`:
+
+```toml
+[embedding]
+mode = "proxy"
+default_endpoint = "http://127.0.0.1:8900/v1/embeddings"
+default_model = "mock-embed-256"
+```
+
+Start XERJ with `--config xerj-proxy.toml` (or set `--embed-mode proxy` and put
+the endpoint in your config). The API key is read from the
+`XERJ_EMBEDDING_API_KEY` environment variable and sent as
+`Authorization: Bearer <key>`.
+
+The wire contract is the standard one:
+
+```
+POST {default_endpoint}
+{"input": ["text one", "text two"], "model": "<default_model>"}
+→ {"data": [{"embedding": [ ... ], "index": 0}, { ... }]}
+```
+
+`mock_embed_server.py` implements exactly this in ~40 dependency-free lines, so
+the example runs offline with no account.
+
+## Point it at a real provider
+
+Any endpoint that speaks the contract above works — just change
+`default_endpoint`, `default_model`, and set `XERJ_EMBEDDING_API_KEY`:
+
+| provider | `default_endpoint` |
+|---|---|
+| OpenAI | `https://api.openai.com/v1/embeddings` |
+| Gemini (OpenAI-compatible mode) | `https://generativelanguage.googleapis.com/v1beta/openai/embeddings` |
+| Local (text-embeddings-inference, LM Studio, llama.cpp server, …) | `http://localhost:PORT/v1/embeddings` |
+
+> Verified in this repo against the reproducible mock, which implements the
+> exact request/response shape XERJ sends. Real providers that speak the same
+> OpenAI `/v1/embeddings` contract are drop-in; confirm your provider's dims
+> match your `dense_vector`/`semantic_text` field and set the key. Endpoints
+> and model names change over time — check the provider's current docs.
+
+## Why this matters for agents
+
+The most common way to get *wrong* RAG results is an embedder mismatch: you
+index with model A and later query with model B, and the vectors are silently
+incomparable. XERJ uses the **one configured embedder for both**, so an agent
+that indexes a corpus and later searches it gets comparable vectors by
+construction — no external inference service to keep in sync, just one endpoint.
diff --git a/docs/examples/external-embeddings/mock_embed_server.py b/docs/examples/external-embeddings/mock_embed_server.py
new file mode 100644
index 000000000..7b1f000b4
--- /dev/null
+++ b/docs/examples/external-embeddings/mock_embed_server.py
@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+"""A tiny OpenAI-compatible /v1/embeddings server, to prove XERJ's external
+embeddings integration end to end without needing a real provider key.
+Deterministic, dependency-free. Logs every call so we can show XERJ hit it."""
+import json, hashlib, http.server, sys
+
+DIM = 256
+CALLS = {"n": 0, "texts": 0}
+
+def embed(text: str):
+ # Deterministic bag-of-trigram hashed vector, L2-normalised — enough for
+ # meaningful nearest-neighbour behaviour in the verification.
+ v = [0.0] * DIM
+ t = text.lower()
+ for i in range(len(t) - 2):
+ h = int(hashlib.md5(t[i:i+3].encode()).hexdigest(), 16)
+ v[h % DIM] += 1.0
+ n = sum(x * x for x in v) ** 0.5 or 1.0
+ return [x / n for x in v]
+
+class H(http.server.BaseHTTPRequestHandler):
+ def log_message(self, *a): pass
+ def do_POST(self):
+ body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
+ texts = body.get("input", [])
+ CALLS["n"] += 1; CALLS["texts"] += len(texts)
+ sys.stderr.write(f"[mock-embeddings] call #{CALLS['n']}: {len(texts)} text(s), model={body.get('model')}\n"); sys.stderr.flush()
+ out = {"object": "list", "model": body.get("model", "mock"),
+ "data": [{"object": "embedding", "index": i, "embedding": embed(t)} for i, t in enumerate(texts)],
+ "usage": {"prompt_tokens": CALLS["texts"], "total_tokens": CALLS["texts"]}}
+ b = json.dumps(out).encode()
+ self.send_response(200); self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(b))); self.end_headers(); self.wfile.write(b)
+ def do_GET(self):
+ b = json.dumps({"calls": CALLS["n"], "texts": CALLS["texts"]}).encode()
+ self.send_response(200); self.send_header("Content-Length", str(len(b))); self.end_headers(); self.wfile.write(b)
+
+if __name__ == "__main__":
+ http.server.HTTPServer(("127.0.0.1", 8900), H).serve_forever()
diff --git a/docs/examples/external-embeddings/run.sh b/docs/examples/external-embeddings/run.sh
new file mode 100755
index 000000000..4c02fad8b
--- /dev/null
+++ b/docs/examples/external-embeddings/run.sh
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+# Reproducible, key-free proof that XERJ embeds through an external
+# OpenAI-compatible /v1/embeddings API. Run from the repo root after
+# `cargo build --release -p xerj-server`.
+#
+# It starts a tiny mock embeddings server, points XERJ at it in `proxy` mode,
+# indexes a few docs, and shows that (a) XERJ called the external API for
+# ingest and (b) a semantic query is embedded by the same API. Swap the mock
+# for any real OpenAI-compatible endpoint (OpenAI, Gemini's OpenAI-compatible
+# embeddings, a local text-embeddings server, …) by editing xerj-proxy.toml.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+XERJ="${XERJ_BIN:-$HERE/../../../engine/target/release/xerj}"
+DATA="$(mktemp -d)"
+
+python3 "$HERE/mock_embed_server.py" & MOCK=$!
+trap 'kill $MOCK $XPID 2>/dev/null || true; rm -rf "$DATA"' EXIT
+sleep 1
+
+# A real provider expects a real key; the mock ignores it.
+XERJ_EMBEDDING_API_KEY="${XERJ_EMBEDDING_API_KEY:-sk-mock}" \
+ "$XERJ" --insecure --data-dir "$DATA" --config "$HERE/xerj-proxy.toml" > "$DATA/xerj.log" 2>&1 &
+XPID=$!
+until curl -s -m2 -o /dev/null localhost:9200/ 2>/dev/null; do sleep 1; done
+
+echo "backend chosen:"; grep -m1 -a "embedding backend" "$DATA/xerj.log" || true
+
+H='Content-Type: application/json'
+curl -s -XPUT localhost:9200/notes -H "$H" \
+ -d '{"mappings":{"properties":{"body":{"type":"semantic_text"}}}}' >/dev/null
+curl -s localhost:9200/_bulk -H "$H" --data-binary $'{"index":{"_index":"notes","_id":"0"}}\n{"body":"the payment service went down when the connection pool was exhausted"}\n{"index":{"_index":"notes","_id":"1"}}\n{"body":"kittens should be vaccinated at eight and twelve weeks of age"}\n' >/dev/null
+curl -s -XPOST localhost:9200/notes/_refresh >/dev/null
+
+echo "external-API calls after ingest: $(curl -s localhost:8900/ | python3 -c 'import json,sys;print(json.load(sys.stdin)["calls"])')"
+echo "semantic query result:"
+curl -s "localhost:9200/notes/_search?filter_path=hits.hits._source.body" -H "$H" \
+ -d '{"size":1,"query":{"semantic":{"field":"body","query":"database outage from too many open connections","k":1}}}'
+echo
+echo "external-API calls after query: $(curl -s localhost:8900/ | python3 -c 'import json,sys;print(json.load(sys.stdin)["calls"])')"
diff --git a/docs/examples/external-embeddings/xerj-proxy.toml b/docs/examples/external-embeddings/xerj-proxy.toml
new file mode 100644
index 000000000..3f2ebd5a4
--- /dev/null
+++ b/docs/examples/external-embeddings/xerj-proxy.toml
@@ -0,0 +1,6 @@
+[embedding]
+mode = "proxy"
+default_endpoint = "http://127.0.0.1:8900/v1/embeddings"
+default_model = "mock-embed-256"
+batch_size = 64
+timeout_ms = 5000