AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
External embeddings — use any OpenAI-compatible API
XERJ can embed semantic_text fields through an external OpenAI-compatible
/v1/embeddings endpoint instead of its built-in lexical or neural backends.
The same embedder is used at ingest and query time, so vectors stay
comparable — you bring the model, XERJ handles indexing, kNN, and hybrid
fusion.
This directory is a key-free, reproducible proof of that path.
Run it
cargo build --release -p xerj-server # once, from the repo root
docs/examples/external-embeddings/run.sh
Expected: XERJ logs embedding backend: external proxy, the mock server logs
one call per document at ingest and one per query, and a semantic search for
"database outage from too many open connections" returns the document about
"the connection pool was exhausted" — proving the query was embedded by the
same external API and matched by vector similarity.
How it's wired
xerj-proxy.toml:
[embedding]
mode = "proxy"
default_endpoint = "http://127.0.0.1:8900/v1/embeddings"
default_model = "mock-embed-256"
Start XERJ with --config xerj-proxy.toml (or set --embed-mode proxy and put
the endpoint in your config). The API key is read from the
XERJ_EMBEDDING_API_KEY environment variable and sent as
Authorization: Bearer <key>.
default_endpoint is not always the request URL. Accepted forms, trailing
slash optional:
| configured | request URL |
|---|---|
http://127.0.0.1:8900/v1 or …/v1/ |
http://127.0.0.1:8900/v1/embeddings |
http://127.0.0.1:8900 or …/ |
http://127.0.0.1:8900/embeddings |
http://127.0.0.1:8900/v1/embeddings |
unchanged |
Any other path refuses to start when mode = "proxy". The error names the
URL that was set and suggests {origin}/v1/embeddings. The string in the
config stays the embedder identity, so switching a corpus from the full path
to the /v1 base is a different embedder.
The wire contract is the standard one, posted to that request URL:
POST {request URL}
{"input": ["text one", "text two"], "model": "<default_model>"}
→ {"data": [{"embedding": [ ... ], "index": 0}, { ... }]}
mock_embed_server.py implements exactly this in ~40 dependency-free lines, so
the example runs offline with no account.
Point it at a real provider
Any endpoint that speaks the contract above works — just change
default_endpoint, default_model, and set XERJ_EMBEDDING_API_KEY:
| provider | default_endpoint |
|---|---|
| OpenAI | https://api.openai.com/v1/embeddings or https://api.openai.com/v1 |
| Gemini (OpenAI-compatible mode) | https://generativelanguage.googleapis.com/v1beta/openai/embeddings |
| Local vLLM, Ollama, text-embeddings-inference, LM Studio, llama.cpp server, … | http://localhost:PORT/v1 or http://localhost:PORT/v1/embeddings |
Verified in this repo against the reproducible mock, which implements the exact request/response shape XERJ sends. Real providers that speak the same OpenAI
/v1/embeddingscontract are drop-in; confirm your provider's dims match yourdense_vector/semantic_textfield and set the key. Endpoints and model names change over time — check the provider's current docs.
Why this matters for agents
The most common way to get wrong RAG results is an embedder mismatch: you index with model A and later query with model B, and the vectors are silently incomparable. XERJ uses the one configured embedder for both, so an agent that indexes a corpus and later searches it gets comparable vectors by construction — no external inference service to keep in sync, just one endpoint.