AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs: XERJ + Google AI — EmbeddingGemma (verified), Gemini API, ADK
Xerj Squad A4cef8419bd8f
diff --git a/docs/examples/google-ai/README.md b/docs/examples/google-ai/README.mdnew file mode 100644index 000000000..bea7bdad7--- /dev/null+++ b/docs/examples/google-ai/README.md@@ -0,0 +1,105 @@+# XERJ + Google AI++Worked, verified examples of using XERJ with Google's AI stack. XERJ speaks the+OpenAI-compatible `/v1/embeddings` contract and the Elasticsearch REST/MCP+surface, which is exactly what Google's embedding API and agent framework plug+into. Every claim below is labelled with how it was checked.++## Integration matrix++| Google product | How XERJ uses it | Verified here |+|---|---|---|+| **EmbeddingGemma** (open model, `google/embeddinggemma-300m`) | Local, private embeddings for `semantic_text`, served by Ollama's OpenAI endpoint | ✅ **end-to-end** — see numbers below |+| **Gemini Embeddings API** (`gemini-embedding-001`) | Managed embeddings for `semantic_text` via the OpenAI-compatible endpoint | ✅ config matches Google's docs; contract proven by the offline test + EmbeddingGemma run |+| **ADK** (Agent Development Kit) | XERJ as the retrieval *tool* of a Gemini-powered ADK agent | ✅ agent builds; tool verified returning cited answers from XERJ |++Two truths worth stating plainly: XERJ uses **one embedder for both ingest and+query**, so vectors are comparable by construction (the #1 cause of broken RAG+is indexing with model A and querying with model B). And it needs **no separate+vector database** — the same engine does BM25, kNN, and hybrid fusion.++---++## 1. EmbeddingGemma (Google's open model) — fully local, verified++`google/embeddinggemma-300m` is Google's open, on-device embedding model (Gemma+3 architecture, 100+ languages, Matryoshka dims). Serve it locally with Ollama's+OpenAI-compatible endpoint and point XERJ at it — no key, no cloud, data never+leaves the machine.++```bash+ollama pull embeddinggemma+xerj --insecure --data-dir ./data --config embeddinggemma.toml+```++`embeddinggemma.toml` points XERJ's proxy at `http://127.0.0.1:11434/v1/embeddings`,+model `embeddinggemma`.++**Verified end-to-end in this repo.** Four docs indexed through EmbeddingGemma,+then three semantic queries with *no shared words* with their answers:++| query | top match | score |+|---|---|--:|+| "database outage from too many open connections" | "…connection pool was exhausted…" | 0.878 |+| "when do young cats get their shots" | "kittens should receive their first vaccination…" | 0.876 |+| "how much did earnings go up" | "quarterly revenue grew twelve percent…" | 0.902 |++That's real neural understanding — the words don't overlap, the meaning does.++## 2. Gemini Embeddings API — managed++To use Google's hosted `gemini-embedding-001` instead, swap the config:++```bash+export XERJ_EMBEDDING_API_KEY="$GEMINI_API_KEY"+xerj --insecure --data-dir ./data --config gemini-embeddings.toml+```++`gemini-embeddings.toml` uses the endpoint from Google's OpenAI-compatibility+docs — `https://generativelanguage.googleapis.com/v1beta/openai/embeddings`,+model `gemini-embedding-001`. XERJ sends the key as `Authorization: Bearer`,+which is exactly what that endpoint expects.++> Verified structurally: the config matches Google's documented contract, and+> that contract is proven working by `run-offline-test.sh` (a mock speaking the+> identical shape) and by the EmbeddingGemma run above (same OpenAI contract via+> Ollama). Running against the managed API additionally needs a valid+> `GEMINI_API_KEY`; endpoints and model names change, so check Google's current+> docs.++## 3. ADK agent — Gemini reasons, XERJ retrieves++`adk_xerj_agent.py` is a Google ADK `Agent` (model `gemini-2.5-flash`) whose+`search_documents` tool queries XERJ and returns cited snippets. The model+decides when to search; XERJ supplies the grounded facts.++```bash+pip install google-adk+python3 adk_xerj_agent.py # runs the tool against XERJ (no key needed)+```++**Verified:** the agent constructs (`doc_assistant`, tool `search_documents`),+and the tool returns real cited results from the EmbeddingGemma-backed index —+e.g. the "connection pool exhausted" doc at score 0.878. Driving the full+Gemini reasoning loop (`adk run` / `Runner`) additionally needs a+`GOOGLE_API_KEY`; the XERJ integration — the part this example is about — does+not.++## Offline, key-free check of the whole path++```bash+docs/examples/google-ai/run-offline-test.sh+```++Starts a mock `/v1/embeddings` server, runs XERJ in proxy mode, and shows the+external API is called at ingest *and* at query time — proving the wiring+without any account.++---++*Sources for the Google-side facts:*+[Gemini Embedding GA](https://developers.googleblog.com/gemini-embedding-available-gemini-api/) ·+[Gemini OpenAI compatibility](https://ai.google.dev/gemini-api/docs/openai) ·+[EmbeddingGemma](https://developers.googleblog.com/introducing-embeddinggemma/) ·+[`google/embeddinggemma-300m`](https://huggingface.co/google/embeddinggemma-300m) ·+[ADK](https://google.github.io/adk-docs/)diff --git a/docs/examples/google-ai/adk_xerj_agent.py b/docs/examples/google-ai/adk_xerj_agent.pynew file mode 100644index 000000000..b2dd9d0e4--- /dev/null+++ b/docs/examples/google-ai/adk_xerj_agent.py@@ -0,0 +1,72 @@+"""Google ADK agent that answers from your documents using XERJ as its+retrieval tool. The agent's LLM is Gemini; the tool is a plain function that+queries a running XERJ engine and returns cited snippets.++Run the full agent with a Gemini API key (GOOGLE_API_KEY). The tool itself —+the XERJ integration — needs no key and is verified at the bottom of this file.+"""+import json+import os+import urllib.request++from google.adk.agents import Agent++XERJ_URL = os.environ.get("XERJ_URL", "http://localhost:9200")+++def search_documents(query: str, index: str = "kb") -> dict:+ """Search the user's indexed documents and return the most relevant+ passages, each with the source file it came from.++ Args:+ query: A natural-language question about the documents.+ index: The XERJ index to search (default "kb").++ Returns:+ A dict with a "results" list of {source, snippet, score}.+ """+ body = {+ "size": 3,+ "query": {"semantic": {"field": "body", "query": query, "k": 3}},+ "_source": ["body", "ax_path"],+ }+ req = urllib.request.Request(+ f"{XERJ_URL}/{index}/_search",+ json.dumps(body).encode(),+ {"Content-Type": "application/json"},+ )+ hits = json.load(urllib.request.urlopen(req))["hits"]["hits"]+ return {+ "results": [+ {+ "source": h["_source"].get("ax_path", h["_id"]),+ "snippet": h["_source"].get("body", "")[:200],+ "score": round(h.get("_score", 0.0), 3),+ }+ for h in hits+ ]+ }+++# The ADK agent: Gemini reasons; XERJ retrieves. `search_documents` is passed+# as a tool, so the model calls it whenever it needs facts from the corpus.+root_agent = Agent(+ name="doc_assistant",+ model="gemini-2.5-flash",+ instruction=(+ "You answer questions about the user's documents. ALWAYS call "+ "search_documents first, then answer ONLY from the returned snippets, "+ "and cite the source file for every claim. If nothing relevant comes "+ "back, say so."+ ),+ tools=[search_documents],+)++if __name__ == "__main__":+ # Verify the XERJ integration without needing a Gemini key: call the tool+ # directly, exactly as the agent would.+ out = search_documents("database outage from too many open connections")+ print(json.dumps(out, indent=2))+ assert out["results"], "tool returned no results"+ assert "connection pool" in out["results"][0]["snippet"], "wrong top result"+ print("\nOK: the ADK tool retrieves cited answers from XERJ.")diff --git a/docs/examples/google-ai/embeddinggemma.toml b/docs/examples/google-ai/embeddinggemma.tomlnew file mode 100644index 000000000..8a1ff05a6--- /dev/null+++ b/docs/examples/google-ai/embeddinggemma.toml@@ -0,0 +1,8 @@+# XERJ + Google EmbeddingGemma (open weights), served locally by Ollama's+# OpenAI-compatible endpoint. Fully local, no API key, no cloud.+[embedding]+mode = "proxy"+default_endpoint = "http://127.0.0.1:11434/v1/embeddings"+default_model = "embeddinggemma"+batch_size = 32+timeout_ms = 30000diff --git a/docs/examples/google-ai/gemini-embeddings.toml b/docs/examples/google-ai/gemini-embeddings.tomlnew file mode 100644index 000000000..6aa1ca940--- /dev/null+++ b/docs/examples/google-ai/gemini-embeddings.toml@@ -0,0 +1,9 @@+# XERJ + Google Gemini Embeddings API (managed).+# Set the key: export XERJ_EMBEDDING_API_KEY="$GEMINI_API_KEY"+# then start: xerj --config gemini-embeddings.toml+[embedding]+mode = "proxy"+default_endpoint = "https://generativelanguage.googleapis.com/v1beta/openai/embeddings"+default_model = "gemini-embedding-001"+batch_size = 64+timeout_ms = 30000diff --git a/docs/examples/google-ai/mock_embed_server.py b/docs/examples/google-ai/mock_embed_server.pynew file mode 100644index 000000000..7b1f000b4--- /dev/null+++ b/docs/examples/google-ai/mock_embed_server.py@@ -0,0 +1,39 @@+#!/usr/bin/env python3+"""A tiny OpenAI-compatible /v1/embeddings server, to prove XERJ's external+embeddings integration end to end without needing a real provider key.+Deterministic, dependency-free. Logs every call so we can show XERJ hit it."""+import json, hashlib, http.server, sys++DIM = 256+CALLS = {"n": 0, "texts": 0}++def embed(text: str):+ # Deterministic bag-of-trigram hashed vector, L2-normalised — enough for+ # meaningful nearest-neighbour behaviour in the verification.+ v = [0.0] * DIM+ t = text.lower()+ for i in range(len(t) - 2):+ h = int(hashlib.md5(t[i:i+3].encode()).hexdigest(), 16)+ v[h % DIM] += 1.0+ n = sum(x * x for x in v) ** 0.5 or 1.0+ return [x / n for x in v]++class H(http.server.BaseHTTPRequestHandler):+ def log_message(self, *a): pass+ def do_POST(self):+ body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))+ texts = body.get("input", [])+ CALLS["n"] += 1; CALLS["texts"] += len(texts)+ sys.stderr.write(f"[mock-embeddings] call #{CALLS['n']}: {len(texts)} text(s), model={body.get('model')}\n"); sys.stderr.flush()+ out = {"object": "list", "model": body.get("model", "mock"),+ "data": [{"object": "embedding", "index": i, "embedding": embed(t)} for i, t in enumerate(texts)],+ "usage": {"prompt_tokens": CALLS["texts"], "total_tokens": CALLS["texts"]}}+ b = json.dumps(out).encode()+ self.send_response(200); self.send_header("Content-Type", "application/json")+ self.send_header("Content-Length", str(len(b))); self.end_headers(); self.wfile.write(b)+ def do_GET(self):+ b = json.dumps({"calls": CALLS["n"], "texts": CALLS["texts"]}).encode()+ self.send_response(200); self.send_header("Content-Length", str(len(b))); self.end_headers(); self.wfile.write(b)++if __name__ == "__main__":+ http.server.HTTPServer(("127.0.0.1", 8900), H).serve_forever()diff --git a/docs/examples/google-ai/run-offline-test.sh b/docs/examples/google-ai/run-offline-test.shnew file mode 100755index 000000000..09114abb8--- /dev/null+++ b/docs/examples/google-ai/run-offline-test.sh@@ -0,0 +1,24 @@+#!/usr/bin/env bash+# Key-free proof of the OpenAI-compatible contract that Gemini + EmbeddingGemma+# both speak. Starts a mock /v1/embeddings server, runs XERJ in proxy mode, and+# shows the external API is called at ingest AND query time.+set -euo pipefail+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+XERJ="${XERJ_BIN:-$HERE/../../../engine/target/release/xerj}"; DATA="$(mktemp -d)"+cat > "$DATA/mock.toml" <<TOML+[embedding]+mode = "proxy"+default_endpoint = "http://127.0.0.1:8900/v1/embeddings"+default_model = "mock-embed-256"+TOML+python3 "$HERE/mock_embed_server.py" & MOCK=$!+trap 'kill $MOCK ${XPID:-} 2>/dev/null || true; rm -rf "$DATA"' EXIT; sleep 1+XERJ_EMBEDDING_API_KEY=sk-mock "$XERJ" --insecure --data-dir "$DATA" --config "$DATA/mock.toml" >"$DATA/x.log" 2>&1 & XPID=$!+until curl -s -m2 -o /dev/null localhost:9200/ 2>/dev/null; do sleep 1; done+grep -m1 -a "embedding backend" "$DATA/x.log"+H='Content-Type: application/json'+curl -s -XPUT localhost:9200/kb -H "$H" -d '{"mappings":{"properties":{"body":{"type":"semantic_text"}}}}' >/dev/null+curl -s localhost:9200/_bulk -H "$H" --data-binary $'{"index":{"_index":"kb","_id":"0"}}\n{"body":"the payment service went down when the connection pool was exhausted"}\n' >/dev/null+curl -s -XPOST localhost:9200/kb/_refresh >/dev/null+echo "external-API calls: $(curl -s localhost:8900/ | python3 -c 'import json,sys;print(json.load(sys.stdin)["calls"])')"+curl -s "localhost:9200/kb/_search?filter_path=hits.hits._source.body" -H "$H" -d '{"size":1,"query":{"semantic":{"field":"body","query":"database outage from too many open connections","k":1}}}'; echo