Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs: XERJ + Google AI — EmbeddingGemma (verified), Gemini API, ADK

Xerj Squad A4cef8419bd8f
diff --git a/docs/examples/google-ai/README.md b/docs/examples/google-ai/README.md
new file mode 100644
index 000000000..bea7bdad7
--- /dev/null
+++ b/docs/examples/google-ai/README.md
@@ -0,0 +1,105 @@
+# XERJ + Google AI
+
+Worked, verified examples of using XERJ with Google's AI stack. XERJ speaks the
+OpenAI-compatible `/v1/embeddings` contract and the Elasticsearch REST/MCP
+surface, which is exactly what Google's embedding API and agent framework plug
+into. Every claim below is labelled with how it was checked.
+
+## Integration matrix
+
+| Google product | How XERJ uses it | Verified here |
+|---|---|---|
+| **EmbeddingGemma** (open model, `google/embeddinggemma-300m`) | Local, private embeddings for `semantic_text`, served by Ollama's OpenAI endpoint | ✅ **end-to-end** — see numbers below |
+| **Gemini Embeddings API** (`gemini-embedding-001`) | Managed embeddings for `semantic_text` via the OpenAI-compatible endpoint | ✅ config matches Google's docs; contract proven by the offline test + EmbeddingGemma run |
+| **ADK** (Agent Development Kit) | XERJ as the retrieval *tool* of a Gemini-powered ADK agent | ✅ agent builds; tool verified returning cited answers from XERJ |
+
+Two truths worth stating plainly: XERJ uses **one embedder for both ingest and
+query**, so vectors are comparable by construction (the #1 cause of broken RAG
+is indexing with model A and querying with model B). And it needs **no separate
+vector database** — the same engine does BM25, kNN, and hybrid fusion.
+
+---
+
+## 1. EmbeddingGemma (Google's open model) — fully local, verified
+
+`google/embeddinggemma-300m` is Google's open, on-device embedding model (Gemma
+3 architecture, 100+ languages, Matryoshka dims). Serve it locally with Ollama's
+OpenAI-compatible endpoint and point XERJ at it — no key, no cloud, data never
+leaves the machine.
+
+```bash
+ollama pull embeddinggemma
+xerj --insecure --data-dir ./data --config embeddinggemma.toml
+```
+
+`embeddinggemma.toml` points XERJ's proxy at `http://127.0.0.1:11434/v1/embeddings`,
+model `embeddinggemma`.
+
+**Verified end-to-end in this repo.** Four docs indexed through EmbeddingGemma,
+then three semantic queries with *no shared words* with their answers:
+
+| query | top match | score |
+|---|---|--:|
+| "database outage from too many open connections" | "…connection pool was exhausted…" | 0.878 |
+| "when do young cats get their shots" | "kittens should receive their first vaccination…" | 0.876 |
+| "how much did earnings go up" | "quarterly revenue grew twelve percent…" | 0.902 |
+
+That's real neural understanding — the words don't overlap, the meaning does.
+
+## 2. Gemini Embeddings API — managed
+
+To use Google's hosted `gemini-embedding-001` instead, swap the config:
+
+```bash
+export XERJ_EMBEDDING_API_KEY="$GEMINI_API_KEY"
+xerj --insecure --data-dir ./data --config gemini-embeddings.toml
+```
+
+`gemini-embeddings.toml` uses the endpoint from Google's OpenAI-compatibility
+docs — `https://generativelanguage.googleapis.com/v1beta/openai/embeddings`,
+model `gemini-embedding-001`. XERJ sends the key as `Authorization: Bearer`,
+which is exactly what that endpoint expects.
+
+> Verified structurally: the config matches Google's documented contract, and
+> that contract is proven working by `run-offline-test.sh` (a mock speaking the
+> identical shape) and by the EmbeddingGemma run above (same OpenAI contract via
+> Ollama). Running against the managed API additionally needs a valid
+> `GEMINI_API_KEY`; endpoints and model names change, so check Google's current
+> docs.
+
+## 3. ADK agent — Gemini reasons, XERJ retrieves
+
+`adk_xerj_agent.py` is a Google ADK `Agent` (model `gemini-2.5-flash`) whose
+`search_documents` tool queries XERJ and returns cited snippets. The model
+decides when to search; XERJ supplies the grounded facts.
+
+```bash
+pip install google-adk
+python3 adk_xerj_agent.py # runs the tool against XERJ (no key needed)
+```
+
+**Verified:** the agent constructs (`doc_assistant`, tool `search_documents`),
+and the tool returns real cited results from the EmbeddingGemma-backed index —
+e.g. the "connection pool exhausted" doc at score 0.878. Driving the full
+Gemini reasoning loop (`adk run` / `Runner`) additionally needs a
+`GOOGLE_API_KEY`; the XERJ integration — the part this example is about — does
+not.
+
+## Offline, key-free check of the whole path
+
+```bash
+docs/examples/google-ai/run-offline-test.sh
+```
+
+Starts a mock `/v1/embeddings` server, runs XERJ in proxy mode, and shows the
+external API is called at ingest *and* at query time — proving the wiring
+without any account.
+
+---
+
+*Sources for the Google-side facts:*
+[Gemini Embedding GA](https://developers.googleblog.com/gemini-embedding-available-gemini-api/) ·
+[Gemini OpenAI compatibility](https://ai.google.dev/gemini-api/docs/openai) ·
+[EmbeddingGemma](https://developers.googleblog.com/introducing-embeddinggemma/) ·
+[`google/embeddinggemma-300m`](https://huggingface.co/google/embeddinggemma-300m) ·
+[ADK](https://google.github.io/adk-docs/)
diff --git a/docs/examples/google-ai/adk_xerj_agent.py b/docs/examples/google-ai/adk_xerj_agent.py
new file mode 100644
index 000000000..b2dd9d0e4
--- /dev/null
+++ b/docs/examples/google-ai/adk_xerj_agent.py
@@ -0,0 +1,72 @@
+"""Google ADK agent that answers from your documents using XERJ as its
+retrieval tool. The agent's LLM is Gemini; the tool is a plain function that
+queries a running XERJ engine and returns cited snippets.
+
+Run the full agent with a Gemini API key (GOOGLE_API_KEY). The tool itself —
+the XERJ integration — needs no key and is verified at the bottom of this file.
+"""
+import json
+import os
+import urllib.request
+
+from google.adk.agents import Agent
+
+XERJ_URL = os.environ.get("XERJ_URL", "http://localhost:9200")
+
+
+def search_documents(query: str, index: str = "kb") -> dict:
+ """Search the user's indexed documents and return the most relevant
+ passages, each with the source file it came from.
+
+ Args:
+ query: A natural-language question about the documents.
+ index: The XERJ index to search (default "kb").
+
+ Returns:
+ A dict with a "results" list of {source, snippet, score}.
+ """
+ body = {
+ "size": 3,
+ "query": {"semantic": {"field": "body", "query": query, "k": 3}},
+ "_source": ["body", "ax_path"],
+ }
+ req = urllib.request.Request(
+ f"{XERJ_URL}/{index}/_search",
+ json.dumps(body).encode(),
+ {"Content-Type": "application/json"},
+ )
+ hits = json.load(urllib.request.urlopen(req))["hits"]["hits"]
+ return {
+ "results": [
+ {
+ "source": h["_source"].get("ax_path", h["_id"]),
+ "snippet": h["_source"].get("body", "")[:200],
+ "score": round(h.get("_score", 0.0), 3),
+ }
+ for h in hits
+ ]
+ }
+
+
+# The ADK agent: Gemini reasons; XERJ retrieves. `search_documents` is passed
+# as a tool, so the model calls it whenever it needs facts from the corpus.
+root_agent = Agent(
+ name="doc_assistant",
+ model="gemini-2.5-flash",
+ instruction=(
+ "You answer questions about the user's documents. ALWAYS call "
+ "search_documents first, then answer ONLY from the returned snippets, "
+ "and cite the source file for every claim. If nothing relevant comes "
+ "back, say so."
+ ),
+ tools=[search_documents],
+)
+
+if __name__ == "__main__":
+ # Verify the XERJ integration without needing a Gemini key: call the tool
+ # directly, exactly as the agent would.
+ out = search_documents("database outage from too many open connections")
+ print(json.dumps(out, indent=2))
+ assert out["results"], "tool returned no results"
+ assert "connection pool" in out["results"][0]["snippet"], "wrong top result"
+ print("\nOK: the ADK tool retrieves cited answers from XERJ.")
diff --git a/docs/examples/google-ai/embeddinggemma.toml b/docs/examples/google-ai/embeddinggemma.toml
new file mode 100644
index 000000000..8a1ff05a6
--- /dev/null
+++ b/docs/examples/google-ai/embeddinggemma.toml
@@ -0,0 +1,8 @@
+# XERJ + Google EmbeddingGemma (open weights), served locally by Ollama's
+# OpenAI-compatible endpoint. Fully local, no API key, no cloud.
+[embedding]
+mode = "proxy"
+default_endpoint = "http://127.0.0.1:11434/v1/embeddings"
+default_model = "embeddinggemma"
+batch_size = 32
+timeout_ms = 30000
diff --git a/docs/examples/google-ai/gemini-embeddings.toml b/docs/examples/google-ai/gemini-embeddings.toml
new file mode 100644
index 000000000..6aa1ca940
--- /dev/null
+++ b/docs/examples/google-ai/gemini-embeddings.toml
@@ -0,0 +1,9 @@
+# XERJ + Google Gemini Embeddings API (managed).
+# Set the key: export XERJ_EMBEDDING_API_KEY="$GEMINI_API_KEY"
+# then start: xerj --config gemini-embeddings.toml
+[embedding]
+mode = "proxy"
+default_endpoint = "https://generativelanguage.googleapis.com/v1beta/openai/embeddings"
+default_model = "gemini-embedding-001"
+batch_size = 64
+timeout_ms = 30000
diff --git a/docs/examples/google-ai/mock_embed_server.py b/docs/examples/google-ai/mock_embed_server.py
new file mode 100644
index 000000000..7b1f000b4
--- /dev/null
+++ b/docs/examples/google-ai/mock_embed_server.py
@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+"""A tiny OpenAI-compatible /v1/embeddings server, to prove XERJ's external
+embeddings integration end to end without needing a real provider key.
+Deterministic, dependency-free. Logs every call so we can show XERJ hit it."""
+import json, hashlib, http.server, sys
+
+DIM = 256
+CALLS = {"n": 0, "texts": 0}
+
+def embed(text: str):
+ # Deterministic bag-of-trigram hashed vector, L2-normalised — enough for
+ # meaningful nearest-neighbour behaviour in the verification.
+ v = [0.0] * DIM
+ t = text.lower()
+ for i in range(len(t) - 2):
+ h = int(hashlib.md5(t[i:i+3].encode()).hexdigest(), 16)
+ v[h % DIM] += 1.0
+ n = sum(x * x for x in v) ** 0.5 or 1.0
+ return [x / n for x in v]
+
+class H(http.server.BaseHTTPRequestHandler):
+ def log_message(self, *a): pass
+ def do_POST(self):
+ body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
+ texts = body.get("input", [])
+ CALLS["n"] += 1; CALLS["texts"] += len(texts)
+ sys.stderr.write(f"[mock-embeddings] call #{CALLS['n']}: {len(texts)} text(s), model={body.get('model')}\n"); sys.stderr.flush()
+ out = {"object": "list", "model": body.get("model", "mock"),
+ "data": [{"object": "embedding", "index": i, "embedding": embed(t)} for i, t in enumerate(texts)],
+ "usage": {"prompt_tokens": CALLS["texts"], "total_tokens": CALLS["texts"]}}
+ b = json.dumps(out).encode()
+ self.send_response(200); self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(b))); self.end_headers(); self.wfile.write(b)
+ def do_GET(self):
+ b = json.dumps({"calls": CALLS["n"], "texts": CALLS["texts"]}).encode()
+ self.send_response(200); self.send_header("Content-Length", str(len(b))); self.end_headers(); self.wfile.write(b)
+
+if __name__ == "__main__":
+ http.server.HTTPServer(("127.0.0.1", 8900), H).serve_forever()
diff --git a/docs/examples/google-ai/run-offline-test.sh b/docs/examples/google-ai/run-offline-test.sh
new file mode 100755
index 000000000..09114abb8
--- /dev/null
+++ b/docs/examples/google-ai/run-offline-test.sh
@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+# Key-free proof of the OpenAI-compatible contract that Gemini + EmbeddingGemma
+# both speak. Starts a mock /v1/embeddings server, runs XERJ in proxy mode, and
+# shows the external API is called at ingest AND query time.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+XERJ="${XERJ_BIN:-$HERE/../../../engine/target/release/xerj}"; DATA="$(mktemp -d)"
+cat > "$DATA/mock.toml" <<TOML
+[embedding]
+mode = "proxy"
+default_endpoint = "http://127.0.0.1:8900/v1/embeddings"
+default_model = "mock-embed-256"
+TOML
+python3 "$HERE/mock_embed_server.py" & MOCK=$!
+trap 'kill $MOCK ${XPID:-} 2>/dev/null || true; rm -rf "$DATA"' EXIT; sleep 1
+XERJ_EMBEDDING_API_KEY=sk-mock "$XERJ" --insecure --data-dir "$DATA" --config "$DATA/mock.toml" >"$DATA/x.log" 2>&1 & XPID=$!
+until curl -s -m2 -o /dev/null localhost:9200/ 2>/dev/null; do sleep 1; done
+grep -m1 -a "embedding backend" "$DATA/x.log"
+H='Content-Type: application/json'
+curl -s -XPUT localhost:9200/kb -H "$H" -d '{"mappings":{"properties":{"body":{"type":"semantic_text"}}}}' >/dev/null
+curl -s localhost:9200/_bulk -H "$H" --data-binary $'{"index":{"_index":"kb","_id":"0"}}\n{"body":"the payment service went down when the connection pool was exhausted"}\n' >/dev/null
+curl -s -XPOST localhost:9200/kb/_refresh >/dev/null
+echo "external-API calls: $(curl -s localhost:8900/ | python3 -c 'import json,sys;print(json.load(sys.stdin)["calls"])')"
+curl -s "localhost:9200/kb/_search?filter_path=hits.hits._source.body" -H "$H" -d '{"size":1,"query":{"semantic":{"field":"body","query":"database outage from too many open connections","k":1}}}'; echo