AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
XERJ Recipes
Use-case-driven guides for building real things with XERJ. Every recipe was verified end-to-end against a live XERJ before being written — the runnable example for each lives under docs/examples/.
These are practical "how do I actually do X" recipes, not an API reference (that follows separately).
| Recipe | What you build | Example |
|---|---|---|
| Zero-config indexing: make any folder AI-searchable | xerj autoindex <folder> — content-sniffed formats, inferred mappings, hostile CSV/HTML/binary handled, a catalog + data map for agents, resumable and idempotent |
built into the xerj binary — no script needed |
| Index your mail: Takeout / Thunderbird / Apple Mail mailboxes | xerj autoindex over an unzipped Google Takeout — the mbox streamed and split on a thread pool, attachments (PDF pages) as their own records, Gmail thread ids, replies_to / attachment_of edges from mail headers; measured on a synthetic mailbox |
built into the xerj binary — generator in scripts/synthetic-takeout.py |
| Zero-config folder → neural semantic search | xerj autoindex a mixed folder, then search the discovered prose by meaning with the built-in neural embedder — structured files stay exactly filterable |
autoindex_semantic.sh |
| All-you-can-eat search: one corpus, five ways | Index once, retrieve as full-text (BM25), semantic, vector kNN, hybrid (RRF), and semantic-scoped-by-filter — from one index, no separate vector DB | all_way_search.py |
| Give an AI agent long-term memory | A memory-backed agent using the /_memory API (store, semantic + keyword recall, metadata filters, forgetting, per-agent isolation) |
agent_memory.py |
| Find the right installed skill by task | xerj autoindex --skills indexes every skill folder it finds, then xerj search --prefix skills "<what you want to do>": a ranked list whose paths name the skill to open, re-runs that index only changed files, --watch |
built into the xerj binary, no script needed |
| Give your agent a second brain it can cite | xerj brain <folder> → a bi-temporal, evidence-carrying link index, then the agent loop over four MCP tools: orient, inspect, assert (with a quote), retire — every past moment replayable via as_of |
built into the xerj + xerj-mcp binaries — no script needed |
| Semantic search & RAG | Retrieval by meaning with semantic_text (auto-embed on ingest, no separate vector DB) |
rag_demo.py |
| Passage retrieval on long docs | semantic_text auto-embeds every overlapping passage; a long doc competes on any one of its sections via best-passage (max-sim) scoring — 98% top-3 vs 32% pooled |
passage_demo.py |
| Log analytics | From raw logs to dashboards — error rates, p95 latency, top services via aggregations | log_analytics.py |
| Vector search (kNN) | Nearest-neighbor similarity search over dense_vector (HNSW), with filters |
knn_demo.py |
| Vector quantization | Opt a dense_vector field into scalar8 (int8_hnsw) — kNN scored from ingest-time 1-byte-per-dim codes at recall@10 ≈ 0.998; the scoring working set is 4× smaller than the f32 vectors (see #392) |
quant_demo.py |
| Hybrid search | Keyword + vector in one query — results neither BM25 nor kNN finds alone | hybrid_search.py |
| Anomaly detection | Statistical _ml detectors that flag spikes in metrics/logs |
anomaly_detection.py |
| Continuous anomaly datafeeds | A live _ml datafeed that re-scores an index on a timer and stores new anomaly records you poll |
datafeed_demo.py |
| Migrate from Elasticsearch | Point your existing ES client at XERJ — same wire, change the URL | migrate_demo.sh |
| Production deployment | The hardened posture: in-process TLS + API-key auth, health-probe & gRPC posture, air-gapped neural-model staging, and copying data out of a live Elasticsearch (scroll + bulk) | inline curl + bash |
| Air-gapped deployment | Stage a release and optional neural model on a connected machine, then run XERJ offline with explicit network and persistence boundaries | inline curl + bash |
| Diagnose CPU and memory with pprof | A private, bounded, symbolized CPU/heap capture with reproducibility manifest, artifact validation, and AI analysis prompts | debug-profiling/ |
Every example is stdlib-only Python 3, plain curl, or the xerj binary itself — no dependencies to install. Start XERJ (./target/release/xerj --data-dir ./data --insecure), then run any example.