Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

5.6 KB
Raw Blame

XERJ Recipes

Use-case-driven guides for building real things with XERJ. Every recipe was verified end-to-end against a live XERJ before being written — the runnable example for each lives under docs/examples/.

These are practical "how do I actually do X" recipes, not an API reference (that follows separately).

Recipe What you build Example
Zero-config indexing: make any folder AI-searchable xerj autoindex <folder> — content-sniffed formats, inferred mappings, hostile CSV/HTML/binary handled, a catalog + data map for agents, resumable and idempotent built into the xerj binary — no script needed
Index your mail: Takeout / Thunderbird / Apple Mail mailboxes xerj autoindex over an unzipped Google Takeout — the mbox streamed and split on a thread pool, attachments (PDF pages) as their own records, Gmail thread ids, replies_to / attachment_of edges from mail headers; measured on a synthetic mailbox built into the xerj binary — generator in scripts/synthetic-takeout.py
Zero-config folder → neural semantic search xerj autoindex a mixed folder, then search the discovered prose by meaning with the built-in neural embedder — structured files stay exactly filterable autoindex_semantic.sh
All-you-can-eat search: one corpus, five ways Index once, retrieve as full-text (BM25), semantic, vector kNN, hybrid (RRF), and semantic-scoped-by-filter — from one index, no separate vector DB all_way_search.py
Give an AI agent long-term memory A memory-backed agent using the /_memory API (store, semantic + keyword recall, metadata filters, forgetting, per-agent isolation) agent_memory.py
Find the right installed skill by task xerj autoindex --skills indexes every skill folder it finds, then xerj search --prefix skills "<what you want to do>": a ranked list whose paths name the skill to open, re-runs that index only changed files, --watch built into the xerj binary, no script needed
Give your agent a second brain it can cite xerj brain <folder> → a bi-temporal, evidence-carrying link index, then the agent loop over four MCP tools: orient, inspect, assert (with a quote), retire — every past moment replayable via as_of built into the xerj + xerj-mcp binaries — no script needed
Semantic search & RAG Retrieval by meaning with semantic_text (auto-embed on ingest, no separate vector DB) rag_demo.py
Passage retrieval on long docs semantic_text auto-embeds every overlapping passage; a long doc competes on any one of its sections via best-passage (max-sim) scoring — 98% top-3 vs 32% pooled passage_demo.py
Log analytics From raw logs to dashboards — error rates, p95 latency, top services via aggregations log_analytics.py
Vector search (kNN) Nearest-neighbor similarity search over dense_vector (HNSW), with filters knn_demo.py
Vector quantization Opt a dense_vector field into scalar8 (int8_hnsw) — kNN scored from ingest-time 1-byte-per-dim codes at recall@10 ≈ 0.998; the scoring working set is 4× smaller than the f32 vectors (see #392) quant_demo.py
Hybrid search Keyword + vector in one query — results neither BM25 nor kNN finds alone hybrid_search.py
Anomaly detection Statistical _ml detectors that flag spikes in metrics/logs anomaly_detection.py
Continuous anomaly datafeeds A live _ml datafeed that re-scores an index on a timer and stores new anomaly records you poll datafeed_demo.py
Migrate from Elasticsearch Point your existing ES client at XERJ — same wire, change the URL migrate_demo.sh
Production deployment The hardened posture: in-process TLS + API-key auth, health-probe & gRPC posture, air-gapped neural-model staging, and copying data out of a live Elasticsearch (scroll + bulk) inline curl + bash
Air-gapped deployment Stage a release and optional neural model on a connected machine, then run XERJ offline with explicit network and persistence boundaries inline curl + bash
Diagnose CPU and memory with pprof A private, bounded, symbolized CPU/heap capture with reproducibility manifest, artifact validation, and AI analysis prompts debug-profiling/

Every example is stdlib-only Python 3, plain curl, or the xerj binary itself — no dependencies to install. Start XERJ (./target/release/xerj --data-dir ./data --insecure), then run any example.