AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs: align README to reality + add public ROADMAP (agent memory, local embeddings, anomaly detection)
xerj-orgd53f2eb146ee
diff --git a/.gitignore b/.gitignoreindex fd3d744b7..42372c8a7 100644--- a/.gitignore+++ b/.gitignore@@ -95,3 +95,4 @@ landing/resources/xerj-sales-kit.zip# Re-include the public architecture doc (Markdown is ignored by default above).!docs/ARCHITECTURE.md+!ROADMAP.mddiff --git a/README.md b/README.mdindex 19c87cf7e..dc8d0acca 100644--- a/README.md+++ b/README.md@@ -135,6 +135,8 @@ XERJ implements the Elasticsearch REST wire protocol. Because it is wire-compati`match_all`, `match_none`, `match`, `match_phrase`, `match_phrase_prefix`, `multi_match`, `term`, `terms`, `range`, `prefix`, `wildcard`, `exists`, `ids`, `bool`, `fuzzy`, `regexp`, `query_string`, `simple_query_string`, `constant_score`, `boosting`, `dis_max`, `geo_distance`, `knn`, `semantic`, `hybrid`.+> **Vector & semantic search notes.** `knn` (dense-vector HNSW) and `hybrid` (BM25 + kNN in one request) run entirely inside XERJ. The `semantic` query resolves query text to a vector at search time via an **external, OpenAI-compatible `/v1/embeddings` endpoint** that you configure (`embedding.default_endpoint` / `embedding.default_model`) — XERJ ships the proxy, not a built-in embedding model. Built-in/local embeddings, auto-embed-on-ingest, and an agent-memory API are on the [roadmap](#roadmap).+**Supported aggregations**`terms`, `stats`, `avg`, `sum`, `min`, `max`, `value_count`, `cardinality`, `range`, `histogram`, `date_histogram`, `percentiles`, `filter`, `missing`, `composite`.@@ -165,7 +167,7 @@ XERJ is a Cargo workspace under [`engine/`](./engine). The crates:| `xerj-fts` | Full-text search: BM25 scoring, analyzer registry, postings lists. || `xerj-vector` | Dense-vector HNSW index for kNN / semantic search. || `xerj-logs` | Columnar log ingestion and retention. |-| `xerj-ai` | Text chunking, embedding proxy, and memory store. |+| `xerj-ai` | Text chunking and an embedding proxy (external OpenAI-compatible API) that powers the `semantic` query. Also houses an experimental memory store not yet exposed over the REST API — see the [roadmap](#roadmap). || `xerj-compress` | Block compression codecs (LZ4, Zstd). || `xerj-common` | Shared types: `Config`, `Schema`, `FieldType`, `XerjError`. || `xerj-cluster` | Embedded Raft consensus for cluster metadata (no external Raft dependency). |@@ -221,6 +223,18 @@ If a test expects a response and XERJ returns something different, that's a bug---+## Roadmap++XERJ ships full-text, aggregation, log-analytics, dense-vector kNN, and hybrid search today. Several AI-adjacent capabilities are **planned but not yet implemented** — they are called out here (and in [`ROADMAP.md`](./ROADMAP.md)) so the README stays honest about what is and isn't built:++- **Built-in / local embeddings & auto-embed on ingest.** Today, `semantic` search proxies to an external OpenAI-compatible embeddings endpoint you configure; there is no bundled embedding model, and the `semantic_text` field type is a mapping placeholder that does not yet auto-embed on ingest.+- **Agent-memory API.** An internal memory store exists in `xerj-ai` but is not yet wired to a REST endpoint (store/recall/forget over the wire).+- **Anomaly detection / ML jobs.** The Elasticsearch `_ml` / `_cat/ml/*` surface is present as compatibility stubs only; there is no detection or forecasting engine behind it yet.++See [ROADMAP.md](./ROADMAP.md) for the full list and status.++---+## ContributingContributions are welcome. Please read [CONTRIBUTING.md](./CONTRIBUTING.md) for how to build, test, and structure changes. A few conventions:diff --git a/ROADMAP.md b/ROADMAP.mdnew file mode 100644index 000000000..28c37db42--- /dev/null+++ b/ROADMAP.md@@ -0,0 +1,58 @@+# XERJ Roadmap++This roadmap tracks capabilities that are **planned but not yet fully implemented**, so the project's public claims stay honest about what ships today versus what is coming. Status is verified against the actual code and by real API requests, not aspirational.++Last reviewed: 2026-07-06 (against `v1.0.0-rc.1`).++## Shipping today (for context)++These are implemented and exercised by the test suite / benchmarks:++- Elasticsearch REST wire compatibility (1,326 / 1,329 ES-YAML conformance cases).+- Full-text search (BM25) and all 38 supported query types.+- The standard aggregation suite.+- **Dense-vector kNN** over HNSW (`knn` query and top-level `knn`), at recall parity with Elasticsearch.+- **Hybrid search** — BM25 + kNN combined in a single request (`{"hybrid": {"query": …, "knn": …}}`).+- Columnar log analytics, bulk / scroll / delete-by-query, aliases, index templates.++## Planned / in progress++### 1. Built-in & local embeddings, auto-embed on ingest++**Status: partial — external proxy only.**++Today the `semantic` query resolves query text to a vector at search time by proxying to an **external, OpenAI-compatible `/v1/embeddings` endpoint** that the operator configures (`embedding.default_endpoint`, `embedding.default_model`). Without that configuration, `semantic` returns a clear error. There is **no bundled/local embedding model**, and the `semantic_text` field type is currently a mapping placeholder — it does not yet auto-embed documents on ingest (indexing into a `semantic_text` field is not wired up).++Planned:+- Auto-embed on ingest for `semantic_text` fields (text in → vector stored, transparently).+- An optional bundled/local embedding model so semantic search works with zero external dependencies.+- An ingest-time embedding pipeline with the existing text chunker (overlapping chunks → per-chunk vectors).++### 2. Agent-memory API++**Status: internal module only, not exposed over REST.**++`xerj-ai` contains a memory-store module (entry model + recall), but it is **not wired to any HTTP endpoint** — there is no `store` / `recall` / `forget` REST API today (all memory paths return `404`).++Planned:+- A REST API to store, semantically recall, and expire agent-memory entries per namespace.+- Recall backed by the existing vector index (kNN over stored memories) plus recency/metadata filters.++### 3. Anomaly detection & ML jobs++**Status: compatibility stubs only.**++The Elasticsearch machine-learning surface is present for wire compatibility (`_cat/ml/anomaly_detectors`, `_cat/ml/datafeeds`, `_cat/ml/trained_models` return empty), but there is **no detection or forecasting engine** behind it — creating an anomaly job (`PUT /_ml/anomaly_detectors/{id}`) is not implemented.++Planned:+- Time-series anomaly detection (baseline + deviation scoring) over indexed metrics/logs.+- Optional forecasting for capacity/write-load signals.++### 4. Other tracked items++- **Distributed clustering maturity** — embedded Raft handles cluster metadata today; multi-node sharding/replication hardening is ongoing.+- **Broader aggregation coverage** — geo/IP/nested/join/span families are partially covered; see the conformance suite for the current surface.++---++Found something claimed but not working? That is a bug in our docs or our code — please [open an issue](https://github.com/xerj-org/xerj/issues). We would rather ship an honest roadmap than an overstated feature list.