AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
docs(rc-2): mark agent-memory, auto-embed semantic_text, and anomaly detection as shipped
xerj-orgcb716d08218d
diff --git a/README.md b/README.mdindex dc8d0acca..4d01fec09 100644--- a/README.md+++ b/README.md@@ -135,7 +135,7 @@ XERJ implements the Elasticsearch REST wire protocol. Because it is wire-compati`match_all`, `match_none`, `match`, `match_phrase`, `match_phrase_prefix`, `multi_match`, `term`, `terms`, `range`, `prefix`, `wildcard`, `exists`, `ids`, `bool`, `fuzzy`, `regexp`, `query_string`, `simple_query_string`, `constant_score`, `boosting`, `dis_max`, `geo_distance`, `knn`, `semantic`, `hybrid`.-> **Vector & semantic search notes.** `knn` (dense-vector HNSW) and `hybrid` (BM25 + kNN in one request) run entirely inside XERJ. The `semantic` query resolves query text to a vector at search time via an **external, OpenAI-compatible `/v1/embeddings` endpoint** that you configure (`embedding.default_endpoint` / `embedding.default_model`) — XERJ ships the proxy, not a built-in embedding model. Built-in/local embeddings, auto-embed-on-ingest, and an agent-memory API are on the [roadmap](#roadmap).+> **Vector & semantic search notes.** `knn` (dense-vector HNSW) and `hybrid` (BM25 + kNN in one request) run entirely inside XERJ. As of **rc-2**, `semantic_text` fields **auto-embed on ingest** using a built-in, zero-config embedder, and the `semantic` query works with no external service; a configured external OpenAI-compatible `/v1/embeddings` endpoint (`embedding.default_endpoint`) is still used at higher quality when set. The built-in embedder is lexical (not neural) — see the [roadmap](#roadmap) and [ROADMAP.md](./ROADMAP.md) for the honest limitation and what's next. rc-2 also adds an **agent-memory REST API** (`/_memory/{ns}`) and **anomaly detection** (`/_ml/anomaly_detectors`).**Supported aggregations**@@ -225,13 +225,13 @@ If a test expects a response and XERJ returns something different, that's a bug## Roadmap-XERJ ships full-text, aggregation, log-analytics, dense-vector kNN, and hybrid search today. Several AI-adjacent capabilities are **planned but not yet implemented** — they are called out here (and in [`ROADMAP.md`](./ROADMAP.md)) so the README stays honest about what is and isn't built:+XERJ ships full-text, aggregation, log-analytics, dense-vector kNN, and hybrid search. **rc-2** added three AI-adjacent capabilities (each conformance-gated and verified by real requests):-- **Built-in / local embeddings & auto-embed on ingest.** Today, `semantic` search proxies to an external OpenAI-compatible embeddings endpoint you configure; there is no bundled embedding model, and the `semantic_text` field type is a mapping placeholder that does not yet auto-embed on ingest.-- **Agent-memory API.** An internal memory store exists in `xerj-ai` but is not yet wired to a REST endpoint (store/recall/forget over the wire).-- **Anomaly detection / ML jobs.** The Elasticsearch `_ml` / `_cat/ml/*` surface is present as compatibility stubs only; there is no detection or forecasting engine behind it yet.+- **Auto-embed on ingest + a built-in embedder** — `semantic_text` works with zero external config; the built-in embedder is lexical (a bundled *neural* model is still future work).+- **Agent-memory REST API** — `POST /_memory/{ns}` store, `/_recall` (kNN or BM25 + metadata filter), list/forget/drop; namespaced and offline.+- **Anomaly detection (`_ml`)** — real statistical detectors with baseline + deviation scoring (on-demand; continuous datafeeds and forecasting are future work).-See [ROADMAP.md](./ROADMAP.md) for the full list and status.+Still planned: a bundled neural embedding model, an ingest-time chunk-embedding pipeline, continuous anomaly datafeeds, and distributed-clustering hardening. See [ROADMAP.md](./ROADMAP.md) for the full status and honest limitations.---diff --git a/ROADMAP.md b/ROADMAP.mdindex 28c37db42..8202894ba 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -15,40 +15,39 @@ These are implemented and exercised by the test suite / benchmarks:- **Hybrid search** — BM25 + kNN combined in a single request (`{"hybrid": {"query": …, "knn": …}}`).- Columnar log analytics, bulk / scroll / delete-by-query, aliases, index templates.-## Planned / in progress+## Landed in 1.0.0-rc.2-### 1. Built-in & local embeddings, auto-embed on ingest+These three shipped in rc-2 (each conformance-gated at 1,326 / 1,329 and verified by real requests). Honest limitations are noted.-**Status: partial — external proxy only.**+### 1. Auto-embed on ingest + a built-in embedder ✅ (rc-2)-Today the `semantic` query resolves query text to a vector at search time by proxying to an **external, OpenAI-compatible `/v1/embeddings` endpoint** that the operator configures (`embedding.default_endpoint`, `embedding.default_model`). Without that configuration, `semantic` returns a clear error. There is **no bundled/local embedding model**, and the `semantic_text` field type is currently a mapping placeholder — it does not yet auto-embed documents on ingest (indexing into a `semantic_text` field is not wired up).+`semantic_text` now works end to end with **zero external configuration**. Indexing a document into a `semantic_text` field auto-embeds its text (previously returned `405`), and the `semantic` query embeds the query text with the same embedder and runs kNN — no external service required. A configured external `/v1/embeddings` proxy is still used, at higher quality, when `embedding.default_endpoint` is set.-Planned:-- Auto-embed on ingest for `semantic_text` fields (text in → vector stored, transparently).-- An optional bundled/local embedding model so semantic search works with zero external dependencies.-- An ingest-time embedding pipeline with the existing text chunker (overlapping chunks → per-chunk vectors).+- **Limitation:** the built-in embedder is a deterministic **lexical** model (feature-hashed word unigrams + character trigrams, L2-normalized) — it captures vocabulary/sub-word overlap, not deep semantics. Paraphrases that share vocabulary rank correctly; truly-synonymous text with no word overlap will not. For production-grade semantics, configure the external proxy. A bundled neural model remains future work.-### 2. Agent-memory API+### 2. Agent-memory REST API ✅ (rc-2)-**Status: internal module only, not exposed over REST.**+A namespaced agent-memory API, backed by regular XERJ indices (reusing document + vector + BM25 + metadata-filter paths), working fully offline:+`POST /_memory/{ns}` (store), `POST /_memory/{ns}/_recall` (kNN by vector or BM25 by text, with optional metadata filter + `k`), `GET /_memory/{ns}` (list), `DELETE /_memory/{ns}/{id}` and `DELETE /_memory/{ns}` (forget / drop). Namespaces are physically isolated.-`xerj-ai` contains a memory-store module (entry model + recall), but it is **not wired to any HTTP endpoint** — there is no `store` / `recall` / `forget` REST API today (all memory paths return `404`).+- **Limitation:** recall is pure relevance (kNN/BM25); recency-blended scoring and semantic dedup from the older internal module are not applied (recency ordering is available via the list endpoint). Single-node.-Planned:-- A REST API to store, semantically recall, and expire agent-memory entries per namespace.-- Recall backed by the existing vector index (kNN over stored memories) plus recency/metadata filters.+### 3. Anomaly detection (`_ml`) ✅ (rc-2)-### 3. Anomaly detection & ML jobs+A real statistical detector replaces the empty compat stubs:+`PUT /_ml/anomaly_detectors/{id}` (create: source index, time field, function `count|mean|min|max|sum`, bucket span, threshold), `GET` (fetch/list — now returns real jobs), `POST /_ml/anomaly_detectors/{id}/_score` (buckets the source over time, builds a moving mean/stddev baseline, flags buckets deviating beyond the threshold with a normalized anomaly score), `DELETE`.-**Status: compatibility stubs only.**+- **Limitation:** on-demand scoring only — no continuous datafeed scheduler, no forecasting, no influencers/model-plot, single-node config registry. `_cat/ml/datafeeds` and `_cat/ml/trained_models` remain valid empty stubs.-The Elasticsearch machine-learning surface is present for wire compatibility (`_cat/ml/anomaly_detectors`, `_cat/ml/datafeeds`, `_cat/ml/trained_models` return empty), but there is **no detection or forecasting engine** behind it — creating an anomaly job (`PUT /_ml/anomaly_detectors/{id}`) is not implemented.+## Planned / in progress-Planned:-- Time-series anomaly detection (baseline + deviation scoring) over indexed metrics/logs.-- Optional forecasting for capacity/write-load signals.+### Neural embeddings & richer ML++- A bundled/local **neural** embedding model (the current built-in embedder is lexical).+- An ingest-time embedding pipeline with the existing text chunker (overlapping chunks → per-chunk vectors).+- Continuous anomaly datafeeds (real-time jobs) and forecasting for capacity/write-load signals.-### 4. Other tracked items+### Other tracked items- **Distributed clustering maturity** — embedded Raft handles cluster metadata today; multi-node sharding/replication hardening is ongoing.- **Broader aggregation coverage** — geo/IP/nested/join/span families are partially covered; see the conformance suite for the current surface.