Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

docs(rc-2): mark agent-memory, auto-embed semantic_text, and anomaly detection as shipped

xerj-orgcb716d08218d
diff --git a/README.md b/README.md
index dc8d0acca..4d01fec09 100644
--- a/README.md
+++ b/README.md
@@ -135,7 +135,7 @@ XERJ implements the Elasticsearch REST wire protocol. Because it is wire-compati
`match_all`, `match_none`, `match`, `match_phrase`, `match_phrase_prefix`, `multi_match`, `term`, `terms`, `range`, `prefix`, `wildcard`, `exists`, `ids`, `bool`, `fuzzy`, `regexp`, `query_string`, `simple_query_string`, `constant_score`, `boosting`, `dis_max`, `geo_distance`, `knn`, `semantic`, `hybrid`.
-> **Vector & semantic search notes.** `knn` (dense-vector HNSW) and `hybrid` (BM25 + kNN in one request) run entirely inside XERJ. The `semantic` query resolves query text to a vector at search time via an **external, OpenAI-compatible `/v1/embeddings` endpoint** that you configure (`embedding.default_endpoint` / `embedding.default_model`) — XERJ ships the proxy, not a built-in embedding model. Built-in/local embeddings, auto-embed-on-ingest, and an agent-memory API are on the [roadmap](#roadmap).
+> **Vector & semantic search notes.** `knn` (dense-vector HNSW) and `hybrid` (BM25 + kNN in one request) run entirely inside XERJ. As of **rc-2**, `semantic_text` fields **auto-embed on ingest** using a built-in, zero-config embedder, and the `semantic` query works with no external service; a configured external OpenAI-compatible `/v1/embeddings` endpoint (`embedding.default_endpoint`) is still used at higher quality when set. The built-in embedder is lexical (not neural) — see the [roadmap](#roadmap) and [ROADMAP.md](./ROADMAP.md) for the honest limitation and what's next. rc-2 also adds an **agent-memory REST API** (`/_memory/{ns}`) and **anomaly detection** (`/_ml/anomaly_detectors`).
**Supported aggregations**
@@ -225,13 +225,13 @@ If a test expects a response and XERJ returns something different, that's a bug
## Roadmap
-XERJ ships full-text, aggregation, log-analytics, dense-vector kNN, and hybrid search today. Several AI-adjacent capabilities are **planned but not yet implemented** — they are called out here (and in [`ROADMAP.md`](./ROADMAP.md)) so the README stays honest about what is and isn't built:
+XERJ ships full-text, aggregation, log-analytics, dense-vector kNN, and hybrid search. **rc-2** added three AI-adjacent capabilities (each conformance-gated and verified by real requests):
-- **Built-in / local embeddings & auto-embed on ingest.** Today, `semantic` search proxies to an external OpenAI-compatible embeddings endpoint you configure; there is no bundled embedding model, and the `semantic_text` field type is a mapping placeholder that does not yet auto-embed on ingest.
-- **Agent-memory API.** An internal memory store exists in `xerj-ai` but is not yet wired to a REST endpoint (store/recall/forget over the wire).
-- **Anomaly detection / ML jobs.** The Elasticsearch `_ml` / `_cat/ml/*` surface is present as compatibility stubs only; there is no detection or forecasting engine behind it yet.
+- **Auto-embed on ingest + a built-in embedder** — `semantic_text` works with zero external config; the built-in embedder is lexical (a bundled *neural* model is still future work).
+- **Agent-memory REST API** — `POST /_memory/{ns}` store, `/_recall` (kNN or BM25 + metadata filter), list/forget/drop; namespaced and offline.
+- **Anomaly detection (`_ml`)** — real statistical detectors with baseline + deviation scoring (on-demand; continuous datafeeds and forecasting are future work).
-See [ROADMAP.md](./ROADMAP.md) for the full list and status.
+Still planned: a bundled neural embedding model, an ingest-time chunk-embedding pipeline, continuous anomaly datafeeds, and distributed-clustering hardening. See [ROADMAP.md](./ROADMAP.md) for the full status and honest limitations.
---
diff --git a/ROADMAP.md b/ROADMAP.md
index 28c37db42..8202894ba 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -15,40 +15,39 @@ These are implemented and exercised by the test suite / benchmarks:
- **Hybrid search** — BM25 + kNN combined in a single request (`{"hybrid": {"query": …, "knn": …}}`).
- Columnar log analytics, bulk / scroll / delete-by-query, aliases, index templates.
-## Planned / in progress
+## Landed in 1.0.0-rc.2
-### 1. Built-in & local embeddings, auto-embed on ingest
+These three shipped in rc-2 (each conformance-gated at 1,326 / 1,329 and verified by real requests). Honest limitations are noted.
-**Status: partial — external proxy only.**
+### 1. Auto-embed on ingest + a built-in embedder ✅ (rc-2)
-Today the `semantic` query resolves query text to a vector at search time by proxying to an **external, OpenAI-compatible `/v1/embeddings` endpoint** that the operator configures (`embedding.default_endpoint`, `embedding.default_model`). Without that configuration, `semantic` returns a clear error. There is **no bundled/local embedding model**, and the `semantic_text` field type is currently a mapping placeholder — it does not yet auto-embed documents on ingest (indexing into a `semantic_text` field is not wired up).
+`semantic_text` now works end to end with **zero external configuration**. Indexing a document into a `semantic_text` field auto-embeds its text (previously returned `405`), and the `semantic` query embeds the query text with the same embedder and runs kNN — no external service required. A configured external `/v1/embeddings` proxy is still used, at higher quality, when `embedding.default_endpoint` is set.
-Planned:
-- Auto-embed on ingest for `semantic_text` fields (text in → vector stored, transparently).
-- An optional bundled/local embedding model so semantic search works with zero external dependencies.
-- An ingest-time embedding pipeline with the existing text chunker (overlapping chunks → per-chunk vectors).
+- **Limitation:** the built-in embedder is a deterministic **lexical** model (feature-hashed word unigrams + character trigrams, L2-normalized) — it captures vocabulary/sub-word overlap, not deep semantics. Paraphrases that share vocabulary rank correctly; truly-synonymous text with no word overlap will not. For production-grade semantics, configure the external proxy. A bundled neural model remains future work.
-### 2. Agent-memory API
+### 2. Agent-memory REST API ✅ (rc-2)
-**Status: internal module only, not exposed over REST.**
+A namespaced agent-memory API, backed by regular XERJ indices (reusing document + vector + BM25 + metadata-filter paths), working fully offline:
+`POST /_memory/{ns}` (store), `POST /_memory/{ns}/_recall` (kNN by vector or BM25 by text, with optional metadata filter + `k`), `GET /_memory/{ns}` (list), `DELETE /_memory/{ns}/{id}` and `DELETE /_memory/{ns}` (forget / drop). Namespaces are physically isolated.
-`xerj-ai` contains a memory-store module (entry model + recall), but it is **not wired to any HTTP endpoint** — there is no `store` / `recall` / `forget` REST API today (all memory paths return `404`).
+- **Limitation:** recall is pure relevance (kNN/BM25); recency-blended scoring and semantic dedup from the older internal module are not applied (recency ordering is available via the list endpoint). Single-node.
-Planned:
-- A REST API to store, semantically recall, and expire agent-memory entries per namespace.
-- Recall backed by the existing vector index (kNN over stored memories) plus recency/metadata filters.
+### 3. Anomaly detection (`_ml`) ✅ (rc-2)
-### 3. Anomaly detection & ML jobs
+A real statistical detector replaces the empty compat stubs:
+`PUT /_ml/anomaly_detectors/{id}` (create: source index, time field, function `count|mean|min|max|sum`, bucket span, threshold), `GET` (fetch/list — now returns real jobs), `POST /_ml/anomaly_detectors/{id}/_score` (buckets the source over time, builds a moving mean/stddev baseline, flags buckets deviating beyond the threshold with a normalized anomaly score), `DELETE`.
-**Status: compatibility stubs only.**
+- **Limitation:** on-demand scoring only — no continuous datafeed scheduler, no forecasting, no influencers/model-plot, single-node config registry. `_cat/ml/datafeeds` and `_cat/ml/trained_models` remain valid empty stubs.
-The Elasticsearch machine-learning surface is present for wire compatibility (`_cat/ml/anomaly_detectors`, `_cat/ml/datafeeds`, `_cat/ml/trained_models` return empty), but there is **no detection or forecasting engine** behind it — creating an anomaly job (`PUT /_ml/anomaly_detectors/{id}`) is not implemented.
+## Planned / in progress
-Planned:
-- Time-series anomaly detection (baseline + deviation scoring) over indexed metrics/logs.
-- Optional forecasting for capacity/write-load signals.
+### Neural embeddings & richer ML
+
+- A bundled/local **neural** embedding model (the current built-in embedder is lexical).
+- An ingest-time embedding pipeline with the existing text chunker (overlapping chunks → per-chunk vectors).
+- Continuous anomaly datafeeds (real-time jobs) and forecasting for capacity/write-load signals.
-### 4. Other tracked items
+### Other tracked items
- **Distributed clustering maturity** — embedded Raft handles cluster metadata today; multi-node sharding/replication hardening is ongoing.
- **Broader aggregation coverage** — geo/IP/nested/join/span families are partially covered; see the conformance suite for the current surface.